Importing data with ElasticSync

This page of the manual describes how to import data using ElasticSync. Importing data is recommended if you want to add new items, customer data or stock etc. to your plentymarkets system and you already have this data available in digital format. ElasticSync allows to carry out data imports by creating so-called syncs. You can import your data manually or automatically.

1. Creating a sync

The steps for creating and importing syncs are identical for all sync types. As such, the steps described in the following apply to all sync types.

Go to Data » ElasticSync to create, edit and manage syncs. Here, you can see an overview of all syncs. With a sync, you can assign the data of a file that is accessible via FTP or HTTP to the data in plentymarkets and import them. When creating a sync, you save information about the data source. Furthermore, it is important that you determine data for the matching and that you assign which data of your file corresponds to the data in plentymarkets. Use the matching to set what data is to be used to try to identify, if a data record already exists or not. Use the mapping to set which column in your file corresponds to which data field in plentymarkets. In the end, set an import interval. Proceed as described below to create a sync.

Set the import interval in the end to avoid that a sync is carried out too early

It is recommended to set the import interval in the end. By doing so, you avoid that a sync is already carried out while you still carry out matching and mapping settings.

Creating a sync:

Go to Data » ElasticSync.
Click on Add sync ().
→ The editing area New sync is displayed.
Carry out the settings. Pay attention to the explanations given in Table 1.
Save () the settings.
→ The sync is created.
→ The matching settings of the sync open.

Table 1. Creating a sync
Setting	Explanation
Basic data
Name	Enter a name. This name is displayed in the overview and can be used for the search.
Interval	Select an interval for the import of syncs. By choosing the option Schedule you can determine how often and at what time the import of syncs shall be started.
Type	Select the type from the drop-down list. Item = Select to import item and variation data. Attribute = Select to import attribute data. Orders = Select to import order data. Properties = Select to import properties. Facets = Select to import facets. Parts compatibility listing = Select to import parts compatibility listing data. Coupon codes = Select to import coupon codes. Manufacturer = Select to import manufacturer data. Campaigns = Select to import coupon campaigns. Categories = Select to import category data. Contacts, companies and addresses = Select to import contact, company and address data. Warehouse = Select to import storage locations. Characteristic = Select to import characteristic data. Newsletter recipient = Select to import newsletter data. Notes = Select to import notes. Stock = Select to import stock data. Incoming items = Select to import incoming item data.
Import
Data type	Select the file format of the import file. Currently, CSV is the only available option.
Character set	Select the character set of the file or the option Detect automatically.
Starting in line	Select a line starting from which the data should be imported.
Header is included	Select whether the import file contains a header.
Use global cache	Select whether the global cache is to be used.
Automatic mapping	Select whether the mapping is to be done automatically. The columns in the CSV file can be automatically mapped with the plentymarkets data fields. To ensure that the mapping works properly, the names of the fields that you selected in the FormatDesigner must not be changed.
Separator
Separator	The separator of the CSV file is automatically detected. This setting is activated by default. Alternatively, you can manually select the field separator that is set in the import file in order to separate the data fields (columns) of a data record from each other. Possible separators are Semicolon, Comma, Tab and Pipe.
Delimiter	Select delimiters within the fields that may contain multiple values. * " = individual values within one field are separated by double quotation marks. * ' = individual values within one field are separated by single quotation marks.
Source
Source	Select Manual upload, FTP, HTTPS or SFTP. Depending on the selected option, the following settings are displayed: Manual upload = Select and upload a local file. After running a sync, the file is automatically deleted from the webspace. This is preset by default. In order to save the file you need to remove the check mark. *Note: storing the file on the webspace is subject to a fee. FTP* = Host, port, user name, password and file name are displayed. Host (mandatory field) = Enter the host of the FTP server. *Tip:* Use your plentymarkets FTP access. Port = Enter the port of the FTP server. If no port is entered, the standard port (22) is used. User name (mandatory field) = Enter the user name for the FTP access. Password (mandatory field) = Enter the password for the FTP access. File name = Enter the path and the name of the file from which the data is to be imported. The file name must end with .csv. *Note:* You can use wildcard characters in the field File name by e.g. replacing a part of the file name with a . Only use wildcard characters in combination with the option Move file to archive when import was successful. Move file to archive when import was successful* = If this option is activated, the import file is automatically moved to the new Archive folder in the FTP directory once the import was successful. HTTPS = Select External URL. External URL = Enter the URL to access the import file. *Tip:* Save the file on your plentymarkets Webspace. Authentication: Select whether an authentication is required to access the file. none = no authentication required HTTP authentication = = an authentication is required for each user and password SFTP = Host, port, user name, password and file name are displayed. Host (mandatory field) = Enter the host of the FTP server. Port (mandatory field) = Enter the port of the FTP server. If no port is entered, the standard port (22) is used. User name (mandatory field) = Enter the user name for the FTP access. Password (mandatory field) = Enter the password for the FTP access. File name = Enter the path and the name of the file from which the data is to be imported. The file name must end with .csv. *Note:* You can use wildcard characters in the field File name by e.g. replacing a part of the file name with a . Only use wildcard characters in combination with the option Move file to archive when import was successful. Move file to archive when import was successful* = If this option is activated, the import file is automatically moved to the new Archive folder in the FTP directory once the import was successful.

2. Setting the import options

The import options can be found in the Matching area. The import options apply both for the matching and for the mapping.

Setting the import options:

Go to Data » ElasticSync.
Click on the sync in the sync overview.
Click on Matching.
Determine the import options. Pay attention to the explanations given in Table 2.
Save () the settings.

Table 2. Import options for syncs
Import option	Explanation
Select an import option	Import new data, update existing = If data can be found by means of the matching fields, a new data record, e.g. a new item or a new order, is added. At the same time, existing data is updated. Only update existing data = Existing data is updated. If no data can be found by means of the matching fields, the data record is not imported. Only import new data = Only new data records are imported, i.e. any data for which no matching data is found based on the synchronisation fields.
Decimal number	Select German spelling or English spelling. German spelling = Decimal numbers are separated by a comma and a full stop serves as thousands separator. English spelling = Decimal numbers are separated by a full stop and a comma serves as thousands separator.
Date format	Enter the date format. If the date in the file corresponds to the standard format Y-m-d H:i:s, it is not necessary to enter another format. Other formats, however, must be entered here. All php-compliant date formats are allowed.

3. Setting the matching

After creating a sync, open it to carry out further settings. First, select the fields that should be used for matching the data. Determine what should happen if no existing data can be found during the matching process.

Matching is optional

Matching fields are optional settings. If you do not select a matching field, the data in your file cannot be assigned to existing data in your system. You can determine one, multiple or all matching fields. The displayed matching fields are different for each sync type.

Setting the matching:

Go to Data » ElasticSync.
Click on the sync in the sync overview.
Click on Matching.
Carry out the settings.
Save () the settings.

4. Creating a mapping

With a mapping, you specify which column of your file corresponds to which data field in your system. Without this mapping, no data can be imported.

Determining the mapping:

Go to Data » ElasticSync.
Click on the sync in the sync overview.
Click on Mapping.
Click on Add mapping ().
Enter a name for the mapping.
Enter a position for the mapping (optional).
Save () the settings.
→ The available mapping fields differ according to the sync type.
Select the data fields for the mapping by placing a checkmark in the fields.
→ The selected data fields are added to the mapping.
Select a column from your file for each data field or select the option Own value.
Save () the settings.

Own value

The option Own value allows to import a fix value for the data field. This fixed value is thus identical for each data record that is imported via this sync.

Mathematical calculations

In the mapping overview, select the option Mathematical calculations in order to adjust values in a CSV file during the import. The options addition (), subtraction (-), multiplication (*), division (/) and the adaptation of percentages ()/(-) are available. The value that is being adapted in the CSV column must be a number.

4.1. Activating lines of a mapping for the import

In the Mapping section, you can activate all lines for the import with just one click. Alternatively, you can decide individually for each line whether you want it to be active or inactive, thus, whether it should be considered in the import or not.

Activating lines for the import:

Go to Data » ElasticSync.
→ The overview opens.
Click on a sync.
→ The Mapping overview opens.
Click on a mapping.
→ The mapping opens.
On the left, click on Activate all in order to consider all lines in the import.
Alternatively, select a line and click on Activate line in order to just consider this line in the import.
Save () the settings.

View of the mapping

On the left side of the mapping you can see the import data of the CSV file that have to be mapped on the plentymarkets page. The plentymarkets data fields are shown on the right. The preview () displays the first 6 values of the CSV file.

4.2. Replacing values with the RegEx function

In order to consider a CSV file during the import as desired, the so-called RegEx function can be applied. Thanks to this function, data is manipulated during the import by using commands, so that the CSV file does not need to be changed anymore.

The following two examples will give an overview of the process.

Example 1:
Input: “24.12.2017”
Search pattern\: /^(\\d+).(\\d+).(\\d+)$/
Replace pattern\: $3﹣$2﹣$1
Result: “2017-12-24”

Example 2:
Input: “Doe, John”
Search pattern\: /^(\\w+), (\\w+)$/ +), (\\w)$/
Replace pattern\: $2 $1 +
Result: “John Doe”

Explanation of the examples:
Right at the beginning, there is the input. In a first step, the pattern to be searched for is inserted as search pattern. Afterwards, it is changed accordingly in the replace pattern, so that it is output during the import as required. Finally, the result is output as desired and will be considered in the CSV file.

Replacing values:

Go to Data » ElasticSync.
Click on a sync.
→ The mapping opens.
In the field Source, select the value Regular expression.
→ The options for Regular expression open.
In the field Value, enter the column name of the CSV file (e.g. Package number, Order ID etc.).
In the field Search, enter the Search pattern.
In the field Replace, enter the Replace pattern.
Save () the settings.
→ The data fields are updated.

Info about RegEx

For further information on RegEx see this page or have a look here. On the latter page, the RegEx engine setting has to be changed to PCRE. This can be done in the dropdown menu on the top right where Javascript is preset.

4.3. Own assignment

With the own assignment you can assign existing values of a CSV file to other values in the system. This enables to define, which value is supposed to be imported for which ID. In addition, for values in the CSV file that are not assigned, you can specify a default value. If no default value is specified, the original value is imported.

Defining an own assignment:

Go to Data » ElasticSync.
→ The overview opens.
Click on a sync.
→ The overview of the mapping opens.
Click on a mapping.
→ The mapping opens.
In the dropdown menu of the Source field, select the option Own assignment.
Fill in the fields. Pay attention to the explanations given in Table 3.
Save () the settings.

Table 3. Own assignment
Field	Setting
Type	Name of the CSV column (e.g. Delivery country)
Default value	Value, in case no assignment was found (e.g. 1)
From	Original value in the CSV file (e.g. DE for Germany)
To	Value, to which the original value shall be mapped (e.g. 1 for DE, 10 for GB etc.)

4.4. Manipulating an import file via the own assignment

In the following section we will demonstrate the usage of the own assignment shown by the example of Activating a shipping profile.
To do so, an item/a variation as well as a shipping profile has to exist.

Figure 1. Shipping profile

In order to link an item/a variation to a shipping profile, the ID is required. Just the name of the shipping profile shown in the import file is not sufficient to create a link.

Figure 2. Item ID

Use the own assignment in ElasticSync, which manipulates the data during the import. As source for the import data, select the value Own assignment.

Figure 3. Own assignment

For type, select the column from the import file. In this case, it is the column Shipping profile.

Figure 4. Select the file format of the import file

Determine which value in the import file shall be changed to which value in order to successfully carry out the import. In this case, DPD shall be changed to the ID of the shipping profile (here ID 7). You can also do this for several shipping profiles by adding further own assignments. Afterwards, carry out the import. The shipping profile is linked.

Figure 5. Determining a value

The own assignment can be used for many other cases. This applies in particular where internal IDs of plentymarkets play a role (e.g. characteristic names, item flags, payment methods, country ISO codes etc.).

5. Filtering import data

You can use filters to limit the imported data. You find the filters in the Mapping area.

Multiple filters are linked with an AND condition

If you use multiple filters, only those data records that apply to all filters are imported. Thus, the filters are linked with an AND condition.

Adding a filter:

Go to Data » ElasticSync.
Click on the sync in the sync overview.
Click on Mapping.
→ An overview of the mapping is displayed.
Open a mapping.
Click on Filter () in the toolbar.
Select the column from your file that you want to filter.
Select an operator.
Enter a value.
Click on Add filter ().
→ The filter is saved.
Add additional filters as necessary.
→ Only data that meets all filter criteria is imported.

6. Report of sync import

A report is available in the overview of a sync. This report shows all imported rows including the success and error messages, which can be opened by clicking on the Message column. Use the filter in order to retrieve specific sync data.

7. Copying syncs

You can copy one sync or multiple syncs at the same time. That way, you can use syncs that you have already configured in order to quickly create a new sync.

Copying syncs:

Go to Data » ElasticSync.
Select one or multiple syncs.
Click on Copy sync.
→ The sync is copied. (copy) is added behind a copied sync.

8. Exporting and importing a sync

You can import and export syncs. Furthermore, you can save syncs locally and import syncs from other users.

Exporting and importing a sync:

Go to Data » ElasticSync.
Select one or multiple syncs.
Click on Export sync ().
→ The sync is automatically downloaded.
Click on Import sync ().
→ The file overview opens.
Click on Upload file () and select a file.
Place a check mark next to the file that you would like to upload and click on Select.
→ A message appears checking whether you want to import the sync.
Click Ok in order to confirm.
→ A message appears explaining that the import originates from another client.
→ The sync has been imported successfully.

Importing syncs from another plentymarkets system

If you would like to import syncs from another plentymarkets system, you have to adjust the matching and mapping settings. It is only possible to import .json files.

Importing and updating syncs

If you want to import and update a sync, a mapping and a matching are required. Otherwise, it is not possible to test or run a sync. In order to save the sync, a data field has to be activated in the mapping overview.
If you just want to import a sync, a mapping alone is sufficient.

9. Uploading files quickly and easily

In order to quickly and easily upload files, you can use the upload button in the File name column.

Uploading files quickly and easily:

Go to Data » ElasticSync.
In the File Name column, click on the button Upload file ().
→ The window My files opens.
Select a file and click on Select.
→ The file is uploaded and the file name is displayed in the column. After running the sync, the file name disappears.

10. Testing the sync and importing data

It is possible to test and carry out syncs. Testing allows you to quickly know whether your file can be imported correctly.

Testing the sync

If you select Test sync, only the first ten lines of the import file are imported.

11. Processing jobs in the queue

The queue enables the import of large amounts of data. Before being imported, the import file is broken down into packages and processed in so-called jobs. A job consists of 750 lines of a CSV file. However, import files with less than 750 lines also make a job. Whenever a sync is started, the CSV file is split up into the respective jobs and is put into the queue. In the queue, new jobs are permanently retrieved and the syncs are processed. There are two different statuses. The status Open jobs means that there are jobs in the queue waiting to be processed. Completed jobs on the contrary have already been processed and the data have been imported.

12. Activating/deactivating mappings

If you manage several mappings in the same sync, you can deactivate or activate specific mappings of the sync in the mapping overview by clicking on the button in the column Active/Inactive. New mappings are active by default.