Importing data with ElasticSync

This manual page describes how to import data using ElasticSync. Importing data is recommended if you want to add new items, customer data or stock etc. to your plentymarkets system and you already have this data available in digital format. ElasticSync allows to carry out data imports by creating so-called syncs. You can import your data manually or automatically. Watch this video and learn about the basics for importing data with ElasticSync.

Get an overview of the tool ElasticSync

1. Assigning user rights for back end users

Users of the type Back end can only access limited areas and menus in the plentymarkets back end. Thus, their access to the system is limited. If users of this type should be able to work with syncs and mappings, a user of the type Admin has to assign the rights listed in Table 1 to those users.

Assigning user rights for back end users:

Go to Setup » Settings » User » Rights » User.
Use the search function () and open the user account that should be edited.
In the Authorisations section, activate the rights as listed in Table 1.
Save () the settings.

Table 1. Assigning rights for the **Data » Import** menu
Setting	Explanation
Data > sync > create	Authorises the back end user to create syncs.
Data > sync > delete	Authorises the back end user to delete syncs.
Data > sync > show	Authorises the back end user to show syncs.
Data > sync > update	Authorises the back end user to update syncs.
Data > sync > mapping > create	Authorises the back end user to create mappings.
Data > sync > mapping > delete	Authorises the back end user to delete mappings.
Data > sync > mapping > show	Authorises the back end user to show mappings.
Data > sync > mapping > update	Authorises the back end user to update mappings.
Data > sync > data log	Authorises the back end user to create, delete, show or update the data log of a sync.

2. Creating a sync

The steps for creating and importing syncs are identical for all sync types. As such, the steps described in the following apply to all sync types.

Go to Data » Import to create, edit and manage syncs. Here, you can see an overview of all syncs. With a sync, you can assign the data of a file that is accessible via FTP or HTTPS to the data in plentymarkets and import them. When creating a sync, you save information about the data source. Furthermore, it is important that you determine data for the matching and that you assign which data of your file corresponds to the data in plentymarkets. Use the matching to set what data is to be used to try to identify, if a data record already exists or not. Use the mapping to set which column in your file corresponds to which data field in plentymarkets. In the end, set an import interval. Proceed as described below to create a sync.

Set the import interval in the end to avoid that a sync is carried out too early

It is recommended to set the import interval in the end. By doing so, you avoid that a sync is already carried out while you still carry out matching and mapping settings.

Creating a sync:

Go to Data » Import.
→ The sync overview opens.
Click on Add sync ().
→ The editing area New sync is displayed.
Carry out the settings. Pay attention to the explanations given in Table 2.
Save () the settings.
→ The sync is created.
→ The matching settings of the sync open.

Table 2. Creating a sync
Setting	Explanation
Basic data
Name	Enter a name. This name is displayed in the overview and can be used for the search.
Interval	Select an interval for the import of syncs. By choosing the option Schedule you can determine how often and at what time the import of syncs shall be started.
Type	Select the type from the drop-down list. Item = Select to import item and variation data. Attribute = Select to import attribute data. Orders = Select to import order data. Properties = Select to import properties. Facets = Select to import facets. Parts compatibility listing = Select to import parts compatibility listing data. Coupon codes = Select to import coupon codes. Manufacturer = Select to import manufacturer data. Campaigns = Select to import coupon campaigns. Categories = Select to import category data. Contacts, companies and addresses = Select to import contact, company and address data. Warehouse = Select to import storage locations. Characteristic = Select to import characteristic data. Newsletter recipient = Select to import newsletter data. Notes = Select to import notes. Stock = Select to import stock data. Incoming items = Select to import incoming item data.
Import
Data type	Select the file format of the import file. Currently, CSV/TXT is the only available option, as only CSV or TXT files can be imported.
Character set	Select the character set of the file or the option Detect automatically.
Starting in line	Select a line starting from which the data should be imported.
Header is included	Select whether the import file contains a header.
Use global cache	Determine whether the global cache should be used for a file or not. If the setting is activated, only modified data rows will be imported. If the setting is not activated, you can import one and the same file again at any time.
Automatic mapping	Select whether the mapping should be done automatically. The columns in the CSV or TXT file can be mapped automatically to the plentymarkets data fields. To ensure that the mapping works properly, the names of the fields that you selected in the FormatDesigner must not be changed.
Separator
Separator	The separator of the CSV or TXT file is automatically detected. This setting is activated by default. Alternatively, you can manually select the field separator that is set in the import file in order to separate the data fields (columns) of a data record from each other. Possible separators are Semicolon, Comma, Tab and Pipe.
Delimiter	Select delimiters within the fields that may contain multiple values. * " = individual values within one field are separated by double quotation marks. * ' = individual values within one field are separated by single quotation marks.
Source
Source	Select Manual upload, FTP, HTTPS or SFTP. Depending on the selected option, the following settings are displayed: Manual upload = Select and upload a local file. After running a sync, the file is automatically deleted from the webspace. This is preset by default. In order to save the file you need to remove the check mark. *Note: storing the file on the webspace is subject to a fee. FTP* = Host, port, user name, password and file name are displayed. Host (mandatory field) = Enter the host of the FTP server. *Tip:* Use your plentymarkets FTP access. Port = Enter the port of the FTP server. If no port is entered, the standard port (22) is used. User name (mandatory field) = Enter the user name for the FTP access. Password (mandatory field) = Enter the password for the FTP access. File name = Enter the path and the name of the file from which the data is to be imported. The file name has to end with .csv or .txt. *Note:* You can use wildcard characters in the field File name by e.g. replacing a part of the file name with a . Only use wildcard characters in combination with the option Move file to archive when import was successful. Move file to archive when import was successful* = If this option is activated, the import file is automatically moved to the new Archive folder in the FTP directory once the import was successful. HTTPS = Select External URL. External URL = Enter the URL to access the import file. *Tip:* Save the file on your plentymarkets webspace. Authentication: Select whether an authentication is required to access the file. none = no authentication required HTTP authentication = = an authentication is required for each user and password SFTP = Host, port, user name, password and file name are displayed. Host (mandatory field) = Enter the host of the FTP server. Port (mandatory field) = Enter the port of the FTP server. If no port is entered, the standard port (22) is used. User name (mandatory field) = Enter the user name for the FTP access. Password (mandatory field) = Enter the password for the FTP access. File name = Enter the path and the name of the file from which the data is to be imported. The file name has to end with .csv or .txt. *Note:* You can use wildcard characters in the field File name by e.g. replacing a part of the file name with a . Only use wildcard characters in combination with the option Move file to archive when import was successful. Move file to archive when import was successful* = If this option is activated, the import file is automatically moved to the new Archive folder in the FTP directory once the import was successful.

3. Setting the import options

The import options can be found in the Matching area. They apply both for the matching and for the mapping.

Before we get to the specific settings of the import options you now have the opportunity to learn more about the data import by watching the following videos. As already mentioned, data can be imported manually or automatically. If you are new to plentymarkets and you want to manually import your data one time, watch this video and learn how to initially transfer product or customer data into your system.

Manually import you data one time

This video shows you how to instruct plentymarkets to automatically import new files and archive old ones.

Automatically import new files and archive old ones

Back to the import options.

Setting the import options:

Go to Data » Import.
→ The sync overview opens.
Click on the sync in the sync overview.
→ The settings open.
Switch to the Matching tab.
Determine the import options. Pay attention to the explanations given in Table 3.
Save () the settings.

Table 3. Import options for syncs
Import option	Explanation
Select an import option	Import new data, update existing = If data can be found by means of the matching fields, a new data record, e.g. a new item or a new order, is added. At the same time, existing data is updated. Only update existing data = Existing data is updated. If no data can be found by means of the matching fields, the data record is not imported. Only import new data = Only new data records are imported, i.e. any data for which no matching data is found based on the synchronisation fields.
Decimal number	Select German spelling or English spelling. German spelling = Decimal numbers are separated by a comma and a full stop serves as thousands separator. English spelling = Decimal numbers are separated by a full stop and a comma serves as thousands separator.
Date format	Enter the date format. If the date in the file corresponds to the standard format Y-m-d H:i:s, it is not necessary to enter another format. Other formats, however, have to be entered here. All php-compliant date formats are allowed.

4. Setting the matching

After creating a sync, open it to carry out further settings. First, select the fields that should be used for matching the data. Determine what should happen if no existing data can be found during the matching process.

Matching is optional

Matching fields are optional settings. If you do not select a matching field, the data in your file cannot be assigned to existing data in your system. You can determine one, multiple or all matching fields. The displayed matching fields are different for each sync type.

Setting the matching:

Go to Data » Import.
→ The sync overview opens.
Click on the sync in the sync overview.
→ The settings open.
Switch to the Matching tab.
Carry out the settings.
Save () the settings.

5. Creating a mapping

With a mapping, you specify which column of your file corresponds to which data field in your system. Without this mapping, no data can be imported. You can create new mappings in the mapping tab. A position is automatically assigned to each new mapping and can be adjusted in the settings tab if necessary.

Creating a mapping:

Go to Data » Import.
→ The sync overview opens.
Click on a sync.
→ The settings open.
Switch to the Mapping tab.
Click on Add mapping ().
→ A window opens.
Enter a name for the mapping.
Save () the settings.
→ The mapping opens. → The available mapping fields differ according to the sync type.
Select the data fields for the mapping by placing a checkmark in the fields.
→ The selected data fields are added to the mapping.
Select a column from your file for each data field or select the option Own value.
Save () the settings.

You want to import a file into plentymarkets, but it’s missing a few key pieces of information? No need to modify the file by hand! This video shows you how to use ElasticSync’s "Own Value" function to add missing information to your sync.

Use the Own value to add missing information to your file

Own value

The option Own value allows to import a fix value for the data field. This fixed value is thus identical for each data record that is imported via this sync.

Mathematical calculations

In the mapping overview, select the option Mathematical calculations in order to adjust values in a CSV or TXT file during the import. The options addition (+), subtraction (-), multiplication (*), division (/) and the adaptation of percentages (+)/(-) are available. The value that is being adapted in the CSV or TXT column has to be a number.

5.1. Activating lines of a mapping for the import

In the Mapping section, you can activate (and also deactivate) all selected lines for the import with just one click. Alternatively, you can decide individually for each line whether you want it to be active or inactive, thus, whether it should be considered in the import or not.

Activating lines for the import:

Go to Data » Import.
→ The sync overview opens.
Click on a sync.
→ The settings open.
Switch to the Matching tab.
Click on a mapping.
→ The mapping opens.
On the left, select all lines you want to consider in the import.
At the top, click on Activate all (). By clicking on Deactivate all (), you can deactivate all selected lines again.
Alternatively, click on Activate row () on the right to activate lines for the import individually.
Save () the settings.

View of the mapping

On the left side of the mapping you can see the import data of the CSV or TXT file that have to be mapped on the plentymarkets page. The plentymarkets data fields are shown on the right. The preview () displays the first 6 values of the file.

Editing position and name

By clicking on Settings () in the mapping, the fields Name and Position become visible and can be edited.

5.2. Activating/deactivating mappings

If you manage several mappings in the same sync, you can deactivate or activate specific mappings of the sync in the mapping overview by clicking on the toggle button in the column Active/Inactive. New mappings are active by default.

5.3. Replacing values with the RegEx function

In order to consider a CSV or TXT file during the import as desired, the so-called RegEx function (Regular Expression) can be applied. Thanks to this function, data is manipulated during the import by using commands, so that the CSV or TXT file does not need to be changed anymore. Watch this video to learn more about the RegEx function.

Modify your data with the RegEx function

The following two examples will give an overview of the process.

Example 1:
Input: “24.12.2017”
Search pattern\: /^(\\d+).(\\d+).(\\d+)$/
Replace pattern\: $3﹣$2﹣$1
Result: “2017-12-24”

Example 2:
Input: “Doe, John”
Search pattern\: /^(\\w+), (\\w+)$/ +), (\\w)$/
Replace pattern\: $2 $1 +
Result: “John Doe”

Explanation of the examples:
Right at the beginning, there is the input. In a first step, the pattern to be searched for is inserted as search pattern. Afterwards, it is changed accordingly in the replace pattern, so that it is output during the import as required. Finally, the result is output as desired and will be considered in the CSV or TXT file.

Replacing values:

Go to Data » Import.
→ The sync overview opens.
Click on a sync.
→ The settings open.
Switch to the Matching tab.
Click on a mapping.
→ The mapping opens.
In the field Source, select the value Regular expression.
→ The options for Regular expression open.
In the field Value, enter the column name of the CSV or TXT file (e.g. Package number, Order ID etc.).
In the field Search, enter the Search pattern.
In the field Replace, enter the Replace pattern.
Save () the settings.
→ The data fields are updated.

Info about RegEx

For further information on RegEx see this page or have a look here. On the latter page, the RegEx engine setting has to be changed to PCRE. This can be done in the dropdown menu on the top right where Javascript is preset.

5.4. Own assignment

With the own assignment you can assign existing values of a CSV or TXT file to other values in the system. This enables to define, which value is supposed to be imported for which ID. In addition, for values in the CSV or TXT file that are not assigned, you can specify a default value. If no default value is specified, the original value is imported.

Defining an own assignment:

Go to Data » Import.
→ The sync overview opens.
Click on a sync.
→ The settings open.
Switch to the Matching tab.
Click on a mapping.
→ The mapping opens.
In the dropdown menu of the Source field, select the option Own assignment.
Fill in the fields. Pay attention to the explanations given in Table 4.
Save () the settings.

Table 4. Own assignment
Field	Setting
Type	Name of the CSV or TXT column (e.g. Delivery country)
Default value	Value, in case no assignment was found (e.g. 1)
From	Original value in the CSV or TXT file (e.g. DE for Germany)
To	Value, to which the original value shall be mapped (e.g. 1 for DE, 10 for GB etc.)

5.5. Manipulating an import file via the own assignment

In the following section we will demonstrate the usage of the own assignment shown by the example of Activating a shipping profile.
To do so, an item/a variation as well as a shipping profile has to exist. The following video gives you an introduction on how the Own assignment works.

Use the Own assignment to trade one piece of information for another

Figure 1. Shipping profile

In order to link an item/a variation to a shipping profile, the ID is required. Just the name of the shipping profile shown in the import file is not sufficient to create a link.

Figure 2. Item ID

Use the own assignment in ElasticSync, which manipulates the data during the import. As source for the import data, select the value Own assignment.

Figure 3. Own assignment

For type, select the column from the import file. In this case, it is the column Shipping profile.

Figure 4. Select the file format of the import file

Determine which value in the import file shall be changed to which value in order to successfully carry out the import. In this case, DPD shall be changed to the ID of the shipping profile (here ID 7). You can also do this for several shipping profiles by adding further own assignments. Afterwards, carry out the import. The shipping profile is linked.

Figure 5. Determining a value

The own assignment can be used for many other cases. This applies in particular where internal IDs of plentymarkets play a role (e.g. characteristic names, item flags, payment methods, country ISO codes etc.).

6. Filtering import data

You can use filters to limit the imported data. You find the filters in the Mapping area.

Multiple filters are linked with an AND condition

If you use multiple filters, only those data records that apply to all filters are imported. Thus, the filters are linked with an AND condition.

Adding a filter:

Go to Data » Import.
→ The sync overview opens.
Click on the sync in the sync overview.
→ The settings open.
Switch to the Matching tab.
Click on a mapping.
→ The mapping opens.
Click on Filter () in the toolbar.
Select the column from your file that you want to filter.
Select an operator.
Enter a value.
Click on Add filter ().
→ The filter is saved.
Add additional filters as necessary.
→ Only data that meets all filter criteria is imported.

7. Error messages and troubleshooting in ElasticSync

ElasticSync offers you various possibilities to identify incorrect syncs beforehand and to perform troubleshooting. Thus, you can for example check syncs for errors before running them or you can view a log of concrete error messages once the sync has been imported. The different possibilities are listed below.

7.1. Checking syncs for errors before the run

With the validation, syncs can already be checked for errors before the run to guarantee as smooth import.

Checking syncs for errors before the run:

Go to Data » Import.
→ The sync overview opens.
Click on a sync.
→ The settings open.
In the settings, matching or mapping tab, click on Validation ().
→ A window opens and shows a table with the respective error messages.
In case an error occurred and you need help in the forum, click on Copy to clipboard () and paste the information in the forum.

7.2. Testing the sync and importing sample data

In the sync overview you can test () and run () syncs. Testing allows you to quickly know whether your file can be imported correctly.

Testing the sync

If you select Test sync, only the first ten lines of the import file are imported.

7.3. Viewing the log of a completed sync

A log listing errors that may have occurred is automatically created for each completed sync.

Viewing the log of a completed sync:

Go to Data » Status.
→ The status overview opens.
Or: Go to Data » Import.
→ The sync overview opens.
Click on Status ().
→ The status overview opens.
Click on a completed sync.
→ The detail view opens.
At the top, click on Log ().
→ A window opens and shows the log of the completed sync.
In case an error occurred and you need help in the forum, click on Copy to clipboard () and paste the information in the forum.

7.4. Viewing success and error messages of syncs

In the Data » Log menu you can see success and error messages of syncs. To find messages of specific syncs, select the reference type elasticSyncId or elasticSyncName and enter the respective reference value. The specific success or error messages can be found in the Message column.

8. Copying syncs

You can copy one sync or multiple syncs at the same time. That way, you can use syncs that you have already configured in order to quickly create a new sync.

Copying syncs:

Go to Data » Import.
→ The sync overview opens.
Select one or multiple syncs.
Click on Copy sync.
→ The sync is copied. (copy) is added behind a copied sync.

9. Importing and exporting a sync

You can import and export syncs. Furthermore, you can save syncs locally and import syncs from other users. This video shows how to use preconfigured syncs in order to safe time when importing data.

Import and export your syncs

Importing and exporting a sync:

Go to Data » Import.
→ The sync overview opens.
Select one or multiple syncs.
Click on Export sync ().
→ The sync is automatically downloaded.
Click on Import sync ().
→ The file overview opens.
Click on Upload file () and select a file.
Place a check mark next to the file that you would like to upload and click on Select.
→ A message appears checking whether you want to import the sync.
Click Ok in order to confirm.
→ A message appears explaining that the import originates from another client.
→ The sync has been imported successfully.

Importing syncs from another plentymarkets system

If you would like to import syncs from another plentymarkets system, you have to adjust the matching and mapping settings. It is only possible to import .json files.

Importing and updating syncs

If you want to import and update a sync, a mapping and a matching are required. Otherwise, it is not possible to test or run a sync. In order to save the sync, a data field has to be activated in the mapping overview.
If you just want to import a sync, a mapping alone is sufficient.

10. Cancelling a sync during the import

Under Data » Status you can cancel a sync import that has already been started. Thus, this function is only available if the sync is currently being processed.

Cancelling a sync during the import:

Go to Data » Status.
→ The status overview opens.
Or: Go to Data » Import.
→ The sync overview opens.
Click on Status ().
→ The status overview opens.
At the very right in the line of the sync that you want to cancel, click on Cancel import ().
→ A window opens and you need to confirm your decision.
Click on Cancel import.
→ The import is cancelled and only those rows that had already been imported when cancelling are in the system.

11. Uploading files quickly and easily

In order to quickly and easily upload files, click on File upload () in the File name column.

Uploading files quickly and easily:

Go to Data » Import.
→ The sync overview opens.
In the File name column, click on Upload file ().
→ Your files open.
Select a file and click on Open.
→ The file is uploaded and the file name is displayed in the column. After running the sync, the file name disappears.

12. Processing jobs in the queue

The queue enables the import of large amounts of data. Before being imported, the import file is broken down into packages and processed in so-called jobs. A job consists of 750 lines of a CSV or TXT file. However, import files with less than 750 lines also make a job. Whenever a sync is started, the file is split up into the respective jobs and is put into the queue. In the queue, new jobs are permanently retrieved and the syncs are processed. There are two different statuses. The status Open jobs means that there are jobs in the queue waiting to be processed. Completed jobs on the contrary have already been processed and the data have been imported.

13. Viewing the status of syncs

The status overview of the syncs shows the current progress of sync jobs and displays details of errors that may have occurred during the import process.

Viewing the status of syncs:

Go to Data » Status.
→ The status overview opens.
Or: Go to Data » Import.
→ The sync overview opens.
Click on Status ().
→ The status overview opens.

In each sync line, there is also a separate status button you can use to open the status of this specific sync.

Table 5. Sync status
Setting	Explanation
Run ID	Each time an import of a sync is started, a Run ID is created in order to better distinguish between the different imports.
Sync-ID	ID of the sync
Name	Name of the sync
Progress	Progress of the sync jobs. As soon as all jobs have been completed, the date and end time are displayed.
Error	Errors that have been occurred in the sync jobs
Imported rows	Number of lines that have been imported from the CSV or TXT files
Total rows	Total number of lines in the CSV or TXT file
Start time	Start time of the sync

Cancellation when too many errors have occurred

In case too many errors occur during the processing of sync jobs, the process is cancelled and an error message appears.

13.1. Filtering for syncs

By using the filter in the status overview, you can filter for syncs of the last 7 days and thus check the status of specific syncs.

Table 6. Filter options
Setting	Explanation
Sync-ID	ID of the sync
Name	Name of the sync
Status	Current status of the sync. Select between In progress or Completed.
Start date from	Select the date as of which you want to filter.
Start time from	Select the time as of which you want to filter. The start time refers to the start of the sync.
Start date to	Select the date up to which you want to filter.
Start time to	Select the time up to which you want to filter. The start time refers to the start of the sync.

13.2. Detail view of sync errors

In the status overview, you can click on a sync to access the detail view where you will find further information on errors that may have been occurred. By clicking on a line of an error in the detail view, a new window opens and displays the name of the table header, the raw data as well es the concrete error message.

Table 7. Details of sync status
Setting	Explanation
Line	Line where the error occurred
Column name	Name of the column where the error occurred
Error	Concrete description of the occurred error
Raw data	Raw data of the error

14. Restoring backups of syncs

If you want to reset sync data to a previous state or restore deleted sync data, you can do so by using the backup function.

Restoring backups of syncs:

Go to Setup » Settings » Data » Backup.
→ The backup overview opens.
In the line of the backup that you want to import, click on Restore backup () on the right.
At the top, select Partially import data base and tick the box ElasticSync.
Click on Restore backup.
→ The backup is restored.