Klevu is a powerful tool used by online retailers to improve their search experience and boost sales. One of the main purposes of using Klevu here is to map the customized data from our product JSON to XML and pass it to the Klevu XML format to get the particular data in Klevu API search, PDP list data, and images.
Problem statement
Klevu has integration with magento but here we’re going to use the XML feed integration due to the customized different types of data in JSON and URLs for media in different servers, so it helps us to combine all the data in one XML file and send it to the Klevu system to sync the data in Klevu server.
Klevu Feed Integration
What is Klevu Feed Sync/synchronization?
The Klevu Feed is a method of providing your store data to be synced with Klevu. You will host an XML file and keep it up to date as your catalog changes. Klevu will then download this XML file from time to time to update the search results that your customers see.
Klevu Feed Usage
For example, if Monday’s feed includes Product A, Klevu will index that product and make it available in the search results. If on Tuesday the feed does not contain Product A, it will be removed from Klevu and will no longer appear in search results.
What is Magento to KlevuFeed Sync?
Magento to Klevu sync is done through XML files. The XML files were generated through some CLI Commands in Magento and CRON jobs as well. We created XML files with the help of Klevu’s sample XML files.
For practical understanding please refer to this link for more details.
Product XML Feed
The product XML file contains the below product data. If any data is not available then it is called blank (some will not be included in the XML, For Example, if UPC is not available for a product then it will not be included in the feed for that particular product).
Example of Product Data Mapping
| Klevu Attribute | Magento Attribute | is_facet | is_searchable | is_return |
| item_type | KLEVU_PRODUCT | null | null | TRUE |
| title | name | null | TRUE | TRUE |
| SKU | SKU | null | TRUE | TRUE |
| id | id | null | null | TRUE |
| short_description | short_description | null | TRUE | TRUE |
| description | description | null | TRUE | TRUE |
| link | product URL | null | null | TRUE |
| availability | In Stock | null | null | TRUE |
| price | price | null | null | TRUE |
| sale_price | price | null | null | TRUE |
| url_key | URL | FALSE | FALSE | TRUE |
Category XML Feed
The categories data was created as an XML file and indexed with Klevu. The mapping of category data will be as shown in the table below.
| Klevu Attribute | Magento Attribute |
| item_type | KLEVU_CATEGORY |
| id | c.’category_id’ |
| title | name |
| link | Category URL |
| image_link | category image URL |
XML File Creation
The product and category XML file creation happens in 3 methods.
- Manual Creation
- Magento CRON Job
- Automatic Creation After PIM Sync
Method 1 – Manual Creation
We can create XML files using CLI commands whenever necessary in this method. This manual option is primarily used while conducting bulk product synchronization from PIM to Magento. We can generate the XML file using the following command to compile the entire product data into XML based on the updated date.
php bin/magento dckap:klevu_feed --updated_date=2024-01-25We can modify the update_date value. If any of the products are updated from that date those product data will be generated as XML data.
All the product data is split into batch files. One batch file contains a maximum of 100K products. Based on the total product count the number of batch files will be defined.
For example:
The first 100k products are in klevu_product1.xml
The second 100k products are in klevu_product2.xml
In this manner, we will generate all the product XML files. Klevu will access these files through the public URL https://yourdomian.com/klevu_product1.xml
Below is the category XML feed creation CLI command. It creates all categories into one XML file.
php bin/magento dckap:klevu_feed --type=categoryAll the category data are created in the klevu_category.xml file
Sample of category XML file: https://yourdomian.com/klevu_category.xml
Once the XML files are created successfully, it automatically indexes by Klevu (if we already indexed those files with Klevu) on the next index process time. Klevu index source files every hour.
Note: Every hour Klevu calls the XML feed URLs and indexes the product data. This will be the default configuration done by the Klevu support team.
If we need to sync immediately, then we can use the demand sync URL which is provided by Klevu. Klevu Feedmonito should run this URL with the necessary API key from the particular store from Klevu KMC.
Refer to the below image to get the API key from Klevu KMC.
Note: We have to send the XML files list to the Klevu support team (support@klevu.com) with a public URL, where those files are generated and helped to index the product and category data in Klevu.
Method 2 – Magento CRON Job
We have already set up a CRON job in Magento to run every hour and it checks the Magento database and generates an XML file (if there is a product data update that happened in the past hour) to update the Klevu product data. All the updated products will be generated in a single file
Method 3 – Automatic Creation After PIM Sync
After the sync is completed from PIM to Magento, then Magneto triggers the klevu_updated_product.xml XML file creation to add the updated product data and it will be processed by Klevu on the next indexing schedule. This file URL also needs to be sent Klevu support team to index the updated product data.
Note: We were maintaining a flag while this method was in progress, so the automatic Magento CRON job will be ignored.
Product Feed Sync Process on the Magento Side
If the products are updated in Magento, we need to update the Klevu feed XML to synchronize the updated product data in Klevu. Dynamically retrieve the updated products from the collection, then update the product’s XML file accordingly.
For single product example:
From id and to id denote the row id of the product.
php bin/magento dckap:klevu_xmlfeed:product --from_id=322270 --to_id=322271After running this command check the particular product XML file, whether it updated the data or not in the XML file.
Then run the merge command: (create one sh file to run these commands according to the product folders.
#remove existing merge file from the server
rm -rf /var/www/html/pub/products_XML/klevu_product_items_1.xml #merge all product XML into a single file
for f in $(ls /var/www/html/pub/products_XML/1/ -tr); do cat "/var/www/html/pub/products_XML/1/$f" >> /var/www/html/pub/products_XML/klevu_product_items_1.xml; done#merge header and footer
cat /var/www/html/pub/header.xml /var/www/html/pub/products_XML/klevu_product_items_1.xml /var/www/html/pub/footer.xml > /var/www/html/pub/klevu_product1_cp.xml #create a backup and change the source file
mv /var/www/html/pub/klevu_product1_cp.xml /var/www/html/pub/klevu_product1.xml
php /var/www/html/replaceXml.php /var/www/html/pub/klevu_product1.xmlThe Final Step is to Sync with Klevu
Create one demand sync sh file to sync the updated products with Klevu and set the cron time also for the sync.
Example
curl https://feedmonitor3.klevu.com/feed-monitor/onDemandSync?searchAPIKey=your-klevu-api-key
curl https://feedmonitor3.klevu.com/feed-monitor/onDemandSync?searchAPIKey=your-klevu-api-key-1
Note: We were maintaining a CRON job for completing the sync process. The updated fields only sync with Kelvu for updating the product data, not the entire product data.
Reference links :
https://www.klevu.com/support-services/
https://help.klevu.com/support/solutions/articles/5000871252-integration-steps-for-magento-2
