Improving the performance of WCS full reindexing

Tips to improve performance of WCS preprocessing


Doing a full re-indexing in WCS could be painful if we have more products and catalogs. If a website has around 100000 products and 60 catalogs it can take unto 4 hours if not properly tuned. This is one of the pain points of many organisations.

There are parameters which we can tune in order to achieve better performance of reindexing job.

Commerce reindexing has two steps. preprocess and build index. Preprocess populates the temporary tables from the base tables and build index will populate the SOLR indexes from these temporary tables. On analysing the job we can see that the main time consumer is preprocessing. The below parameters can be tuned to make preprocessing faster.

1. Increase the batchSize

           Increasing the batch size will make the job to process more data in a single batch. The default value of the batchSize parameter will be 500. This is not sufficient for organisations with more products and catalogs. Tune this to a higher value to get a better performance. 100000 is a good value to start with.

          The change needs to be done in the preprocess xmls located in 

          WC_InstllDir\search\pre-processConfig\MC_10001\<DB>

          

          open the xml and change the batchSize="500" to a desired value

    2. Increase the java heap allocated

          

        When the batch size is increased it would require more memory to  process the data. In case if we don't increase the heap size it is very likely to hit an OutOfMemoryException. Also increasing the allocated heap will improve the performance of the job as well. 

        Preprocessing is achieved by calling WC_InstllDir\bin\di-preprocess.sh. Open the script                       and edit the below line 

        "${JAVA_EXE?} -Xms256m  -Xmx1024m" to a higher value. 

          setting minimum heap as 1024 and maximum as 3072 should suffice.

         "${JAVA_EXE?} -Xms1024m  -Xmx3072m"

Note : Make sure that you have enough memory in the box before allocating. 

  3. Reduce the number of custom tables

        It is a normal requirement that we would need to index custom data in SOLR and hence will need new tables. It is better to design it in such way that the total number of custom temporary tables created during pre-process job is minimised.

For example, If we are maintaining a flag of product per esite and want this in SOLR,  There are two ways to do it. One is to create a table per esite which would look like below.

Table : XI_Esite1

CATENTRY_ID          CUSTOMFLAG

============          ============

12458                            J

14257                            L

14789                            Y

Table : XI_Esite2

CATENTRY_ID          CUSTOMFLAG

============          ============

12458                            N

14857                            O

14789                            Y

With this approach pre-process job has to create table for all the esite for the same catentrys.

The better approach would be to create a single table with multiple coulumns. Something like below

Table : XI_CUSTOMFLAG

CATENTRY_ID    CUSTFLAG_ESITE1  CUSFLAG_ESITE2

============    ================  ===============

12458                                  J                                   N

14257                                  L                                  NULL

14789                                  Y                                  Y

14857                                  O                                  NULL 

This will help in better performance of buildIndex as well as it will try to do a join on all the custom tables.