Configuring Batch Processing

• Click on three dots () on the right-hand corner of the Workflow landing page. The batch processing option is displayed.
• Click on batch processing option, and the following popup is displayed.
  
  
  
  
  

• The default chunk size is 1000.  The batches are created depending on the chunk size.
• The data page will be displayed as follows
  
  
  
  

• The records are processed as per mentioned batch size. When you explore the output, you will see all the processed data together. 
  
• The data is processed internally in the batches and hence is applicable for the tasks wherein batch operations can be handled.
  
• The tasks which are not supporting batch execution will still execute with the regular execution.

Advantages

• Processing becomes faster. It improves the speed of many ETL operations like Missing value Imputation, expression, Cleansing, and model testing.
• The parallel processing becomes faster. Consider the following pipeline.
  • Dataset reads the data from the file.
  • You build an expression and save it in the output file in one process.
  • In another process you build the Model.
  • When you apply batch processing, both processes run concurrently.
  • Following figure explains the parallel processing.

• You are allowed to explore the model when it is processing.

Limitations

• Apply Batch processing only on pipeline.
• Don’t apply Batch processing in the case of entire column operations like average, and totals.
• Batch processing is not allowed on the datasets generated using the following techniques
  • SSAS RDBMS
  • Twitter
  • JSON file format
  • Google news