Today’s digital work processes often require moving large volumes of data in batches from one application to another — let’s say, from a file containing sales leads into a marketing automation platform, or building ETL pipelines for a data warehouse.
Occasionally the upstream app publishing data has a larger batch size than the batch size of the downstream app consuming the data. When there’s a mismatch, complex logic is required to manage the data flow.
Challenges with managing mismatch in batch sizes
It typically requires complex logic to take the input from the upstream application and convert it into smaller batches for the downstream applications. The resizing of the batch requires looping through the upstream application data one row at a time, and composing the batch for the downstream application.
This results in inefficient processing when the batch sizes are large, and also requires special error handling to ensure data integrity (i.e. no duplicate data across batches). Workato’s recipes make it easier to set up this logic, but there was still some burden on the creation and maintenance of recipes.
Taking the work out of batch processing
The newly introduced “Repeat in Batches” feature lets you easily process data in batches across applications without having to deal with complex logic.
To control the data flow, users can configure the batch size (from one row to thousands). The Repeat in Batches feature will automatically parse large data sets into smaller batches for transfer, optimizing throughput.
Watch a short demo of this new feature.
Repeat in Batches: Step-by-step instructions
Let’s see this feature at work. In this example, we are loading sales lead data from a CSV file on Amazon S3 into Salesforce. The CSV file contains 50,000 leads, but Salesforce can only ingest 200 leads at a time.
First, we’ll create a new recipe that triggers on a specified interval (drawing on the Scheduler by Workato app). This recipe is set up so that once a week all lead data from Amazon S3 will be pulled (though this pull schedule could be as often as every 5 minutes).
Start the recipe by specifying the source of the data to be extracted
Next, select the “CSV by Workato” app and use the “Parse CSV” action. This action allows you to parse the data from S3 into lines that you can batch.
Use the “CSV by Workato” app to parse the CSV file into rows
This next step is where the new feature comes in. Choose the “Repeat Action” option and select the input as the “lines” from the parsed CSV (i.e. the output from Step 3). Then, select “Batch of items” in the “Repeat mode” dropdown menu. This seemingly simple option contains powerful logic that takes care of everything needed to moderate the data stream. All you have to do at this point is select the batch size.
The “Batch of items” selection is what kicks off the “Repeat in Batches” action
Finally, you simply choose the app where you want to load the data. In this example, Salesforce is chosen and the action “Create records in batches” is selected. Note that this displays the red “Batch” tag, which indicates it will be a batch load, rather than a row-by-row load. Different apps may have other actions, so make sure that you select the “batch” action for your destination app.
The final step is to select the requisite batch load action in the target app —Marketo in this case.
The red “Batch” tag indicates that you are using the “batch” action of your destination app
And that’s it! Here’s how that Recipe looks both with and without Repeat in Batches. Notice that the workflow is simpler and the result is shorter, more manageable Recipes.
The “Repeat in Batches” feature reduces the number of setup steps required to create a batch process
Quicker, easier, safer data handling
Hopefully, you’ll find that this new feature streamlines the handling of bulk data, which is key to today’s data-intensive business processes. Repeat in Batches shrinks the time and effort needed to build Recipes, lowers the ongoing maintenance burden, reduces the risk of bugs cropping up in loops, and helps boost throughput for batch data processing. It’s just one of the many new features and improvements Workato is constantly rolling out.
Want to see Workato in action? Click here to request a custom demo. For an explanation of how our intelligent platform can integrate anything and automate everything across your organization, click here. Or for live, real-time demos and presentations on other handy features, check out our Product Hour videos.