Transferring large files in Enterprise
Enterprises often have the need to transfer large files of few hundred megabytes from one system to another. Some popular use cases include:
- Move daily data dump files from database or ERP (such as NetSuite) into a file storage system (such as Amazon S3)
- Transfer files from multiple sources into one place for another app to process. For example, consolidating data files of sales and marketing into a SFTP folder, then import to Salesforce
- Back up the company’s files and save into another file storage system
Traditional approaches to file transfer usually have a small file size limit or require large server for processing. With Workato, the File streaming mechanism allows you to transfer large files between different systems easily, without any setup.
Streaming: Transferring large files easily
Streaming is Workato’s solution to transfer large files between 2 applications. Let’s see how it works.
First, let’s take a look at the traditional approach to file transfer. The Integration Server downloads the whole file from the Source App, saves it in memory, then uploads the file to Destination App.
Figure 1: Traditional File Transfer Mechanism
The disadvantage of this method is that file size will be limited by server’s memory. And because 1 server handles multiple file transfers at one time, it may only allocate memory as low as 20MB for each file. To transfer large files of few hundred megabytes, you will need to pay for a much larger server!
Workato solves this problem by cutting a large file into smaller chunks, then “streaming” 1 chunk at a time to the destination. This allows Workato server to technically transfer files of unlimited size, and you don’t need to pay that extra cost to move your data.
Figure 2: Workato File Streaming Mechanism
How to use Streaming in Workato
At the moment, we have implemented Streaming for transferring file from Amazon S3 to SFTP. The Streaming mechanism happens under the hood, so you don’t need to do any extra setup. Try out this sample recipe with files of up to 300MB.
Figure 3: Workato recipe to transfer file from Amazon S3 to SFTP
We are rolling out Streaming to the rest of file triggers & actions in Workato. The upcoming ones are SFTP, Box, Dropbox, Service Now, etc. Note that in this first implementation of Streaming, we have a job run time limit of 30 minutes, so you can transfer files of up to about 300MB.
In the future iterations, we aim to remove this file size limit completely by dividing the chunks into several jobs. By then you will be able to transfer file of unlimited size between any app.
Stay tune for future product updates. And head out to our Workato community page for more automation use cases!