SAP Cloud Platform Integration helps you achieve integration with OData services. One of the key elements for integration is processing and exchange of large amounts of data. To ensure that these scenarios are executed without any issues, especially related to memory, you can use the Process in Batches option provided by the OData V2 adapter/SuccessFactors OData V2 adapter in SAP Cloud Platform Integration. You need to use this adapter in a Local Integration Process and invoke that process using a Looping Process Call to use the Process in Batches option.
This option enables the adapter to fetch only a limited number of records in one message processing cycle. This number is defined by the value that you provide in the Page Size field of the adapter configuration.
1- You must specify the Page Size based on the number of records that you want the system to fetch in one page of results.
2- This option enables you to fetch records in batches. For example, if you have 100 records and you set the Page Size as ’10’, the adapter fetches data in batches of 10 records each.
Important Consideration for Using Process in Batches
To use this option, you must ensure that you are using the OData adapter in a Local Integration Process that is invoked by a Looping Process Call. The looping process triggers the message processing multiple times depending on the configuration and ensures that all the pages are retrieved from the receiver system. For example, let us assume that there are 100 records that you want to fetch, in batches of 10 each. You configure the Page Size in OData receiver adapter as ’10’ and configure the Looping Process Call to run a minimum of 10 iterations to ensure that all the records are fetched.
Advantage of Using this Approach
It is recommended that you use the Process in Batches with Looping Process Call option while handling heavy loads of data. By doing so, you ensure that the memory consumption is optimized and the scenarios run without any issues. The overall load on the tenant will also be moderate by using this approach while working with large amount of data.
Example Integration Flow
Here’s a scenario (using OData V2 adapter) where you can understand how to use this approach to fetch large amounts of data from a OData service. Please note that this screenshot is being used for indicative purpose only. You can add other flow steps after the Looping Process Call based on your requirement and complete the modeling of the integration flow. You can also use a sender system with an adapter in the sender channel if required.
This integration flow is designed to fetch all the records from OData service in batches (pages). The Looping Process Call will invoke the Local Integration Process, which will fetch one page of record from the OData service and send it to Receiver 2 through mail adapter. This process is executed repeatedly till the Condition Expression is satisfied or Maximum No. of Iterations is completed.
After you deploy the integration flow, only the Looping Process Call step is executed repeatedly. Please note that the integration process will only be executed once and you see only one message in the Message Monitoring tab. For each iteration of the Looping Process Call, one page of records is fetched from the OData service.
If you want to manipulate each page of data (mapping, convert into another format, push data to target system, etc.), you must add these integration flow steps inside the Local Integration Process. The Integration Process (main process) will get only the last page of the records fetched.
You can choose and configure the adapters in the Integration Process and Local Integration Process based on your requirement. In the OData receiver adapter that you have used in Local Integration Process, ensure that you select Process in Batches option. In the Page Size field, provide the value for the number of records that you want in one message.
Here are the values that you can use in the Looping Process Call after you assign the Local Integration Process 1 to it using Select:
Expression Type: Non-XML
Condition Expression: ${property.Receiver1.OData.hasMoreRecords} contains ‘true’
Max. Numbers of Iterations: <total number of iterations required to fetch all records>
The syntax of condition expression is ${property.<receivername>.<channelname>.hasMoreRecords} contains ‘true’.
Significance of Max. Number of Iterations
1. By default, the value of Max. Numbers of Iterations is set to 100. If you wish to manually provide this, you can calculate this value by dividing total number of records by Page Size.
2. It is mandatory to have a value for Max. Numbers of Iterations to ensure that the integration flow is not stuck in an endless loop in case of any issues with the looping.
3. If you are unsure about the maximum iterations value, enter the value as 99999 (upper limit).
4. If Condition Expression evaluation fails, loop will be only executed once. Ensure values for <receivername>, <channelname> etc are provided accurately.
Additional Information About This Scenario
What data is returned to the main integration process from local integration process in this integration flow?
When you deploy this integration flow, the Local Integration Process will send only the last page of the records fetched from the OData service to the Integration Process.
For example, consider this scenario:
Here, a receiver system is accessed via Mail adapter (Mail 2) in Integration Process. If you configure the adapter to send an email to your email ID, you will get the last page of records fetched from the OData service. If you consider a service that contains 75 records, and you have specified the Page Size as 10, you will receive eight emails from Mail 1 adapter and one email having 5 records from Mail 2.
Please note that this is under the assumption that no further records are added to the OData service when the data is being fetched from it by the OData adapter.
What happens when pagination fails?
If there is a failure in processing a page of data (communication error in fetching the 4th page out of 10, or any flow step within the Local Integration Process throws an error, issue in the adapter configuration of Mail 1, etc.), the entire message processing will fail.
How to handle exceptions?
If you want to handle the exceptions which might occur in processing the data in pages inside Local Integration Process, you can use an Exception Subprocess. In the example below, if there is any error in the Local Integration Process, a message will be sent to the Receiver via Mail adapter.
