Bulk Extract API is used to efficiently process large data requests and retrieve massive amounts of data in batches, each containing a user-specified number of records. Bulk Extract requests can be based on one the following:
- Queries created with the BillingPlatform Query Language
- The RETRIEVE method that is better suitable for retrieving a large amount of data on a specific entity in your enterprise billing system.
The retrieved data can be formatted as JSON or as CSV with the user-defined delimiters, qualifiers, and end-line format characters. You can select either response format for QUERY requests or for requests based on the RETRIEVE method.
To extract large datasets from the Mediation Data Lake (MDL), use the MDL EXTRACT method. This method extracts data asynchronously, one request at a time, which allows for handling complex data retrieval scenarios, providing customizable data formatting options, and ensuring efficient tracking and management of extraction processes. This method supports only CSV format output.
NOTES:
- The Bulk Request Processing Process must be active in order to process a Bulk Extract API request. Otherwise the request remains in a PENDING status and is not processed. The Bulk Request Processing process can be activated (or inactivated) on the Process Console page by a user with sufficient privileges.
- Entities Activity and Collector_Fields can be set to save deleted entity log records. These records can later be extracted by the Bulk Extract API. Please contact your BillingPlatform representative to activate this functionality.
To send a Bulk Extract request using the QUERY method, issue a POST /rest/2.0/bulk_api_request call with the following parameters:
| Field | Data Type | Description |
|---|---|---|
| RequestName | TEXT | Descriptive name of the new request. |
| RequestBody | TEXTAREA | The request payload created using the BillingPlatform Query Language. The payload length cannot exceed 4000 characters. |
| RecordsPerBatch | NUMBER | Number of records returned in one batch. For example, prior to running a request that is bound to return about a million records, consider splitting this amount into smaller batches of 50 thousand records each for easier processing. |
| RequestMethod | TEXT |
|
| ResponseFormat | TEXT | JSON |
URL: https://my.billingplatform.com/myorg/rest/2.0/bulk_api_request
Method: POST
{
"brmObjects":{
"RequestName":"Request Name",
"RequestBody":"select AccountId as AccountId,
AccountPackageId as AccountPackageId,
AccountProductId as AccountProductId,
ActivityCollectorId as ActivityCollectorId,
ActivityDate as ActivityDate,
BillingIdentifierId as BillingIdentifierId,
...
RateSource as RateSource from activity where rownum <= 100000",
"RecordsPerBatch":"200000",
"RequestMethod":"QUERY",
"ResponseFormat":"JSON"
}
}To send Bulk Extract request using the RETRIEVE method, issue a POST /rest/2.0/bulk_api_request call with the following parameters:
| Field | Data Type | Description |
|---|---|---|
| RequestName | TEXT | Descriptive name of the new request. |
| RequestBody | TEXTAREA | When applying the RETRIEVE method, this parameter can be a standard ANSI SQL query, for example Id > 2. |
| RecordsPerBatch | NUMBER | Number of returned records in one batch. For example, prior to running a request that is bound to return about a million records, consider splitting this amount into smaller batches of 50 thousand records each for easier processing. |
| RequestMethod | TEXT | RETRIEVE |
| ResponseFormat | TEXT | CSV |
| RetrieveEntityName | TEXT | The name of the target entity. Make sure to indicate the internal entity name as it appears in Setup → Development → Entities → [entity details] → Entity Name. This name cannot include spaces or start with a number or punctuation mark. |
| CSVDelimiter | TEXT | (optional) The character used as the column-delimiter in a CSV file. |
| CSVQualifier | TEXT | (optional) The characters placed around each field to signify that it is the same field. Most commonly those are double quotes ("). |
| CSVEndLineFormat | TEXT | (optional) The character used to signify the end of line in a CSV file: CR, LF, or CRLF. |
| Columns | TEXT | (optional) The fields that should be included in the extracted data. If this field is defined, only the specified columns will be extracted. If omitted, the API returns all available columns. |
URL: https://my.billingplatform.com/myorg/rest/2.0/bulk_api_request
Method: POST
{
"brmObjects":
{
"RequestName": "RetrieveCsvExample",
"RequestBody":"Id > 2",
"RecordsPerBatch":"1000000",
"RetrieveEntityName":"ACCOUNT",
"Columns": ["Status", "ActivityDate"],
"RequestMethod":"RETRIEVE",
"ResponseFormat":"CSV",
"CSVDelimiter":",",
"CSVQualifier":"\"",
"CSVEndLineFormat":"CRLF"
}
}A successful POST request returns the ID assigned to the new request. You can use this ID to search for the request in the Setup > Data Management > Bulk API Request section in the BillingPlatform console.
{
"createResponse": [
{
"ErrorCode": "0",
"ErrorText": " ",
"ErrorElementField": " ",
"Id": "number"
}
]
}As soon as the request status changes to PROCESSED, opening it in the Setup > Data Management > Bulk API Request section displays its details split into four categories:
- Request Specification: The request settings provided in the API call. In addition, this category includes the Number of batches field that represents the total number of batches retrieved by the request.
- Processing Log: The total number of processed records and the request processing log. The log contains details about how the request was parsed, when its processing started, whether batch execution was successful, etc.
- Results: The data retrieved by the request split into batches in the required format. You can click on each batch to view its details and download it (see also Direct file download).
- System Information: The request ID, the date and time of its creation, etc.