# Bulk API and Imports Outreach API supports bulk operations on several models. It also supports import of data from CSV files to bring data from external sources into Outreach. ## Bulk actions through API All the supported bulk actions are found under **Outreach REST API reference > Batch > [Batch Actions](/api/reference/batch).** ### Making requests Bulk requests are shaped as follow: ``` POST https://api.outreach.io/api/v2/batches/actions/ ``` You can use `actionParams` to the query as you would for [other actions](/api/making-requests/#other-actions-on-a-resource). The action will always support the `skipConfirmation` parameter, we will talk more about this [here](#skipping-confirmation). In the next section we will see how to apply filters with the `filter` parameter. Browse Batch definition in [the API-Reference](/api/reference/batch) to see the supported parameters for each bulk action. If you are using an OAuth token, make sure to have `batches.read`, `batches.write` and the targeted resources `write` scopes, e.g: `accounts.write`, `prospects.write`... ### Applying filters As mentioned in the previous section, all bulk actions support the `filter` parameter. You can chain multiple filters and the syntax should respect the [newFilterSyntax](/api/making-requests/#new-filter-syntax). Here is an example of a bulk action on accounts: ``` POST https://api.outreach.io/api/v2/batches/actions/accountsBulkModify?actionParams[filter][id][]=1&actionParams[filter][id][]=2&actionParams[filter][id][]=3&actionParams[filter][owner][id]=123 ``` The bulk action will be triggered on the exact same records you would get with the corresponding collection GET request: ``` GET https://api.outreach.io/api/v2/accounts?filter[id][]=1&filter[id][]=2&filter[id][]=3&filter[owner][id]=123&newFilterSyntax=true ``` In the absence of any filter the action would trigger on **ALL** the records it applies to. ### Building the request's body Make sure to read about ["classic" POST requests](/api/making-requests/#create-a-new-resource). The data passed to the bulk actions is given in the attributes. Here is an example of `bulkModify` on accounts: ``` POST https://api.outreach.io/api/v2/batches/actions/accountsBulkModify?actionParams[filter][id][]=1&actionParams[filter][id][]=2&actionParams[filter][id][]=3 ``` ```json5 { "data": { "attributes": { "companyType": "Public Company", "custom1": "edit from bulk", ... } } } ``` Check the documentation of each action to see which attributes are accepted. You may notice some bulk actions, such as `bulkDelete`, that don't require attributes. Those do not require any payload to be passed. ### Understanding the result The result given is a batch. This is the model used to track the bulk requests. It will look like this: ```json5 { "data": { "type": "batch", "id": 1156, "attributes": { "action": "bulk_modify", "canceledAt": null, "confirmCount": 2, "confirmedAt": null, "confirmedCountRequired": null, "contextId": 0, "contextType": "", "createdAt": "1701880061", "failures": 0, "finishedAt": null, "pending": 0, "startedAt": null, "state": "pending", "summary": null, "total": 0, "updatedAt": "1701880061" }, "relationships": { "batchItems": { "links": { "related": "https://api.outreach.io/api/v2/batchItems?filter%5Bbatch%5D%5Bid%5D=1156" } }, }, "links": { "self": "https://api.outreach.io/api/v2/batches/1156" } } } ``` At first, the result can have 2 states: either `pending` or `confirming`. We will talk about the later in the [next section](#confirming-bulk-request). When it is `pending` you can keep track of the progress by querying the `/batches/` endpoint with the id of the returned batch, e.g: ``` GET https://api.outreach.io/api/v2/batches/1156 ``` The response will be the same as above but some fields might have been updated. Take a look at the batch definition in the [API Reference](/api/reference/) for more details about the batch model. From pending state the batch can move to either `finished`, `failed` or `canceled`. In some cases, the `summary` might be updated with information related to the failures during the process (some failures are minor and won't trigger the whole batch to fail). If no failure happens, the `summary` stays `null`. Additionally, the detail of each item that was processed can be found through the `batchItems` endpoint. Ideally, you will want to filter by the batch you are dealing with, e.g: ``` GET https://api.outreach.io/api/v2/batchItems?filter[batch][id]=1156 ``` If you are only interested in the failures you can filter based on the state: ``` GET https://api.outreach.io/api/v2/batchItems?filter[batch][id]=1156&filter[state]=error ``` ### Confirming bulk request A bulk gets into confirming state when the confirmation hasn't been skipped (see [skipping confirmation](#skipping-confirmation)). This means that an additional action will be required to trigger the action to start. When receiving a batch in confirming state, it will look like this: ```json5 { "data": { "type": "batch", "id": 1156, "attributes": { // other attributes ... "confirmCount": 2, "confirmedCountRequired": true, // can also be null or false "state": "confirming", }, } // relationships, links ... } ``` In case `confirmedCountRequired` is `true`, you can call: ``` POST https://api.outreach.io/api/v2/batches/1156/actions/confirm?actionParams[confirmedCount]=2 ``` Otherwise there is no need to specify the `confirmedCount` (there is no harm in putting it though): ``` POST https://api.outreach.io/api/v2/batches/1156/actions/confirm ``` Note that you can also use the `/cancel` action on the batch (without params). It will move the batch state to `canceled` and stop processing of items if it ever started. ### Skipping confirmation When using bulk actions in automation, you may want to avoid the confirmation step. For that, when [starting a new bulk](#making-requests), you can pass the `skipConfirmation` action parameter with `true` value: ``` POST https://api.outreach.io/api/v2/batches/actions/accountsBulkModify?filter[id]=1,2,3&actionParams[skipConfirmation]=true ``` ## Importing CSV through API Our system has been designed to allow you to upload a CSV file to our S3 bucket and then access this file via its reference (so called `storageKey`). This replicates the feature you may have seen on Outreach Client. To use the import feature with an Oauth token you will need the `imports.write` and `imports.read` scope. ### Uploading the file First of all you will need to upload your CSV file to our S3 bucket. The first step is to generate the link, we use AWS Presigned URLs. Let's take a look at the API call: ``` POST https://api.outreach.io/api/v2/imports/actions/generateUploadLink ``` The response body is slightly unconventional as the model doesn't have an id and it will look like this: ```json5 { "data": { "type": "upload", "attributes": { // presigned AWS S3 URL "uploadUrl": "https://outreachimportsdata-us-east-2.s3.us-east-2.amazonaws.com/bento/folder1/folder2/folder3/data.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&otherParams...", "storageKey": "adfaaf30-4514-49ff-ac0f-0c5bafb918b2" } } } ``` Carefully save the `uploadUrl` and the `storageKey` as you won't be able to get them again. You can then upload the file to the `uploadUrl` using PUT method. Make sure to read the AWS documentation. ### Validating the file After the upload is done, the file is in our S3 bucket but is not yet ready to be used. You will need to validate the file to ensure that it isn't corrupted. This is done by calling the following action: ``` POST https://api.outreach.io/api/v2/imports/actions/validateUpload?actionParams[storageKey]=adfaaf30-4514-49ff-ac0f-0c5bafb918b&actionParams[hash]=abcde... ``` The `hash` is the SHA512 hash of the file you uploaded. The response will look like this: ```json5 { "data": { "type":"validateUpload", "attributes": { "headerValue": [{ "value":"object name", "header":"name", }], "recordCount": "123" } } } ``` The header value is a list with the headers of the CSV file and the values in the first row that Outreach API was able to extract. The `recordCount` is the amount of records Outreach API found in the CSV file. You will need this in the next step. ### Starting the import You will again need the `storageKey` here. This is how we will find which file to use. You will also need to send us the amount of records in the CSV file as `recordCount`. You can then call the import action of your choice to process the uploaded CSV. You can find those by browsing the [import's API Reference](/api/reference/import): ``` POST https://api.outreach.io/api/v2/imports/actions/accountsImport ``` ```json5 { "data": { "attributes": { "storageKey": "adfaaf30-4514-49ff-ac0f-0c5bafb918b", "recordCount": "123", "dupeMethod": "skip", "mappings": { "companyName": "name", "": "", } } } } ``` Your imported CSV file may have duplicated records with what Outreach has. Those are identified by unique fields that are identical. `id` is one of them for all models. Other fields may be unique as well for each specific model (e.g id, email...). This is described in the import action itself (listed under the [import's API Reference](/api/reference/import)). The `dupeMethod` is used to treat those clashing records. It can either be: - `skip` (default) : we will skip duplicates and no data will be stored from that row - `missing`: we will update empty fields in the record by their imported value - `overwrite`: we will update all the fields with their imported value All other non-duplicated records are created in DB. The `mappings` are used to map the headers of your CSV file to the Outreach field names. To ensure consistency, provide mappings for all the fields. You may leverage the `headerValue` from the previous step. The response you will receive looks like this: ```json5 { "data": { "type": "import", "id": 639, "attributes": { "createdAt": "2024-03-07T13:21:47.000Z", "dupeMethod": "skip", "dupes": 0, "errorReason": null, "externalId": null, "externalName": null, "externalType": null, "failures": 0, "fileName": "account.csv", "fileSize": null, "frequency": "once", "loadFromPlugin": false, "mappings": { "Id": "id", "Title": "title" }, "pluginId": null, "prospectOwnerId": null, "recurring": false, "reportInstanceId": null, "scheduledAt": null, "source": null, "stageId": null, "state": "progressing", // or finished or failed "stateChangedAt": "2024-03-07T13:21:49.000Z", "syncedUntil": null, "timeZone": null, "total": 1, "type": "ImportAccount::Csv", // special case: Import::Csv for prospects "updatedAt": "2024-03-07T13:21:49.000Z" } } } ``` Similarly to batches you can track the `state` of the import using: ``` GET https://api.outreach.io/api/v2/imports/639 ``` NOTE: Beneath the hood, imports are running in batches. You can find the corresponding batch and batch items using the following: ``` GET https://api.outreach.io/api/v2/batches?filter[contextId]= GET https://api.outreach.io/api/v2/batchItems?filter[batchId]= ``` ## Bulk Upsert Bulk upsert is an alternative approach to CSV import for syncing records to Outreach. Instead of uploading a CSV, bulk upsert allows the records to be passed directly to Outreach as API v2 JSON objects. Contrary to CSV import, however, request size is limited to a maximum of 2MB. This section describes the use of the bulk upsert API. To use the bulk upsert feature with an Oauth token you will need the `imports.write` and `imports.read` scope. ### Starting a bulk upsert To start a bulk upsert call the `/bulkUpsert` import action (or `/customObjectBulkUpsert` action for custom objects) passing an array of API v2 objects in the `records` array attribute. The type of the import will be infered from the typeof the collection, so all records must belong to the same API v2 resource. Records must be valid API v2 JSON for their respective resource. For example, to start a bulk upsert of products, the individual records should be valid [POST](/api/making-requests/#create-a-new-resource) or [PATCH](/api/making-requests/#update-an-existing-resource) requests depending on if the record is intended for insertion or update. The following example shows an upsert of products intended to overwrite dupes based on ID with new attribute values and owner relationship. Bulk upsert supports all API v2 objects that are upsertable (having both create and update actions) and includes custom objects. See [Bulk Upsert](/api/reference/import) and [Custom Object Bulk Upsert](/api/reference/import) for REST API reference documentation. #### Example bulk upsert of Products ``` POST https://api.outreach.io/api/v2/imports/actions/bulkUpsert ``` ```json5 { "data": { "attributes": { "dupeMethod": "overwrite", "records": [ {"data": { "id": 1, "type": "product", "attributes": { "name": "Candlestick Holder", "classification": "Own" }, "relationships": {"owner":{"data":{"type":"user","id":1}}}}}, {"data": { "id": 2, "type": "product", "attributes": { "name": "Brass Buttons", "classification": "Complementary" }, "relationships": {"owner":{"data":{"type":"user","id":1}}}}}, // ...additional records ] } } } ``` Similar to CSV import, your records may duplicate what already exists in Outreach. At this time, bulk upsert identifies dupes by `id` only. Supported dupe methods are: - `skip` (default): we will skip duplicates and no data will be stored from that row - `missing`: we will update empty fields in the record by their imported value - `overwrite`: we will update all the fields with their imported value - `overwriteNoCreate` (bulk upsert only): we will only update fields on duplicate records without creating any records ### Source-destination mapping When performing a bulk upsert, you may want to correlate newly created records in Outreach with the records in your source system. This can be accomplished through including metadata on the individual API v2 resource objects, which allow for the presence of a `meta` field. Outreach will persist individual records along with metadata as `batchItems` which will include the Outreach ID once the item has been processed. In the following example, multiple accounts with source IDs are included for bulk upsert: ```json5 { "data": { "attributes": { "dupeMethod": "skip", "records": [ {"data": { "type": "account", "attributes": { "name": "Fixtures & Furnishings" }, "meta": { "sourceId": "1234" }}}, {"data": { "type": "account", "attributes": { "name": "Ben's Burnishing" }, "meta": { "sourceId": "2345" }}} ] } } } ``` Once the import has started, you can query for batch items where `[batchId]` equals the ID of the batch. The Outreach ID (`itemId`) and original JSON object (`data`) which includes the metadata information you provided will be present on the batch items of your import. The following example demonstrates filtering batch items for this purposes. (Note that `batchId` can be retrieved by filtering batches on contextId) ``` GET https://api.outreach.io/api/v2/batches?filter[contextId]= GET https://api.outreach.io/api/v2/batchItems?filter[batchId]=&fields[batchItem]=itemId,data ``` ```json5 { "data": [ { "type": "batchItem", "id": 101, "attributes": { "data": "{\"data\":{\"type\":\"account\",\"attributes\":{\"name\":\"Fixtures & Furnishings\"}},\"meta\":{\"sourceId\":\"1234\"}}", "itemId": 1 } }, { "type": "batchItem", "id": 102, "attributes": { "data": "{\"data\":{\"type\":\"account\",\"attributes\":{\"name\":\"Ben's Burnishing\"}},\"meta\":{\"sourceId\":\"2345\"}}", "itemId": 2 } } ] } ``` ### Notification via webhook You may subscribe to be notified when your import is finished using the [webhook API](/api/reference/webhook) and subscribing to the `finished` action and `import` resource. ### Bulk Upsert data retention Bulk upsert batch data will be retained for 30 days to allow time for batch results (e.g. number of failures, duplicates, successes, failure details) to be reviewed and any issues investigated. Only upsert batch data will be removed: Batch data for other actions including CSV imports will be retained. ## Bulk rate limiting ### Per request For now, bulk requests can be triggered on a maximum of 100 000 items. If it exceeds this amount, it won't start and will be moved to `failed` state. Also, together with the Outreach Client, there is a limit of 5 Millions of records per day that can be processed through bulk actions. Bulk requests through API v2 that exceed their daily batch item limit will receive a 429 "Too Many Requests" error. Batch item allowance is refreshed throughout the day incrementally until the 5 million maximum is reached.