Incremental update is the selective fetching of data from one system to another. It usually means only extracting / loading the data that was created or updated since the last time data was fetched. The main purpose of incremental updates is to reduce the system overhead required to fetch already existing or unmodified data.
To fetch data incrementally you need to query the endpoint and apply a filter by a specific date field to get data on or after a certain date/time. Refer to the special query parameters syntax for dates for more details on the supported filtering capabilities.
Most of the API resources will contain createdAt and updatedAt properties, or something similar, which can be used to perform incremental updating:
-
createdAt: can be used to fetch new records that were created after a certain date. For example,
GET /entity?sort=createdAt&createdAt[gt]=2021-01-01will return all new records that were created since last fetching date (1st Jan 2021) -
updatedAt: can be used to fetch new records that were updated after a certain date. For example,
GET /entity?sort=updatedAt&updatedAt[gt]=2021-01-01will return records that were created or modified since last fetching date (1st Jan 2021). Please note thatupdatedAtdoes not reflect updates on children models.
List of resources that can be pulled incrementally by using createdAt or updatedAt fields:
- get-daily-productions
- get-monthly-productions
- get-ownership-qualifiers
- get-projects
- get-projects-daily-productions
- get-projects-monthly-productions
- get-project-wells
- get-forecasts
- get-forecast-outputs
- get-scenarios
- get-wells