Faster Monthly Records Download

Guidance
1

Hi,

I am trying to figure out the most efficient way of downloading monthly records. I have working python script that creates the monthly export url based on the project and then reads each record and stores it. This process seems very slow (15+ minutes for large projects) when compared to download the csv files straigth from ComboCurve (1-2 minutes for large projects), so just curious if there is an option to start a bulk download or can someone post their code on how they are parsing through the monthly records quicker.

Thanks,

Eduardo

1 Like
2

You can add the take parameter to your route. For production data, it should be set to 20000:

account = ServiceAccount.from_file(<file>)
auth = ComboCurveAuth(account, <apikey>)

url = f'{API_BASE_URL}/projects/{project_id}/monthly-productions'
headers = auth.get_auth_headers()
params = {'take': 20_000}

response = requests.get(url, headers=headers, params=params)
3

I was referring to the monthly-exports request. I tried changing the take to 20,000 but I am getting a bad request. Here is the URL structure:

‘https://api.combocurve.com/v1/projects/{project_id}/scenarios/{scenario_id}/econ-runs/{econ_id}/monthly-exports/{econrun_id}?take=20000’

4

@dfulford basically this but without having to paginate:

5

In the docs, the take limit for monthly-exports is 100. There is also a concurrency parameter, which I’ve had success with setting to 10.

Ultimately, this route just seems to be pretty slow.

6

It is extremely slow - if I was using this for anything other than loading into our database I would have gone a more efficient path - but it takes around 3 min run and we have it syncing nightly with the rest of our ETL endpoints.

7

We can launch an investigation into the endpoint. I think it might just be due to the sheer amount of data but we should be able to continue to optimize the fetch.

8

Having a similar issue. It’s taking about 10 minutes to pull in all data from a single scenario. Not a big deal if doing a nightly database update, but I’m embedding my script into a Spotfire project to dynamically pull data while running different scenarios.

Maybe adding a “date” parameter to the endpoint will help to bring down the volume of data. For me, I typically only care about the next 3-5 years of monthly data. I’m not sure exactly what that does under the hood, but if it limits the data on your side before sending over, it could help.

1 Like
9

I think this will be extremely helpful

10

Have you tried using the new route for monthly-exports? We have a route for monthly-econ-results that fetches the same information as the monthly-exports without having to run an export first.

11

Could you point me in the direction of this new route?

12

@Austin.Franks you can access the route here

https://api.combocurve.com/v1/projects/:projectId/scenarios/:scenarioId/econ-runs/:econRunId/monthly-econ-results

13

is it possible to add a filter for combos for the new monthly-exports route?

14

Hey @Jose.RodriguezZavala!
We just allow filtering on the columns, start date, and end date.