Get daily production not getting all records

Guidance
1

Hello, I’m trying to pull down all of the daily production data from a project. Below is my script. I’m using the pagination example here (combocurve-api-v1 · PyPI), with a few modifications to dump it into a pandas dataframe.

My problem is the script is only returning a fraction of the daily production. My project has ~128K records, and I’m only getting ~39K. Any suggestions on what I’m doing wrong?

from combocurve_api_v1 import ServiceAccount, ComboCurveAuth
import requests
import pandas as pd
from combocurve_api_v1.pagination import get_next_page_url

url_daily_production = f'https://api.combocurve.com/v1/projects/{projectid}/daily-productions'

url = url_daily_production + f'?take={20000}'

response_list = []

has_more = True

# Keep fetching while there are more records to be returned
while has_more:
    
    response = requests.get(url, headers=auth_headers)

    response_list = response_list + response.json()

    url = get_next_page_url(response.headers)
    has_more = url is not None

df_with_all_records = pd.DataFrame(response_list)
2

Hey @Sevans_Callon welcome to the forum!
You are correct. This is a bug that is affecting daily and monthly productions using the cursor.

Here is another post about the issue:

We already have a PR with a fix in place and it should be going out in the next release. In the meantime, you can use skip / take pagination for these endpoints as a workaround.

1 Like
3

Thanks, the skip/take method worked. Attaching is my script for anyone needing a python solution for the daily production.

skip = 0
take = 20000
response_count = 1

url = f'https://api.combocurve.com/v1/projects/{projectid}/daily-productions?skip={skip}&take={take}'

response_list = []

# Keep fetching while there are more records to be returned
while response_count != 0:
    
    response = requests.get(url, headers=auth_headers)
    response_count = len(response.json())
    response_list = response_list + response.json()
    
    skip = skip + take

    url = f'https://api.combocurve.com/v1/projects/{projectid}/daily-productions?take={take}&skip={skip}'

df_with_all_records = pd.DataFrame(response_list)

And just an FYI. I was using the daily production csv export in CC to test if I was getting an accurate number of records back. For some reason the API was returning a few hundred records short. It turns out I imported some production data that did not match any of the wells in the wellheader table in my project. The csv export in CC returns all daily production regardless if it has a well header match, and it looks like the API only returns records that have a match in the wellheader table.

CC Team, please correct me if I have that wrong.

4

Hey, I just wanted to reply and let you know that a fix for this issue has been released. You should now be able to use cursor pagination on daily and monthly productions.

1 Like