Question - how do I set my take on GET requests to all?
urltwo = (
"https://api.combocurve.com/v1/projects/61a92c8f34254c0013cacf3e/scenarios/61a93338b763c20015f3f68f/econ-runs/"
+ econId
+ "/monthly-exports/"
+ econRunId
+ "?skip=0"
+ "&take=200"
)
right now I am just using a number I know is bigger than my rows, but I want to see if there is a way to set it to grab all the values.
Thanks!
@michael.tanner good question.
There’s a maximum allowed value for every take parameter. On this endpoint in particular it’s 200, like you have in your snippet. This is mainly because there’s only so much data is recommended to transfer in a single HTTP request + response.
So the expected is that caller do pagination. Which means repeating the request with a fixed take, but incrementing skip until you get no records back.
Pseudo-code would be roughly something like:
MAX_TAKE = 200
skip = 0
has_more = True
while has_more:
response = make_request(skip, MAX_TAKE)
process_response(response)
skip += len(response)
has_more = len(response) == MAX_TAKE # if len(response) < take then this is the last page
Hopefully we will add helpers for this in Python in the near future, like the ones we have in the C# client to do this pagination work for you.
I hope that helps.
@maikel.ruiz thanks! Wow I got lucky with picking 200.
Will add this to my code here - working on summing up a few different output rows (all the taxes) into one column.
Haha, got it.
We have a guide for this topic that will be coming in the near future with more explanation:
@michael.tanner check the latest version (0.2.3) combocurve-api-v1 · PyPI
We just added the get_next_page_url helper. Example:
from combocurve_api_v1.pagination import get_next_page_url
# See Authorization section
auth_headers = combocurve_auth.get_auth_headers()
# Additional filters are allowed, it is preferred not specify skip if its value is 0
url = 'https://api.combocurve.com/v1/wells?take=200'
# First request
has_more = True
# Keep fetching while there are more records to be returned
while has_more:
response = requests.get(url, headers=headers)
# Process response
url = get_next_page_url(response.headers)
has_more = url is not None
On the surface this mainly manages the skip parameter for you. But this is in fact more robust and will also transparently allow other more optimized pagination methods in the future.
just pip upgraded to new version, worked great
this is exactly what I need to finish this project - you team rocks per usual
Glad to hear, and thanks. We appreciate your collaboration to improve our tooling and documentation.
Ok little help here. The below code is my first attempt…it never exits the while loop so I added the print statement…not sure how to say has_more go to false?
# Reautenticated client
auth_headers = combocurve_auth.get_auth_headers()
# set new url with econRunID, skipping zero
urltwo = (
"https://api.combocurve.com/v1/projects/61a92c8f34254c0013cacf3e/scenarios/61a93338b763c20015f3f68f/econ-runs/"
+ econId
+ "/monthly-exports/"
+ econRunId
+ "?take=200"
)
has_more = True
while has_more:
# same as above, parsing as JSON string
response = requests.request("GET", urltwo, headers=auth_headers)
urltwo = get_next_page_url(response.headers)
has_more = urltwo is not None
print(len(response.text))
That’s weird. I have this working code, I only removed the authentication part, and it’s not very different from yours:
import requests
from combocurve_api_v1.pagination import get_next_page_url
'''
TODO: Authentication steps: instantiatie `combocurve_auth`
'''
BASE_URL = 'https://api.combocurve.com'
url = f'{BASE_URL}/v1/projects?take=200'
has_more = True
# Keep fetching while there are more records to be returned
while has_more:
print(url) # see the URL for each request
headers = combocurve_auth.get_auth_headers()
response = requests.get(url, headers=headers)
# Process response here
# print(len(response.json()))
url = get_next_page_url(response.headers)
has_more = url is not None
Maybe it takes a while inside the while because it’s actually making more requests.
You can also give it a try in a different resource, with fewer records, like in this example using /projects.
@maikel.ruiz hmmm thanks for the sample code! Yeah - its only a 4 well test but for the actual cash flow run I am pulling will have 130 wells.
I wonder, letting it run for a little see if it pops at some point.
I noticed earlier I tried with one-liner. Now when I tried with the cash flows It’s indeed looping forever so you may have found a bug in the API for this particular endpoint. Looking into this on our side.
@maikel.ruiz you rock!! I am seeing if there is anything I can do on my side as well
@michael.tanner we deployed earlier the fix we were working on for the cashflows pagination and I just verified it is working as expected now.
Thanks for you patience 
@maikel.ruiz nice!! Yes…it works!! Pops out of the loop
Here is my issue…how do I store the data from each of the URL’s…
urltwo = (
"https://api.combocurve.com/v1/projects/61a92c8f34254c0013cacf3e/scenarios/61a93338b763c20015f3f68f/econ-runs/"
+ econId
+ "/monthly-exports/"
+ econRunId
+ "?take=200"
)
has_more = True
while has_more:
response = requests.request("GET", urltwo, headers=auth_headers)
urltwo = get_next_page_url(response.headers)
has_more = urltwo is not None
jsonStr = response.text
print(len(jsonStr))
# print(jsonStr)
dataObj = json.loads(jsonStr)
dataResults = dataObj["results"]
dataResults is length 43, which is great, should have 243 records…but I am missing the part where I save the first 200 to a table.
Thanks for getting on this so quickly!! I was working through pagination youtube and had something simlinar to your psuedo code but it was much bulkier and still struggled with the saving each request
Yeah, expected, you have the print outside the while, so you are only printing for the last value response had, and not for every response.
Something like this will print the length for every page:
urltwo = (
"https://api.combocurve.com/v1/projects/61a92c8f34254c0013cacf3e/scenarios/61a93338b763c20015f3f68f/econ-runs/"
+ econId
+ "/monthly-exports/"
+ econRunId
+ "?take=200"
)
has_more = True
while has_more:
response = requests.request("GET", urltwo, headers=auth_headers)
urltwo = get_next_page_url(response.headers)
process_page(response.json())
has_more = urltwo is not None
def process_page(response_json):
results = response_json["results"]
# TODO: process `results`, write to DB or append to total result
print(len(results))
Also note how response.json() saves a few steps.
Ah ok make sense! Thanks - will work this solution here. @maikel.ruiz