CFHTTP Issue: Unwanted concurrent calls to API
I'm building an app that pulls data from a 3rd party service and am running into an issue where CF is executing unwanted multiple concurrent calls to the same API endpoint. The API has a few limitations:
- No concurrent calls to the same endpoint for a specific API user account
- A limit of 500 results returned by the API per call, with a nextPageUrl value passed back for pagination
The data I'm trying to retrieve contains over 5000 records, so I'm pulling in 500 records at a time until the nextPageUrl returns null (per the API docs). I setup a page with the following code to test out the process:
<cffunction name="getAPIdata" access="public" returntype="boolean" output="yes">
<cfset local.result = false>
<cfset local.apiUrl = "[INITIAL API CALL URL HERE]">
<cfset local.apiCallComplete = false>
<cfloop condition="local.apiCallComplete IS FALSE">
<cfhttp url="#local.apiUrl#" method="get" result="local.res" throwOnError="no">
<cfhttpparam type="header" name="Content-Type" value="application/json">
<cfhttpparam type="header" name="Credentials" value="[API USER CREDENtIALS HERE]">
</cfhttp>
<cfset local.data = deserializeJSON(local.res.fileContent)>
<cfquery>
<!--- Loop over local.data array and insert into db --->
</cfquery>
<!--- Write call results to log file and increment count (for testing) --->
<cfif isDefined("local.data.nextPageUrl")>
<cffile action="append" addnewline="true" file="#expandpath('../log.txt')#" output="Count: #count#">
<cffile action="append" addnewline="true" file="#expandpath('../log.txt')#" output="Next page: #urlDecode(local.data.nextPageUrl)#">
<cffile action="append" addnewline="true" file="#expandpath('../log.txt')#" output="-------------------------------------------------------------------------------------------">
<cfset count++>
<cfset local.apiUrl = urlDecode(local.data.nextPageUrl)>
<cfelse>
<cffile action="append" addnewline="true" file="#expandpath('../log.txt')#" output="Count: #count#">
<cffile action="append" addnewline="true" file="#expandpath('../log.txt')#" output="Next page: NONE">
<cffile action="append" addnewline="true" file="#expandpath('../log.txt')#" output="-------------------------------------------------------------------------------------------">
<cfset local.result = true>
<cfset local.apiCallComplete = true>
</cfif>
</cfloop>
<cfreturn local.result>
</cffunction>
I initialize the count and call this function on the same page with:
<cfset count = 1>
<cfset getAPIdata()>
So in theory, it should loop - returning 500 records at a time - until all records have been retrieved. The weird thing is that as I watch the log.txt file, the count and next oage url values are logged twice:
Count: 1
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 1
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 2
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 2
Next page: https://...
-------------------------------------------------------------------------------------------
etc.
I immediately get a warning email that I had exceeded the API thread threshold limit of one call to the same endpoint for this user, and duplicate records are inserted into the database. I tried putting the code inside the conditional loop in a cflock to see if that would keep this from happening, but while watching the log it would start to increment properly and then after the count hit 3 or so, it would add another incrementing series of calls starting at 1 in the middle of the existing loop, like:
Count: 1
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 2
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 3
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 1
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 2
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 4
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 3
Next page: https://...
-------------------------------------------------------------------------------------------
etc.
again triggering the warning email and duplicated database entries. My development stack is CF2018 behind Apache 2.4 with MySQL. Any ideas what may be causing this? I have also tried adding the function above to a CFC component and invoking it from a separate page. Same issue.
