CFHTTP Issue: Unwanted concurrent calls to API

[Alex26362256ko9t]

I'm building an app that pulls data from a 3rd party service and am running into an issue where CF is executing unwanted multiple concurrent calls to the same API endpoint. The API has a few limitations:

1. No concurrent calls to the same endpoint for a specific API user account
2. A limit of 500 results returned by the API per call, with a nextPageUrl value passed back for pagination

The data I'm trying to retrieve contains over 5000 records, so I'm pulling in 500 records at a time until the nextPageUrl returns null (per the API docs). I setup a page with the following code to test out the process:

<cffunction name="getAPIdata" access="public" returntype="boolean" output="yes">

<cfset local.result = false>
<cfset local.apiUrl = "[INITIAL API CALL URL HERE]">
<cfset local.apiCallComplete = false>

<cfloop condition="local.apiCallComplete IS FALSE">

<cfhttp url="#local.apiUrl#" method="get" result="local.res" throwOnError="no">
<cfhttpparam type="header" name="Content-Type" value="application/json">
<cfhttpparam type="header" name="Credentials" value="[API USER CREDENtIALS HERE]">
</cfhttp>

<cfset local.data = deserializeJSON(local.res.fileContent)>

<cfquery>
<!--- Loop over local.data array and insert into db --->
</cfquery>

<!--- Write call results to log file and increment count (for testing) --->
<cfif isDefined("local.data.nextPageUrl")>
<cffile action="append" addnewline="true" file="#expandpath('../log.txt')#" output="Count: #count#">
<cffile action="append" addnewline="true" file="#expandpath('../log.txt')#" output="Next page: #urlDecode(local.data.nextPageUrl)#">
<cffile action="append" addnewline="true" file="#expandpath('../log.txt')#" output="-------------------------------------------------------------------------------------------">
<cfset count++>
<cfset local.apiUrl = urlDecode(local.data.nextPageUrl)>
<cfelse>
<cffile action="append" addnewline="true" file="#expandpath('../log.txt')#" output="Count: #count#">
<cffile action="append" addnewline="true" file="#expandpath('../log.txt')#" output="Next page: NONE">
<cffile action="append" addnewline="true" file="#expandpath('../log.txt')#" output="-------------------------------------------------------------------------------------------">
<cfset local.result = true>
<cfset local.apiCallComplete = true>
</cfif>

</cfloop>

<cfreturn local.result>
</cffunction>

I initialize the count and call this function on the same page with:

<cfset count = 1>
<cfset getAPIdata()>

So in theory, it should loop - returning 500 records at a time - until all records have been retrieved. The weird thing is that as I watch the log.txt file, the count and next oage url values are logged twice:

Count: 1
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 1
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 2
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 2
Next page: https://...
-------------------------------------------------------------------------------------------

etc.

I immediately get a warning email that I had exceeded the API thread threshold limit of one call to the same endpoint for this user, and duplicate records are inserted into the database. I tried putting the code inside the conditional loop in a cflock to see if that would keep this from happening, but while watching the log it would start to increment properly and then after the count hit 3 or so, it would add another incrementing series of calls starting at 1 in the middle of the existing loop, like:

Count: 1
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 2
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 3
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 1
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 2
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 4
Next page: https://...
-------------------------------------------------------------------------------------------
Count: 3
Next page: https://...
-------------------------------------------------------------------------------------------


etc.

again triggering the warning email and duplicated database entries. My development stack is CF2018 behind Apache 2.4 with MySQL. Any ideas what may be causing this? I have also tried adding the function above to a CFC component and invoking it from a separate page. Same issue.

[Alex26362256ko9t]

UPDATE:

I tested the original code on my production server running IIS and it works perfectly. Looks like this is an issue with Apache (possibly url rewrites) or something else in my dev environment rather than an issue with ColdFusion.

[BKBK]

Two things:

1. You should transfer the initialization code

<cfset count = 1>

to the function. For example, place

<cfset local.count = 1>

before the loop, and change the remaining names from count to local.count.

2. In place of

<cfif isDefined("local.data.nextPageUrl")>

use something more rigorous, like

<cfif structKeyExists(local.data, "nextPageUrl") and trim(local.data.nextPageUrl) is not "">

[BKBK]

Do the 2 suggestions solve the problem?