script to evaluate many different variables

Report · Sep 29, 2009

Hi all

I have about 10000 items to evaluate every month. Each item has over 300 variables. These variables are matched against benchmark data. These data may be numeric, strings, dates, time .... etc.

I have written a script of CFIF/ELSE or CASE for each of these variable. Anytime one of these variables is FALSE, the item is rejected and it looks at the next item. The whole generation times 30 mins or more. I am not sure whether this is the best way to script. Is there a more intelligent way to script for this problem. Each item takes a few seconds for the script to run through, therefore with 10000 items, it takes foreever.

Would appreciate any suggestions. Thanks

Watson

Report · Sep 29, 2009

Possible, if we had some idea of what your script looks like.

How many tests per variable?

How do you loop over the tests?

How do you loop over the variables?

How do you loop over the items?

Have you organized your operations to quickly evalute the most common failures so desicions can often be reached faster?

Report · Sep 29, 2009

I wrote this software about 10 yrs ago. it is a relatively complex software for rostering of flight crew. I tried to stick to best practises and made the script as efficient as possible. However it still took a long time to process and to come up with the monthly roster. I am now trying to rescript the whole software. The script is just too complex to show here. However, at the core of the script, I have these variables that are evaluated over and over again for correctness. Those that are likely to fail are put right at the top of the loop so that it does not waste time if the loop is to fail.

However, if and elseif only evaluates one variable at a time. So with over 300 variables for each item, each item took sometime. Is there any new development where based on a pattern, all 300 variables can be evaluated for TRUE in one go.

Thanks for any suggestions

Report · Sep 29, 2009

Ultametaly, on one CPU in one process, NO.

A computer can only make one comparison at it time, but it can do that comparison very quickly.

But at a higher abstraction layer, maybe you can speed up these process somewhat.

It would be a much more complex logic and will take some real planning to make it all fit together correctly.

If you are on the latest version of ColdFusion, you may be able to make use of the new <cfthread....> functionality to allow for multi-threading. Then, if this process is running on a relatively modern, powerful machine, it might be able to bring more resources to bear on the problem at one time.

But sometimes, large complex problems take a lot of time to process and that is just the nature of the beast.

And sometimes ColdFusion is not the best choice for large and complex problems. What ColdFusion is ever becoming better at is being a glue between more powerful processors for large and complex problems.

I.E. the idea here maybe that this core process in the heart of your process could be more efficiently done in Java. But the nice thing here is that you wouldn't need to replace the entire software with a Java version. Just create a single Java module for the core, heavy lifting part and then use ColdFusion's abilty to leverage Java to tie that into the rest of the application.

P.S. Another idea might be to look at the gateway functionality in modern ColdFusion servers. This would be another way to achieve asynchronous processing where you could get multiple parts of the application to be run in parallel at the same time. And could possibly be leveraged together with either the <cfthread...> and|or the Java options.