Json - faster parsing (scripting)

[SimonBjörk]

I need to parse large json files via scripting in After Effects. Currently I use the json2 library.

It works well but it's very very slow. Something that is almost instant in Python can take several minutes. I've searched around and I haven't found a solution except for using an eval call on the string which I'm not to happy with as it's a potential security risk.

Are there any solutions to this or any other third-party libraries that are faster? If not, is there any way to make the "eval" call safer?

[Mylenium]

If not, is there any way to make the "eval" call safer?

I don't think so. By it's nature it's supposed to just interpret the code directly as working script or binary data. Any check would happen after that like e.g. browsers do byte checks in their rendering engines or limit the scope to specific DOM and data types. that's why I also don't think you'll have much luck finding faster or better alternatives due to how this is supposed to work. You'd have to replace the script engine rather than the code feeding it.

Mylenium

[Mathias Moehl]

The eval workaround is also the only one I am aware of for ExtendScript. Would love to hear of better solutions!

[SimonBjörk]

I tried to find any alternative libraries, such as json_sans_eval but the result was similar. It's horribly slow. I have a json file that is about 10mb and it's mostly very long arrays (more then 10.000 items). Currently it takes about 25 minutes to parse this. In python it's done in a couple of seconds.

I also found out that the json2 library is using eval as well, but does a bit of security checks first. It should be fast though. Any idea why this is? I'm assuming it's becuse of the long lists?

There's also other libraries that I found, for example fast-json but I'm not sure how this works (can't find the actual code). Are these really compatible to use in After Effects scripting?

Also, is there any info on adding a native json parser?

[Mathias Moehl]

If you want to dig that deep, maybe you can figure out somehow, which parts of the parsing process take that much time. When creating very long text files line by line with ExtendScript, I noticed that accululating the final text with

result += nextLine;

many many times was MUCH slower than pushing all lines to an array and then doing an array.join at the end.

The longer "result" is, the slower the += becomes and this is only the case for ExtendScript and not for modern JavaScript implementations.

Not sure how json2 does implements JSON.stringify, but if it uses + to concatenate the values, maybe it could be optimized in the same way. For parsing, I guess contacenating strings is not the issue, but you could investigate which other operations are performed and if any of those becomes slow on huge strings. 

[SimonBjörk]

I guess it's the regex stuff that's slow? Becuse I think it pretty much is just regex checks and then an eval.

Btw, for stringify it seems to work well to use obj.toSource() and then just some string manipulation (adding quotes). Much faster.

[SimonBjörk]

-