Json - faster parsing (scripting)

Forum|Forum|3 years ago
June 5, 2022
3 replies
845 views

I need to parse large json files via scripting in After Effects. Currently I use the json2 library.

It works well but it's very very slow. Something that is almost instant in Python can take several minutes. I've searched around and I haven't found a solution except for using an eval call on the string which I'm not to happy with as it's a potential security risk.

Are there any solutions to this or any other third-party libraries that are faster? If not, is there any way to make the "eval" call safer?

Scripting

This topic has been closed for replies.

SimonBjörkAuthor

Known Participant

-

Mathias Moehl

Community Expert

The eval workaround is also the only one I am aware of for ExtendScript. Would love to hear of better solutions!

Mathias Möhl - Developer of tools like BeatEdit and Automation Blocks for Premiere Pro and After Effects

SimonBjörkAuthor

Known Participant

I tried to find any alternative libraries, such as json_sans_eval but the result was similar. It's horribly slow. I have a json file that is about 10mb and it's mostly very long arrays (more then 10.000 items). Currently it takes about 25 minutes to parse this. In python it's done in a couple of seconds.

I also found out that the json2 library is using eval as well, but does a bit of security checks first. It should be fast though. Any idea why this is? I'm assuming it's becuse of the long lists?

There's also other libraries that I found, for example fast-json but I'm not sure how this works (can't find the actual code). Are these really compatible to use in After Effects scripting?

Also, is there any info on adding a native json parser?

Mathias Moehl

Community Expert

If you want to dig that deep, maybe you can figure out somehow, which parts of the parsing process take that much time. When creating very long text files line by line with ExtendScript, I noticed that accululating the final text with

result += nextLine;

many many times was MUCH slower than pushing all lines to an array and then doing an array.join at the end.

The longer "result" is, the slower the += becomes and this is only the case for ExtendScript and not for modern JavaScript implementations.

Not sure how json2 does implements JSON.stringify, but if it uses + to concatenate the values, maybe it could be optimized in the same way. For parsing, I guess contacenating strings is not the issue, but you could investigate which other operations are performed and if any of those becomes slow on huge strings.

Mathias Möhl - Developer of tools like BeatEdit and Automation Blocks for Premiere Pro and After Effects

Mylenium

Legend

If not, is there any way to make the "eval" call safer?

I don't think so. By it's nature it's supposed to just interpret the code directly as working script or binary data. Any check would happen after that like e.g. browsers do byte checks in their rendering engines or limit the scope to specific DOM and data types. that's why I also don't think you'll have much luck finding faster or better alternatives due to how this is supposed to work. You'd have to replace the script engine rather than the code feeding it.

Mylenium

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded