Random expression errors that solve themselves and not rendering in AME

[TASan]

I am very confused at the moment. I have been working long hours on a project that is an hour long, that uses data from two CSV-files to render different UI-elements. These are going to be set on top of another video later. There are issues, however.

After Effects is complaining about random expression errors that solve themself by just jumping a couple of frames forward. Also, frame 1 of the video shows the number 14 when the CSV has the number 45 at that index. It's correct on frame 0 and on frame 2, and seems correct for the rest of the video.

Lastly, the expressions are not evaluated when exporting with AME. Lossless directly from AE works, but I can't work with 200 mb per second files. Also, dynamic link just hangs a lot of times when I try to render in AME.

What would be the best way for me to give you all of the information needed to solve this issue? I don't have all the errors in my head, because they randomly appear and disappear. I could share the whole project, but preferably not publicly because I don't want anyone to just download and copy my work.

[Mylenium]

Start by telling us exactly where the expressions are used, what the data is, whether or not the data file has actually been added to the timeline, render settings, system info, possibly language and font settings and so on. Screenshots usually tell a lot. Some limitations have to be accepted, others may be avoidable be smartening up the code or using different procedures.

Mylenium

[TASan]

Thank you!

I’ll try to break it down and explain as best I can. This happens on two different systems. One powerful workstation and a weaker laptop. Both run Windows 10 and AE-version 18.1.0 (Build 38). I am using the English version and all my data have English characters.

The expressions are bringing in data from two different CSV-files. I am trying to build a workflow where I can just swap the CSV-files and the data will change. Therefore, I am referring to them as footage rather than dropping them on the timeline. If dropping them on the timeline is better, and I can still just swap the files out, please do tell.

The CSV-files are numerical and textual data where each row corresponds to 1 elapsed second. That way I can floor time and use that to bring out the correct data. I have set the composition length to be the same length in seconds as the CSV to avoid any problems there.

Here are my expressions:

On a source text layer to display a timer:

let sek = Math.floor(time);
footage("workout.csv").dataValue([5, sek]);

On a sound layer to play a 6 second sound clip when the timer hits 5:

if (Math.floor(time) >= 1) {
    let sek = Math.floor(time)-1;
    let currentCountDownTime = footage("workout.csv").dataValue([5, sek]);  
            if (currentCountDownTime <=6) {
                        (time%1)+6-currentCountDownTime;
            } else {
                        0;
            }
} else {
            //This else block seemed to fix one of the returning errors
            0;
}

This is on a shape layer with trim paths to animate a stroke:

let sek = Math.floor(time);
let st = footage("workout.csv").dataValue([8,sek]);
let tot = footage("workout.csv").dataValue([8,sek]) + footage("workout.csv").dataValue([9,sek]);

linear(time, tot, st, 100, 0);

And then I just have a lot of these expressions on text layers to display different things:

let sek = Math.floor(time);

"SPEED: " + footage("workout.csv").dataValue([6, sek]).toUpperCase();

Problem 1: Wrong data shown (solved)

Everything seemed to work fine when previewing and rendering in After Effects, except one thing. That is, I get all the data I expect, but for one frame in one layer. This expression is used:

let sek = Math.floor(time);
footage("workout.csv").dataValue([5, sek]);

I checked that “sek” was 0 on both working frames and the offending frame. I also checked that hardcoding index 0 gave the correct data on the offending frame. But the combination gave the wrong data, specifically 14 instead of 45. The default on the text layer was 60.

I have tried clearing all cache and turning everything on and off again. Nothing worked. And then suddenly after not having worked on it for a few hours it works again. I am just listing the error here for reference.

Problem 2: Random expression errors

I still seem to be getting some errors like this:

This one can happen for a lot of the layers with expressions. Note that "awesomeSek" is just from me trying different variable names because I thought I might have some collisions. This one can happen at the very start as well as the very middle of the timeline. I don't know why, because the length of the composition is the same as the CSV file. I don't quite understand it either. Is it saying that it tries to get index 3601, but that there are only 3600 present?

This one is fun, I think! It expects a semicolon, but the semicolon is already present. The expression also works perfectly fine.

This is the last one I have managed to get a screenshot of. Something is missing here, apparently, but not on the next line where the expression is basically exactly the same?

I don’t quite understand all these errors, and they appear and disappear randomly. I find it hard to pinpoint a cause. They can be there for one layer, and if I hop a few frames forward they disappear. Later the same error, or a different one, appear for a different layer, or the same.

Problem 3: No expressions are rendering in AME

When I do test renders losslessly, or in fact using any codec, straight from After Effects, everything shows up. When I try to do it from AME, nothing seems to work. The layers are stuck on their default settings (expressions not executing) or layers are just gone. Sometimes AME just hangs forever. It can hang while trying to connect to dynamic link with a progress bar at the bottom, but it can also hang without any progress bar and just nothing happening. I am trying to render using h.264.

Here is an example of expected result and actual render the few times I get it to work.

It can also be that the layer is visible, but static on the default value because the expressions are not evaluating.

Problem 4: How to render this in an efficient way?

Lastly, I am wondering what is the best way to render such a thing? It’s just an overlay that changes data every second, but the files get huge if the quality is to be good. Is it possible to render in a way that only each second gets rendered with "new data"? I am thinking almost like vector data that just changes. I am going to import this into Premiere Pro and render it on top of another video. I want to avoid double compression in addition to huge file sizes.

Thank you very much for your time in this matter!

[Mylenium · Accepted solution]

Well, you have a fundamental misunderstanding there. All this "let" stuff is pretty much useless and superfluous. Expressions strictly evaluate in the order the code is written and excessively declaring data objects and variables will cause unnecessary jumping around. That alone may explain 80% of your errors. Get rid of it and structure your code in a way that it evaluates linearly in the correct order. The rest appear to be simple "out of bounds" errors because your time value returns too large results or wrong ones. You may need to implement safeguards like limiting the maximum value or an explicit conversion to an integer value.

Mylenium

[TASan]

Thanks for clearing that up!

Is it so that expressions are sandboxed per expression window, layer or composition in a way? From where to where are they linearly evaluated? Where does a variable stop being "global"?

[TASan]

Sorry, I am just trying to understand. Isn't this an oxymoron?

"Expressions strictly evaluate in the order the code is written and excessively declaring data objects and variables will cause unnecessary jumping around."

If they strictly evaluate in the order the code is written, how can this cause unnecessary jumping around? I see that declaring the variable "sek" and then using it once might be superflous, but if using it more than once in an expression, the end result is better performance or at worst the same performance? It's also makes the code easier to read.

I might add that all the expressions I have posted are for different properties. I found this online and if it's correct, my declaring should not be causing any issues:

"An expression is a little piece of software — much like a script — that evaluates to a single value for a single layer property at a specific point in time."

You also write "Get rid of it and structure your code in a way that it evaluates linearly in the correct order.".

Get rid of what exactly? Do you mean I should change this:

let sek = Math.floor(time);
footage("workout.csv").dataValue([5, sek]);

To this?

footage("workout.csv").dataValue([5, Math.floor(time)]);

I can't see how that would make any difference in terms of errors here. It evaluates linearly in both cases.

I agree on the out of bounds errors though, and will set up some safe guards. But I would love to learn if I have misunderstood something fundamental about expressions, evaluation of them, and JavaScript in general.

[Mylenium]

You can declare variables using the usual kaboom=Math.kaboom(X,Y) syntax. This can of course be concatenated and reused into ping=kaboom*foo and whatever you fancy. However, there is no point in explicitly instanciating objects with let or using var. Expressions are strictly bound to the property stream they are applied to. That is all their scope and all references to outside data therefore are explicit as you already know. Of course the engine will honor jumps within the code, but in 20 years of AE I always found it best to not confuse the engine too much. So to recap: Keep it simple and only write the bits that are necessary. As per your examples something like

sek=Math.floor(time);
st=footage("workout.csv").dataValue([8,sek]);

will be more useful than more complex code constructs. If you still have issues, you may further harden your code with the usual thisComp.layer() stuff or to make yoru references even more explicit just like you may want to use the built-in time conversion methods like framesToTime(). Depending on your project, some simple math methods return a wrong time due to AE's internal concept of "fluid time" and rounding errors making a mess. Finally, you may also want to comb through your data cells/ nodes in your CSV in a for() loop to ensure that a) you force AE to look for the correct value and b) you don't get values from lines that don't exist. And of course the usual: Add the data source as a layer to the comp, make sure the files are okay and so on...

Mylenium

[TASan]

Got it, keep it simple! 🙂

All of the errors disappeared after ensuring I did not get out of bounds. I still see the value in just keeping it simple though. The last problem of rendering this in a good way was solved by simply dragging the CSV-files into the layer panel. I did not change any code to reference them in any other way than I already did.

But this seemed to add them to the project so that I could use dynamic link in Premiere. The project file went from 1 Mb to 114 Mb, so something happened. Then I can render in AME and get just one render of a good enough quality and I don't have to render two times!