Render Problems - CSV files and After Effects

[Blending 5FD0]

Hello!

I need help to better explore the functionality of linking CSV files with After Effects projects.

I have some projects that I need to render very often, the video design of these projects is exactly the same, what changes is the text information and in some projects I also need to change the backgroundcolor.

With that in mind, I decided to explore the potential of After Effects template creation further and use a CSV file to change the text information and create some conditionals to change the background color according to what is written in the CSV file.

This allows me to create these videos much faster and put to render directly in Media Encoder or using a .bat file to render it with aerender, so I don't even need to open the project.

The issue is that for some reason in some videos the CSV information is lost in the middle of rendering and the video texts start to glitch and switch frantically between the information in the CSV file and the information that was in the Text Layer...

This forces me to check all the videos in full and put the videos that had a problem to re-render...

This CSV functionality is extremely cool and can speed up many workflows, but it definitely needs some bug fixing to work flawlessly.

Would the Adobe team please take a look at this?

A suggestion on my part would be to add a conditional in the programming that behaves as follows:

- IF the Layer is linked to a CSV file, get the necessary information in the first frame of that layer and store it.

That way, the information would not be linked during the entire rendering process, which would possibly avoid any connection problems between After Effects and the CSV file.

If this works as expected, it will certainly help many professionals and greatly expand the possibilities.

Additional Information

- These template projects have only one composition in the project.

- Projects are rendered in H.264.

- I render in Media Encoder by copying the project to a watched folder

- Using the .bat command in Aerender the glitch problems are even bigger

- The bug does not occur in all videos, which indicates that probably in some renderings there is a bug that causes the link with the CSV file to be lost and continuously reconnected

[Mylenium]

None of this makes any sense to anyone if you don't provide proper system info, examples of the CSV files, info on how they are linked, screenshots, possibly project files. Just saying that you have problems is of no use to anyone without the pertinent details.

Mylenium 

[Blending 5FD0]

If you carefully read the huge text that I published, where I explain the problem in detail and then you simply try to throw CSV file into an After Effects project, you will probably be able to understand it, but you have to try.

Basically, the problem is that every now and then the link to the CSV file is lost in the middle of rendering.

As I said, you just create a project, create a text layer and link the source text to a CSV spreadsheet.

But since I believe I can't count on you to do this, I created the project for you and the CSV file.

They can be downloaded here: AE + CSV | Bug Demonstration

Change the information in the CSV file and play it in Media Encoder to render, with each render change the CSV file information again.

You will see that information is lost from time to time.

As for my project, unfortunately I can't share it because it contains sensitive information and code that I don't want to share.

[Blending 5FD0]

Important detail:
Once you download the project you must open it, relink the CSV file and save the project.

Even not asking to relink the file, for some reason when the project goes to another folder it is necessary to relink so that the Media Encoder render takes the information from the CSV.

Run the tests using a watch folder.

[JohnColombo]

Hi @Blending 5FD0,

Thanks very much for reporting this issue. We have had some issues opened in the past regarding scenarios where rendering through AME or aerender did not correctly pick up changes to CSV footage between renders, which sounds similar to what you've reported. Would you mind answering a few questions to provide additional detail for our team to investigate?

1. Are you on macOS or Windows? Your mention of .bat files seems to point to Windows, but asking just in case.
2.  When you copy the project to the watched folder for AME, do you copy/move the CSV as well?
3. Do you ever see mixtures between different values from the CSV, e.g. from a past value, or only between current CSV values and the original Text layer values?
   
   It's possible the bug has to do with either moving the project file (the CSV cannot be found for some frames) or possibly something to do with the frame caching. Since your expression link to the CSV footage does not change over time, perhaps try adding `posterizeTime(0)` to the start of the expression so that it uses the same value for the whole comp. This would be similar in practice to what you've suggested as a fix and will also help us identify the possible point of failure.

Thanks again for reporting this issue and for any further information you can provide,

- John, After Effects Engineering Team 

[Blending 5FD0]

Hi John,

First I want to thank you for making yourself available to understand the problem and for proposing some initial solutions.

It's great to have a direct channel where we can interact with the development team and help with improvements and bug fixes.

About problem with CSV files

To be more illustrative, I put the project of one of my clients to render today several times and so I managed to reproduce the bug, I had some files that had the bug, but how to publish them could imply problems with copyright due to the videos used, I decided to use some royalty free images.

Attached to the publication are 4 videos:

Video 1 - This video rendered correctly, did not show any bugs

Video 2 - This video was rendered right after the first one and presented the bugs reported in the publication, see that the synchronization with the CSV file was lost during rendering and the text presented glitches when the synchronization was lost. In this case, the background didn't keep changing color because in the project it is defined that the default color is blue, otherwise the background would have presented glitches as well.

Video 3 - This video was rendered right after video 2, which presented problems, notice that this video was rendered correctly and did not present any problems.

Video 4 - This video was rendered right after video 3, in this case I even changed the video file, to do that I replace the video file that is linked with the After Effects project with another file with the same name. See that this video was also rendered correctly, which shows that the problem really seems to happen because for some reason sometimes the link with the CSV file is lost.

Answering your questions

1. I'm on Windows 11 Pro, I tested it in 3 different configurations using Windows 11 Pro, in all of them I had this problem with CSV files.

2. No, the CSV file is in a folder inside the project folder.

3. Mixtures do not happen with previously typed values, only between what is in the Text Layer and what is in the CSV file, so I believe it is not a cache related bug, it seems that the link with the file is lost and this causes cause After Effects to use the text on the Text Layer.

Remembering, to render these videos I just copied the After Effects projects to a watch folder, changing the information in the CSV at each render.

As I commented initially, my idea with these projects is to explore the potential of After Effects template creation to the fullest, and as you can see, the workflow is almost functional, this possibility of generating videos with information from CSV files is fantastic!

However, due to these current bugs, the process ends up not being as scalable, on average, one out of every ten renders has a problem.

I believe that with some change in the way After Effects gets information from CSV files this can work perfectly.

If I can help with anything else to troubleshoot this issue, please let me know, I'll be happy to test and help with that.

[Blending 5FD0]

Yesterday I took some more time to test some renders and managed to reproduce the problem even more clearly.

Below are 2 videos demonstrating the link loss problem:

Video 1 - notice that the video started to render correctly, but in second 21 the render loses the link and starts to have glitches between what is in the CSV and what is in the project, this can be seen more clearly through the alternation of background colors and final signature screen.

Video 2 - This video started rendering wrong and then the link was established and ended up rendering with the CSV information.

My CSV file has the following structure:

Title,Description,Color

If I can help with any more information, please let me know.

[Blending 5FD0]

Any chance the After Effects and Media Encoder team will look into this issue and try to resolve it soon? Or should I not count on it?

[JohnColombo]

Hi @Blending 5FD0,

I was able to reproduce what you described on a Windows system using the AME watch folders; I got one or two good renders, but they quickly started showing frames that behaved as if the file didn't exist or wasn't referenced properly. I'll be showing this to our engineers to see what can be done. Thanks again for the detailed description of the issue and for the files to reproduce it.

That said, in looking at your project, I saw your expressions are not using the proper declaration keyword for your variables. Contrary to many highly-shared examples on the internet, declaration keywords actually do make a difference for expressions, both in performance and stability. See here for more details: https://www.w3schools.com/js/js_variables.asp.

As an experiment, I add the "let" keyword to the `Ref` variable as follows in the expression for the color:

let Ref = thisComp.layer("Adobe Forum....

After adding the declaration for the variable, a newly saved project no longer showed the glitching frames or pre-expression values when rendered repeatedly with different .csv values through AME. Would you give this a try and see if it makes a difference on your system as well?

Cheers,

John, After Effects Engineering Team

[Blending 5FD0]

Perfect, John!
In the meantime, I'm going to adjust my expressions with the declaration keywords and do new tests. I usually use them whenever I'm creating something in Javascript, but I really had no idea that the declarations had an effect in After Effects, it's really good to know, I'm going to start using it in all my projects.

Thank you very much for your attention and patience in understanding the problem, I will be happy to follow up on this issue and help test future versions of AME and AE.

If I can help with anything else, let me know.

Yours sincerely,
Tyrone Hirt Stachera

[Blending 5FD0]

Hello John!
Just bringing some updates regarding this issue.

I'm still having several problems with desynchronization of CSV information when rendering projects through AME (or AE Render).

I tweaked all my projects with the declaration keyword to make sure that wasn't the problem, but I still have glitched renders.

If you can show this to your AE and AME engineering colleagues I would be very happy, I believe making some changes to make sure that the information from the CSV file is read before starting the render and that the initial information is maintained until the finalization, it can work perfectly.

It's probably not as simple as it seems, but I really believe that this mechanism has great potential and that the development team has the competence to solve it quickly.

Thanks for all the attention so far!

Tyrone Hirt Stachera