Can't add CSV file to timeline

[FUKU5C88]

Hello.

I've got a 3.6MB CSV-file(attached) which I want to add to my timeline in order to export the footage with media encoder. The CSV file is not corrupted since I can add it to my project and access it's data correctly in my animations. 

Only adding that file to my timeline does not work. I've even created a new project with basically nothing and the results were the same:

When I drag the file to my timeline the RAM usage increases. First with steps of 50mb but later only with 1mb steps. I've let that process run for about 2h and ended up with a memory consumption of 13gb. AE is not responding during that whole process. 

Json doesn't work either. With a simple converter the resulted file was 20mb and the process was the same...

Do you have any ideas?? This problem really suxs... 

System specs:

- Ryzen 3950x

- 4x 16GB 3200/cl16 RAM

- GTX 1080ti (drivers up to date)

- two different nvme drives for cache and project files

- win10 (up to date)

- AE (up to date)

[Roland Kahlenberg -]

Your last option is to use TSV.
And how large is the dataset in the CSV? Have you tried another CSV that's worked previously?

[FUKU5C88]

Haven't tried TSV yet.

The file is attached. It has about 66.000 lines and about 10 columns or so. 

If I shorten this file it works. But that can't be a solution...

I can even use 14MB CSV files in my project - this works fine but adding a 3.6MB file to the timeline doesn't... =\

[FUKU5C88]

Oof I have somehow choosing my reply as working answer while trying to edit it 😂 not sure if I can undo that... Can't find an option on smartphone... 

I wanted to correct: shorting it down to 10.000 or 5.000 lines*

[FUKU5C88]

Okey found the button.. weird that I can't edit my own posts 😂

[Roland Kahlenberg -]

That's a quick way to earn a bonus point! 😄

Try validating your CSV here – https://csv.openbridge.dev/

If the CSV is still an issue, file a Bug Report together with the CSV.

On troubleshooting, try splitting up the CSV into two parts and see if this works - of course, validate the CSVs first.

[FUKU5C88]

Haha yes 😄 I'm new in here... 😇

I'll give that a try but I don't think the CSV files are faulty. They are exported through a Java CSV library and they work while being in the project. I can access each and every value in my animations but I can't add the file to the timeline which is necessary for media encoder export. 

We will see 🙂

I'd love to open that bug report then 😂 but I have no idea where to do that.

[Roland Kahlenberg -]

To send a Bug Report, AE's Help Menu has an item named "Provide Feedback". A similarly named link (to the right edge of the page) is available in the Main Page of this forum.

Do update us over here.

[FUKU5C88]

So i have tested my CSV File on https://csv.openbridge.dev/ :

I havent tested TSV but it must be the same.... i mean its basically the same as CSV

[FUKU5C88]

So i have tried TSV - same result as CSV. 

I have tried JSON with arrays  (so only 5MB file instead of the previous 20mb) - same result.

I have tried splitting up the records of that one 3.6MB file. Resulted in 1 file per column -> about 240kb file -> same result. I've even uploaded a video about that problem: https://www.youtube.com/watch?v=TGy52imsbIg (it's possible that this video is still being processed - Endresult is 1440p60)

Gonna open that bug feedback thingy now... =/

[Roland Kahlenberg -]

That's a PITA. Please send in the BR.
And how are you intending to complete this project?

My thoughts are to try and split the JSON in 1/2. Then bake the Expressions with  the Easy Bake script from aescripts.com. Then load the second half and do the same. Or perhaps two 1/2 sized CSVs might work instead of one full-sized CSV?

[FUKU5C88 · Accepted solution]

With timestamps and sliders.

I've imported the small CSV (and other format) files. When putting them into my timeline AE creates a new "line" for each value.... This is what takes all the time and is not very effective.

Sliders instead do basically the same. They store a value bound to a key frame. And very interestingly its a TEXT-Format that is being copied when you select multiple timstamps of a slider and copy them. In addition to that you can copy and paste the timestamps of multiple sliders at once:

So my Data-Basis is still CSV but instead of filtering (calculating additional values) and converting that thing to CSV i just print a text file which i then copy and paste onto the sliders.... 🙂

This works very nice and even has a few advantages:

+ If the project is 50 FPS and my data was only recorded at 10hz it automatically smoothes the values in between. Previously i did this through code expressions.

+ No need to have After effects opened in order to have updating values in premiere pro*

+ After effects -> Media encoder exporting works now

+ Premiere Pro* -> Media encoder exporting works now

*This project of course uses the AE Composition 😉

[FUKU5C88]

 Oh i forgot to add the BR:
https://adobe-video.uservoice.com/forums/911311-after-effects/suggestions/42111511-problem-with-addi...

Rendering well... 😂 20 Min-Video = 24h(increasing) Thanks to Shadow/Highlight effect... :

First 4 threads are not attached to the rendering process.

[Roland Kahlenberg -]

I think it's stuff like these that let to talk on developing a JSONMoGraph thingy but it's been about 3 years since and I've not heard much about it. And thanks for posting the BR.