AE 26 Advanced 3D Render Stops Rendering with Out Of Memory Error (Mac and Windows)

Issue

Rendering a composition using the Advanced 3D Renderer causes memory usage to steadily increase throughout the export until After Effects reaches its memory limit and fails with the following error: "After Effects error: After Effects has unexpectedly run out of memory and cannot complete this export." The issue reproduces consistently on both macOS and Windows and appears to affect all versions of After Effects after 25.4. In addition to the render failure, After Effects does not appear to fully release memory after the error occurs, leaving the application in an unstable state until it is restarted.

Steps to Reproduce

1. Create a composition using the Advanced 3D Renderer.

2. Add multiple 3D layers that are pre-compositions.

3. Have those prec-ompositions contain additional nested pre-compositions.

4. Configure the nested prec-ompositions so they reference different sections of a large master composition.

5. Apply Geometry Options > Curvature to some or all of the 3D layers.

6. Render the composition to QuickTime ProRes 422.

7. Monitor memory usage during the export.

Link to project file below contains a simplified version that reproduces the issue.

Expected Result - The render completes successfully and memory usage remains stable throughout the export.

Actual Result - The render stops partway through the export and displays the following error: "After Effects error: After Effects has unexpectedly run out of memory and cannot complete this export." The failure typically occurs between frames 200 and 1000, with the supplied sample project usually failing around frame 311.

After the error occurs:

- Memory usage remains unusually high.

- Purging Memory and Disk Cache only partially releases the allocated memory.

- Subsequent renders often fail immediately or within a few frames.

- After Effects must be quit and restarted before rendering can continue.

Memory Behavior Observed - During rendering, After Effects memory usage steadily increases frame after frame and does not appear to be released as frames are completed.

Observed behavior:

- RAM usage continually climbs throughout the export.

- Memory usage eventually reaches the configured After Effects memory limit.

- The render then fails with the out-of-memory error.

- Memory usage remains elevated after the failure.

- Purging Memory and Disk Cache only partially reduces memory consumption.

- Subsequent render attempts usually fail immediately or very quickly.

- Restarting After Effects restores normal memory usage and allows another render attempt.

This behavior is consistent with a memory leak or memory management regression within the Advanced 3D Renderer.

After Effects Version - After Effects 26.2.1 (Build 2) - Reproduced in all tested versions after 25.4

Operating Systems - macOS- macOS 26.3, Windows - Windows 11 Pro, Version 10.0.26200 ,Build 26200

GPU Driver Windows- NVIDIA Game Ready Driver 591.86

The issue reproduces regardless of platform.

Output Format: QuickTime, Apple ProRes 422, Trillions of Colors

Comparative Testing

Renderer Comparison:

- Advanced 3D Renderer: Fails with out-of-memory error.

- Classic 3D Renderer: Completes successfully.

Version Comparison:

- After Effects 25.4: Completes successfully.

- After Effects 25.5 and later: Fails with out-of-memory error.

Multi-Frame Rendering:

- Enabled: Fails.

- Disabled: Fails.

Disabling Multi-Frame Rendering changes the frame at which the failure occurs but does not prevent the issue.

Project Complexity

Increasing project complexity causes the failure to occur sooner. Factors that appear to accelerate the problem include:

- Higher layer counts

- Additional nested precompositions

- Larger master compositions

- FBX assets

- More Advanced 3D geometry operations

Additional Notes

- The issue occurs in both newly created projects and existing projects.

- The issue reproduces on both macOS and Windows.

- Available system RAM is not exhausted prior to failure; rather, After Effects memory consumption continuously increases until reaching its internal limit.

- Because the same projects render successfully in 25.4 and in the Classic 3D Renderer, this appears to be a regression specific to the Advanced 3D Renderer introduced after 25.4.

Sample Project File: Dropbox Link