Why do INDD files bloat over time?

[James Gifford—NitroPress]

Is there any simple reason that ID files slowly bloat over time, sometimes to absurd sizes? I know most "intelligent" doc formats keep adding app data like undo content, macros, etc., and Word is absolutely terrible at ever cleaning out or removing any of it. Cutting the content to a new file is often the only solution.

But I just had an experience with an INDD file that shocked me. Working on a long-time, episodic project (the book in the footer, actually), I started getting frustraing problems with EPUB export. Tried every fix including a wholesale cut and drop into a new doc. No go. So I did a laborious move of content to a new INDD file, which of course meant time rebuilding the TOC and some other details... but bingo. All of the quirky problems went away. Time well spent.

Then I compared the files. The new one is about 440k. The older one was a whopping 9.5M! What does ID do to keep adding junk to the basement when a document is edited and maintained over a long period of time?

[TᴀW · Accepted solution]

The extra stuff is the "undo" information, which is stored with the document (even though it's unusable as soon as the doc is closed).

Doing a "Save As" will clear out this bloat.

It's been like this forever AFAIK.

Ariel

[James Gifford—NitroPress]

I figured undo was a big part of it. However, I 'Save As' at regular intervals to archive development versions, and in trying to debug the glitches I also did an IDML export/open. I'm not sure if any of those fixed the file size but none fixed the weird glitches.

I'll keep an eye on Save As results next time.

[rob day]

Hi @James Gifford—NitroPress, There also can be a problem with placed images bringing Photoshop ancestor metadata into the document. If that’s happening the metadata has to be cleared out of the placed files.

Ancestor metadata bloat usually is more extreme than your example—see this thread which includes a metadata clearing script, in it I posted an example of a single page ID doc, with one flattened PSD link, saving at 256MB:

https://community.adobe.com/t5/indesign-discussions/is-there-a-script-to-clear-indesign-metadata-red...

[James Gifford—NitroPress]

I have a few images in this doc (actually just one, placed at several different scalings) (plus a few small logo files) but it's not large and it's not heavy on metadata. The same image is in the newly rebuilt file as well.

I just don't think I've ever seen a doc file do a ~20X bloat, not even the most corrupt, circulated Word doc. 🙂

[rob day]

but it's not large and it's not heavy on metadata

The bloated metadata is ancestor metadata coming from Photoshop and isn’t always obvious in the InDesign file. The problem you are experiencing might not be related, but there are a number threads in the Photoshop forum on the ancestor problem.

This 8-bit blank 1920 x1080 pixel canvas weighs 65MB—the problem metadata is buried in the Raw Data:

When this file gets placed, the Raw Data can replicate itself in the ID file:

After running the ID script:

[James Gifford—NitroPress]

Got all that; thanks for the detailed explanation.

However, does this apply to anything but placed PSDs? (Or perhaps TIFFs?) In a case such as mine, with a ~100k JPEG that's been exported to web standards from Photoshop, is there any place where this crypto data could be pulled into the ID document? As noted, the cleaned up document, with the same image (just a placeholder image of a jewel, used for scaling discussions) is ~400k. If there's not extra data in there, would subsequent edits or manipulation pull it in?

(All noted for when I do work with more demanding images at high res and cycled through PS manipulation.)

[rob day]

It can happen with any image format, but the only way to know if it’s affecting your document would be to run the ID script I posted:

https://assets.adobe.com/public/9a2cbd1d-bd8d-4fdc-5232-98128f54348b

In the example I’m showing above, I can delete the placed image files, save the blank page and the ID file size is still 453MB:

Here’s a blog post from @Stephen_A_Marsh :

https://prepression.blogspot.com/2017/06/metadata-bloat-photoshopdocumentancestors.html

And a Photoshop thread from 2017:

https://community.adobe.com/t5/indesign-discussions/file-size-is-too-big/td-p/9370587/page/3

[Creamer Training]

Besides the undo's and metadata, I wonder if some of the bloat is a "fast save" feature. Acrobat used to have this problem; in order to save faster, the software did not rewrite the file during save, but simply recorded the changes. 

An example I used to use in my classes was to have a PDF with 100 pages and deleting 99 of them. A proper save would leave just 1 page--a "fast save" remembers the 100-99 pages even though once can't access the deleted pages. (A proper save would be the equivalent of a save-as.)

[Derek Cross]

IDML cleans up the document and rebuilds it, Save As doesn't, it just adds an extra layer of data. So if you want to clean up your file and start with a blank slate and avoid the build up of crud that can cause document corruption, Export your file as an IDML document.

[James Gifford—NitroPress]

I haven't fully explored Bob's information above, but here's my results, from experimenting before I made this post:

• Original file, with EPUB export issues: 9400k
• IDML export: 430k
• IDML save-as: 430k
• Save-As: 4600k

None of the save-as'es fixed the glitchy issues, so I moved the content to a new file (in sections; a wholesale move didn't fix things), which started out in the ~500k range again.

After a day of fairly minor work, text editing mostly, it's back to 7500k. But without glitches (again) (yet).

Sigh.

[rob day]

If it is a case of metadata coming from links, a Save As to IDML doesn’t help much:

[rob day]

Can you share a packaged document?

[James Gifford—NitroPress]

It's proprietary.

However, I did the following experiment. On a copy, I deleted all the graphics (3 JPGs, one used several times). Nothing left in the Links panel even though I am sure there are no "forgotten" placed elements.

• Export to IDML: 400k.
• Save stripped file: 7500k
• Open IDML, save as new INDD: 3500k.
• Do some random editing (cut and paste a few paragraphs), save... 6500k.

If this has anything to do with images and metadata, it's being damned persistent about it. 🙂

[rob day]

Right, that is one of the symptoms of ancestor metadata, I posted an example above with all the content deleted weighing 453 Megabytes. Have you tried running the script? Run it on a packaged copy with the images.

[James Gifford—NitroPress]

I ran the script. It appeared to reduce the INDD file from 7 to 4 MB. But a little trivial editing immediately expanded the file to 7 MB again.

The three images involved are 100k and two about 4-5k. They simply do not have any excess metadata in them, and I don't believe PS — under Windows, at least — stores historical data for work files separately.

This isn't a problem, per se; it's not causing me any workflow problems and I'm not concerned about it, other than that something (probably pointless) is going on under the hood to create these huge file expansions (from 0.5MB to 4MB to 7MB with essentially the same document file).

—

[rob day]

I was only suggesting the metadata bug was a possible cause—sorry the script didn’t work. The bug has been discussed and is listed in Adobe’s bug history—see January 2019 (version 20.0.2) release on the bug fix page:

https://helpx.adobe.com/photoshop/kb/fixed-issues-history.html

I don't believe PS — under Windows, at least — stores historical data for work files separately

Here‘s an example with 65MB of metadata—look at File Info>Raw Data—if I place this file 5x, the ID doc will bloat to over a Gigabyte:

https://shared-assets.adobe.com/link/7ea0589d-b86a-4fdf-65cb-e9add7cc119a

 

[Tana Reiff]

I'm two years late to the conversation and not techie enough to even ask for a solution. Just wanted to add to the conversation that I've had a possibly related problem with InDesign files exporting to incredibly bloated PDFs after replacing .jpg images with smaller files. The InDesign files were smaller but the PDFs were 5x larger - from 96mb to over 500mb. I did a Save As every time, sometimes more than one, and it still happened. The PDFs did not increase in size each time. They simply quintupled. I was trying to submit a high-quality PDF to KDP (Kindle Direct Publishing) and followed all their guidelines but KDP came up with error messages on the upload, or the upload succeeded but the file processing failed. The numerous bloated PDFs look fine. I finally went back to the 96mb PDF without all the replaced photos, even tho it would print a lot better if I could replace the photos. If anyone else has experienced this, maybe I could benefit for the next time.

[James Gifford—NitroPress]

I'd say 90% of this has to do with your PDF export settings. Smaller files in ID don't necessary translate to reduced file sizes in PDF if you have high export PPI selected, and other things.

It's very much an end to end process.

[Tana Reiff]

Right. I believe settings were the same  - no downsampling and 300dpi. I didn't want to sacrifice any quality.

[rob day]

Right. I believe settings were the same - no downsampling and 300dpi. I didn't want to sacrifice any quality.

Just to clarify your image export Compression settings—if you’ve turned off downsampling, you can’t specify an export resolution. With downsampling turned off, the images would export at their Effective Resolution, which could be considerably higher than 300ppi depending on how you have scaled the images.

For example this image has a starting Actual Resolution of 300ppi, but has been scaled down to 31%, so its Effective Resolution is 943ppi, and that‘s the resolution that will export when Do Not Downsample is selected:

If I inspect the image in Acrobat I can see there is considerably more resolution (and file size) than needed:

If you are scaling your images down, your Compression settings would account for the overly large PDF file sizes.

[rob day]

The PDFs did not increase in size each time. They simply quintupled.

Hi @Tana Reiff , You can check a PDF’s space usage in AcrobatPro via Tools>Optimize PDF>Advanced Optimization>Audit Space Usage. If you have a metadata problem a large percentage of space will be in Document Overhead. Try discarding user data and saving an optimized PDF:

[Tana Reiff]

That is helpful - thanks.

[Tana Reiff]

p.s. I just checked that. You've nailed it. I went through advanced optimization on a 504mb file and a 96mb file. The larger one had the replaces images (smaller than what they replaced). In both PDFs, 99% of the file size was images. I discarded user data and the percentage stayed the same but the final file size of the huge PDF went down to 96mb. No degradation of quality. Sure wish I had posted my question sooner but I'll know for the next time. Many thanks.

Just one more question. Do you know if there's a way to discard user data in InDesign before exporting to PDF?

[rob day]

From your captures It’s not clear to whether excess metadata is the problem, but you shouldn’t see such a large increase exporting the same document multiple times.

The link to the metadata script I posted above no longer works, here’s an updated link:

https://shared-assets.adobe.com/link/9da08cd8-3bbe-4ba2-6631-79dd8b5e46c0