Files larger then 2GB with Mac OS X

[Thomas_Rauscher]

Hello,

I am trying to write a BigTIFF plugin, and of course the whole point (compared to TIFF) is to write files > 2GB. The plugin works fine on Windows but on Mac (tested with CS3/CS5 32bit/64bit) I always get the error message "Could not write <filename> because the document exceeds the 2.00G limit for saving <plugin name> files". The same error happens with the "SimpleFormat" plugin.

How can I lift this file limit on the Mac? Is there a flag somewhere that I have to set to make this work?

MfG, Thomas.

[Chris Cox]

Change the PiPL to tell Photoshop that your format has a larger limit.

Without a file limit PiPL entry, Photoshop assumes a 2 Gig limit (for legacy plugins).

[Thomas_Rauscher]

Hello Chris,

of course the PiPL was the place I looked, but I could not and still can not find a setting that would allow this. If I search the documentation for "limit" nothing shows up. Would you mind being more specific and tell me which setting I have to change? I already have

     PlugInMaxSize { 2147483647, 2147483647 },

     FormatMaxSize { { 32767, 32767 } },

set, but this doesn't do the trick.

Mfg, Thomas.

[Chris Cox]

You're telling Photoshop that your plugin is limited to 2 billion or 32767 pixel dimensions.

You didn't set the file size limit.

[Thomas_Rauscher]

Hello Chris,

so how can I set the file size limit?

MfG, Thomas.

[Chris Cox]

Hmm, I know I added a PiPL property for that at one point... now I can't find the @%$@#% documentation for it.

[qu1j0t3]

I didn't recall such a property either, Chris. I think what actually happened is that Photoshop switched to the new Fork calls, which are free of the limit. Perhaps the OP is not using Fork calls to read/write?

I have sample code which deals with large files in plugins, if interested.

[Chris Cox]

There is an explicit check in the Photoshop code for files over the format size limit, and by default plugins have a limit of 2 Gig.

I know I added a property for that, but I can't find the documentation.  I have a bad feeling that I added it in a branch that didn't get integrated correctly, and because none of our plugins seem to go over 2 Gig, we failed to notice.

Grrrrr, ok, tomorrow I do a full code search at the office (including all my branches).

Unfortunately, if the property is not there, then I won't be able to add it until the next major version.

[qu1j0t3]

Chris,

Is it necessary to add a selector? Why can't Photoshop just skip this check in all cases?

[Chris Cox]

We wanted error checking, but maybe it's better to skip the checks on plugins.

[Thomas_Rauscher]

If you can provide me with sample code that CAN write files larger then 2GB I would be more then happy.

I use 64bit offsets with FSReadFork,FSWriteFork, FSSetForkPosition etc. and the same code works outside of Photoshop. The file is written correctly but at the very end Photoshop decides it is larger then 2GB and deletes it . Reading 10GByte BigTIFFs also works just fine.

Btw: why is the limit not enforced for reading, and also why does the same plugin work on Windows?

[qu1j0t3]

Thomas

I was mistaken about writing. If I attempt to write a > 2GB file, even using forks, Photoshop CS3 on Mac aborts the attempt after 2GB.

Reading works fine.

--Toby

[ilvar]

We actually have been supporting reading BigTIFF through our Geographic Imager plugin for quite a while and yeah, there is no size limit on read.  However at the moment we are in process of testing our writer, and I guess it would be pretty much a showstopped for the BigTIFF Mac writer support as the whole point of this format is to handle files that ARE larger than a couple of gigs. Chris, so did you have a chance to look at the code? Is there really _any_ workaround for this check?

[Chris Cox]

Currently, there is no workaround, sorry.

We appear to have lost some code in an integration SNAFU, and nobody noticed until now.

[Thomas_Rauscher]

Any chance that this will be fixed, and if so, in which way?

[Chris Cox]

At this point it would have to be fixed in the next major version of Photoshop.

The changes are a bit risky for a dot release, and aren't addressing a crash or major customer problem (just a potential problem).

[paulheckbert]

Thomas Rauscher:

It sounds like you're trying to create a BIGTIFF "file format plugin".  I haven't tried that type of plugin, but I've done some experiments with Export plugins and was able to modify the code provided for the export plugin called Outbound, so that it can write huge Photoshop Raw files.  As provided, the Outbound source code is limited to pictures with width and height less than or equal to 32,767, because it (archaically) uses 16 bit variables for image size.

I went to

http://www.adobe.com/devnet/photoshop/sdk.html

got the CS5 SDK for Photoshop plugins,

building with xcode,

modified the code in samplecode/export/outbound/common/Outbound.{cpp,h,r} as follows:

in DoPrepare():

  don't limit yourself to .5 MB of scratch space (sheesh - what year is this, 1990?) but instead much more,

  add the statement "gStuff->PluginUsing32BitCoordinates = true;",

removed casts to (short),

don't use gStuff->imageSize or gStuff->theRect or Rect, but instead use gStuff->imageSize32 and gStuff->theRect32 and VRect,

in Outbound.r, modified PiPL section, adding the lines:

+    // max size this plugin works with,
+    // plugin name will be grayed in export list on images larger than this
+    PlugInMaxSize {0x7fffffff, 0x7fffffff},

With these changes, I was able to export a 5GB file from Photoshop CS4, using the Outbound plugin.  It's still limited by Photoshop's (also archaic) 300,000 pixel cap on image width and image height, unfortunately.  So I don't know about file format plugins, but it appears possible, today, to write an export plugin that could write BIGTIFF files larger than 4GB.

[Thomas_Rauscher]

Thank you Paul for that tip!

It is a ugly work around to save files with the "Export" function and you lose built in logic but it works.

Btw: I guess we both have the same use case (GPixel panoramas) for this functionality. I think in my case the final solution may be putting the tiling into the Photoshop export plugin and skip using BigTIFF as an intermediate.

I also hope that CS6 lifts the 300,000 pixel limit in times where you can shot >10GPixel with equipment cheaper then a Photoshop license.

Mfg, Thomas.

[Chris Cox]

I also hope that CS6 lifts the 300,000 pixel limit in times where you can shot >10GPixel with equipment cheaper then a Photoshop license.

It takes a lot of time to test and validate all the features with documents even at 300k pixels.

To increase the limit, we need a good reason to redo all that testing.  And "we want to set a new record panorama size" isn't a good reason.

[Joergen Geerds]

Thomas,

I tried your bigtiff plugin and was also surprised that it wouldn't write more than 2 GB on a mac, but I was also surprised that it was a 32bit plugin, which didn't make much sense for CS5/64bit, when the sole purpose of a bigtiff is handling large files. So why not deliver it as a 64bit plugin?

it would be great to get the bigtiff properly working under OSX, and even add layer support, since the PSB format (with adjustment layers) is bound to use an awfully slow, single threaded compression (according to Chris Cox), which makes saving moderate to really large files absolutely no fun (on average, saving a PSB is about 10-15x slower than saving it as a regular tiff). having a working bigtiff would be great, because it would (hopefully) save large files much faster.

Chris:

while I do know that you think that larger than 300k files have no future or market, there are enough panoramas out there now that have exceeded the 300k size. the latest release of ptgui (9.01) has also dropped the 300k limit and allows larger files to be written as PSB (even though PS can't open them, but other apps (like krpano) can at least use them. getting back to the issue above, I think if you would lift the compression requirements of the PSB, or make it multi-threaded, or allow for custom settings (like ptgui lets you select packbits yes/no, but doesn't force LZW like PS does), testing larger projects would actually become a possibility. I can understand that nobody would want to test/save a super large project as a PSB written at 10MB/s, it would take forever.