PS Custom Format Plugin - Issue Reading Files > 2GB & < 4GB

[echeraud]

Is this issue fixable within our format plugin code, or is it  a photoshop limitation?

I am new to Photoshop & Photoshop Plugins.  I have reviewed the sdk photoshop documentation & searched web on this subject.

I am working with a photoshop format plugin written by my predecessor who is no longer here.

The plugin is modeled after the simpleFormat plugin.

The plugin is being used on systems running Photoshop CS6 & CC on 64bit Windows 7 with 8GB or more memory.

The plugin allows a file of our format to be opened, read and written.

--------------------------------------------------------------------------------------------------------------------------------------------------------

ISSUE:

The plugin works fine reading  files < 2GB or > 4GB.  For files between 2GB and 4GB, the call to allocate memory fails.

--------------------------------------------------------------------------------------------------------------------------------------------------------

In the plugin's PIPL structure:   the FormatMaxSize {32767, 32767} and PluginMaxSize {PlugInMaxSize { 2147483647, 2147483647 }

----------------------------------------------------------------------

In the plugin code;  the DoReadStart() method opens the file and reads the file header information.  This works fine.

Next, in the DoReadContinue() method: SPsBuffer->New(&bufferSize, buffersize) is returning NULL.     see below.

void PluginMain (const int16 selector,

                                                                         FormatRecordPtr formatParamBlock,

                                                                         intptr_t * data,

                                                                         int16 * result)

{

...

    gFormatRecord = reinterpret_cast<FormatRecordPtr>(formatParamBlock);

          gPluginRef = reinterpret_cast<SPPluginRef>(gFormatRecord->plugInRef);

          gResult = result;

          gDataHandle = data;

...

sSPBasic = ((FormatRecordPtr)formatParamBlock)->sSPBasic;

...

if (gCountResources == NULL ||

            gGetResources == NULL ||

            gAddResource == NULL ||

			gFormatRecord->advanceState == NULL)
		{
			*gResult = errPlugInHostInsufficient;
			return;
		}

		// new for Photoshop 8, big documents, rows and columns are now > 30000 pixels
		if (gFormatRecord->HostSupports32BitCoordinates)
		gFormatRecord->PluginUsing32BitCoordinates = true;

...

}

static void DoReadPrepare()

{

     gFormatRecord->maxData = 0

}

static void DoReadContinue (void)

{

          int32 done;

          int32 total;

          int32 row;

          VPoint imageSize = GetFormatImageSize();

          /* Set up the buffer & progress variables. */

          done = 0;

          total = imageSize.v;

    Ptr pixelData        = NULL;

    Ptr rawData          = NULL;

    Ptr uncompressedData = NULL;

    int64* offsetTable   = NULL;

/* allocate the pixel buffer. */

          unsigned32 bufferSize = gFormatRecord->planes * imageSize.v * imageSize.h;

          pixelData = sPSBuffer->New( &bufferSize, bufferSize );            <======   This allocation fails for file sizes > 2GB & < 4GB.

          if (pixelData == NULL)

          {

                    *gResult = memFullErr;

                    //return;

                    if(*gResult == memFullErr) { goto ReadContinueCleanUp; }

          }

[Guest]

I'm not sure what kind of file you are trying to open or what version of Photoshop you are working with. Need more information!

[echeraud]

Hi TLL,

Thanks for responding so fast ☺

In my forum discussion I have stated that:

1. this is a format plugin modeled after the simple format plugin ( the one included with the adobe photoshop sdk).

2. This format is used with CS6 & CC.

The file I am reading is my companies file format. I believe it may be a modified TIFF type format.

[Guest]

I just saw the forum you were posting in after I hit reply so I'm most likely out of my league here. I was curious if you are working with geospatial data, as there are existing PS plugins for working with (opening and saving-as) tif imagery bigger than 2GB...

[echeraud]

No problem ☺

[echeraud]

Some examples of files that are successfully read and files that are not successfully read are shown below:

Filenames that contain "nok" as part of the name, would not open and an out of RAM error occurs.

in the ReadContinue method, shown in an earlier post on this thread.

Filenames	12x12ok	12x24ok	ok	nok	nok	nok	ok
hdots	17280	17280	23160	23180	30001	32768	32768
vdots	8640	17280	23160	23180	30001	32768	32768
hdots * vdots	149299200	298598400	536385600	537312400	900060001	1.074E+09	1.074E+09
hdots  + vdots	25920	34560	46320	46360	60002	65536	65536
hdpi,vdpi	1440:720	1440:720	1440:720	1440:720	1440:720	1440:720	1440:720
#channels	4	4	4	4	4	4	5
#bpp	2	2	2	2	2	2	2
compression	Packbits	Packbits	Packbits	Packbits	Packbits
fileLength(bytes)			8593776	8647536	14365054	17063936	21258240
filesize (MB)	356	712	8.19	8.24	13.7	16.27	20.27
filesize (GB)	0.348	0.69	0.008	0.008	0.0134	0.016	0.0198

[Tom Ruark · Accepted solution]

1) Requesting anything above 2 gig is risky. Try to break your problem down into smaller chunks.

2) You are over flowing unsigned32. That max's out at 4 gig. Your 5 gig request is really 1 gig.

3) We have a new suite New64 that takes an unsigned64. Against point #1 but available for those that dare.

[echeraud]

Thanks Tom, I really appreciate your feedback!!!!!

[drhe]

Hi Tom,

Where could I find this new suite New64 ?

Cheers

[Tom Ruark]

PIBufferSuite.h

[drhe]

Thanks Tom, I switched to using sPSBuffer64 but it didn't resolve my issue.

I am curious why this limitation was never removed in the latest cc 2017 sdk.

We're having to work on 40k+ latlongs/panos and it would be quite useful to have quick way to allow us to export bigger resolutions

any other files I should be looking into in order to remove this limitation ?

[Tom Ruark]

The New64 if failing? What is your pluginmaxsize in your PiPL?

[drhe]

I left it at : PlugInMaxSize { 2147483647, 2147483647 }

anything higher than that and I don't see the plugin appear in my file format options when saving out a file in photoshop

I've also tried looking to make the rows and columns in PIFormat.h    int64, but it then crashes photoshop on export

there's got to be a piece i'm missing. i did switch everything to

... sPSBuffer64->New64( &rowBytes, rowBytes ) ... so there must be something else i'm not seeing

[Tom Ruark]

Right. That is pixels width and height. How about this one? FormatFileSizeLimit

/**

* The maximum size on disk the format can read or write. Photoshop defaults to 2 gigabytes. If

* your format can read and write files larger then set this property accordingly.

* key value is 'fslm'.

* This property key reflects the \c FormatFileSizeLimit property in the PiPL resource file.

* The data for the property has type \c int32 and is number of gigabytes.

* Photoshop uses this field to allow file sizes larger than 2 gigabytes.

*/

#define PIFmtFileSizeLimitProperty 0x66736c6d /* 'fslm' \<int32 \> Max size on disk your format can read and write. In gigabytes. */

And you are doing something link this:

// new for Photoshop 8, big documents, rows and columns are now > 30000 pixels

if (gFormatRecord->HostSupports32BitCoordinates)

gFormatRecord->PluginUsing32BitCoordinates = true;

And using those coordinates.

[drhe]

I did have an old PIFormat.h

so i got the new one from sdk 2017 and changed it from :

#define PIFmtFileSizeLimitProperty 0x66736c6d /* 'fslm' \<int32 \> Max size on disk your format can read and write.*/

to:

#define PIFmtFileSizeLimitProperty

0x7FFFFFFFFFFFFFFF

and i added the  FormatFileSizeLimit property in the PiPL resource file, but I cannot seem to be able to compile it again now...

7>.\..\..\Output\Objs\OpenEXRFormat\Release64\OpenEXRFormat.rr(244194): error : ParsePiPLBody: Unexpected token ('

FormatFileSizeLimit { 2147483647, 2147483647 },' : 21)

[Tom Ruark]

Don't edit API files. In your PiPL add the property and set the values accordingly and the number is in gigabytes.

[drhe]

I cannot add the FormatFileSizeLimit property to the equivalent of the  SimpleFormat.r 

I can see in PIPL,r that i have a case for it :

		case FormatFileSizeLimit:
				longint = '8BIM';
				key longint = 'fslm';
				longint = 0;
				longint = 4;
				longint;

but I'm just not sure where to set that property since it won't let me in OpenEXRFormat.r  which is my equivalent to SimpleFormat.r

[Tom Ruark]

samplecode\format\layerformat\common\LayerFormat.r:152:         FormatFileSizeLimit { 4 }

You will need all the .r and cnvtpipl.exe (for windows) from the new SDK.

I would recommend moving your plug-in to use the new SDK.

[drhe]

thanks for all your help Tom.

If anyone is interested the solution was a combination of three things

1) make sure to use the latest cc 2017 sdk

2) crank the FormatFileSizeLimit { 10 }      // for 10gb files

3) make sure you're not using imageSize (which is deprecated) and use instead imageSize32 in the SimpleFormat.cpp file

anyway, i can export fine now

cheers