Poor RAM management in Photoshop CS4 - need help!

[Overfocused]

I have done numerous hours of research over the years only to give up and come back to this subject later on. I will try again!

I must ask: is there ANY way to get Photoshop CS4 to automatically flush used RAM after closing image files? I'm tired and annoyed of CS4 eating more and more RAM without freeing anything after an image has been closed. An image shouldn't still eat up memory when it isn't even open anymore. It seems to be terrible RAM management of the program itself and how Adobe has forgotten to write that little snippit of code to clear the RAM after an image has been closed.

I know this is not because of:

Cache levels

Setting too large of a saved history state; purging history cache or any kind of cache still never frees RAM

A lack of updates; I have been up to date with CS4 very punctually as they were released

The computer I am on; this poor RAM management has been around since CS2 and it's the same on the last 4 computers I have used

Does CS5 still do this too?

[D Fosse]

What problem?

Photoshop uses the memory you have made available in Preferences. That memory will get reused when you close the file, but it will not be released, until you exit the application.

This is by design. It is not a flaw, or a problem.

Problems happen if you set the allocation too high in preferences. Depending on how much you have installed, the ideal range is 70-80%. Any more and you will choke the whole system.

Problems also happen if you don't have enough scratch disk space. With raster image editing there is generally no such thing as "enough RAM", which is why you have a scratch disk. The disk is always in use - that's Photoshop's main memory, while RAM is more like a cache to hold the most current data.

[Hudechrome]

Yeah, like 16 terabytes!

[Noel Carboni]

Overfocused wrote:

@Noel - That's what I'm talking about. I edit on large images, or a ton of RAW images at one time for batch procesing, and then it starts reaching some slump point where its not quite as it should be and it gets iffy.

It's possible OpenGL, in my case, was at fault.  OpenGL has always been somewhat flaky, and I do remember switching to Basic mode did make things better for me.   I know you said you have OpenGL disabled.

-Noel

[Noel Carboni]

Overfocused is not wrong about his observations.

I have to say - and this is VERY subjective - that when I moved up to Photoshop hCS4 from CS3 way back when (I now use CS5) that I felt that occasionally I needed to close down Photoshop from time to time because it just seemed to "load up" and get balky.  I remember thinking it was a change from before, when I could just edit and edit and edit without ever closing the app.  And Edit - Purge with Photoshop CS4 never seemed to make it all better.

For what it's worth, I can't say I've felt this way with Photoshop CS5 in the past year, though to be fair the mix of things I work on has kind of changed, and I'm running Windows 7 x64 now instead of Vista x64.  Sometimes I work on big, complex astroimages for quite a long time, and I just don't think CS5 performance feels like it degrades like CS4 did over time.  It's as though something has been fixed - I just assumed it was growing pains for the first 64 bit version.

-Noel

[Chris Cox]

Overfocused is not wrong about his observations.

He probably is seeing something cause a slowdown.

But his supposed cause and reasoning are completely, and verifiably, wrong.

[Stellar_sincerity15B9]

Doesn't seem like, takes 6GB of ram, "I had settle to used until 4,4GB" and it doesn't care and uses 6GB instead, I close all the documents and open a new small one and now is in 7GB, im totally lost with your meaning of "very good memory management"

[j_iliz]

As somebody who has been involved with performance analysis and tuning of the largest business application systems in the world I was interested in seeing Chris’s description of the memory management philosophies as implemented in Adobes CS* products.

Chris has outlined management strategies that are consistent with SAP’s high volume business processing platforms. The old school thinking of allocate and de-allocating memory has been outdated for many years. SAP abandoned these old school techniques close to 20 years ago with the advent of their R/3 product. They were inspired to do this to help minimize support costs which stabilizes the processing engine, which in turn keeps the customers happier and good word of mouth in the end helps them to sell more software. Everyone wins…

Smaller scale users usually never have to confront these high resource intensive systems/designs, but here is one such case where the education process does needs attention to take these users into the 21 century of high quality software design.

Chris’s finger pointing to other areas possibly causing issues definitely have merit. However, even as SAP has learned, they do uncover glitches in their products now and then. In this case, a different type of support is needed to pinpoint and resolve the issue and I really doubt an internet chat form like this is the most effective way to tackle things. I guess we will see soon enough.

[Hudechrome]

An interesting discussion about memory management. I would like to ask i-Iliz if he/she has any suggestions for small users in troubleshooting/tuning our systems should we suspect memory problems of the nature described here. As Noel mentioned, there was problems in the past, resolved by purging or exiting and reopening the program that is no longer a problem today at least on Win7 64. Yet things apparently are not perfect.

[j_iliz]

Hudechrome wrote:

An interesting discussion about memory management. I would like to ask i-Iliz if he/she has any suggestions for small users in troubleshooting/tuning our systems should we suspect memory problems of the nature described here. As Noel mentioned, there was problems in the past, resolved by purging or exiting and reopening the program that is no longer a problem today at least on Win7 64. Yet things apparently are not perfect.

Well I cant offer much in terms of words of wisdom as I know nothing about the processing architecture of the CS* software and how it interplays with the OS.

If it were me trying to better understand my environment though, the first thing I would do is boot the computer into Safe Mode and do my testing from there. If I remember correctly, Safe Mode only starts the vital portions of the OS thus eliminating other elements, especially 3^rd party software that might be affecting my tests or results. If the stuttering stopped in this environment I would know some extraneous software would most likely be helping to cause the issues.

[Hudechrome]

Thanks.

Agreed about Safe Mode.

[j_iliz]

In catching up in this thread, it seems there were two main issues.

1)      You overallocated your PS RAM and needed to be scaled back.

2)      Your historically conditioned to believe software apps should swap data a standard way. The way your using the system is probably not the most effective processing for your workflow. It probably is though for a high % of other users that use the application. You just have to live with it… or jump ship.

>>> This is why I want every MB I can scrape out of my machine. Anyone who thinks paging is a good thing for non-emergency operations can scratchdisk my arse haha (joke)

You don’t have enough experience with these topics. This is actually a true statement. Most companies with systems large and small have to think about what it costs to process specific jobs. If one in a thousand jobs swap, a decision must be made to know how much it would cost to retrofit a very expensive environment to keep that one job swap from occurring. Most times it is better to let the swap happen for this non-mission critical item than pay $50,000 to upgrade all systems.

>>> There's no excuse for a crappy 'advanced memory algorhithms not taught in universities!"

Your po-pooing stuff you don’t know much about. Your blaming the world for your self inflicted woes. Look from within grasshopper.

[Overfocused]

Your historically conditioned to believe software apps should swap data a standard way. The way your using the system is probably not the most effective processing for your workflow.

-  I have hundreds of TIFFS that take 60-120MB each in processing, and 8gb  of RAM. The more total RAM actually usable that I can use the better.  How the heck can using less of what is actually available RAM enhance my  workflow? Standard are standards for a reason. The ink industry for  printing is non-standardized and people waste $3-5 billion+ a year only  increasing on unused ink that is still remaining in their cartridges, because there is no standard way of manufacturing ink, filling cartridges, or monitoring how much ink is actually left in a cartridge. I'd love for more standards in more markets. Unless natively stable, all other programs on my computer work flawlessly with said demonized traditional methods and standards, even in high bandwidth applications a hell of a lot more intensive than  Photoshop CS4 will EVER be. Currently people keep disregarding the fact that in my situation CS DOESNT FREE THE RAM.  CS4 is balloon. Balloon inflate. Never deflate even when emptied. Overflow into other rooms of critical OS space where balloon shouldn't be when it fill the room full. How many analogies must I give? Must I use sign language? Binary?

You don’t have enough experience with these topics. This is actually a
true statement. Most companies with systems large and small have to
think about what it costs to process specific jobs. If one in a thousand
jobs swap, a decision must be made to know how much it would cost to
retrofit a very expensive environment to keep that one job swap from
occurring. Most times it is better to let the swap happen for this
non-mission critical item than pay $50,000 to upgrade all systems.

I said NON EMERGENCY SITUATIONS.  Emergencies are once in a while. You're repeating what I said like a parrot, but in different longer sentences with more words telling me I don't get what I'm talkinga bout. You're missing the full picture of what I am actually saying. That 1 in 1000 overflow would be exactly true in my situation with 7GB used, but it's not. Why not? Because the CS4 balloon will not deflate itself or reuse the empty space inside the balloon when it's needed, and the OS isn't given back any of the empty space to work with to know it doesn't need to page anything. In turn CS4 is going to try and scratch too when the OS is telling it that is has no more RAM left when indeed it does, but its hiding it in its own reserve balloon from itself. If a company needed just 700MB more room and either they spend $50k to upgrade to that, or get CS4 to release some space once in awhile, they'd sure as hell bother Adobe or write their own mod to force CS4 to release unused allocated RAM to get it.

Self inflicted only because I want to push the limit of what is actually usable on my machine, not some 'safe zone' with high tolerance. My system is tuned for a fine line of performance and I know what it can hande. I know what it uses and it does not spike usage for random OS operations. It can handle 7GB to 7.2GB of RAM of freely used RAM to one active program if it needs that big of a workspace to work. If CS4 isn't going to reuse the hollow areas in that balloon, then that's not my fault. My OS can handle the fine tuning, and CS4 is what cannot, obviously, since it can see that much RAM but not use the last %10 correctly.

[Jeff Schewe]

Overfocused wrote:

My system is tuned for a fine line of performance and I know what it can hande.

Hum...as far as I can tell, you've never actually bothered to mention what system nor OS you are using nor really basic troubleshooting like what % of ram you are allowing Photoshop to use.

You spout plenty of opinion, but little actual info.

If you simply want to attract attention, fine...but your rants aren't actually helping you, right?

To be precise, if you want help, you might want to provide basic system and troubleshooting and actually try to work with Chris rather than argue.

Ya see, Chris knows more about Photoshop performance than most everybody else on the planet.

You want help? Cool, ask...you want to Piss&Moan™, fine but you won't be getting much help. You really are only showing your own shortcomings...have fun flailing about...

[Overfocused]

Hum...as far as I can tell, you've never actually bothered to mention what system nor OS you are using nor really basic troubleshooting like what % of ram you are allowing Photoshop to use.

Good Lord you're thick. If you were actually paying attention and wanted to help out instead of trying to get attention yourself, you'd have all the system info you need. Granted I did not say my OS, I've said every other detail about my RAM configurations. I was working with Chris but no one sees what is happening on my computer no matter how many analogies I give.

I'm not going to let you be a lazy bum and ignore the whole conversation when I've conveyed plenty of information to Chris throughout.

Now I've shown that Photoshop is reusing allocated memory.

Like hell you have! I kept track of what was going on as I experienced it and you still don't even acknowledge that. How many times do I have to tell you ITS NOT REDUCING THE AMOUNT OF RAM THAT IT IS USING WHEN MUCH LESS IMAGE DATA IS OPEN THAN WHAT PHOTOSHOP IS TAKING FROM THE WINDOWS OS.

And that includes a machine that locks up because of a high memory limit set in one application?

Yes, in Photoshop CS4 and CS4 ALONE. You take so much out of context it gets annoying. Tunnel vision to prove your own opinions blinded by your experience with PS, even though I clearly have the situation happen to me in front of my face after I explained it as it was happening, which no one gave a single neuron to try and actually process what I said because you don't believe me and you don't want to believe me because its not in your superprofessional realm of thinking.

If it was as simple of a problem as to what OS I was using or how big my page file is, I wouldn't need to ask the highest level of tech support for answers to such a base level problem.

I know setting PS to 7.2 GB locks it up after doing enough processing and compiling, but it wouldn't if CS would free the damn RAM for Windows that its not even using after its done with it. It stays holding 7.2GB no matter how many images I close and leaves Windows presssed against the wall.

And you're ignoring the fact that the system should keep running even with 200% of RAM allocated by applications, thanks to virtual memory and the pagefile.  It might slow down if more than 95% of the RAM is active, but it shouldn't stop.

And you're ignoring the fact that after I close 4GB of image data that if only 3gb of image data is open it shouldnt be slowing down at all or using 7.2 GB still.

[Hudechrome]

Pass the popcorn!

Jeff, saw some pictures of Lake Shore Drive after the blizzard. Beautiful but deadly.

[Overfocused]

Yeah that was some nasty stuff. Around 300 cars was it?

Funny you mention the blizzard, the last freeze up I had with PS was while editing some large panoramas taken during a trip out into the blizzard... somewhere close to Chicago. Lol.

[Hudechrome]

I used to live there  and it seems to me the last big one in Chi-Town was right around when I left for good.

Wish I could contribute more but I never used CS4, jumping from 3 to 5.

Chris is really good at this stuff. Work with him.

One thing I would have tried  long ago is a memory test, like mem test86:

http://www.memtest86.com/

Memory is evasive in giving up it's troubles. I added 4G to mine a few days ago and promptly got Blue screened. The messages I was getting didn't point to exactly what the problem was, and it wasn't bad memory. It was misconfigured and I suspected it right off, but even setting some functions in BIOS to Auto made it worse!

It's fine now. A long talk with AMD support confirmed what I saw a being reasonable givin the starting configuration. I'm still a mite suspicious though!

Good luck, I am really interested in what you find.

Please, suspend disbelief for a bit.

[Overfocused]

One thing I would have tried  long ago is a memory test, like mem test86:

Thanks for the suggestion, but I ran this on the most thorough setting and let it cycle 30+ times when I bought the 8GB to test for DOA. The RAM I use is good.

Yes, we know - and I've already explained why that is perfectly normal, expected, and why it won't cause problems:  because Photoshop is reusing the memory, and will free memory when the OS needs it.

But its not doing that for the OS here! If paging it is the solution to managing BLANK RAM, then thats just ridiculous.

[Chris Cox]

ITS NOT REDUCING THE AMOUNT OF RAM THAT IT IS USING WHEN MUCH LESS IMAGE DATA IS OPEN THAN WHAT PHOTOSHOP IS TAKING FROM THE WINDOWS OS.

Yes, we know - and I've already explained why that is perfectly normal, expected, and why it won't cause problems:  because Photoshop is reusing the memory, and will free memory when the OS needs it.

And you're ignoring the fact that after I close 4GB of image data that if only 3gb of image data is open it shouldnt be slowing down at all or using 7.2 GB still.

Correct on the shouldn't be slowing down part - which is why we are trying to help you figure out what is wrong with your system.

But Photoshop should still be using all the memory it has allocated (up to the limit you set in preferences), until that memory is needed by a plugin or the OS.

You need to be looking at your system and figuring out why normal to moderately heavy operation is causing it to lockup.

Even if Photoshop has 7.5 GB allocated, and is only using 3GB, the OS is free to page out the idle memory (even image data you aren't currently using). Photoshop also will page the idle memory and image memory out to scratch so it'll free up faster if a user operation or the OS needs it.

And even if all the RAM is allocated, the OS should not lockup due to simple paging.

I clearly have the situation happen to me in front of my face after I explained it

No, you jumped to several conclusions about your situation, which have already been shown to be false.  Most of that was based on a lack of knowledge about memory management and how operating systems handle memory -- and we've tried to fix some of that lack of knowledge by explaining how things actually work.  Continuing to repeat that black is white, will not make it so.

Your valid observations are: 

* Photoshop doesn't reduce memory usage unless it has to (true, by design)

* That your system was running slowly when most of the RAM was in use (yes, that is likely)

* That your system locked up while paging heavily when most of the RAM was in use (ok, not normal, but we don't know the cause)

Now, we can work with you to solve that last mystery, but you have to work with us.

Shouting and repeating things that you know not to be true isn't going to solve that.

[Chris Cox]

Overfocused: You're just digging yourself deeper now.  You've proven that you don't know memory management beyond the basics. We understand -- all of us had to learn at one point or another.  You've proven that you are jumping to conclusions.  Ok, we've probably all done that at least a few times.

Now I've shown that Photoshop is reusing allocated memory.

And I've told you (though it's difficult to show) that Photoshop will detect system paging and scale back it's memory usage.

But you keep claiming otherwise.

And you're ignoring the fact that the system should keep running even with 200% of RAM allocated by applications, thanks to virtual memory and the pagefile.  It might slow down if more than 95% of the RAM is active, but it shouldn't stop.

I want to push the limit of what is actually usable on my machine

And that includes a machine that locks up because of a high memory limit set in one application?

Again, you've got something else wrong with your system if that is happening.

We're trying to help you figure out what is causing your system to run slow (mostly allocating too much RAM to one app), and give you ideas about what might be causing the lockup (which we still don't know).

Are you going to listen, and work with us, or continue to repeat things that you know not to be true?

[Chris Cox]

even in high bandwidth applications a hell of a lot more intensive than  Photoshop CS4 will EVER be.

Considering that the common operations in Photoshop are DRAM or bus bandwidth limited, that Photoshop frequently exposes motherboard problems before dedicated testing tools, and that computer makers use Photoshop to stress test their memory and support systems... I kind of doubt that.

[Hudechrome]

Well, PS is included in standard testing for performance, but I never heard of using it for stress testing. I thought prime 95 was the go to program. Is there some particular component that PS streeses? How is it configured for the test?

[j_iliz]

Hudechrome wrote:

I thought prime 95 was the go to program. Is there some particular component that PS streeses? How is it configured for the test?

Prime 95 is correct.

I am not sure what PS stresses though....

[Chris Cox]

I am not sure what PS stresses though....

Photoshop does a lot of sustained highly optimized memory transfers, plus some optimized scatter/gather operations.

These tend to stress different levels of cache, plus the DRAM controllers and DRAM chips.

This can expose power supply issues, timing issues, pulse skew issues, malformed pulse problems due to unexpected access patterns, etc. in the RAM and support chips (all the way back to the CPU).

Photoshop catches a lot of problems that dedicated RAM testing software misses.

From a historical perspective:  http://forums.adobe.com/thread/375773?tstart=0

Hardware design, hardware testing, and extensive software testing on those motherboards missed the problems.

Yet Photoshop pretty reliably reproduced the problems: all traced to hardware design defects (decoupling cap too small, traces too small, regulator too small, etc.).

[Hudechrome]

Very interesting, Chris.

Do the tests highlighting decoupling caps also tell which ones? If so, I want to know how it's configured and how it is reported!

Traces. Two types immediately come to mind: Power and Controlled Impedance, balanced and unbalanced. If your test points to power supply difficulties I would assume any traces implied would be included. How about the cables from the PS?

As for the controlled impedances, some rather dedicated high speed oscilloscopes, usually sampling, are used to ferret out problems like bad terminations, variations in characteristic impedance due to changes in board dielectric constant etc. Tricky stuff I used to love to do!

Actually, I did suggest that perhaps Photoshop (then CS4) might be a good addition to our testing methods on that overclock utility, especillay for very long term distress tests. Wasn't sure how it would have been configured but it was nixed. (I think that certain people suspected an ulterior motive on my part!)

But I ran tests anyway, not to stress but to see how the cpus handled CS4.

BTW, an interesting factoid about stress test. A big one is simply OS bootup!