Poor RAM management in Photoshop CS4 - need help!

[Overfocused]

I have done numerous hours of research over the years only to give up and come back to this subject later on. I will try again!

I must ask: is there ANY way to get Photoshop CS4 to automatically flush used RAM after closing image files? I'm tired and annoyed of CS4 eating more and more RAM without freeing anything after an image has been closed. An image shouldn't still eat up memory when it isn't even open anymore. It seems to be terrible RAM management of the program itself and how Adobe has forgotten to write that little snippit of code to clear the RAM after an image has been closed.

I know this is not because of:

Cache levels

Setting too large of a saved history state; purging history cache or any kind of cache still never frees RAM

A lack of updates; I have been up to date with CS4 very punctually as they were released

The computer I am on; this poor RAM management has been around since CS2 and it's the same on the last 4 computers I have used

Does CS5 still do this too?

[Chris Cox]

Do the tests highlighting decoupling caps also tell which ones? If so, I want to know how it's configured and how it is reported!

No, Photoshop just causes the system to fail.  Someone still has to hook up a scope and see exactly where it failed.

Yes, high speed scopes are a must - but you need software to exercise the system to get into the failure state.

[Hudechrome]

Thanks. Business as usual. (Ihope you realize that the questions were a bit tongue -in-cheek, as I would not expect actual readouts, but hey! What do I really know here! )

Final question: What does PS exercise that primje 95 doesn't? and how is it configured? Does it run a battery of filters on a given size file on a continuous basis?

I would appreciate any parameters here you can share.

Thanks!

[Chris Cox]

Final question: What does PS exercise that primje 95 doesn't? and how is it configured? Does it run a battery of filters on a given size file on a continuous basis?

Each company has their own test suite.    We gave them all some scripts that run large images with a variety of filters. (which script depends on which year they talked to us)

I know that one in particular did a lot of simulation and analysis and settled on 4 things that really pushed the DRAM system.

Another runs a "representative" suite of tests on a variety of image sizes.

And I haven't heard details from several of the companies -- we just hear from them when scripting changes, or they need help decipering a problem. (never thought I'd need quite so much EE in this job)

[Hudechrome]

(never thought I'd need quite so much EE in this job)

Never thought I would have to know so much damn software in mine!

[Hudechrome]

I spent the last hour watching the resource monitor in Win7 display the RAM usage and it's distribution between"In Use", "Standby" "Free" and % Used Physical Memory" After startup, I see 16% Used PM, and a very small standby. In Use is about 1200MB or so. As I open successive programs which include PS, Bridge and DXO, I note a very small increase in %PM, but the distribution changes dramatically. Now, opening FF additionally % in Use PM Jumps to 23%. Then I close everything and the only thing that returns to the value after bootup is %Used PM. Standby started at 459MB after boot, and settled at1304 upon closing all programs.

I haven's shown all the hard numbers, but from what I can deduce from Standby and Free summations, allowing no more than 90% in PS Performance tab should give me a comfortable margin. This is where I decided final settings.

Hoefully, my deduces aren't wild!

[j_iliz]

Overfocused wrote:

The more total RAM actually usable that I can use the better.

But you ARE using it and you've told PS how much you actually want dedicated to it. You only did not understand what you were doing and you've choked off your OS then thereafter looking to blame something else than your own lack of understanding with what you were doing.

Overfocused wrote:

Currently people keep disregarding the fact that in my situation CS DOESNT FREE THE RAM.

You don't get it. You have asked PS to reserve a certain amount of RAM so that others processes in your system dont get in your way of your photo processing. The way you would like it to run is to let other processes grab more and more memory when PS releases it and eventually choking off your photo processing.

Professional businesses use PS and this is the best way to keep multi users from choking each other off from a memory perspective. Like I said, GOOD SW DESGIN. The way you prefer it, BAD SW DESIGN.

It is obvious much of this banter is damaging your ego. We've all gone through stuff like this in our own lives. I guess now it is your turn. The best thing you can do from here is keep control of your emotion because in your subsequent posts your not doing your reputation any personal favors.

[Noel Carboni]

j_iliz wrote:

You don't get it. You have asked PS to reserve a certain amount of RAM so that others processes in your system dont get in your way of your photo processing. The way you would like it to run is to let other processes grab more and more memory when PS releases it and eventually choking off your photo processing.

Though the use of charged words like "choking off" creates bias, you're right.  That's exactly what should be happening.  Saying that preallocating RAM is a "21st century" technique yielding "higher quality" results gives off a certain smell.  It's the smell of poorly designed code.

A person populates a computer system with a certain amount of RAM to be able to support all of the expected computing tasks (note: plural).

That Adobe wants Photoshop to just take control of a big block of it in a more or less static fashion is, frankly, a bit like a kid in the playground who does not play nice and takes all the toys away from the other kids.  Sure, Adobe says he'll give them back if someone wants them, yet this does not bear out in actual observed operation.

Photoshop is not the center of everyone's computing universe, and should be able to be set to just use memory dynamically per the operating system's capabilities - just like the other applications do.  This way it can both share resources and get all that are available.

If the mix of operations on the computer happens to slow Photoshop down, then so be it - it's the user's choice to run other stuff.  If the user chooses to run only Photoshop, then so be it.  Adobe should not try to decide this ahead of time!

And - no disrespect to Chris - but I'm willing to postulate that Adobe's memory management prowess just might not exceed that of a major operating system manufacturer.

Before you argue further, you should know I'm a career software architect with 35 years experience, so I can smell what's being shoveled.

But...

Given that this behavior is not likely to change in a product that's reached version 12.0, the only workable solution in a practical sense seems to be to put more toys in the playground - drop another 4 to 8 GB of RAM in the system and just assume Photoshop will be hogging a big block of it.

-Noel

P.S., Chris, have you ever just tried carving out the buffer suite and replacing it with malloc() / free() to see how it would work?

[Chris Cox]

P.S., Chris, have you ever just tried carving out the buffer suite and replacing it with malloc() / free() to see how it would work?

Yep, every cycle we try it.  And it always blows.  We have to actively manage the memory, and reduce the OS allocation/free calls to keep performance at a reasonable level.

Photoshop is not trying to hog memory, we're just doing what we have to do to maintain performance for our users.

Ww try to stay out of the way of other applications (like monitoring paging status to back off our usage, and allowing the user to set the maximum usage).

but I'm willing to postulate that Adobe's memory management prowess just might not exceed that of a major operating system manufacturer.

It's pretty close actually.  But the OS is designed for a general case, while Photoshop is designed for specific usage -- we know more about how we use the memory than the OS can (even using madvise calls. And yes, we have tried that too).

Nothing being shovelled here -- just a lot of experience and results from extensive testing being applied.

Yes, if you don't have the background and experience, some of the decisions probably seem counterintuitive.

But when we talk to other experienced memory management experts, they agree with our design.

[Noel Carboni]

So Photoshop has been optimized at the expense of other applications on the system.  Fair enough - you're not in the business of running those other applications well.

Chris Cox wrote:

a lot of experience and results from extensive testing being applied.

And that, I will agree, cannot be challenged.

However, it is clearly experience with a lot of givens - such as Photoshop's basic architecture, and the business goal of making Photoshop work best, as noted above.

-Noel

[Chris Cox]

So Photoshop has been optimized at the expense of other applications on the system.

That is not what I said.

[Noel Carboni]

Perhaps that was a bit harsh; I'm sorry.  I guess I'm just feeling frisky today. 

But in all seriousness I wouldn't be here debating this if I didn't happen to agree with OverFocused that Photoshop CS4 was a rather poorly behaved child in the playground.  But given that, I did what I suggested - put in more RAM - which works.

-Noel

[Marceppy]

Sorry - I had to reboot my system ... where were we?

[Overfocused]

Perhaps that was a bit harsh; I'm sorry.  I guess I'm just feeling frisky today. 

But in all seriousness I wouldn't be here debating this if I didn't happen to agree with OverFocused that Photoshop CS4 was a rather poorly behaved child in the playground.  But given that, I did what I suggested - put in more RAM - which works.

-Noel

Since I can't add more and this sytem is maxed, I've gone the other way and gone to safer point and set the max to 6.8GB. I will manually adjust to the 7.2 for the biggest jobs I process since it works smoothly the first time or two it fills to the 7.2 mark in a session.

j_iliz - I think your frame of mind is in single user environments

It is, this is a single user environment

Chris - I am sorry for giving you such crap, however I do because regardless of my lack of pinpoint technical knowledge from a programming standpoint of RAM management, I have been using computers  since I was 4 years old and building them since I was 11. My technical troubleshooting intuition is throwing a red flag that something is wrong, and not with my OS configuration; the OS is behaving exactly as it should. Something is happening though, directly linked between CS and the OS when the RAM fills up. Its like its not just switching to paging. I assume it's possible for the OS to try and page while CS is trying to use the scratchdisk at the same time? Could a double overlapping page action be a cause?

The bully/poorly behaving child terminology mentioned earlier to me seems to describe the problem I'm having. Its like the red line at school by the street around the playground that kids get in trouble (or run over) for crossing. CS4 has the whole playground, but it wants whats past the red line too and fights for it when really it should back off after its retrived the lost ball that went into the street.

[Hudechrome]

So far as the paging interaction is concerned, if the paging file and the scratch disk are the same HD, or even separate partitions of the same HD, yes, you probably will experience problems with multi-tasking, or even PS alone as the OS looks after itself. Maybe you already know this, but I hate to assume.

[Overfocused]

So far as the paging interaction is concerned, if the paging file and the scratch disk are the same HD, or even separate partitions of the same HD, yes, you probably will experience problems with multi-tasking, or even PS alone as the OS looks after itself. Maybe you already know this, but I hate to assume.

Yeah, it happens while PS is doing its thing alone while other programs sit idle and aren't asking for more resources.

However, if there is a dual page/scratch action attempt going on when PS is trying to extend the reserve physically past the 7.2GB mark, it obviously isn't working out so well.  If it went back to use what empty space is inside of it rather than shrinking the available RAM that the OS can see, I think it would quit pushing the OS to use the pagefile while PS is trying to scratch. The OS only sees an ever shrinking available phsyical RAM, even when PS has 4GB free in its reserve, the OS can only see the reserve is trying to expand more. I obviously haven't found a simpler way of explaining it yet, lol.

[j_iliz]

The performance monitoring tools allow graphic capabilities where you can track the memory usage for both PS and other processes real-time. How about posting a graphic showing the memory behavior every… 3 seconds or so. OS kernel process can also be superimposed on the graphic. This might help people understand what it is your experiencing.

[Chris Cox]

when PS is trying to extend the reserve physically past the 7.2GB mark, 

Photoshop will not allocate RAM beyond the limit set in preferences.   There are some cases where a third party plugin or an OS routine may do  so temporarily - and then Photoshop will try to bring it's usage back  down under the limit.  (plugins are supposed to advise Photoshop about their memory needs, but some get sloppy)

If it went back to use what empty space is inside of it

Again, Photoshop does reuse it's memory.  If Photoshop has memory available to reuse, it does that first, always.

[PECourtejoie]

"plugins are supposed to advise Photoshop about their memory needs, but some get sloppy"

Chris, we saw in another thread that the Mondo plug-ins do manage their own VM. (the MVM files). If PS has already 7.2Gigs out of 8, and huge Panoramas are being built, couldn't that be the issue?

[Chris Cox]

Chris, we saw in another thread that the Mondo plug-ins do manage their own VM. (the MVM files).

If PS has already 7.2Gigs out of 8, and huge Panoramas are being built, couldn't that be the issue?

I don't think so, since Mondo plugins manage their memory inside Photoshop's limits.

And PhotoMerge doesn't use a plugin, but a JavaScript.

[Chris Cox]

So far as the paging interaction is concerned, if the paging file and the scratch disk are the same HD, or even separate partitions of the same HD, yes, you probably will experience problems with multi-tasking, or even PS alone as the OS looks after itself.

Yes, both will run faster if the OS paging file(s) and Photoshop scratch file(s) are on separate physical devices (less head thrash, lower controller overhead).

That's one of the first things we advise for building a high performance Photoshop system.

[Chris Cox]

the OS is behaving exactly as it should.

What you have described is the OS or hardware misbehaving.

I assume it's possible for the OS to try and page while CS is trying to use the scratchdisk at the same time?

Yes, which is why we free up RAM if the OS starts to page heavily.  We want to minimze double swapping if at all possible.

But double swapping just slows things down - it does not lockup, hang, or crash.

Photoshop does back off it's RAM usage when the system needs it, and does reuse it's RAM.  The approach is more "we'll take what we need, and until someone else needs it we keep reusing it".

So something is going wrong on your system once paging starts to get relatively heavy.

My bet is still on a disk issue (but it could be a driver, or power supply).

[j_iliz]

Noel Carboni wrote:

function(){return A.apply(null,[this].concat($A(arguments)))}

j_iliz wrote:

You don't get it. You have asked PS to reserve a certain amount of RAM so that others processes in your system dont get in your way of your photo processing. The way you would like it to run is to let other processes grab more and more memory when PS releases it and eventually choking off your photo processing.

Though the use of charged words like "choking off" creates bias, you're right.  That's exactly what should be happening.  Saying that preallocating RAM is a "21st century" technique yielding "higher quality" results gives off a certain smell.  It's the smell of poorly designed code.

This is exactly the root why we don’t meet up 100% on this. It has to do with the general concept of mission criticality, what jobs should be deemed more important in multi user/task environments. I think your frame of mind is in single user environments. Mine is in multi-user environments and Adobe sells a product that is supposed to support both worlds.

Your analogy where one kid in the playground takes most the toys so others can’t play with them assumes the kids play nice and don’t mind sharing. Once it is proven the kids cant place nice with each other mom and dad have to segregate the environment and say “this is your toy” mom and dad decide whom gets what.

It should be up to management to decide who gets what resources and enforce that rule. The traditional world you speak of only opens the door to a battle royal and then both competitors both work less effectively than possible.

I do agree that if there is only one dedicated user not competing for mission critical resources, the traditional OS style of memory management is superior. I suspect that Adobe at one time was confronted with a decision to optimize their performance and minimize their support efforts for multi-user environments and the most effective way to do that is give people their own dedicated playgrounds. By not doing this, the company effectively creates battle royals between high resource types users.

My background is in multi-user environments and it is no mistake that this is the way the big boys do things, Oracle, SAP etc. It is actually a MUST or your business get eaten alive by support costs, your product reliability trashed, then your whole business goes down the tubes.

[Noel Carboni]

j_iliz wrote:

This is exactly the root why we don’t meet up 100% on this. It has to do with the general concept of mission criticality, what jobs should be deemed more important in multi user/task environments. I think your frame of mind is in single user environments. Mine is in multi-user environments and Adobe sells a product that is supposed to support both worlds.

I understand completely the concepts and what you're saying, though I don't think Photoshop literally needs to support any multi-user environments (in the "multiple users simultaneously running it on one computer" sense).  But yes - multiple different user installations and of course multi-tasking with any number of other apps.  And certainly Photoshop has to work in both Apple and PC environments.  In this vein it surprised me that Adobe was willing to take on the OpenGL Medusa - but that's another story.

What we're talking about is:

Real world defensive implementation vs. ideal implementation.

What software engineers actually produce vs. what could be produced in a perfect world.

Business decision involving risk vs. technical decision involving concepts.

Take over the part of the system that traditionally causes problems so that it can be kept under better control.  I get it.  But it still doesn't mean it's the best implementation for a particular user or in a particular situation.  And all it does is push contention problems elsewhere, as referenced by this thread.  Elsewhere is good when someone else is supporting that.

I sense an improvement in Photoshop CS5 vs. its predecessor.  It's good to know the memory management strategy is continuing to be refined.

-Noel

[j_iliz]

Noel Carboni wrote:

I understand completely the concepts and what you're saying, though I don't think Photoshop literally needs to support any multi-user environments (in the "multiple users simultaneously running it on one computer" sense).

I don't think so.

I speculate there is plenty of marketplace out there for multi-user on multiCPU systems. But that is just me.

I think only Adobe knows the answer and they have clearly made a business decision to capture that revenue.

Money talks

[Noel Carboni]

Are you thinking of something other than a PC or Mac system?  These are not multi-user systems as far as I know, at least not in the "multiple users can use Photoshop simultaneously" sense... 

While multiple users can indeed log into a Windows terminal server system and run apps independently, the necessity of having accelerated graphics needed by Photoshop just isn't there (Photoshop will not use accelerated OpenGL when running across RDP), and you can't tie more than one physical console (monitor/keyboard/mouse) directly to a PC.

I'm less confident that multiple consoles cannot be tied to a Mac.

I'm genuinely curious what multi-user system you have in mind here.

-Noel