Kernel Crash during render with CUDA and Nvidia Quadro 4000 - total frustration!

[LindyHop1943]

I just want to throw the computer and Premiere out of the window. I'm so frustrated that I want to stop doing video work for good! Once a year I have to edit large instructional DVDs for a dance camp. This year I'm having major issues that kill my life and my health. I have spent more than 3 weeks on that issue without any result.

Problem: When I export a section of the Premiere timeline the kernel graphics card driver crashes. About 1 to 10 minutes into exports (direct export, no queued AME) the screen goes black and Windows reports a NVidia driver kernel error. The driver is recovered and Premiere hangs. This only happens on CUDA export not with the software option of Mercury. The error is not related to a certain clip or portion of it. If it crashes on a certain clip and I restart Premiere this clip will be rendered fine the next time. The crashes happen always after a certain but varying time somewhere during the rendering.

Since the source material is 2K I need to scale it for DVD output. Therefore I need the CUDA support for the output quality. Premiere uses the better Lancos scaling algorithm only with CUDA support and when exported directly (not via AME—at least not with CC2014). Without the CUDA support the scaled down pictures looks not sharp enough. DVD already has a low resolution. Unfortunatey Premiere has no option of sharpening the output.

For troubleshooting I have tried:
- Premiere Pro 7, 7.2, 8.0, 8.0.1 and the actual 8.2
- various nVidia drivers from one year ago to the actual one
- different user account
- different render formats (for the project I need to output to MPEG-DVD)
- newly created projects and timelines
- checked the temperature with GPU-Z
- exported to external and internal drives

- source files form external and internal drives

- the latest step was a complete reinstall of the system (Win7, updates, CC) incl. BIOS update

Nothing stopped the kernel crashes when using the CUDA.

I contacted PNY and NVIDIA. I run stress tests and other tests on the card. It shows no problem at all. The graphics driver doesn't crash on any other usage.The card does not show any issues.

I have a ASUS P6T board with i7 920, 12 GB RAM, NVIDIA  PNY Quadro 4000. Win 7 Ultimate SP1, all updates and patches installed. The source material of this year is from a Sony F7 in 2K.

It looks to my very much as if Premiere has a bug that causes the driver to crash. It worked fine until last year's project. The only changes are the constant OS and software updates. No hardware has been changed.

The randomness of the crashes would normally suspect thermal issues, but since the computer doesn't show any thermal issues during other work or stress tests and only with CUDA I rather suspect an issues of the parallel programming within Premiere when addressing the CUDA.

Does anyone have another clue?

[RoninEdits]

haha, this keeps getting stranger and stranger. if it was the usb cable, and its slightly damaged and working now, it may act up again later and cause the error again. i guess if you really wanted to test things, you could try the old quadro card in the system again, but i might just leave well enough alone though.

[LindyHop1943]

Yes, I was thinking about getting the Quadro back, but now I first have to finish my project.

[LindyHop1943]

It would have been a miracle if Premiere would not come up with other issues. I needed to re-render some clips. Instead of kernel driver crashes I now receive Premiere crashes of the same nature as in https://forums.adobe.com/thread/1489720?start=40&tstart=0.

They have the same randomness as the kernel driver crashes before. If I render the same clip again that crashed before it works fine the second time. Maybe the issues is caused by AutoSave, too. It would explain a lot.

[RoninEdits]

that auto save crash is a new one for me. did you try turning off auto-saves and did that help?

[LindyHop1943]

I just run across that other post. I'm on the road now for 4 weeks. Unfortunately I had no time anymore to run a test before I left. As I said, suddenly I received the Premiere crashes instead of the kernel crashes of the graphics driver. I normally have AutoSave on with a 5 minutes interval, due to the frequent Premiere crashes. I had lost too often too much work.

Marcus

[RoninEdits]

premiere still has to communicate with windows and cuda, so while the links aren't related to premiere, i thought there was some chance they could still help. at least something else to try before buying a new video card. it will be interesting to see if the new video card helps.

im not sure about the cuda sdk's and premiere. the only thing i've seen is the ray-tracing feature has been removed in maxwell. i've seen several post the benchmark results bill mentions, suggesting premiere is working fine with the newer cards.

[ECBowen]

I have seen this before when you have media in the timeline that is non standard resolution for video with a different aspect ratio. CC 2014 has been much better at dealing with that but previous versions of Adobe often had issues with the memory management of those buffers. The fix when this occurred was to turn the GPU acceleration off and just run the software MPE mode. However you are stating that is not an option because of the scaling quality. So my recommendation is take that timeline into AE and let the compositor do the software scaling since they are far better at that with CPU processing.

Eric

ADK

[LindyHop1943]

This sounds that even a different graphics hard won't make a difference. So the hyped professional CC became just less professional!

In my last deperation I'm trying to upgrade to Win 8.1. Maybe the different drivers work better. I created a disc image, so I can go back in case I need to.

Going to AE might be an option. Is the software scaling really different (meaning better) than the one in Premiere?

I have about 220 clips of instruction that need to be rendered separateley for DVD/Encore. At the moment I'm using AutoHotKey to automatically go from clip to clip and render it out directly. This allows me to have the computer run over night and then have all clips ready. (I automated most tasks with AutoHotKey and also created a program that creates all titles form a text file). I would need to get everything into After Effect, get the full length SD clip back and then change the timeline setting to SD. Then I would be able to render the single clips per software only. A lot of crappy work because Adobe doesn't fix their bugs and makes their software more stable (every CC release had me experience more issues than all the CS version together).

[ECBowen]

This is not a bug. This is the Nvidia driver crashing because of what the hardware acceleration is requesting and the memory management ie memory management profiles for GPU acceleration to even function. It could possibly be a bad video card but I doubt it based on the other information provided. You can run 3D Mark and push the video card to the same extent Adobe is pushing it and verify if the video card is an issue. However if 3D mark doesn't show the same problem then the card is not the problem as far as hardware defect. The 970 and 980 GTX cards would have different resource management ie memory management profiles so could potentially fix the issue just because of that. However I have seen this issue before going back to the 600 series GPU's when it's occurred and it always revolved around non standard video resolution media and the buffers required to scale that to the other media.

Yes AE is far better at scaling via software ie CPU processing since the render engine  ie player is different and built towards compositing. You points on batch encoding though I cant argue with. You can render queue out these as different Comps and jobs in AE. Beyond that I don't have further ideas for you.

Eric

ADK

[RoninEdits]

you said previous versions of premiere had memory management issue, but it was improved, so is it not a premiere software issue? besides different resource management of nvidia maxwell, would faster memory interface and/or more video memory help?

[ECBowen]

It's still a Premiere centric issue. Keep in mind because it's GPU acceleration with media involved, there are many elements to this. Video drivers, resource allocation by the bios on the board, Memory management of the application and the video driver, amount of system ram available, What player versions are installed, FX used, etc, etc. Most of this boils down to ram buffers though and how Premiere is allocating for GPU acceleration buffers based on what media is used and what the application has to do processing that media and FX. All of this is in play and I would need to see which variable is the trigger element.

Eric

ADK

[LindyHop1943]

The system finished installing Win 8. The bug is still there. The Nvidia driver is apparentlly the same for Win 8 as 8.1 as 7.

So the only option is to have a new card or card and board and hope that the issue doesn't come up again. It is strange that there is no way to log the system Nto get more information about where the source of the crash is. Neither Nvidia nor PNY support new about such an option.

[RoninEdits]

that was a response to Eric's previous post about premiere, the info above the post should show it was in response to ECBowen.

if you watch gpu-z while rending, how is the gpu usage and memory usage? maxed or what? and how is the cpu usage and how much of the 12gb ram is used?  also, the 700w should be enough, even if its lower quality, they usually are within 50-100w of rating.

I tried different footage and the problem happened, too. There is no extra hardware and not third party plug-ins. The timeline only has a fade in and out on each clip plus a title and an overlay video at the begining (animated lower third) of each section. As I wrote the error occurs randomly and when I render the same clip again it normally works on the second try, very seldom I needed a third try.

have you tried removing the fade's, title, and animated lower thirds? or trying the media alone on a new timeline? if this works, does gpu-z show any change in usage?

[LindyHop1943]

So it comes down to the point that I just waisted three weeks of my time again and my only option is to spend more money to get another component that might be as buggy or not. Maybe it will work now, but in a few weeks any update can make it useless again. What a wonderful world we are living in. I guess that the code that spys on our usage and scans our data is more tested than anything else (sarcasm).

There should be the same rules for goods as for software. If I buy a TV and it doesn't work as it should I have the option to demand repair in normal time (2 weeks), a price reduction or a I can give it back and get a full refund for at least 1 year. If for any bug that has been found and that is not fixed in an appropriate time frame we can demand a price reduction it would finally be cheaper for software companies to create working software than buggy banana software. Especially with subscription you keep paying if it works or not. I hate that world!

[ECBowen]

If you figure out the trigger you can likely get the resolution. It's up to you whether that is worth the time or not. BTW there are exponential amount of variables involved with GPU acceleration and those change based on hardware, software, configuration for both, media, FX, API/MPI updates and more and that is before you even consider workflow and project differences. Adobe can't test them all or customers would never see the product because the hardware would change before the product even released.

Eric

ADK

[LindyHop1943]

I totally agree that it is very difficult and that it is hard to pinpoint the exact cause, but just because it is like that the software vendors would need to have higher standards in development, error recovery, and quality control. Because of this standard procedures should exist and tools should be integrated that easily allow to pinpoint the cause and provide developers in short time the necassary information to find the source of a problem. None of that exists.

Instead all the big companies work with the 5% rule. If a bug is known, but less than 5% of the user base are affected it won't be fixed. That is a problem. I remember the Encore bug that corrupted manually entered subtitles and rendered hours and days of work useless was know and not fixed for 4 consequtive versions.

Even it is called a "service". The advertisement and the manual of the service defines the features and usage. Anything that does not do as the manual describes is a bug.

-------------------------

I just tried to render a piece to mp4 to try out a render that doesn't scale down the 2K footage. The driver still crashed.

[RoninEdits]

was that render with a clean timeline without the other title and animations, just the footage?

when exporting, do you have "use render maximum quality" checked? if you uncheck/check does it make a difference?

do you have the cpu/mbd or ram overclocked? if so can you set them back to defaults?

[LindyHop1943]

The render was a section of my timeline with one title and video. Once I tested new projects, timelines, and other footage before and had the same issues. Second, as I said, the crash doesn't occur on a specific time marker. If it crashes for one clip it will work on the next run. So it is not the clip.

Maximum render quality is not turned on. It actually doesn't matter, because this checkbox is irrelevant when using CUDA. Premier does not use this setting when CUDS is enabled.

Nothing is overclocked. BIOS is at default settings as it was before the BIOS update.

[RoninEdits]

some more threads relating to the nvidia error, several different fixes that worked for some.

https://social.technet.microsoft.com/Forums/windows/en-US/69013365-60c4-4b89-87b1-e2866e143817/nvidi...

http://www.tomshardware.com/answers/id-1755834/nvidia-display-driver-stopped-responding-recovered-er...

http://www.tomshardware.com/answers/id-2581920/display-driver-nvlddmkm-stopped-responding-successful...

some found the problem was ram, some bad, some underclocked to get the error to stop. is your 12gb ram 3x4gb or 6x2gb? are they all from the same kit or mixed brand/model?

[LindyHop1943]

Ronin, the 12 GB RAM are 6x2GB, all the same brand and type.

[RoninEdits]

many share your viewpoint with software, and for some reason software gets a free pass. yes, its complicated and there are many variables, but everything now days is complicated and has to work with other equipment. it was pointed out by another here, that when adobe switched to the subscription model, their lawyers were able to classify the software as a "service" and not a finished "product". therefore its very difficult to sue or get any compensation for a faulty product.

[LindyHop1943]

Hi Ronin, where did you read from my posts that I stated that previous versions of Premiere had memory management issues.

[ECBowen]

I don't think throwing more hardware at this is going to resolve it until you figure out the trigger. If you want to contact me direct, I will remote in and take a look. What Ronin is asking you to do is a good way to isolate the media element trigger and a good suggestion. Ronin's memory management question was directed at me btw.

The Geforce cards have different drivers and bios. That can drastically change the memory management especially with GPU acceleration. The number  of Cuda cores can drastically change the GPU acceleration memory management because buffers cannot be shared across cuda cores processing different threads if required. They have to be duplicated then. Vram amount and bandwidth will effect the caching models used by the application and driver. Those can effect this as well. However I still think that's unnecessary until we define the trigger and see what the options are.

Eric

ADK