RENDER PROBLEM - AME and Premiere Pro CC causing random horizontal lines on H.264 Renders

[JesseSchluntz]

I just got the new mac pro with freshly installed, up to date software, so of course something isn't working right...

When exporting batches of H.264s from Premiere Pro CC using AME, I get random horizontal lines on screen for most videos. (see below image for a clear example of one of these random horizontal bugs. the line below the logo is the issue)

Tried restarting the app a few times, which gave me a clean render here and there, but mostly the issue persists and it's taking me forever to produce clean renders. I notice this problem less when exporting straight from the timeline in Premiere Pro, but alas it still happens.

This random issue is catastrophic for my workflow. Help?!

[Text formatting corrected.  Please type your posts directly into the forums, as copy/paste often leaves unwanted code throughout.]

Message was edited by: Jim Simon

[Kevin-Monahan]

I see it now, never mind.

Kevin

[Kevin-Monahan]

Hi Jesse,

Can you check the internal heating of your Mac Pro? There are free utilities out there to check this. Perhaps the 8 core is giving you the same trouble we had with the 12 cores at NAB. Here's an article on the topic: Fixing an overheating Mac Pro - no expense required - Alx Klive's BlogAlx Klive's Blog

Thanks,

Kevin

[leffty24]

After going through this entire thread (skimming near the middle), I'd like to chime in that I'm reasonably certain this is a heat issue with the Firepro D700's. We've experienced this issue from day 1 on our Mac Pro exporting multiple codecs (h.264, prores) from multiple cameras (RED, Blackmagic prores/cinema DNG), using different applications (Davinci, REDCINE-X, Premiere, Media Encoder). There's only one common factor: exports that were done with a low-temp gpu had no errors (green horizontal single-frame lines), and as the temp increased, the errors increased. It's a very consistent progression that can be tracked through a batch of rendered files.

My specs:

Mac Pro late 2013 8 core

Dual Firepro D700

32gb ram

For those testing different combinations of applications, OS X, and drivers, you'll run yourself mad thinking you found a working combo if you happen to test it when it's sufficiently cool to avoid producing errors, only to have it show up again when the GPU is hotter. This is why it's been such a difficult issue to track. The issue may not be as bad (or exist at all) in FCPX because Apple may have been able to tune it in such a way to avoid overheating the GPU (just a theory). The fact that this issue shows up for Davinci, REDCINE-X, and Premiere tells me that Apple may have exceeded the thermal management capabilities of its new Mac Pro (despite designing the ENTIRE system around this concept) with these high-end cards, and are purposely underclocking it in its own pro applications to avoid issues.

This would explain why:

1) Those downgrading to less powerful cards are not experiencing the issue anymore

2) Switching off hardware rendering in certain applications yields fewer errors.

3) The errors are difficult to reproduce at the Genius Bar (because you have to turn off your computer and drive it to the store, during which time the computer completely cools off)

4) The errors exist across all application and OS X versions. The errors maybe slightly worse or better in different OS X versions, but each OS version will have tweaks to the heat/GPU handling. But this points to a very tedious situation with borderline overpowered cards for this system.

Kevin, I appreciate you engaging on this topic for quite some time. I don't see this as an adobe issue, so I'm not going to yell "PLEASE FIX". I hope the solution is not to simply under clock the GPU in Adobe applications. I don't know what's an acceptable operating temperature for a pro GPU (I assume all produce errors at some point), but this doesn't seem to be a proper implementation of these cards on Apple's part. The temp threshold for errors is too low.

If it's helpful, during renders that produce errors the GPU diodes are maxing at 187 degrees. The CPU's are over 200. Fan is maxed near 1900 RPM.

I'm heading to a store today to get it checked out, and I'll report anything of interest.

I hope we're close to a solution. If you've got data that contradicts my theories, I'd love to know if I'm on the wrong track.

Thanks,

David

[LMProductions]

Thanks for sharing David. I concur 100%. In my 14 months or so going through two different 2013 MacPros, both with Dual D700s, I narrowed it down to heat issues with the GPUs as noted in my earlier posts.

My Co-worker has the exact same machine and he has never had any issues, but he is not pushing the machine much and therefore it does not heat up like mine does. I've installed smcFanControl and am monitoring things more intelligently now to gauge what temps I hit before I start having issues. I was up to 135 degrees Fahrenheit yesterday but no signs of green lines yet. When I do see them, I will crank up the fan speed to cool off the machine and see if they go away.

Now the question is.. is this a GPU problem? Or a problem with how the new MacPro's are monitoring/implementing the GPUs? I believe there were a few reports on this thread of people with non-Apple machines with D700s reporting problems -- so that would indicate it is strictly an AMD problem. A better question yet would be, is AMD aware of this and how are they going to fix it? Or when will they issue the re-call? Im not holding my breath. It will likely be years before either company acknowledges the issue if they ever do at all. The good news is, knowing what the problem is, we can control the issue to some degree by monitoring the temps and implementing measures to keep the computers cool.

[leffty24]

That's exactly what I was wondering. Apparently the equivalent on the PC side is the Firepro W9000, so I'd be interested if that card has issues.

I'd love to know if fan control is helping you avoid it. I'm about to get a small a/c just to pump cool air into the bottom of it. Nobody's built a liquid cooling system for the new Mac Pro yet have they? That's probable impossible. I just found that you can turn off one of the cards in REDCINE-X, so maybe the fan can keep up with just one running. It's so freaking fast transcoding 6K RED footage with them both on though .

[nl_muc-postpro]

Well, we tested an export from a 12 core 64 GB Dual D700 in an xMac Pro Server Rack in a cooled Environment today actually and it doesn't make a difference at all. I wish it did but cooling obviously doesn't do the job..

[leffty24]

Thanks for sharing your test. That's really interesting. In digging a little deeper, I've tested by doing exports in REDCINE-X with one card switched off, and it seems only one of the D700's is causing the issue. When I solo the first card, I see glitches, and when I solo the other, I have none. I've only done a few tests where this is the case, so it's not enough to be certain only one of the cards is the issue. But if so, it would be kind of nice to know that only certain cards are bad and they could be replaced. So far 2 exports with one card yielded glitches, and three exports immediately following with the other card gave no glitches.

[whitediamondvid]

Hey leffty24 how did you turn off your graphics card? (I have mac pro 2013).

In your successful test (without glitches) which graphics card did you turn off, A or B?

[JesseSchluntz]

I'm on a mobile edit on my laptop until next week. I'll start monitoring the trash can when I get back to my studio for sure. If you can direct me to which utility I need to do this I'd appreciate it.

[Kevin-Monahan]

Hi Jesse,

If you can direct me to which utility I need to do this I'd appreciate it.

The link contains the utility I recommend for fan control: SMCFanControl

For monitoring heat (and other things), try a trial of iStat Menus, a Mac app by Bjango

Thanks,

Kevin

[nl_muc-postpro]

Hi Kevin,

we have a setup of one 12 Core dual D700, one 8 Core dual D700 and one 6 Core Dual D500. All late 2013 mac Pros and a few iMacs with GTX 780M Cards. The iMacs are fine, so is the 6 Core D500 MP.

Both, 8 & 12 Core with the D700s are having the same issues. We have tried literally everything from switching monitors, re-installing the systems, cooling them with SMC and fans but still have the "lines issue".. One User in this thread seems to have had luck by only using his 4k Monitor when exporting but I tried the same with only attaching one screen but still had problems, so I guess he was just lucky..

Our red frames are gone since we are currently in a beta program (NDA). Unfortunately we have to export all our longer edits on the 6 Core machine, shorter clips on the two D700s through Software only, to not block a system for exports only.. I have spoken to AMD, Apple and Adobe..Nobody seems to be aware of this issues! One technician from a partner company claims he has heard it's an issue with the "Adobe" Render Engine not working properly with the D700s and OpenCL.

This has been a real headache for a long time now and is causing a lot of issues in our Production, as well as additional expenses since we are losing a lot of time and resources. Obviously also when booking editors that have to push overtime!

Since you guys are still in testing phase I assume we still can't expect a fix in the coming update shown at NAB, correct?

Was really hoping to see this issues finally being resolved..

Read this below and assumed it might have done the job..

What to look for in the AMD Booth #SL9324 (south hall lower level)

• New fully integrated and redesigned OpenCL AMD accelerated features for Adobe Premiere Pro CC (part of Adobe Creative Cloud) allow editors to be more creative with color than before. In addition to OpenCL streamlined color features, Adobe will showcase an all new OpenCL powered Morph Cut feature for making, previously impossible, jump cuts natural and seamless.
• The Foundry’s NUKE node-based compositor, MODO 3D modeling, and MARI 3D paint tool powered by the Apple Mac Pro and the AMD FirePro D700 GPU;

I feel like I own a Ferrari I can only drive 30mph in first gear... Please fix this issue asap!! Really been long enough now..

If there is anything we can help with or test, please feel free to contact me.

best regards

[pavlesavic@me.com]

Same issues with the 6 Core and D700 : /.

I tried to install a fan ap and it doesnt change anythin.

Any news from anywhere to fix this ?

Thanks

Pavle

[agarabaghi]

So i just skimmed through this whole issue and want to add in what ive noticed and maybe this is where i should be looking for help?

Computer Specs:

Windows 8.1 x64

i7 4770k

2x R9 290

SSDs

Asus Maximus VI

32gig DDR3 1600

Symptom:

Amd Driver 14.11.2 beta (opencl 1.2) OpenCL GPU accelerated playback and rendering both work.

Amd Driver 15.4 Beta (or 14.12 Omega) (opencl 2.0) OpenCL GPU playback does not work and just shows a black screen (audio and timeline move just fine). If i go to software acceleration video comes back. I can queue the render in adobe media encoder and select opencl engine for the render and that works.

So whats up with the opencl 2.0 playback support?

[Kevin-Monahan]

Hi AGarabaghi,

Sorry you're having trouble.

Computer Specs:

Windows 8.1 x64

This thread is about the 2013 Mac Pro, and exporting H.264 with GPU acceleration enabled with dual GPUs. You're issue might sound similar, but it is not the same issue. Feel free to create your own post describing your issue.

Thanks,
Kevin

[Evett_Rolsten]

I have been battling this issue for the past 2 weeks trying to figure it out.  I'm not getting any banding but I'm seeing all kinds of artifacts and blocking going on in the shadows.  I've tried exporting in ProRes and in H.264 with same results.

Is anyone getting footage like this?  Look in the shadows on the tree within the first 10 seconds and you'll see what I'm talking about.

Private Video on Vimeo

password - premiere

Computer specs FYI

[Derjis]

Looks like it's just an issue with compression settings; definitely not the same issue being discussed on this thread.

If you want to start a new thread in the discussions, or shoot me a PM, I can help you out with the compression settings.

Cute dog, BTW!

[Hasenbau]

‌Probably it's "just" the bad osx opencl driver issue other software companies have to deal with???

http://preta3d.com/os-x-users-unite/

Had again the same issues on my last job.

EOS 1DC 4k footage. Luckily it was a green screen job and i rendered in AE not ME or PCC.

I did a quick keying test in PCC, but i needed keylight for better keying results.

The test in PCC again went crazy with lines and glitches.

I bought Apple Care one month ago because the cans first birthday was a few days ago.

No one knows if it's a Adobe, AMD or Apple Problem. Looking forward to the next 365 days of getting entertained by a watch company, a company that probably buys a freaking good software company (foundry) and ?AMD?.

D700s, 6 core, 64gb ram, 10.9.5, ironic user.

[APB_Post]

At work we were using GPU rendering on our Mac Pros for months before we started having issues. I don't believe our VFX guys have been having issues with After Effects, but us editors have been having better luck using the late 2013 iMacs using CUDA than the Mac Pros using Open CL, which is a disgrace. We have currently have 3 6-core late 2013 Mac Pros w/dual D500's for editorial and all 3 of them are having the horizontal banding issue when exporting using Open CL.

I've recently put SMC on one of the Mac Pros. This made me realize how horrible Apples heat management system is. Its great that they designed this new compact/high air flow case, but if by default the fan never creeps about 1000 rpm even when the temperature gets up to above 80c then how good of airflow can it really have? They love to make things super silent, but for a professional editing system it's all about airflow and performance. As soon as I turned up the fan speed to about 1300-1500 rpms the temperature immediately dropped about 3 degrees. Granted at the moment no one was doing any heavy rendering, but it still has me hopeful this might help our problem. The edit bays the mac pros are in tend to run warmer than the room the iMacs are in so I'm hoping our problem is more on the heat side and less on the Open CL issue, but only time will tell.

The strange thing about this all is that we had the Mac Pros for several months before this export problem started cropping up. I don't know if it's because the MPs were running hot the whole time and are now damaged or if it's because of an update that happened at some point that caused Adobe and OS Xs Open CL drivers to not play well. What ever the issue is for the moment disabling Open CL in Premiere and Media Encoder has eliminated the banding issue, but severely limited the MPs computing power. We still seem to have issues with Premiere crashing though so I have no idea what the issue is. I have been telling editors to disable Open CL and monitor their temperature so I will watch and wait for a Open CL fix from someone.

[Mac_Bauer]

There's been a lot of talk about GPU temps on this forum, I ran a few tests and here's what I found.  On my Mac Pro (6 core, 64 RAM, 512 GB, D700, one DP monitor) I'd start getting lines around 150-160 in media encoder.  I thought maybe Final Cut has some kind of feature that pays closer attention to GPU temps and adjusts performance accordingly to keep temps low... I was wrong.  Even when I manually turned the fan down (to artificially raise temps), Final Cut was still performing perfectly at 190 degrees (F).  The test was using 4k H.264 footage from a GH4 with tons of effects added on (keyed, color correction, tons of other stuff across multiple layers).  Next, I moved on to a couple of intensive games, maxed them out to the point where the frame rate was very low and the computer was running hot... but it never crashed.

This seems to be an issue with Premiere / Media Encoder.  Could there be driver issues? Could there be quality control issues with the cards?...sure, but why does everything else run fine and Premiere / Media Encoder (and DaVinci) have problems?  Kevin, have you guys ever compared notes with the Final Cut guys on how the cards are being used?  Maybe you are exposing a quality control or diver issue with the cards, but maybe there is also a more efficient way you could be using the GPUs (whatever Final Cut is doing).  I realize you are competitors in a way, but they are also part of Apple and I'm guessing Apple would be interested in replacing less $5-10k computers.

[Derjis]

Thanks for running those temp tests, Mac; I did a spit take when I read "190^o", and then was, like, "oohhhhhhhhhhhh, right. Farenheit". I think 190F is close to max rating for GPUs, but doesn't exceed max (couldn't find any specs specifically for the D500/700s). 160F is still really comfortable, in terms of temps. Interesting results!

[heyitsvo1]

I just did a similar test using FCPX to render a 20 minute 4k Red sequence to a 1080p H264 file. FCPX did NOT exhibit artifacting of any nature BUT the Premiere edit of the exact same specs had the green artifacts dancing all over the place after the 12minute mark in the file.

I looked at the console when premiere launched and it shows an error in regards to OpenCL 2.0

Last I checked Yosemite is OpenCL 1.5

[APB_Post]

The more I'm hearing peoples problems and researching on my own the more this seems to be an issue with Mac's Open CL driver. Why is there no Open CL 2.0 for Mac? If their high end systems use Open CL cards why don't they run the latest drivers? My only thought is  that Apple trying to discourage people from using Adobe on their Pro systems and pushing people back to FCP X. Not trying to point any fingers, but Apple has a history of forcing it's users to bend to their will and use their programs.

[OMS_AK]

Honestly Kevin, maybe you should consider buying out one or two Mac Pros from Adobe customers who are experiencing the issue (I'll remove myself from the pool of potential sellers, I just want to see this fixed).  It'd give Adobe a chance to observe and begin to troubleshoot the issue, as well as generate some good well by showing us that you're actually interested in fixing the problem.  There's enough people in this forum who have lost business from this over the last year, and maybe it's time for Adobe to have a little skin in the game.

[Kevin-Monahan]

Hi OMS_AK,

Honestly Kevin, maybe you should consider buying out one or two Mac Pros from Adobe customers who are experiencing the issue (I'll remove myself from the pool of potential sellers, I just want to see this fixed).

Yes, we're already doing that.

...that you're actually interested in fixing the problem.

Are you new to the thread? We've been trying to reproduce this issue for months. We can't, but we've been trying.

Thanks,

Kevin

[OMS_AK]

Kevin-Monahan wrote:

Honestly Kevin, maybe you should consider buying out one or two Mac Pros from Adobe customers who are experiencing the issue (I'll remove myself from the pool of potential sellers, I just want to see this fixed).

Yes, we're already doing that.

So you've purchased entire machines that are having the issue?  Or just the graphic cards?

We can't reproduce the error. We are still testing the faulty D700s (GPUs not the entire machine). No results yet.

If you can't duplicate it, why not start with the whole machine and work your way down?

Kevin-Monahan wrote:

Are you new to the thread? We've been trying to reproduce this issue for months. We can't, but we've been trying.

Not new.  This has been going on for 16 months and you can't duplicate it... how hard are you trying?  Are the faulty machines magically fixing themselves en route to your testing facility?