[FIXED] Watchdog DPC Violation Error with Multiple NVIDIA GPUs
Written on May 11, 2018 by Ken ColorossiIntroduction
Recently we have been seeing an issue in our support department with a specific hardware combination: the Asus X99E-10G WS motherboard and triple or quad NVIDIA GPU configurations. When running a workload that is heavy on both the CPU and GPUs, systems with this hardware can crash and give the DPC_WATCHDOG_VIOLATION error or "blue screen of death" (BSOD).
We have created an instructional video to guide users through resolving this problem by reverting to an older NVIDIA graphics driver and then preventing Windows from automatically updating to a newer version. We are also currently coordinating with NVIDIA on a permanent fix. Updates will be published to the article as they become available.
Updates
Final Update 9/10/2018 [SOLUTION]
We have identified a solution that mitigates the DPC errors and while it does not completely solve the problem, in all testing the systems are significantly more stable with fewer crashes. If you need to update to the newest NVIDIA driver in order to run a particular application then this adjustment will help you. If you do NOT need to update in order to run an application we would advise against deploying the adjustment. The adjustment involves editing the registry so at this time we are not releasing a detailed guide however we are releasing an executable that will perform the registry edits for you.
Please navigate HERE and download the files to your affected computer. Once downloaded, run the application to perform the adjustment.
Alternatively, if you have any questions or prefer our Puget Systems Support staff perform the adjustment please don't hesitate to reach out; we are here to help!
Update 6/13/2018
Nivida has stated that the watchdog BSOD w/ multi GPU on X99-E 10G WS is now a "blocking issue", which means they cannot release a new driver unless it contains a fix.
Update 5/24/2018
With the release of new drivers Nvidia has removed the 382.53 driver from their list. You can access the download from the Nividia database here.
Update 5/15/2018
Currently Nivida has been able to replicate this issue in their lab and are working on a solution.
How to install NVIDIA Graphics Driver v382.53
The video is embedded below but if you would prefer to read the instructions rather than watch it you can click here to see the script text.
Video Script:
Today, we will demonstrate how to find, install, and hide the older driver for the Nvidia GPU problem with the DPC_WATCHDOG_VIOLATION. The issue is with the Asus X99E-10G WS motherboard and a triple or quad Nvidia GPU stack. When running a GPU and CPU heavy workload, it will cause the system to crash and give the DPC_WATCHDOG_VIOLATION, which looks like this. (SHOW SCREENSHOT).
Step 1: Revert to Older Graphics Driver
So we are going to go to “nvidia.com” then “click on support” then select “Download Drivers” then down in the “Beta, Older drivers and more” section select “Beta and Older Drivers” For this system, we are going to use the defaults because that is all correct. However, if you have GTX1080s or Titans, you’ll need to change the Product drop down to match your system.
From there, we are going to click “Search.” We are going to download the the driver version “382.53” by clicking on “GeForce Game Ready Driver,” then click “Download” and finish by clicking “Agree & Download.”
Once the download is complete. Close your browser, open file explore and navage to the downloads folder
Now, we are going to double click on the file, select yes to the “Do you want the app to make changes to your desktop?” Once you’ve done that, select “OK” on the “Specify the folder where the driver files are to be saved” that is going to unpack where it needs to for installation and start the installer. Next click Agree and Continue to the “License Agreement” then select “Custom” and “NEXT.” We are doing this so we can select “Perform a clean installation” This is very important. Also we are going to un select “NVIDIA GeForce Experience.” This is for gamers and isn’t needed for what we are doing.
Then click “NEXT” This will remove all remanence of the new driver and install the older 382.53 driver. Now that it is done with the removal we will restart the system by clicking “Restart Now” and when we come back it will automatically start the installer. Now that the driver is finished installing we will restart by clicking “Restart now”.
Step 2: Installing Wushowhide
Now that the driver is installed, let’s download and run a program from Microsoft to hide the NVIDIA driver from Windows Update so it doesn't try and force install the newer driver again. We are going to do a Google search for wushowhide (windows update show hide). We want the one what shows it’s from Microsoft. We are going to scroll down to the section “For Windows 10 v1607 (Anniversary Update) and click on “Download the “Show or hide updates” troubleshooter package now” .
Now we are going to go back to our Downloads folder and open the wushowhide program.
This may need to be run a few times until the NVIDIA driver update shows up. So we are going to double click on the file to run it. We are going to click on “Next”. It’s going to look for pending updates. Then we are going to click on “Hide Updates” Then we are going to select the NVIDIA Display driver and click “NEXT”. It will end at “Troubleshooting has completed” then click “Close” and you are done.
Conclusion
We hope you found the video above helpful!
If you own a Puget System and would like further assistance with this issue, please go here to submit a support request.
If you do not own a Puget System, but are experiencing this issue and need assistance, please go here to submit a help request with NVIDIA directly. You may also want to contact the manufacturer of your system for additional support.
Yeah unfortunately this is not a good fix when you use software that requires the newest Nvidia drivers.
I'm affected by this issue too. Unable to use Vray RT and Octane Render. Both need newer driver than 382.53. Very sad this problem is going on for 6 months now. Nvidia please wake up. Thanks Puget to make Nvidia acknowledge the problem.
Thank Puget Systems for taking care of this problem and giving us this update! I really hope the problem will be fixed soon !
Any update on this? New nVidia driver does not contain the fix and still lists this issue as Unresolved.
New Nvidia Driver has been released and the bug is still in the Known Issue. So apparently Nvidia didn't think it was a "blocking issue". That is really frustrating.
Hello. Nvidia did publish a new driver that did not resolve the issue. However, we have been conducting internal testing and have found that the new driver + a specific registry edit DOES resolve the issue. We are working on putting together detailed instructions for implementing this fix. Please stay tuned in to the article for an update in the next couple of days
Hi!, if you are talking about TDR Delay it didn“t fix the issue for me
Thank you Ken. Looking forward to test it.
Thanks Kens. Really hope this works!
Ken, do you have any update? We're are hanging on your words
Thank you for your Inquires everyone. We are still working on our internal testing and hope to have something for you very soon. Our current solution is not yet stable enough for public release.
I need a favor from someone using an x99-e 10g ws motherboard. I have been dealing with an sli issue on this particular motherboard for awhile and I do believe it is related to the PLX chip found on the board. I have rma'd the videocards, the motherboard and the memory. At this point I am ruling the memory and the videocards out. As for my request, can someone please SLI two (no more, no less) nvidia cards on this motherboard on slots one and five (cannot be any other slots) using the MOST RECENT drivers and tell me if they get any crashes, errors etc. I only ask for this to be done as I currently have two GTX 1070s in slots one and three (liquidcooled and the linking block only spans that far) and I am wondering if I sli the two cards across slots one and five (hence each card will be operating on a different PLX chip) will the issue cease. Currently I am limited to using only nvidia driver 382.53 (its the only one that works, no crashes, errors, BSODs). Without the latest drivers installed a lot of current games will not recognize my second videocard, some games wont even start (GTA V). Also I am using Windows 10 64 bit. Thanks guys.
The registry script above did not set my GPUs correctly in MSI mode. I found the MSI utility v2 linked here doing a better job.
https://forums.guru3d.com/t...
It did not fix my problem nonetheless.