I know exactly what you're thinking... "Has Kevin gone mad? Why is he talking about Windows, and Windows Server at that?" Well, the truth is, I use and support Windows on a daily basis, even Windows Server. Today's post may not be for everyone, and that's completely ok. Truth be told, it's primarily for me. I love it when someone else gets something out of my posts, but in some cases, the only person that may get something out of a post is me. Documenting processes and resolutions is a key part of my business, and that can sometimes leak out to the public.
Today's post is in response to a problem I encountered last week. I had several Windows Server 2012 R2 servers to update. When I checked for updates, there were over 200 available updates, nearly all of which were "critical". They obviously needed to be applied, so I began the process. At 99% completion, I was told that one of the updates had failed (impossible to know which one), and all pending updates were going to be undone. After about an hour of "undoing changes", the servers restarted. That's when things went from bad to really bad. The following screen appeared, and stayed there, even after hours of waiting, with no progress being made:
Obviously, a serious problem had occurred, and it was time to resolve the issue on these production servers. Before I get into the fix, you will notice that this is a virtual machine running on a Microsoft Hyper-V platform. With that in mind, here are the steps that I took to resolve the issue.
1) Turn off VM. In most cases, I recommend gracefully shutting down a Windows system. It's gentler on the OS than simply pulling the plug. In this case, you will likely need to just pull the virtual plug (using Hyper-V's "Shut Down" utility.
2) Adjust Autostart Settings (set a 5-second delay). In order to continue, we will need to boot the VM into Safe Mode. In order to do this, we need a little more time to press the F8 key when we turn on the VM. To add a 5-second delay, highlight the VM then go into the VM settings and adjust the "Automatic start delay" option.
3) Power on VM in Safe Mode. To do this, open the VM's console, power it up and press/hold the F8 key. The following menu will appear:
4) From here, we will want to choose "Troubleshoot" then "Command Prompt"
5) When prompted, you will need to enter the password for a local administrator account.
6) Next, we need to run the following command to get a list of all available drives:
wmic logicaldisk get name
7) Next, we need to locate the drive that contains the Windows directory. This may or may not be the C: drive listed. The easiest way to confirm is to check each drive letter for the Windows directory by using the "dir" command. The following instructions assume the Windows directory exists on the C: drive. Yours may differ. If it does, simply replace "C" with your drive letter.
8) Next, we need to navigate to the C:\Windows\WinSxS directory by using the following command:
cd Windows\Winsxs
9) Next, we need to list all files beginning with "Pending". To do this, we can use the following command:
dir pending*
10) Next, we need to rename all "Pending" files to append "_old" to the end of the file name. Here is a sample command that will rename "Pending.xml" to "Pending_old.xml" (pay close attention to the syntax):
ren Pending.xml Pending_old.xml
11) Next, we need to create a new blank "Pending.xml" file, which will be required steps to come later. We do this with the following command:
echo > pending.xml
12) Next, we need to navigate to the C:\Windows directory. To do this, we will use the following command (which simply navigates up one level):
cd..
13) Next, we need to create a new directory within Windows called "scratch", which is required for the next step. To do this, we will use the following command:
mkdir scratch
14) Next, we need to run a DISM command to remove/revert all pending actions from the damaged image. In my case, these are the failed updates which are attempting to install. To do this, we will use the following command:
DISM /Image:C:\ /Cleanup-Image /RevertPendingActions /scratchdir:C:\Windows\Scratch
15) Next, we need to rename the "SoftwareDistribution" folder to "SoftwareDistributionOLD". To do this, we will use the following command:
ren C:\Windows\SoftwareDistribution C:\Windows\SoftwareDistributionOLD
16) Next, we need to run a system file checker tool to repair missing or corrupt system files. To do this, we will use the following command:
sfc /scannow
17) Lastly, we need to restart the VM. The process may take slightly longer than normal, but after 30-60 seconds, you will see something looking like this:
Once this process has completed, Windows Server will be back to normal!
Easy enough, right? ;-)
