A few weeks ago, we encountered a problem with the orchestrated creation of virtual machines where “VMM Error 801: VMM cannot find VirtualHardDisk <GUID>” was being reported. The Orchestrator runbook was terminating, failing to create the VM. The service request would fail in turn.

I did a quick check up on Orchestrator and Service Manager, no misconfiguration or issues were evident on the runbooks or the request offerings. This was expected since the error was being returned by VMM. I created several test VMs to get some insight into the problem and noticed that the creation was failing on only one of the Hyper-V farms in our environment. The GUID was always the same, it never changed between test runs.

Further checking showed that only one node on the cluster was causing the issue. Whenever a VM is created on this node, the creation would fail. Creating the virtual machine directly from the template in VMM fails as well with the same error. Since this node had the lightest load, it was given the best star rating for placement. The node was healthy otherwise and was not reporting any issues. I migrated all roles off the node and created a test VM, it was created successfully. The problem had to be caused by one the hosted VMs.

I checked the VMM logs for any errors that popped up when the roles were being migrated. VMM had failed to refresh one of the VMs.

The GUID was the same. I created another test VM on the node that now hosted the roles, the creation failed with the same error. VMM was unable to refresh the problematic VM and this was causing it to be unable to create any new virtual machines on the node hosting this VM. When I tried checking this virtual machine properties through VMM, the console was unable to display the properties and it crashed.

I checked the VM through Hyper-V Manager. It was running on a differencing VHD with a direct link to the root VHD.

This VM was supposed to have a snapshot which was taken around November 2013 when the application was updated. Hyper-V Manager was showing that the virtual machine had no snapshots though. It seemed VMM was having problems refreshing the virtual disks of this VM or viewing its properties when the snapshot configuration is missing. I backed up the VMM database and executed the following SQL query to remove orphaned objects as detailed in this Technet article.

BEGIN TRANSACTION T1

DECLARE custom_cursor CURSOR FOR
SELECT VHDId, VDriveId from
dbo.tbl_WLC_VDrive WHERE [VHDId] NOT IN
(SELECT VHDId from dbo.tbl_WLC_VHD WHERE VHDID IS NOT NULL)

DECLARE @VHDId uniqueidentifier
DECLARE @VDriveId uniqueidentifier

OPEN custom_cursor
FETCH NEXT FROM custom_cursor INTO @VHDId, @VDriveId

WHILE(@@fetch_status = 0)
BEGIN
if(@VHDId is NOT NULL)
DELETE FROM dbo.tbl_WLC_VDrive
WHERE VDriveId = @VDriveId
FETCH NEXT FROM custom_cursor INTO @VHDId, @VDriveId
END
CLOSE custom_cursor
DEALLOCATE custom_cursor

COMMIT TRANSACTION T1

The query ran successfully, however, it did not update any rows and the problem remained.

I shut down the VM thinking that a merge operation would take place but Hyper-V did not initiate any such operation. This VM is hosted on a Windows Server 2008 R2 Hyper-V farm by the way, versions prior to Server 2012 do not support online VHD merge.

The disks had to be manually merged. Disk editing allows you to manually merge a differencing VHD back to its parent with a simple procedure.

From within Hyper-V Manager, launch Edit Disk. Browse to the location of the differencing disk then choose the Merge option. Finally, choose the option to merge the differencing disk back to its parent disk. When the merge process was finished, a new virtual machine was created and the newly merged VHD was assigned to it.

Virtual Machine Manager was now able to successfully refresh the properties of the new VM and the orchestrated virtual machines creation returned to a fully operational state.

Caution: In case you have more than one snapshot and they are chained serially, be careful with the options during the merge wizard. Always merge a differencing VHD back to its immediate parent and not the root unless it is the last (oldest) AVHD in the chain. Start from the newest snapshot and work your way back.