No Virtual Mahines were found on this server

Right… This has been driving me crazy today. Not really sure why it happened but I’ve managed to sort if out now and wanted to share!

Our hyper-v server hosting our domain controller and a couple of Linux machines decided today that it didn’t want to admit to having any machines hosted on it. When opening hte hyper-v management tools on my Win 7 laptop I was getting the message “No Virtual Mahines were found on this server”. That was quite a worry and made me reluctant to re-boot the host machine through fear of not being able to control the hosted machines any more.

My first hurdle was getting access to the Hyper-v R2 Event Log. For some reason the firewall was not configured to allow me to access this. This was a simple fix of opening the fiewall on the hyper-v server to allow WMI access. The alternative is to drop the fire wall for the duration of this fix (netsh firewall set opmode disable) and re-enable it after (netsh firewall set opmode enable).

Ok the real crux of the problem. For some reason the security permissions on the hyper-v configuration files had been lost or reset. It took me an age to fix this but finally managed to piece together enough information to fix it.

The Virtual Machine Management Service uses some configuration files from the directory %SYSTEMDRIVE%\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\. The xml files in this directory are in fact symlinks and should be named the same as the xml configuration files for the virtual machines them selves.

We store our virtual machines in the directory v:\VMs. So for example our machine test-01 has it’s xml file stored as V:\VMs\test-01\Virtual Machines\<guid>.xml.

Each virtual machine on the host has a similar set up. To fix the permissions you need to recreate the symlinks in the directory %SYSTEMDRIVE%\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\. To do that you need to execute the following commands for each machine.

mklink “%SYSTEMDRIVE%\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\<vm_guid>.xml” “V:\VMs\win2k8r2\Virtual Machines\<vm_guid>.xml”

icacls “%SYSTEMDRIVE%\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\<vm_guid>.xml” /grant “NT VIRTUAL MACHINE\<vm_guid>”:F /l

The one part that I missed initially was the “/l” on the end of the icacls command. This is very important and tells icacls to apply the security permission to the link rather than the destination file.

If any of your VM’s have snapshots you need to repeat a similar procedure for the snapshots directory.

mklink “%SYSTEMDRIVE%\ProgramData\Microsoft\Windows\Hyper-V\Snapshots\<snapshot_guid>.xml” “V:\VMs\test-01\Snapshots\<snapshot_guid>.xml”

icacls “%SYSTEMDRIVE%\ProgramData\Microsoft\Windows\Hyper-V\Snapshots\<snapshot_guid>.xml” /grant “NT VIRTUAL MACHINE\<vm_guid>”:F /l

When you have repaired all the files in the directories simply restart the Virtual Machine Management Service” by using the following two commands:

net stop vmms

net start vmms

You should now be able to view the files in the Hyper-v management console.

Linux IC components for CentOS

I’ve not posted for a while and I’ve just had something that people may be interested in.

I think I’ve posted before that we are using Microsoft Hyper-V server to host our virtual machine infrastructure at work. Over the weekend I thought I’d run an update on our linux machines (not been updated for approx 3 months). There was something like 300MB of updates to install so I started the update (by connecting to the console of the machine) and letting it run over the weekend.
When I came into work monday morning I figured it would be best to restart the machine and boot into the new Kernel… only to find that I got a visit from Kernel Panic. Damn I hate Kernel Panic! He’s a pain in the arse.
Anyway… turns out that the new kernel had some issues with the hard disks in use. I used the install CD to get into the recovery mode and changed in /boot/grub/grub.conf the line for the new kernel to remove the hda=noprobe hdb=noprobe. On re-boot the disks were probed and the system booted ok.
Problem now was the Linux Integration Components for hyper-v were compiled against the wrong kernel sources. To fix this it is a simple process of re-installing the IC components. (We are running Hyper-V R2 now so make sure you have the correct version of components installed.
If people want some steps to install the IC components post some comments and I’ll write another post and put together the steps there.

Apache running webs on a network drive…

Ok… here’s a tricky one I’ve just had to work through.

I have apache, Mysql, PHP running on my Windows machine. So that my development files are stored in a backed up location I wanted to move my files to a mapped network drive. This causes some problems which I’ve just worked through.
When you map a network drive this drive gets mapped under your user account. The apache service runs under the computer system account so has no visibility to the mapped drive and as such using w:\…. in your DocumentRoot directive will not work. Only local drives can be accessed in this manner. As a mapped drive this should have a UNC path associated with it. E.g. \\domain.local\Development\MyWebs. This is the style of path required for Apache.
When entering this path into the httpd.conf file you need to ensure that you use “/” instead of “\”. So the above would be “//domain.local/Development/MyWebs”.
You will then need to change the user that the Apache service runs as. I’ve set it to run under my domain account. This account I know has NTFS permissions to the mapped drive/UNC path.
One final step is to give the new account that the service runs as access to the logs directory under the server root (or move your logs to a location that has permission). After all that you should be able to have a local Apache server running webs stored on a network drive.

Split site DNS

Another problem… unfortunately not yet resolved.

The environment:
We have 2 windows servers in two separate sites in Active Directory. Server 1 is a Small business server so is forced to the be Primary Domain Controller. This is fine. Server 2 is a Server 2008 Standard acting as a secondary domain controller in our satellite office. We have two sites set-up in Active Directory for each physical location. This means the local clients connect to their nearest domain controller (based on IP subnets).
Our internal domain is “company-private.local” (for example). Externally we have a web presence using “company-public.com”. We have a split DNS set-up for accessing crm and email web access internally (to name but a few).
The Problem:
With the server in the satellite office we are implementing a replicated version of our crm software (utilising MySQL master-master replication). We have an external DNS record pointing to the head office implementation of the CRM software at crm.company-public.com. At the head office we have an internal DNS record pointing to the same server using an internal IP address.
What we want is to be able to set-up a scenario where we have the head office clients pointing to their closest copy of the CRM software, our satellite office pointing to their local copy of CRM and external clients able to connect to the head office all using the address crm.company-public.com.
The head office and external clients is simple. Setup a split DNS so the internal clients point to the internal address and the external clients point to the external address of the same machine.
The problem here is that with a Windows Server domain the dns records are kept in sync accross the sites meaning ALL clients get directed to the same machine.
I think I can set it up so internal clients connect to the correct sites by using crm.company-private.local and external clients connect to crm.company-public.com (Due to company-private.local being our active directory domain), but I’m wanting to set-up all to access the public address…..
Any readers got any ideas??? Please help!!!! lol

Microsoft Hyper-v Server

We’re in the process of doing a server upgrade (hence the lack of posts recently!). Things are just coming back to normal here so I’ve got some time to post some problems we’ve encountered. I’ll probably do them as a series of posts over the next few days.

The main problem we have faced has been the unexpected RAM hungriness of the Hyper-V server. We have one server running Windows Server 2008 SBS (which in it’s self demands at least 6 gig of ram) and a Linux email gateway server (with 1 gig assigned). What I didn’t realise first off was that when you assign this ram to the virtual machine it takes all of it away from the host OS. This has left our host hyper-v server with 1 GIG of ram.
According to the info we have found on the web people are saying that this is enough but I can state from our experience that this is not the case.
Recently we shutdown the SBS server to do some maintenance on it’s virtual Hard Disks and when it came to restarting it again the Hyper-v server stated that it did not have enough memory to complete the request. Basically… as soon as the SBS server shutdown the Hyper-v server grabbed the released RAM to complete it’s normal operations.
We’re now considering putting yet another 8 gig of ram into the host server just for it’s self to play with giving it 9 gig for it’s own operations, 6 gig to SBS for it’s domain and file server responsibilities and 1 gig to the email gateway server.
Lesson from this… when building a virtualisation server… give it as much RAM as your budget will stretch to… it’ll need it.

Why do things always go wrong at the same time?

I’ve had one of those days today where everything goes wrong at the same time. Our ecommerce system interfacing with sagepay (used to be protx) has decided to stop taking 3D secured transactions. Investigations are still on going here!

The other problem has been our exchange server. First thing this morning I get a call saying that the exchange server has stopped recieving connections again! Now this was the second time in as many days and shouldn’t be happening. On goes my investigative hat! It appears that the mailbox store on exchange is limited to 18GB by default and we’ve just reached this limit! I found this out by the hundreds of error messages in the system event log! Apparently you can make the store bigger but need to doso via the registry. After fixing this it appears to work again but think this will be a recuring problem… peoples mailboxes are going to continue to expand so I may have to investigate a mailbox archiving program/method at some point.

As for working on something that’s not been fixing things… I’m about to start looking into the Zend Framework MVC engine. Quite excited about this as the other parts of hte Zend Framework I’ve used so far has been brilliant! Very easy to use and delivers some quite complex functionality with minimal effort.

I’ll post again with the outcome of my investigations!