Imaging a Hard Drive to use in Unraid as a Virtual Machine

August 25, 2020 | By greg | Filed in: Servers, Technology, Unraid.

As I have been writing about recently, I have been in the process of upgrading the hardware for my home lab. I might not have been super clear in explaining what the process of migrating everything has been, so I will start by covering that. I’ll then explain how I imaged the SSDs in my trading computer to transfer to Unraid, so I could create a Manjaro Linux VM to run these images, just as they were on the separate computer. This would probably work with Windows as well, but as I have never tried it, I can’t say for sure although I’d bet there’d be problems along the way.

I purchased an HP ML350P along with a Cisco C220 to replace my current Unraid server and HP DL360 Gen 7 respectively. The Unraid server is built around an FX-8350, with 32GB RAM and maxed out SATA ports. I was running out of disk space, and I initially was going to purchase an LSI SAS RAID card and flash to IT mode. After taking into consideration I cannot add more memory and this just isn’t the platform to rely on for all my data, I decided to find an enterprise grade server to replace it with. I found the HP a few hours away and also snagged the Cisco rack mount along the way as well.

My plan is to use the old Unraid box as my new trading computer and get rid of the FX-6300 driven machine I am currently using.  First, I needed to test Unraid on the HP, as there are all kinds of posts across the web of people having issues with the HP servers and Unraid, software RAID, PCI passthrough, and even installing PCI cards in general; basically anything but typical enterprise use.  After installing the drives and creating a new Unraid flash drive, I got everything up and running on the new server and started testing, then got into dismantling the FX-6300 machine. I pulled the GPUs, an RX580 and FirePro W4000, and installed in the old Unraid server that will become my new trading computer.  Since I still needed to have a desktop PC to trade on while testing the HP, I ended up creating images of the SSDs to mount in Unraid as run as a virtual machine.  I could then use the GPUs passed through to the VM to output to my three monitors and with the images running on installed SSDs, it would be pretty near bare metal performance.

As I mentioned, the trading PC had two small SSDs installed in it.  One of these contained the boot partition, Manjaro root, and the swap partition while the home directory was exclusively located on the second.  I attached these to the Unraid server before running dd on each of them to image each drive completely to a gzipped image file.  The dd command to perform this:

dd if=/dev/sda/ bs=64K conv=sync,noerror status=progress| gzip -c manjaro_os.image.gz

The 64K block size parameter is the sweet spot of speed and RAM usage as I have found in previous testing.  My gzipped images were placed in an NFS share I created before hand on the server specifically for moving these images around.  I then used fdisk to create an info file containing the pertinent information to recreate these on a new drive if needed, such as the locations of the partitions on the disks.  The command and info file from the main drive is listed below.

Command:

sudo fdisk -l > manjaro_os.txt

Output to manjaro_os.txt file:

Disk /dev/sdb: 119.25 GiB, 128035676160 bytes, 250069680 sectors
Disk model:
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 1723D6A3-7EDE-42C5-B806-4C8294F10870

Device Start End Sectors Size Type
/dev/sdb1 2048 3715068 3713021 1.8G BIOS boot
/dev/sdb2 3715072 229144572 225429501 107.5G Linux filesystem
/dev/sdb3 229144576 250068986 20924411 10G Linux swap

That’s really all there is to imaging the drives.  It amazes me how so many people, especially “professionals” in the IT and MSP world, waste time pulling drives to clone in docks or trying to use Ghost or other GUI utilities when you can literally boot any Linux distro or even Windows and run a single command with dd to clone or image any drive or partition in minutes.  It’s one of the things that irritated me working for a cut rate IT shop, especially when the clients trusted us a professionals to do a job as efficiently as possible, only to pay four hours labor for a one hour job.  It is surprising how many people in IT know so little and can BS their way into sounding smart, but that’s a topic for another day.

Anytime I create and image, whether from a complete drive or just the partition, I always like to mount and test them.  Since these images were compressed, I needed to create a copy to uncompress.  I wanted to keep both images around as backups, so once a copy was made I placed the original into my backup share for storage.  I then extracted the copy with the gunzip command (gunzip manjaro_os.img) and then mounted each to Unraid via GUI for testing.  I renamed the extensions from {file}.image given by gzip to {file}.img so Unraid could recognize it in the VM Template GUI.  Both image files were moved to my cache drive, which is a 500GB SSD and I could begin to work on setting up the template.  You can see how I set this up below:

This is how the template looks using the image files in place of vdisk or qcow2 files and with the GPU attached to VM.

As you can see in the “Other PCI Devices” section, I have set this server up to pass the entire USB controller through to the VM rather than the USB devices themselves.  The main reason for doing this is so the USB devices can be “hot plugged” without having to restart the VM but also because it keeps the VM operating as close to a normal bare metal PC as possible.  I will create an instruction on how to set all of this up on another day as there are many different methods and configurations for this.

There is still a little fine tuning left to do in the XML template but I am currently trading on this computer with Thinkorswim and I haven’t had any issues.  The HP isn’t liking the GPU I added for testing so I will most likely need use a hacked iLO firmware or create a PWM controller to put in between the board and the fans.  Stay tuned for an article on that coming up shortly and thanks for reading!


Leave a Reply

Your email address will not be published. Required fields are marked *