Sometimes when you’re using KVM guests to test something, perhaps like a Ceph or OpenStack Swift cluster, it can be useful to have SSD and NVMe drives. I’m not talking about passing physical drives through, but rather emulating them.
QEMU supports emulating NVMe drives as arguments on the command line, but it’s not yet exposed to tools like virt-manager. This means you can’t just add a new drive of type
nvme into your virtual machine XML definition, however you can add those
qemu arguments to your XML. This also means that the NVMe drives will not show up as drives in tools like
virt-manager, even after you’ve added them with
qemu. Still, it’s fun to play with!
QEMU command line args for NVMe
Michael Moese has nicely documented how to do this on his blog. Basically, after creating a disk image (raw or qcow2) you can add the following two arguments like this to the
qemu command. I use a numeric drive id and serial so that I can add multiple NVMe drives (just duplicate the lines and increment the number).
-drive file=/path/to/nvme1.img,if=none,id=NVME1 \ -device nvme,drive=NVME1,serial=nvme-1
libvirt XML definition for NVMe
To add NVMe to a libvirt guest, add something like this at the bottom of your virtual machine definition (before the closing
</domain> tag) to call those same
<qemu:commandline> <qemu:arg value='-drive'/> <qemu:arg value='file=/path/to/nvme1.img,format=raw,if=none,id=NVME1'/> <qemu:arg value='-device'/> <qemu:arg value='nvme,drive=NVME1,serial=nvme-1'/> </qemu:commandline>
virt-install for NVMe
If you’re spinning up VMs using virt-install, then you can also pass these in as arguments, which will automatically populate the libvirt XML file with the arguments above. Note as above, you do not add a
--disk option for NVMe drives.
--qemu-commandline='-drive file=/path/to/nvme1.img,format=raw,if=none,id=NVME1' --qemu-commandline='-device nvme,drive=NVME1,serial=nvme-1'
Confirming drive is NVMe
Your NVMe drives will show up as specific devices under Linux, like
/dev/nvme0n1 and of course you can see them with tools like
nvme tool listing the NVMe drive in a guest.
sudo nvme list
This should return something that looks like this.
Node SN Model Namespace Usage Format FW Rev ------------- ------- --------------- --------- ----------------------- -------------- ------ /dev/nvme0n1 nvme-1 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 512 B + 0 B 1.0
SSD drives are slightly different. Simply add a drive to your guest as you normally would, on the bus you want to use (for example, SCSI or SATA). Then, add the required
set command to set rotational speed to make it an SSD (note that you set it to
1 in qemu, which sets it to
0 in Linux).
This does require you to know the name of the device so it will depend on how many drives you add of that type. Although it generally follows a format like this, for the first SCSI drive on the first SCSI controller,
scsi0-0-0-0 and for SATA,
sata0-0-0, but it’s good to confirm.
You can determine the exact name for your drive by querying the guest with
virsh qemu-monitor-command, like so.
virsh qemu-monitor-command --hmp 1 "info qtree"
This will provide details showing the devices, buses and connected drives. Here’s an example for the first SCSI drive, where you can see it’s
dev: scsi-hd, id "scsi0-0-0-0" drive = "drive-scsi0-0-0-0" logical_block_size = 512 (0x200) physical_block_size = 512 (0x200) min_io_size = 0 (0x0) opt_io_size = 0 (0x0) discard_granularity = 4096 (0x1000) write-cache = "on" share-rw = false rerror = "auto" werror = "auto" ver = "2.5+" serial = "" vendor = "QEMU" product = "QEMU HARDDISK" device_id = "drive-scsi0-0-0-0" removable = false dpofua = false wwn = 0 (0x0) port_wwn = 0 (0x0) port_index = 0 (0x0) max_unmap_size = 1073741824 (0x40000000) max_io_size = 2147483647 (0x7fffffff) rotation_rate = 1 (0x1) scsi_version = 5 (0x5) cyls = 16383 (0x3fff) heads = 16 (0x10) secs = 63 (0x3f) channel = 0 (0x0) scsi-id = 0 (0x0) lun = 0 (0x0)
QEMU command for SSD drive
When using qemu, add your drive as usual and then add the
set option. Using the SCSI drive example from above (which is on
scsi0-0-0-0), this is what it would look like.
libvirt XML definition for SSD drive
Similarly, for a defined guest, add the
set argument like we did for NVMe drives, that is at the bottom of the XML, before the closing
<qemu:commandline> <qemu:arg value='-set'/> <qemu:arg value='device.scsi0-0-0-0.rotation_rate=1'/> </qemu:commandline>
If your machine has NVMe drives specified also, just add the
set args for the SSD, don’t add a second
qemu:commandline section. It should look something like this.
<qemu:commandline> <qemu:arg value='-set'/> <qemu:arg value='device.scsi0-0-0-0.rotation_rate=1'/> <qemu:arg value='-drive'/> <qemu:arg value='file=/var/lib/libvirt/images/rancher-vm-centos-7-00-nvme.qcow2,format=qcow2,if=none,id=NVME1'/> <qemu:arg value='-device'/> <qemu:arg value='nvme,drive=NVME1,serial=nvme-1'/> </qemu:commandline>
virt-install command for SSD drive
When spinning up machine using
virt-install, add a drive as normal. The only thing you have to add is the argument for the qemu
set command. Here’s that same SCSI example.
Confirming drive is an SSD
You can confirm the rotational speed with
lsblk, like so.
sudo lsblk -d -o name,rota
This will return either 0 (for rotational speed false, meaning SSD) or 1 (for rotating drives, meaning non-SSD). For example, here’s a bunch of drives on a KVM guest where you can see
/dev/nvmen0n1 are both SSDs.
NAME ROTA sda 0 sdb 1 sr0 1 vda 1 nvme0n1 0
You can also check with
smartctl, which will report the rotational rate as an SSD. Here’s an example on
/dev/sda which is set to be an SSD in KVM guest.
smartctl -i /dev/sda
This shows a result like this, note
Rotational Rate is
Solid State Device.
=== START OF INFORMATION SECTION === Vendor: QEMU Product: QEMU HARDDISK Revision: 2.5+ Compliance: SPC-3 User Capacity: 107,374,182,400 bytes [107 GB] Logical block size: 512 bytes LU is thin provisioned, LBPRZ=0 Rotation Rate: Solid State Device Device type: disk Local Time is: Wed Dec 18 17:52:18 2019 AEDT SMART support is: Unavailable - device lacks SMART capability.
So that’s it! Thanks to QEMU you can play with NVMe and SSD drives in your guests.