KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9 / Kernel & Hardware (2024)

THIS GUIDE IS SLIGHTLY OUTDATED AND I'M NOT MAINTAINING IT ANYMORE
PLEASE CHECK OUT ALEX'S BLOG FOR AN UP TO DATE GUIDE

This is intented for people who wish to pass-through a GPU to a virtual machine using the KVM hypervisor, QEMU and vfio-pci

NOTE: AMD RADEON 5xxx, 6xxx, 7xxx, 2xx and NVIDIA GEFORCE 7, 8, 4xx, 5xx, 6xx, 7xx 9xx have been reported working with this, passing though an intel IGD is not supported YET

NOTE: THIS IS EXPERIMENTAL SO IT MIGHT NOT WORK ON YOUR SYSTEM

This is the result on my radeon 6950:

This is the result on my geforce 470 gtx:

Recommended reads

Alex Williamson's blog and FAQ for detailed technical information
OVMF whitepaper.
GPU passthrough database by noctavian

At this moment there are two ways to achieve this using KVM: legacy VGA via Seabios or non-VGA (UEFI) via OVMF

The main practical advantage of using OVMF is that on intel systems you no longer need the i915 VGA arbiter patch which also disables DRI on the host

Requirements

• AMD-VI/VT-D enabled and working (On intel systems both your MB and you CPU must support it, to find out if you CPU has VT-D support go here: http://ark.intel.com/, you also need to boot with intel_iommu=on to enable it)

• At least 2 GPU's, one primary boot device and the card you wish to pass-through

• Qemu=>2.0

• Aditional kernel patches might be required if you're using an Intel CPU: ACS override patch and i915 VGA arbiter patch, you can find a kernel package with these patches included on AUR: linux-vfio

Aditional requirements for OVMF:

• An UEFI compatible GPU (Most modern gpus support this, SEE: Does my graphics card ROM support EFI?

• An UEFI compatible GUEST (ex: Windows 7 and up)

• OVMF: Gerd Hoffmann's rpms or OVMF-svn from AUR

If you're building your kernel manually this option is required:

CONFIG_VFIO_PCI_VGA=y

Preparing your System

On intel system the i915 VGA arbiter patch is required if:

1. You're using the IGD on your host

2. You're doing VGA assignment

NOTE: Enabling this will disable DRI!

If you wish to avoid this, then then you should try OVMF

To enable it add this kernel parameter to your bootloader:

i915.enable_hd_vgaarb=1

Next, we need to prepare the GPU for vfio:

1. Blacklist radeon or nouveau or nvidia or fglrx on /etc/modprobe.d/blacklist.conf

   Example, blacklisting the opensource radeon module:

   echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf 

2. Use pci-stub

   In my case since i have 2 radeon cards blacklisting the radeon module is not an option, so i use pci-stub

   NOTE: If pci-stub was built as a module, you'll need to modify /etc/mkinitcpio.conf and add pci-stub in the MODULES section, after that you need to update your initramfs like this

   mkinitcpio -p linux-mainline

   lspci

   07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] <-- radeon 695007:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] <-- radeon 6950 audio

   lspci -n

   07:00.0 0300: 1002:6719 <-- radeon 695007:00.1 0403: 1002:aa80 <-- radeon 6950 audio

   Now add this kernel parameter to your bootloader:

   pci-stub.ids=1002:6719,1002:aa80

   dmesg | grep pci-stub

   [ 2.096151] pci-stub: add 1002:6719 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000[ 2.096160] pci-stub 0000:07:00.0: claimed by stub[ 2.096165] pci-stub: add 1002:AA80 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000[ 2.096174] pci-stub 0000:07:00.1: claimed by stub[ 2.096178] pci-stub: add 1B21:1042 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000

Setting up vfio and kvm modules

*********************************

This is only required if you get this message:

vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform

If your board doesn't enable interrupt remapping, you need to add this to your bootloader:

vfio_iommu_type1.allow_unsafe_interrupts=1

Or if vfio-pci was built as a module ( default on arch )

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf

*********************************

Some applications like Passmark Performance Test and SiSoftware Sandra crash the VM without this:

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

Binding a device to vfio-pci

We'll use this script to make life easier:

#!/bin/bashmodprobe vfio-pcifor dev in "$@"; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_iddone

Save it as /usr/bin/vfio-bind

chmod 755 /usr/bin/vfio-bind

Bind the GPU and GPU audio:

vfio-bind 0000:07:00.0 0000:07:00.1

Systemd service:

[Unit]Description=Binds devices to vfio-pciAfter=syslog.target[Service]EnvironmentFile=-/etc/vfio-pci.cfgType=oneshotRemainAfterExit=yesExecStart=-/usr/bin/vfio-bind $DEVICES[Install]WantedBy=multi-user.target

cat /etc/vfio-pci.cfg

DEVICES="0000:00:11.0 0000:04:00.0 0000:05:00.0 0000:06:00.0 0000:07:00.0 000:07:00.1"

Testing if its working out

Seabios:

OVMF:

Note:

• kvm=off will hide the kvm hypervisor signature, this is required for NVIDIA cards, since its driver will refuse to work on an hypervisor and result in Code 43 on windows (unless you're using a QUADRO)

• x-vga=on is required for vga assignment

• -vga none disables the default vga device on QEMU, it is also required for vga assignment

You should see a black qemu window on your main display, and seabios/ovmf ouput on your monitor from your passthru'd card saying it cant find anything to boot

If you're using an intel cpu and nothing happens try this:

modprobe -r kvm_intelmodprobe kvm_intel emulate_invalid_guest_state=0

NOTE: There might be some problems using nvidia/fglrx/nouveau drivers on the host gpu, see the ISSUES section below on how to solve this

At this point you're ready to create your vm, you could use libvirt + virt-manager to create your vm using a nice graphical interface, there are
also some interesting examples in this thread, otherwise if you prefer to use the command line continue reading

DISK:

Creating a disk image:

Create a 30Gb raw image:

dd if=/dev/zero of=windows.img bs=1M seek=30000 count=0

Using a VIRTIO controller:

We create our virtio controller for qemu:

 -device virtio-scsi-pci,id=scsi

Attaching our hdd:

-drive file=/home/nbhs/windows.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk

Attaching a cdrom:

-drive file=/home/nbhs/windows.iso,id=isocd,if=none -device scsi-cd,drive=isocd

To install windows, you'll need to download the virtio drivers iso from here

Attaching the virtio iso:

-drive file=/home/nbhs/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd

Windows will complain it cant find a disk to install to, just click browse, find the folder for your os/arch, select "RED HAT VirtIO pass-through controller" then click next.

For more info check out the Archlinux wiki's QEMU section

Using a physical disk or partition:

-drive file=/dev/sdb,id=disk,format=raw,if=none -device scsi-hd,drive=disk

You might need to change the bus depending on the controller you use

If you wish to use a physical partition and be able to read its contents later on, you can follow these guides:

http://fds-team.de/cms/articles/2013-12 … yer-u.html
https://wiki.archlinux.org/index.php/QE … disk_image

Passing a sata disk controller to qemu:

If you dual boot like me you can pass an entire sata controller (in ahci mode) and all the drives attached, and seabios will boot from them.

lspci

00:11.0 SATA controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)

Make sure all volumes are unmounted, then bind the controller to vfio:

vfio-bind 0000:00:11.0

Now we pass the controller to the vm:

-device vfio-pci,host=00:11.0

USB:

Its also possible to passthrough a usb controller, in my case an ASMEDIA USB3 Controller:

lspci

04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller05:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller06:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller

Bind them to vfio:

vfio-bind 0000:04:00.0 0000:05:00.0 0000:06:00.0

Pass them to the vm:

-device vfio-pci,host=04:00.0 \-device vfio-pci,host=05:00.0 \-device vfio-pci,host=06:00.0

Or a specific device:

lsusb

...Bus 004 Device 010: ID 045e:00e1 Microsoft Corp. Wireless Laser Mouse 6000 Reciever <-- mouseBus 004 Device 011: ID 045e:074b Microsoft Corp. <-- keyboard...

We pass them to the vm:

-usb -usbdevice host:045e:00e1 -usbdevice host:045e:074b

NETWORK:

Please see https://wiki.archlinux.org/index.php/QEMU#Networking

AUDIO EMULATION:

To hear the sound from the vm on your host speakers you'll need to add this lines:

-soundhw hda

You might need to start qemu like this:

QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64...

alsa:

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa qemu-system-x86_64...

Note: these are the settings i found that work great on my system, if you get crackling/skipping audio you might want to try different settings

To see the available drivers and audio options:

qemu-system-x86_64 -audio-help

LOADING YOUR CARD ROM FROM A DUMP/FILE:

modify this line:

-device vfio-pci,host=07:00.0,...... \

change it to this:

-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \

ISSUES:

• Using the latest NVIDIA drivers on the guest will result in code 43, but there's a workarround

• Using the latest NVIDIA drivers on the guest using hv enlightenments will result in code 43, see: https://forums.geforce.com/default/topi … 8/#4314318

• Using AMD proprieraty drivers on the host, see: https://bbs.archlinux.org/viewtopic.php … 2#p1273412 the only solution atm is using the opensource radeon driver

PERFORMANCE IMPROVEMENTS:

• SEE: https://bbs.archlinux.org/viewtopic.php … 1#p1270311

• Pin vcpu to pcpu, see http://www.linux-kvm.com/content/tip-ru … cific-cpus

• Use cgroups, specifically cpuset, see https://access.redhat.com/site/document … puset.html

• Use Hugepages, see http://pic.dhe.ibm.com/infocenter/lnxin … ttunhp.htm

• Use Hyper-V enlightenments, see: http://blog.wikichoon.com/2014/07/enabl … h-kvm.html

Last edited by nbhs (2015-07-11 16:23:26)