This is going to be a nice long post. And it could get a bit heady since I'm not the greatest of teachers but I figured I'd help out a linux bro or 2 that goes through the pain of dual booting like I used to. A big plus to this method is you can use systemd to boot windows at the same time as your linux. You can also do some fun things with steam in home streaming.
First thing's first. You need some extra hardware. This works best with nvidia/intel hardware.
You need:
A motherboard AND processor that supports VT-D. I'm running an i7 5820k with an MSI X99A Raider. Asus boards have issues with iOMMU grouping so stay away from them (more on that later)
2 video cards (Nvidia is king here unfortunately) I'm running a GTX 960 for Linux, and a 970 for passthrough.
A big help is a PCIE lan card (intel is king here) but it's not expressly needed.
You will want to use Arch Linux or at the least Fedora Rawhide. More on that later.
The GUIDE
So I break my system a lot. Which is why I love btrfs and it's snapshot feature. I highly recommend setting this up since if you fuck something up it makes it super easy to boot back into a workable system. This guide is going to assume you are using Arch but should be similar for any distro really.
I'm using GRUB2 with UEFI.
First thing you want to do is enable vt-d. To do this you need to edit your grub.cfg. To do this simply open your terminal of choice and input
sudo nano /etc/default/grub
Look for the line that says "GRUB_CMDLINE_DEFAULT="" and addintel_iommu=on" to what is inside of that line.
Example:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
Then generate your grub.cfg with your changes with
sudo grub-mkconfig -o /boot/grub/grub.cfg
Reboot.
Once your back it's a good idea to check that VT-D is now actually running. First thing is to make sure the PCIE bus the card is installed in is actually on. You can do this with
lspci -nnk
The important info should look like this
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1131]
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_drm, nvidia
03:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1131]
Kernel driver in use: snd_hda_in
Kernel modules: snd_hda_intel
My 970 is installed in the 3rd PCIE slot so 03:00.0 is correct. 03:00.1 is the audio bus for the card and will also need to be passed through for clean audio. More on that later.
Next we need to see if the 970 falls into another pci-e bus's iommu group and if that will conflict. To find this out you look in /sys/bus/pci/devices/YOUR_BUS/iommu_group/devices/. If you do not have an iommu_group folder then vt-d was not enabled. Good luck figuring that one out.
Here's what mine looks like
[uruzzero@arcturus ~]$ ls -lha /sys/bus/pci/devices/0000\:03\:00.0/iommu_group/devices/
total 0
drwxr-xr-x 2 root root 0 Jun 19 23:39 .
drwxr-xr-x 3 root root 0 Jun 19 23:39 ..
lrwxrwxrwx 1 root root 0 Jun 19 23:39 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:02.0/0000:03:00.0
lrwxrwxrwx 1 root root 0 Jun 19 23:39 0000:03:00.1 -> ../../../../devices/pci0000:00/0000:00:02.0/0000:03:00.1
As you can see the card and it's audio bus are in one group. This is what you NEED TO SEE. If you see multiple devices in this group then read the link titled "Fixing IOMMU grouping" at the bottom of the post.
Next you need to install some software. And this is where Arch really shines. Assuming you've understood everything this far I'll skip explaining what AUR is and what it can do since you likely already know.
First thing you want to install is qemu. Specifically qemu git.
So use
yaourt -S qemu-git
Next go here and download edk2.git-ovmf-x64-###.noarch.rpm. This is the UEFI bios qemu will use to boot your system. You can use OMVF-git from aur but this is what I did and what worked for me.
Next install RPMextract
sudo pacman -S rpmextract
Then extract the files and move them to where they should go according to the rpm
[uruzzero@arcturus Downloads]$ ls
edk2.git-ovmf-x64-0-20160616.b1894.gd8d217c.noarch.rpm
[uruzzero@arcturus Downloads]$ rpmextract.sh edk2.git-ovmf-x64-0-20160616.b1894.gd8d217c.noarch.rpm
[uruzzero@arcturus Downloads]$ ls
edk2.git-ovmf-x64-0-20160616.b1894.gd8d217c.noarch.rpm usr
[uruzzero@arcturus Downloads]$ sudo cp -R usr/share/* /usr/share/
[uruzzero@arcturus Downloads]$ ls /usr/share/edk2.git/ovmf-x64/
OVMF_CODE-pure-efi.fd OVMF_CODE-with-csm.fd OVMF-pure-efi.fd OVMF_VARS-pure-efi.fd OVMF_VARS-with-csm.fd OVMF-with-csm.fd UefiShell.iso
Next you need to create a script to bind your GPU to vfio. I use this one
#!/bin/bash
modprobe vfio-pci
for dev in "$@"; do
vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
device=$(cat /sys/bus/pci/devices/$dev/device)
if [ -e /sys/bus/pci/devices/$dev/driver ]; then
echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
fi
echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done
I suggest putting it in /usr/bin/vfio-bind. Remember to chmod it.
Now use your newly created script to bind your GPU. You also need to bind the audio bus.
sudo vfio-bind 0000:0#:00.0 0000:0#:00.1
Replace the # with your pci bus id. Also here is where you want to pass through your pcie network adapter for easier networking but if you dont have one dont worry about it. Qemu sets up a network by default so you'll have internet. You just may need some extra configuration if you want to communicate between your VM and host machine.
Now you need the VirtIO drivers so windows can detect your passed through hdd during installation. You can get them here
Now you want to create a script to start your VM. Here's mine but check the QEMU wiki and make sure to tweak what's needed for your hardware.
#!/bin/bash
echo “Starting Windows 10”
xrandr --output DVI-I-1 --off
# xrandr --output HDMI-0 --off
cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/my_vars.fd
qemu-system-x86_64 \
-enable-kvm \
-name Windows10 \
-m 16384 \
-cpu host,kvm=off -smp 12,sockets=1,cores=6,threads=2 \
-vga none \
-soundhw hda \
-usb -usbdevice host:1532:0214 -usbdevice host:1038:1384 -usbdevice host:0909:001c \
-device vfio-pci,host=03:00.0,multifunction=on \
-device vfio-pci,host=03:00.1 \
-device vfio-pci,host=08:00.0,multifunction=on \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/tmp/my_vars.fd \
-device virtio-scsi-pci,id=scsi \
-drive file=/home/uruzzero/.kvm/win10.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \
-drive file=/dev/nvme0n1,id=ssd,format=raw,if=none -device scsi-hd,drive=ssd \
-drive file=/home/uruzzero/.kvm/virt.iso,id=virtiocd,if=none,format=raw -device ide-cd,bus=ide.1,drive=virtiocd
xrandr --output DVI-I-1 --mode "1920x1080" --rate 60 --left-of HDMI-0
# xrandr --output HDMI-0 --mode "1920x1080" --rate 60 --right-of DVI-I-1
You want to pass through an entire HDD for best performance in the VM. Make sure to replace -usbdevice host with your keyboard and mouse USB id.
Once you run the script your display should show "Press any key to boot from disk..." make sure and press it. If you miss it wait till you get to the uefi prompt. Type exit and navigate to boot manager and select your windows 10 iso. When you get to the disk selection screen it will prompt you for a driver. Navigate to the virtio iso, virtscsi folder, Windows 8.1, x64. This also works for windows 10. Go ahead and install your video card driver, steam, games etc at this point.
Troubleshooting:
"My sound sounds like an old man shitting in a pringles can!" See here
"How do I use my mouse and keyboard to control my host and guest at the same time?" A. Run windows without a display and use steam in home streaming, or B. install a neat little program called Synergy. It's behind a $10 paywall but here's a protip: Google "Synergy nightly builds". They are free.
I'm going to copy and paste this several other places but I figured it might be useful to SOMEONE here. Since I was talking to a few Linux bros on Server 1 yesterday and they were asking me how I got this working. I'll add a youtube video at some point. It's pretty nifty to be able to tun multiple operating systems at the same time with full gpu acceleration. If anyone has any linux questions feel free to talk to me about it either here or in game. I'm a Linux Foundation certified SysAdmin so this is kinda a passion of mine.
