See notation of virt/kvm/kvm_main.c in linux kernel

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

/*
 * Kernel-based Virtual Machine driver for Linux
 *
 * This module enables machines with Intel VT-x extensions to run virtual
 * machines without emulation or binary translation.
 *
 * Copyright (C) 2006 Qumranet, Inc.
 * Copyright 2010 Red Hat, Inc. and/or its affiliates.
 *
 * Authors:
 *   Avi Kivity   <avi@qumranet.com>
 *   Yaniv Kamay  <yaniv@qumranet.com>
 *
 * This work is licensed under the terms of the GNU GPL, version 2.  See
 * the COPYING file in the top-level directory.
 *
 *

I got some questions

• what means kernel-based
• what is VT-x
• emulation? binary traslation?
• who is Avi Kivity
• is there any user-mode hypervisor?

Kernel-based

Kernel-based Virtual Machine (KVM) is a virtualization module in the Linux kernel that allows the kernel to function as a hypervisor. It was merged into the mainline Linux kernel in version 2.6.20, which was released on February 5, 2007. ^[1]

its available under linux/virt

VT-x

1
2

This module enables machines with Intel VT-x extensions to run virtual
machines without emulation or binary translation.

According to the code notation, Intel VT-x extensions is metioned.

Previously codenamed “Vanderpool”, VT-x represents Intel’s technology for virtualization on the x86 platform. On November 13, 2005, Intel released two models of Pentium 4 (Model 662 and 672) as the first Intel processors to support VT-x. The CPU flag for VT-x capability is “vmx”; in Linux, this can be checked via /proc/cpuinfo, or in macOS via sysctl machdep.cpu.features.^[2]

for example, on centos 7.6

1
2

[root@test ~]# cat /proc/cpuinfo | grep vmx | head -1
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat

or on Intel CPU MacBook Pro (2020)

1
2

➜  ~ sysctl machdep.cpu.features | grep -i vmx
machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX EST TM2 SSSE3 FMA CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC MOVBE POPCNT AES PCID XSAVE OSXSAVE SEGLIM64 TSCTMR AVX1.0 RDRAND F16C

vmx is available

“VMX” stands for Virtual Machine Extensions, which adds 13 new instructions: VMPTRLD, VMPTRST, VMCLEAR, VMREAD, VMWRITE, VMCALL, VMLAUNCH, VMRESUME, VMXOFF, VMXON, INVEPT, INVVPID, and VMFUNC.[21] These instructions permit entering and exiting a virtual execution mode where the guest OS perceives itself as running with full privilege (ring 0), but the host OS remains protected.^[2]

note: virtual execution mode is a important concept

refer to paper kvm: the Linux Virtual Machine Monitor KVM is designed to add a guest mode, joining the existing kernel mode and user mode

In guest-mode CPU instruction executed natively but when I/O requests or signal(typically, network packets received or timeout), exit guest-mode is required and kvm than redirect those I/O or signal handling to user-mode process to emulation device and execute actual I/O. After I/O handling finished, KVM will enter guest mode to execute its CPU instructions again.

For kernel-mode handling exit and enter is basic task. And user-mode process calls kernel to enter guest-mode until it interrupt.

Emulation & Binary translation

In computing, binary translation is a form of binary recompilation where sequences of instructions are translated from a source instruction set to the target instruction set. In some cases such as instruction set simulation, the target instruction set may be the same as the source instruction set, providing testing and debugging features such as instruction trace, conditional breakpoints and hot spot detection.

The two main types are static and dynamic binary translation. Translation can be done in hardware (for example, by circuits in a CPU) or in software (e.g. run-time engines, static recompiler, emulators).^[3]

Emulators mostly used to run softwares or applications on current OS where those softwares or applications are not support. For example, https://github.com/OpenEmu/OpenEmu a multiple video game system. This is advantage of emulators.

Disadvantage is that binary translation sometimes require instructino scan, if its used for CPU instruction translations, it spends more time than native instruction. More details in Translator-Internals will be talked in next blogs.

Avi Kivity

Mad C++ developer, proud grandfather of KVM. Now working on @ScyllaDB, an open source drop-in replacement for Cassandra that’s 10X faster. Hiring (remotes too).

from https://twitter.com/avikivity

Avi Kivity began the development of KVM in mid-2006 at Qumranet, a technology startup company that was acquired by Red Hat in 2008. KVM surfaced in October, 2006 and was merged into the Linux kernel mainline in kernel version 2.6.20, which was released on 5 February 2007.

KVM is maintained by Paolo Bonzini. ^[1]

Virtualization

For hardware assisted virtualiztion, VMM running below ring0 Guest OS, user application can directly execute user requests, and sensitive OS call trap to VMM without binary translation or Paravirtualization so overhead is decreased.

But for older full virtualization design

Guest OS runs on Ring1 and VMM runs on Ring0, without hardware assist, OS requests trap to VMM and after binary translation the instruction finally executed.

KVM Details

Memory map

From perspective of Linux guest OS. Physical memory is already prepared and virtual memory is allocated depend on the physical memory. When guest OS require a virtual address(GVA), Guest OS need to translate is to guest physical address(GPA), this obey the prinsiple of Linux, and tlb, page cache will be involved. And no difference with a Linux Guest running on a real server.

From perspective of host, start a Guest need to allocate a memory space as GPA space. So every GPA has a mapped host virtual address(HVA) and also a host physical address(HPA)

So typically, if a guest need to access a virtual memory address

GVA -> GPA -> HVA -> GPA

at least three times of translation is needed.

Nowadays, CPU offer EPT(Intel) or NPT(AMD) to accelerate GPA -> HVA translation. We will refer that in after blogs.

vMMU

MMU consists of

• A radix tree ,the page table, encoding the virtual- to-physical translation. This tree is provided by system software on physical memory, but is rooted in a hardware register (the cr3 register)
• A mechanism to notify system software of missing translations (page faults)
• An on-chip cache(the translation lookaside buffer, or tlb) that accelerates lookups of the page table
• Instructions for switching the translation root inorder to provide independent address spaces
• Instructions for managing the tlb

As referred in Memory map GPA -> HVA should be offered by KVM.

If no hardware assist, use shadow table to maintain the map between GPA and HVA, the good point of shadow table is that runtime address translation overhead is decrease but the major problem is how to synchronize guest page table with shadow page table, when guest writes page table, the shadow page table need to be changed together, so virtual MMU need offer hooks to implement this.

Another question is context switch. Shadow page tables based on the fact that guest should sync its tlb with shadow page tables so that tlb management instruction will be trapped. But the most common tlb management instruction in context-switch is invalidates the entire tlb. So the shadow page tables need to be synced again. Causes bad performance when vm runs multi processes.

vMMU is implement in order to improve guest performance which caches all page tables during context switch. This means context swtich could find its cache from vMMU directly, invdalidates tlb has no influence on context-switch.

• [1] https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine
• [2] https://en.wikipedia.org/wiki/X86_virtualization#Intel_virtualization_(VT-x)
• [3] https://en.wikipedia.org/wiki/Binary_translation
• [4] https://www.kernel.org/doc/ols/2007/ols2007v1-pages-225-230.pdf

之前的文章里，我们讨论了现存的virtio-networking架构，包括基于内核的（vhost-net/virtio-net）以及基于用户态DPDK的（vhost-user/vhost-pmd），现在我们需要转移我们的注意力到一个目标是让virtio-networking架构给VM提供有线连接速度的架构

本文将会涵盖构成这个架构的数据面以及控制面组件。我们将会介绍SR-IOV技术，以及这个技术如何提升网络性能。还会介绍virtio的硬件方案以及vDPA（virtual data path acceleration）带来的巨大好处。最后通过比较这些virtio-networking架构来做一个总结。

本文主要是为了那些有兴趣想了解不同virtio-networking架构的（包括vDPA），但不那么深入细节的人。当然后面也会提供一个技术细节的分享以及一个实践教程。

Data plane and control plane for direct access to NIC

在之前的vhost-net/virtio-net和vhost-user/vhost-pmd架构里，网卡都是接入在OVS kernel或者OVS-DPDK里的，而virtio的后端接口则是从OVS的另外一个port出去的

为了提升网络性能，直接把网络连到guest里，和之前的virtio架构类似，我们拆分了网卡的控制面和数据面：

1. 控制面。提供网卡和guest之前的配置修改和特性协商功能，用来建立和销毁数据面通道
2. 数据面。用来在guest和网卡之间传输数据包。当直接把网卡连接到guest的时候，实际上是要求网卡要支持virtio ring layout的

这个架构如下图所示：

笔记：

• 如果需要知道KVM，libvirt以及Qemu进程的额外信息，可以看前面的文章
• 数据面直接从网卡到guest，实际上是通过guest提供一个网卡可访问的共享内存实现的，并且并不经过host kernel。这个意味着网卡和guest都需要使用完全一致的ring layout否则就需要做地址翻译，地址翻译意味着性能损耗
• 控制面的实现则可能设计host kernel或者qemu进程，这个取决于具体的实现

SR-IOV for isolating VM traffic

在vhost-net/virtio-net和vhost-user/virto-pmd架构里，我们是用了软件交换机（OVS）可以让一个网卡对接到物理端口上，然后分发数据包到不同的虚拟机的端口上。

把网卡挂在虚拟机上最简单的方法就是硬件透传，也就是直接把一个网卡提供给guest kernel的驱动。

问题是我们需要在服务器上有一个单独的通过PIC暴露的物理网卡，接下来的问题就是我们如何在物理网卡上创建“虚拟端口”？

SR-IOV（Single root I/O virtualization)是一种PCI设备规范，允许共享一个物理设备给多个虚拟机。换言之，这个功能允许不同的虚拟环境里的虚拟机共享一个网卡。这意味着我们能够拥有一个类似把一个物理网卡拆分为多个以太网接口的功能，帮我们解决了上面提到的“虚拟端口”的创建问题。

SR-IOV有两个主要功能

1. Physical Functions，即PCI设备的完整功能，包括发现，管理和配置功能。每个网卡都有一个对应的PF能提供整个网卡设备的配置
2. Virtual Functions，是单个PCI功能，可以控制设备的一部分，并且是PF的子集。同一个网卡上能有多个VF

我们需要在网卡上配置VF，PF，VF相当于是虚拟接口，PF相当于是网络接口，举个例子，我们有一个10GB网卡有一个外部接口和8个VF。那么这个外部端口的速度以及双工是取决于PF的配置而频率限制则是VF的设置

hypervisor提供了映射virtual function到虚拟机的功能，每个VF都可以被映射到一个VM（一个VM可以同时有多个VF）

然后来看看SR-IOV是如何映射到guest kernel，用户态DPDK或者是直接到host kernel的吧

1. OVS和SR-IOV： 我们使用SR-IOV给OVS提供多个物理面端口，比如配置多个单独的mac地址，虽然我们只有一个物理网卡，但可以通过VF实现。并且给每分配一段内核内存到特定到VF（每个VF都有）
2. OVS DPDK和SR-IOV：跳过物理机内核，通过SR-IOV直接从物理网卡到OVS-DPDK。映射host用户态内存给网卡的VF
3. SR-IOV + guests：映射guest内存到网卡，跳过所有物理机环节。注意：使用设备透传，ring layout在物理网卡和guest之间是共享的，因此特定网卡才能被使用，因为这个逻辑一定是网卡厂商提供的。

注意：当然还有不是很常见的第四个方案，就是透传设备给guest里的用户态DPDK应用。

SR-IOV for mapping NIC to guest

重点说一下SR-IOV到guest的情况，这里存在一个问题就是在直接映射内存到网卡的场景下，如何更高效的发包收包。

我们有下面两个方法解决这个问题：

1. 使用guest kerel驱动：这个方法就是使用网卡厂商提供的kernel驱动，即直接映射IO内存，这样的话硬件设备就能够直接访问guest kernel的内存了
2. 在guest里使用DPDK-pmd驱动：这个方法，就是使用网卡厂商提供的DPDK pmd驱动，运行在guest的用户态，能够直接映射IO内存，因此硬件设备也能够直接访问用户态的特定进程

这一段我们重点看看DPDP pmd驱动的方案，整合起来就是下面这个图：

笔记：

• 数据面是和厂商挂钩的直接访问VF
• 对SRIOV，网卡厂商的驱动需要在host和guest都装
• host内核驱动以及guest的pmd驱动并不直接互相访问，PF/VF的驱动是通过其他接口配置的（比如libvirt等）
• 厂商提供的VF-pmd需要负责网卡VF的配置而PF驱动则负责在host内核上管理好物理网卡设备

总结一下这个方案，我们可以通过SR-IOV + DPDK PMC的方式给Guest提供一个很好的网络性能，不过这个方法还是挺麻烦的。因为这个方法是和厂商绑定的，因此需要在guest和host跑一样的驱动，并且特定网卡。这意味着网卡硬件升级，虚拟机应用驱动也需要升级。如果网卡被替换成了另外一个厂商的网卡，那么guest也需要装一个新的pmd。同时迁移虚拟机则会要求host上配置完全一致。也就是说网卡需要版本一致，物理位置需要一致，并且厂家还要提供迁移支持。

因此我们要处理的问题就是如何使用标准接口实现SR-IOV的性能提升，最好是只需要标准驱动，把这个驱动问题从整个架构中解耦出来。

下面的两个方案就是来解决这个问题的

Virtio full HW offloading

第一个方案是virtio的硬件替代方案，把virtio的控制面和数据面都转移到硬件上，也就是说网卡（当然还是通过VF来提供虚拟接口），支持virtio控制面的标准，包括发现，特性协商，以及建立/销毁数据面，等等。这个设备也支持virtio rang layout，因此一旦内存在网卡和guest之间被映射了，他们就能够直接通信了。

这个方案里，guest能够直接和网卡使用PCI通讯吗所以没有必要使用额外的驱动。然而需要网卡厂商提供支持virtio标准的网卡，包括控制面的软件实现，一般来说都是操作系统实现的，这个情况就是需要网卡自己实现。

下面是硬件架构的图：

笔记：

• 实际上控制面需要的操作是非常负责的主要是和IOMMU以及vIOMMU，下篇文章里会说
• 实际上在host kernel，qemu进程还有guest kernel都涉及这个流程，图里面简化了
• 当然也可以吧virtio数据面控制面放到kernel里而不是用户态（和SRIOV的场景一致），也就是直接使用virtio-net驱动来和网卡通讯（而不是使用virtio-pmd）

vDPA - standard data plane

Virtual data path acceleration (vDPA) 是一个通过virtio ring layout和放置一个SRIOV在guest，来标准化网卡SRIOV数据面将这个网络性能改善和厂家实现解耦的方案，通过增加一个通用的控制面以及阮家架构来支持vDPA。提供一个抽象层，在SRIOV之上，并且给未来的可拓展IOV打好基础。

和virtio的硬件方案类似，数据面直接建立在网卡和guest之间，都使用virtio ring layout。然而每个网卡厂家可能就会提供各自的驱动了，然后一个通用的vDPA驱动就被添加到了kernel里面来完成常见网卡驱动或控制面之间的virtio控制面翻译工作。

vDPA是一个灵活性更高的方案，相比硬件方案来说，网卡厂家支持virtio ring layout的成本更小了，并且也能够达到性能提升的目的。

示例图如下：

笔记：

• 实际上在host kernel，qemu进程还有guest kernel都涉及这个流程，图里面简化了
• 类似SRIOV和virtio全硬件方案，数据面控制面都是在guest内核里而不是用户态（优劣和之前提到的一样）

vDPA有潜力成为一个权威的给虚拟机提供以太网接口的方案：

1. 开源的标准：任何人可用，并且贡献标准，而不被特定的厂商限制
2. 优异的性能：接近SRIOV，中间没有翻译成本
3. 可以支持未来的硬件平台拓展技术
4. 独立于特定厂商的标准驱动，意味着只需要配置一次驱动，而不用太关心网卡和版本
5. 传输保护，guest直接使用单个接口。从host角度容易发现，并可以做好切换
6. 在线迁移，提供不同网卡不同版本的在线迁移
7. 提供一个标准的容器加速接口
8. 裸机，提供标准的网卡驱动mvirtio网卡驱动可以被作为一个裸机驱动，当时用vDPA软件架构的时候，驱动这个驱动来适配不同网卡硬件

Comparing virtio architectures

总结一下之前的系列里我们学到的内容，包括四种提供给vm以太网络的架构，vhost-net/virito-net, vhost-user/virito-pmd, virtio full HW offloading 和 vDPA。

然后来比较一下他们的优劣：

总结

这篇文章我们包含了四种提供以太网接口的virtio-networking架构概览，包括速度慢的（virtio-net）到比较快的（vhost-user）还有最快的（virtio硬件方案和vDPA）

我们强调vDPA和SR-IOV相对其他技术来说的优势，也提供了四种技术的对比，接下来会更加深入的使用virtio硬件方案以及vDPA。

CPU aceess memory

CPU core -> MMU(TLBs, Table Walk Unit) -> Caches -> Memory(Translation tables)

CPU VA -> MMU find PTE(Pysical table entry) -> TLB -> L1 cache -> L2 cache -> L3 cache

note: pretend a architecture with TLB between CPU and L1 cache.

TLB is a some cache form VA-to-PA translaction and formed by PTE blocks.

if TLB miss, CPU find PA from L1 and so on until PA is find and then put the PTE into TLB.

what is TLB?

TLB definition from wiki: A translation lookaside buffer (TLB) is a memory cache that is used to reduce the time taken to access a user memory location. It is a part of the chip’s memory-management unit (MMU). The TLB stores the recent translations of virtual memory to physical memory and can be called an address-translation cache. A TLB may reside between the CPU and the CPU cache, between CPU cache and the main memory or between the different levels of the multi-level cache. The majority of desktop, laptop, and server processors include one or more TLBs in the memory-management hardware, and it is nearly always present in any processor that utilizes paged or segmented virtual memory.

note:

1. TLB stores recent translations that means not all address translation entry is stored in TLB, take care about cache miss.

2. TLB may reside between the CPU and the CPU cache, between the CPU cache and primary storage memory, or between levels of a multi-level cache.

3. virtual addressing met cache miss or physical addressing, CPU always uses TLB to find and store it into cache.

4. cache strategy LRU or FIFO

5. The CPU has to access main memory for an instruction-cache miss, data-cache miss, or TLB miss, but compare to others the third case TLB miss is too expensive.

6. freqently TLB misses occur degrading performance, because each newly cached page displacing one that will soon be used again. Where the TLB acting as a cache for the memory management unit (MMU) which translates virtual addresses to physical addresses is too small for the working set of pages. TLB thrashing can occur even if instruction cache or data cache thrashing are not occurring, because these are cached in different sizes. Instructions and data are cached in small blocks (cache lines), not entire pages, but address lookup is done at the page level. Thus even if the code and data working sets fit into cache, if the working sets are fragmented across many pages, the virtual address working set may not fit into TLB, causing TLB thrashing.

TLB-miss handling

Two schemes for handling TLB misses are commonly found in modern architectures:

• With hardware TLB management, the CPU automatically walks the page tables . On x86 for example, use CR3 register to walks page tables if entry exists, bring back to TLB and TLB tries and access will hit. Or raise a page fault exception which need to be handled by operation system and load correct physical address to TLB(page swap in/out). CPU change do not cause loss of compatibility for the programs.
• With software-managed TLBs, a TLB miss generates a TLB miss exception, and operating system code is responsible for walking the page tables and performing the translation in software. The operating system then loads the translation into the TLB and restarts the program from the instruction that caused the TLB miss. As with hardware TLB management, if the OS finds no valid translation in the page tables, a page fault has occurred, and the OS must handle it accordingly. Instruction sets of CPUs that have software-managed TLBs have instructions that allow loading entries into any slot in the TLB. The format of the TLB entry is defined as a part of the instruction set architecture (ISA).

note:

1. hardware TLB management TLB handling the lifecycle of TLB entries.
2. hardware TLB management throws page fault that OS must handling and OS should bring the missing table entry of physical address into TLB cache. And than the program resume.
3. hardware TLB management maintain TLB enties is invisible to software.
4. hardware TLB management can change from CPU to CPU, but without causing compatibility for the programs. In other words, CPU should obey the rules of TLB management so there is always any page fault exception require OS to handle
5. software TLB management throws TLB miss exception and OS owns the responsibility to walk page tables and translation in software. Then OS loads TLB table and restart programs (attention! not resume but restart).
6. compare hardware and software TLB management, according to 2 CPU finds TLB and throw page fault exception when hardware, but in sofware situation, the CPU’s instruction sets should have instruction to load TLB to anywhere and TLB entry can be used directly by CPU instruction

In most cases, hardware TLB management is used. But according to wiki, some of the architectures using software TLB management.

Typical TLB

These are typical performance levels of a TLB:

• Size: 12 bits – 4,096 entries
• Hit time: 0.5 – 1 clock cycle
• Miss penalty: 10 – 100 clock cycles
• Miss rate: 0.01 – 1% (20–40% for sparse/graph applications)

The average effective memory cycle rate is defined as m + (1-p)h + pm cycles, where m is the number of cycles required for a memory read, p is the miss rate, and h is the hit time in cycles. If a TLB hit takes 1 clock cycle, a miss takes 30 clock cycles, a memory read takes 30 clock cycles, and the miss rate is 1%, the effective memory cycle rate is an average of 30 + 0.99 * 1 + 0.01 * 30 (31.29 clock cycles per memory access)

note: research more of TLB performance

use perf test TLB miss

1

perf stat -e dTLB-loads,dTLB-load-misses,iTLB-loads,iTLB-load-misses -p $PID

if a high TLB missing rate exists in your OS, try to use huge page to decrease the table entries in TLB which will cut down the miss rate. But some application is not siutable for huge page and more details need to be change before use this solution.

Address-space switch

After process context switches, some TLB entries’ virtual address to physical address mapping is invalid. In order to clean thoes invalid entires, some strategies is required.

1. flush all entries after process context change
2. mark the entries with its process so the process context change do not matter
3. some architecture use a sinlge address space operating system, all process use the same virtual-to-pysical mapping
4. some CPU have a process register and hardware uses TLB entries only the current process ID matches

note:
flushing TLB is an important security mechanism for memory isolation. Memory isolation is especially critical during switches between the privileged operating system kernel process and the user processes – as was highlighted by the Meltdown security vulnerability[2]. Mitigation strategies such as kernel page-table isolation (KPTI) rely heavily on performance-impacting TLB flushes and benefit greatly from hardware-enabled selective TLB entry management such as PCID.

Virtualization and x86 TLB

With the advent of virtualization for server consolidation, a lot of effort has gone into making the x86 architecture easier to virtualize and to ensure better performance of virtual machines on x86 hardware

EPT required.

reference

1. https://en.wikipedia.org/wiki/Translation_lookaside_buffer
2. https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)