Introduction

Virtio memory ballooning is a technique that adjusts memory allocation in virtualized environments. The hypervisor can add or remove memory from a virtual machine based on demand, using a balloon driver in the guest operating system. When demand is high, the balloon driver inflates and the guest operating system releases memory. When demand is low, the balloon driver deflates and the guest operating system can use more memory.

This technique optimizes memory usage and reduces the risk of memory exhaustion, making it useful in cloud computing environments. However, it also has trade-offs to consider. Inflating the balloon driver can cause performance issues if the guest operating system can’t release memory quickly enough. It may also struggle with high memory pressure. Understanding these limitations is key to making informed decisions about using virtio memory ballooning.

Overview of Virtio Memory Ballooning

Based on wiki memory ballooning is a technique used to eliminate the need to overprovision host memory used by a virtual machine. To implement it, the virtual machine’s kernel implements a “balloon driver” which allocates unused memory within the VM’s address space into a reserved memory pool (the “balloon”) so that it is unavailable to other processes on the VM. However, rather than being reserved for other uses within the VM, the physical memory mapped to those pages within the VM is actually unmapped from the VM by the host operating system’s hypervisor, making it available for other uses by the host machine. Depending on the amount of memory required by the VM, the size of the “balloon” may be increased or decreased dynamically, mapping and unmapping physical memory as required by the VM.

According to the Virtio v1.2 specification, Virtio Memory Ballooning follows the Virtio protocol. Including:

Feature bits

• VIRTIO_BALLOON_F_MUST_TELL_HOST (0): Host must be notified before balloon pages are used.
• VIRTIO_BALLOON_F_STATS_VQ (1): A virtqueue is present for reporting guest memory statistics.
• VIRTIO_BALLOON_F_DEFLATE_ON_OOM (2): Balloon deflates when guest is out of memory.
• VIRTIO_BALLOON_F_FREE_PAGE_HINT (3): The device supports free page hinting. The configuration field free_page_hint_cmd_id is valid.
• VIRTIO_BALLOON_F_PAGE_POISON (4): The driver will immediately write poison_val to pages after deflating them. The configuration field poison_val is valid.
• VIRTIO_BALLOON_F_PAGE_REPORTING (5): The device supports free page reporting. A virtqueue is present for reporting free guest memory.

Memory Statistics Tags

• VIRTIO_BALLOON_S_SWAP_IN (0): Amount of memory swapped in (in bytes).
• VIRTIO_BALLOON_S_SWAP_OUT (1): Amount of memory swapped out to disk (in bytes).
• VIRTIO_BALLOON_S_MAJFLT (2): Number of major page faults that have occurred.
• VIRTIO_BALLOON_S_MINFLT (3): Number of minor page faults that have occurred.
• VIRTIO_BALLOON_S_MEMFREE (4): Amount of memory not being used (in bytes).
• VIRTIO_BALLOON_S_MEMTOT (5): Total amount of memory available (in bytes).
• VIRTIO_BALLOON_S_AVAIL (6): Estimate of available memory (in bytes) for starting new applications.
• VIRTIO_BALLOON_S_CACHES (7): Amount of memory (in bytes) that can be quickly reclaimed without I/O.
• VIRTIO_BALLOON_S_HTLB_PGALLOC (8): Number of successful hugetlb page allocations in the guest.
• VIRTIO_BALLOON_S_HTLB_PGFAIL (9): Number of failed hugetlb page allocations in the guest.

Free page hinting

Free page hinting is used during migration to determine which pages within the guest are not being used. These pages are then skipped over while migrating the guest. The device will indicate it is ready to start hinting by setting the free_page_hint_cmd_id to one of the non-reserved values that can be used as a command ID. The driver is notified of the following reserved values:

• VIRTIO_BALLOON_CMD_ID_STOP (0): any previously supplied command ID is invalid. The driver should stop hinting free pages until a new command ID is supplied, but should not release any hinted pages for use by the guest.
• VIRTIO_BALLOON_CMD_ID_DONE (1): any previously supplied command ID is invalid. The driver should stop hinting free pages and release all hinted pages for use by the guest.

When a hint is provided, it indicates that the data contained in the given page is no longer needed and can be discarded. If the driver writes to the page, this overrides the hint and the data will be retained. Any stale pages that have not been written to since the page was hinted may lose their content. If read, the contents of such pages will be uninitialized memory.

Page Poison

Page Poison is a feature that lets the host know when the guest is initializing free pages with poison_val. When enabled, the driver immediately writes to pages after deflating and pages reported as free will retain poison_val. If the guest is not initializing freed pages, the driver should reject the VIRTIO_BALLOON_F_PAGE_POISON feature. If the feature has been negotiated, the driver will place the initialization value into the poison_val configuration field data.

Free Page Reporting

Free Page Reporting is a method similar to balloon inflation, but without a deflation queue. Reported free pages can be reused by the driver after the request is acknowledged, without notifying the device.

The driver initiates reporting by gathering free pages into a scatter-gather list, which is then added to the reporting_vq. The exact timing and selection of free pages is determined by the driver.

Once the driver has enough pages available, it sends a reporting request to the device, which acknowledges the request using the reporting_vq descriptor. After acknowledgement, the driver can reuse the reported free pages by returning them to the free page lists in the guest operating system.

The driver can continue to gather and report free pages until it has reached the desired number of pages.

Comparison to Other Memory Management Techniques

Virtio memory ballooning is just one of several memory management techniques available in virtualized environments. Here are some other techniques that are commonly used:

Overcommitment

Overcommitment is a technique that allows virtual machines to use more memory than physically available. This is useful when memory demand is highly variable. However, overcommitment can cause performance issues if the host system runs out of memory and needs to swap memory pages to disk.

KVM hypervisor automatically overcommits CPUs and memory. This means that more virtualized CPUs and memory can be allocated to virtual machines than there are physical resources. This saves system resources, resulting in less power, cooling, and investment in server hardware while still allowing under-utilized virtualized servers or desktops to run on fewer hosts.

Memory Compression

Memory compression compresses memory pages to free up memory in high demand situations. However, this technique can lead to performance problems if the compression algorithm is slow or if memory demand is high.

Zram, zcache, and zswap advance in-kernel compression in different ways. Zram and zcache, both found in the staging tree, have improved in design and implementation, but they are not stable enough for promotion into the core kernel. Zswap proposes a simplified frontswap-only fork of zcache for direct merging into the MM subsystem. While simpler than zcache, zswap is entirely dependent on still-in-staging zsmalloc and has limitations. If zswap is merged, it remains to be seen if it will ever be extended adequately.

Hypervisor Swapping

Hypervisor swapping is a technique in which the hypervisor swaps memory pages between the host and guest operating systems in order to optimize memory usage. This can be useful in situations where there is a high demand for memory or when the host system is running low on memory. However, hypervisor swapping can also lead to performance issues if the guest operating system can’t release memory quickly enough.

Compared to these techniques, virtio memory ballooning has some unique advantages. It optimizes memory usage within the guest operating system itself, reducing the risk of memory exhaustion and improving performance. However, it also has some trade-offs to consider, such as the potential for performance issues if the guest operating system can’t release memory quickly enough.

How to use Virtio Memory Ballooning on linux

Environment

On host side we use libvirt to setup a vm.

The memory tag means: The maximum allocation of memory for the guest at boot time.

The currentMemory tag means: The actual allocation of memory for the guest.

1
2
3

<maxMemory slots='16' unit='KiB'>1524288</maxMemory>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>

And add memballoon virtio device in vm xml:

1

<memballoon model='virtio'>

To use Virtio Memory Ballooning on Linux guest, you’ll need to ensure that your kernel has support for the virtio_balloon driver. You can check for this by running the following command:

1

lsmod | grep virtio_balloon

If the virtio_balloon driver is not listed, you may need to load it manually by running the following command:

1

modprobe virtio_balloon

We can do some test to confirm balloon driver is working:

Basic usage

Explaination from

libvirt/virsh.rst at master · libvirt/libvirt

1
2
3
4
5
6
7
8
9
10
11
12

# virsh dommemstat YOUR_VM_NAME          
actual 8388608          # Current balloon value (in KB)
swap_in 7011156         # The amount of data read from swap space (in kB)
swap_out 664776         # The amount of memory written out to swap space (in kB)
major_fault 234565      # The number of page faults where disk IO was required
minor_fault 84722778    # The number of other page faults
unused 6291308          # The amount of memory left unused by the system (in kB)
available 8388044       # The amount of usable memory as seen by the domain (in kB)
usable 6349618          # The amount of memory which can be reclaimed by balloon without causing host swapping (in KB) *
last_update 1682566755  # Timestamp of the last update of statistics (in seconds)
disk_caches 116620      # The amount of memory that can be reclaimed without additional I/O, typically disk caches (in KiB)
rss 8529188             # Resident Set Size of the running domain's process (in kB)

with memory balloon we can get details about guest usage which matches the Memory Statistics Tags we metioned above.

And from dominfo we can see the memory usage directly

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# virsh dominfo YOUR_VM_NAME
Id:             7
Name:           1970b0ef25e44adc834767fe81f155d5
UUID:           1970b0ef-25e4-4adc-8347-67fe81f155d5
OS Type:        hvm
State:          running
CPU(s):         4
CPU time:       214084.1s
Max memory:     8388608 KiB
Used memory:    8388608 KiB
Persistent:     yes
Autostart:      disable
Managed save:   no
Security model: none
Security DOI:   0

Shrinking memory

At first, check the unused memory of your guest

1
2

# virsh dommemstat YOUR_VM_NAME | grep unused
unused 2868704

then we try to set memory to a size we want

Simply,

1

use actual - unused = 8388608 - 2868704 = 5519904

Then we use setmem

1

# virsh setmem YOUR_VM_NAME --size 5519904KiB --current

Check the shrink take effects:

1
2
3
4
5
6
7
8
9
10
11

# virsh dommemstat YOUR_VM_NAME
actual 5519904
swap_in 0
swap_out 2592
major_fault 6236
minor_fault 181380396
unused 140212
available 5139400
usable 3424496
last_update 1682567978
rss 5583008

actual changed to 5519904 and we check the guest on the other side

1
2
3
4

# free -hm
              total        used        free      shared  buff/cache   available
Mem:           4.9G        862M        134M        299M        3.9G        3.3G
Swap:          7.9G        3.5M        7.9G

Total memory changed even smaller than 5519904 ~= 5.26G about 7% memory missing and almost same with available 5139400

Expanding memory

To increase the memory allocation of a virtual machine using virtio memory ballooning, you can use the virsh setmem command. For example, to increase the memory allocation to 8GB, you would run:

1

virsh setmem YOUR_VM_NAME --size 8G --current

This will increase the memory allocation of the virtual machine to 8GB. However, it’s important to note that the guest operating system must have support for virtio memory ballooning in order to take advantage of this feature.

In addition, it’s important to monitor the memory usage of virtual machines to ensure that they have enough memory to operate effectively. This can be done using tools like virsh dommemstat to monitor memory usage statistics.

1
2
3
4
5
6
7
8
9
10
11

# virsh dommemstat YOUR_VM_NAME
actual 8388608
swap_in 0
swap_out 2592
major_fault 6236
minor_fault 181827159
unused 3008116
available 8008104
usable 6293140
last_update 1682571788
rss 7545844

Inside guest

1
2
3
4

# free -hm
              total        used        free      shared  buff/cache   available
Mem:           7.6G        862M        2.9G        299M        3.9G        6.0G
Swap:          7.9G        3.5M        7.9G

With 8GB memory from qemu side, guest have total 7.6G memory. There is still a 5% missing.

Industry Practices

Proxmox

Dynamic memory management shows that KSM and memory balloon works on windows and linux guest, a memory range from min and max will be required and guest’s memory will dynamicly changed between the range to impelement memory ballooning.

Google cloud

Dynamic resource management Memory ballooning is an interface mechanism between host and guest to dynamically adjust the size of the reserved memory for the guest. A virtio memory balloon device 
 is used to implement memory ballooning. Through the virtio memory balloon device, a host can explicitly ask a guest to yield a certain amount of free memory pages (also called memory balloon inflation), and reclaim the memory so that the host can use the free memory for other VMs. Likewise, the virtio memory balloon device can return memory pages back to the guest by deflating the memory balloon.

Compute Engine E2 VM instances that are based on a public image
 have a virtio memory balloon device , which monitors the guest operating system’s memory use. The guest operating system communicates its available memory to the host system. The host reallocates any unused memory to other processes on demand, thereby using memory more effectively. Compute Engine collects and uses this data to make more accurate rightsizing recommendations.

In Linux kernels before 5.2, the Linux memory system sometimes mistakenly prevents large allocations when the balloon device is present. This is rarely an issue in practice, but we recommend changing the virtual memory overcommit_memory setting to 1 to prevent the issue from occurring. This change is already made by default in all Google-provided images published since February 9, 2021.

To fix the setting, use the following command to change the value from 0 to 1:

1

sudo /sbin/sysctl -w vm.overcommit_memory=1

To persist this change across reboots, add the following to your /etc/sysctl.conf file:

1

vm.overcommit_memory=1

Nutanix

Squeeze even more memory of your HCI

Memory overcommit allows more memory to be assigned to VMs than is physically present in the server hardware. Unused memory allocated to a VM can be reclaimed by the hypervisor and made available to other VMs on the host. AHV adjusts memory usage for each VM according to its usage, allowing the host to use excess memory to satisfy the requirements of other VMs. This reduces hardware costs for large deployments or increases the utilization of an existing environment that can’t be immediately expanded with new nodes. VMs without memory overcommit will operate with their pre-assigned memory, and can coexist with overcommit enabled VMs. Nutanix uses a multi-tier approach combining ballooning and hypervisor-level swap to optimize performance. Metrics are presented to the administrator in Prism Central to indicate the gains achieved through overcommit and its impact on VM performance. Memory overcommit may not be appropriate for performance-sensitive workloads due to its dynamic nature.

Limits of Memory Overcommit

Memory overcommit has the following limitations:

• You can enable or disable Memory Overcommit only while the VM is powered off.
• Power off the VM enabled with memory overcommit before you change the memory allocation for the VM.
  For example, you cannot update the memory of a VM that is enabled with memory overcommit when it is still running. The system displays the following alert: InvalidVmState: Cannot complete request in state on.
• Memory overcommit is not supported with VMs that use GPU passthrough and vNUMA.
  For example, you cannot update a VM to a vNUMA VM when it is enabled with memory overcommit. The system displays the following alert: InvalidArgument: Cannot use memory overcommit feature for a vNUMA VM error.
• Memory overcommit feature can slow down the performance and the predictable performance of the VM
  For example, migrating a VM enabled with Memory Overcommit takes longer than migrating a VM not enabled with Memory Overcommit.
• There may be a temporary spike in the aggregate memory usage in the cluster during the migration of a VM enabled with Memory Overcommit from one node to another.
  For example, when you migrate a VM from Node A to Node B, the total memory used in the cluster during migration is greater than the memory usage before the migration.
  The memory usage of the cluster eventually drops back to pre-migration levels when the cluster reclaims the memory for other VM operations.
• Using Memory Overcommit heavily can cause a spike in the disk space utilization in the cluster. This spike is caused because the Host Swap uses some of the disk space in the cluster.
  If the VMs do not have a swap disk, then in case of memory pressure, AHV uses space from the swap disk created on ADSF to provide memory to the VM. This can lead to an increase in disk space consumption on the cluster.
• All DR operations except Cross Cluster Live Migration (CCLM) are supported
  On the destination side, if a VM fails when you enable Memory Overcommit, the failed VM fails over (creating the VM on the remote site) as a fixed size VM. You can enable Memory Overcommit on this VM after the failover is complete.

Limitations and Challenges

Guest should support virtio memory ballooning, if the balloon driver not available there is no effective way to do it.

Not all situations are suitable for memory ballooning. Frequent expansion and contraction of memory can be harmful if the memory usage changes dynamically.

Future Development

https://www.linux-kvm.org/page/Projects/auto-ballooning The auto ballooning project was initiated in 2013. The hypervisor and Linux kernel need to be updated to support the project, which has not been upstreamed yet.

Real-World Implementation Case Study

• Lessons Learned Building a Production Memory-Overcommit Solution

Conclusion

Virtualization is important in modern computing for flexible and efficient resource allocation. Memory management is challenging in virtualized environments when multiple virtual machines run on a single physical server. Virtio memory ballooning optimizes memory usage by dynamically adjusting guest memory reservation. It improves performance and reduces the risk of memory exhaustion. This article explains how to use virtio memory ballooning on Linux, compares it to other memory management techniques, and discusses industry practices, limitations, and future developments.

References

• libvirt/virsh.rst at master · libvirt/libvirt
• virtio-v1.1-csprd01/virtio-v1.1-csprd01.html#x1-2790005
• Dynamic memory management
• Dynamic resource management
• Squeeze even more memory of your HCI
• Limits of Memory Overcommit
• https://www.linux-kvm.org/page/Projects/auto-ballooning
• Lessons Learned Building a Production Memory-Overcommit Solution

Powered by Notion AI

qemu quorum block filter

Based on the code design of blkverify.c and blkmirror.c, the main purpose is to mirror write requests to all the qcow images hanging in the quorum, and the read operation is to check whether the number of occurrences of the qiov version meets the value set by the threshold through the parameters set by the threshold. Then it returns the > value of the result with the highest number of occurrences, if the number of occurrences i is less than the threshold then it returns the quourm exception and the read operation returns -EIO.

The main use of this feature is for people who use NFS devices affected by bitflip errors.

If you set the read-pattern to FIFO and set the threshold to 1, you can construct a read-only first disk scenario.

block-replication

The micro checkpoint and COLO mentioned in the introduction to the QEMU FT solution will continuously create checkpoints, and the state of the pvm and svm will be the same at the moment the checkpoint is completed. But it will not be consistent until the next checkpoint.

To ensure consistency, the SVM changes need to be cached and discarded at the next checkpoint. To reduce the stress of network transfers between checkpoints, changes on the PVM disk are synchronized asynchronously to the SVM node.

For example, the first time VM1 does a checkpoint, it is recorded as state C1, then VM2’s state is also C1, at this time VM2’s disk changes start to cache, VM1’s changes are written to VM2’s node through this mechanism, if an error occurs at this time how should it be handled?

Suppose we discuss the simplest case of VM1 hanging, then because the next checkpoint has not yet been executed, VM2 continues to run the state of C1 for a period of time and the disk changes are cached, at this time it is only necessary to flush the cached data to VM2’s disk single point to continue to run or wait for FT reconstruction, which is the reason for the need to do SVM disk changes caching (here the data (including two copies, one is to restore to VM2 last checkpoint cache, the other is to VM2 in C1 after the cache of changes)

The following is the structure of block-replication:

1. The block device on the primary node mounts two sub-devices via quorum, providing backup from the primary node to the secondary host. The read pattern (FIFO) is extended to meet the situation where the primary node will only read the local disk (the threshold needs to be set to 1 so that read operations will only be performed locally)
2. A newly implemented filter called replication is responsible for controlling block replication
3. The secondary node receives disk write requests from the primary node through the embedded nbd server
4. The device on the secondary node is a custom block device, we call it an active disk. it should be an empty device at the beginning, but the block device needs to support bdrv_make_empty() and backing_file
5. The hidden-disk is created automatically, and this file caches the contents modified by what is written from the primary node. It should also be an empty device at the beginning and support bdrv_make_empty() and backing_file
6. The blockdev-backup job (sync=none) will synchronize all the contents of the hidden-disk cache that should have been overwritten by nbd-induced writes, so the primary and secondary nodes should have the same disk contents before the replication starts
7. The secondary node also has a quorum node, so that the secondary can become the new primary after the failover and continue to perform the replication

There are seven types of internal errors that can exist when block replication runs:

1. Primary disk I/O errors
2. Primary disk forwarding errors
3. blockdev-backup error
4. secondary disk I/O errors
5. active disk I/O error
6. Error clearing hidden disk and active disk
7. failover failure

For error 1 and error 5, just report block level errors directly upwards.

For 2, 3, 4, and 6 need to be reported to the control plane of FT for failover process control.

In the case of 7, if the active commit fails, it will prompt a secondary node write operation error and let the person performing the failover decide how to handle it.

colo checkpoint 

colo uses vm’s live migration to achieve the checkpoint function

Based on the above block-replication to achieve disk synchronization, the other part is how to synchronize the running state data of virtual machines, here directly using the existing live migration, that is, cloud host hot migration, so that after each checkpoint can be considered pvm and svm disk/memory are consistent, so need to be in This event depends on the time of live migration.

First, let’s organize the checkpoint process, which is divided into two major parts

Configuration phase

This part will be executed mainly when the colo is first set up, we know that by default at the beginning we will configure the disk synchronization of pvm and svm, but the memory is not actually synchronized yet, so at the beginning we will ask the svm to be pused at first after the startup, and then submit two synchronization operations from the pvm side

1. Submit the drive-mirror task to mirror the contents of the disk from the pvm to the remote svm’s disk (embedded nbd is used here, which is also the target disk of the block replicaton later) to ensure that the pvm’s contents are consistent with the svm’s
2. Submit a migration task to synchronize memory from pvm to svm, and since both pvm and svm are required to be paired at this point, you actually wait until both pvm and svm are synchronized, then you need to cancel the drive-mirror task, start block replication, and continue running vm
   Of course, the paused state mentioned in 2 has been changed to be similar to hot migration after the improvements made by intel. After the drive-mirror task is submitted, the id of the task and the information of the block replication disks are used as parameters for the colo migration, which will actually be automatically changed when migrating in the line of online migration. After the migration is completed, the drive-mirror task is automatically cancelled and block-replication is automatically started before running vm, which simplifies the steps a lot.

After the configuration, you need to manually issue a migrate command to the colo pvm, and the checkpoint will enter the cycle of monitoring after the first migrate.

Start the checkpoint

The checkpoint mechanism consists mainly of a loop, and the code flow of qemu is as follows:

Combined with this picture we explain the more important parts inside

Process phase

COLO-FT initializes some key variables such as migration status, FAILOVER status and listens to the internal checkpoint notify (triggered from COLO compare)

After the first successful migrate, the discount state is initialized and the migration state is changed to COLO

After receiving a request for a checkpoint, a series of checkpoint creation processes are performed

Colo Status

For the COLO-FT virtual machine, there are two important states

One is the MigrationState, which on the COLO-FT virtual machine is MIGRATION_STATUS_COLO corresponding to the string “COLO”, which is a prerequisite state to allow checkpointing, and the cloud host must have established the COLO-FT mechanism. FT mechanism, that is, through the above configuration phase to complete the configuration and the first checkpoint, will enter this state and the main loop

Another state is failover_state, which is a global variable defined in colo_failover.c, which is accessed by colo.c through failover_get_state(), and this parameter is set to FAILOVER_STATUS at the start of the checkpoint loop _NONE, which means that failover is not needed. The bottom half of qemu mounts the mechanism for modifying this state, so it can be triggered by user state commands, so you need to pay attention to whether failover is triggered or not when actually doing checkpoint

Communitaion

COLO communicates through messages to get the status of the SVM, as well as to send and confirm the start and completion of the checkpoint, and the message process inside has the following main steps

1. Sending COLO_MESSAGE_CHECKPOINT_REQUEST
2. After the SVM receives the message, pause the SVM and send COLO_MESSAGE_CHECKPOINT_READY
3. PVM starts saving and live migration of VMSTATE
4. SVM gets the migrated information and does the CPU synchronization and VM state LOAD locally.
5. SVM will wait for a check message from PVM after the migration is completed, and PVM will send a message after the live migration is completed.
6. PVM sends COLO_MESSAGE_VMSTATE_SIZE with the size of VMSTATE sent via QIOChannelBuffer
7. SVM receives the message and checks if the size received locally is the same as the size sent, if it is, it replies COLO_MESSAGE_VMSTATE_RECEIVED
8. After confirming the VMSTATE transfer, the SVM will do some migration and subsequent synchronization and cleanup.
9. After completion, the SVM executes vm_start() and sends COLO_MESSAGE_VMSTATE_LOADED.
10. After the PVM receives the message that the SVM has successfully loaded, the PVM will also execute vm_start().

The logic of suspend, migrate and resume the operation of PVM SVM is realized through the message collaboration between PVM and SVM

Existing problems, because the current checkpoint are notified to each other through the message, once the corresponding packet is sent and not returned, the next wait may always exist, can not be closed, assuming that at this time from the bottom half (bottom half) to send a request also did not do to clean up the wait state.

It should be noted that: the default checkpoint once the failure occurs, the vm will be a direct exit, requiring the rebuilding of COLO-FT, so the establishment of COLO-FT failure needs to be analyzed from two parts

Whether the configuration phase migration has failed
Whether the configuration is complete (migration has become colo state) but the checkpoint failed (the above process failed) resulting in COLO-FT exit

colo proxy

colo proxy as the core component of COLO-FT, this article mainly focuses on the functionality of colo proxy in QEMU

When QEMU implements the net module, it actually treats the actual device in the guest as a receiver, so the corresponding relationship is as follows

TX RX

qemu side network device (sender) ——————-→ guest inside driver (receiver)

Combined with the code, the filter will be executed before actually doing transimission, and then go to sender processing.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

NetQueue *queue;
size_t size = iov_size(iov, iovcnt);
int ret;

if (size > NET_BUFSIZE) {
    return size;
}

if (sender->link_down || !sender->peer) {
    return size;
}

/* Let filters handle the packet first */
ret = filter_receive_iov(sender, NET_FILTER_DIRECTION_TX, sender,
                         QEMU_NET_PACKET_FLAG_NONE, iov, iovcnt, sent_cb);
if (ret) {
    return ret;
}

ret = filter_receive_iov(sender->peer, NET_FILTER_DIRECTION_RX, sender,
                         QEMU_NET_PACKET_FLAG_NONE, iov, iovcnt, sent_cb);
if (ret) {
    return ret;
}

queue = sender->peer->incoming_queue;

return qemu_net_queue_send_iov(queue, sender,
                               QEMU_NET_PACKET_FLAG_NONE,
                               iov, iovcnt, sent_cb);

The filter-mirror and filter-redirect in the network-filter implemented by colo act as the forwarding function of the proxy

The classic process for a network card is as follows:

The host device receives the network packet and sends it to the guest

1. First execute the first filter-mirror, for qemu is transimission, so execute mirror action, the network packet mirror a copy sent off through outdev (chardev), and then call the next filter (because it is TX, so other filters will not be executed, so pvm on (the packet is sent directly to the guest)
2. SVM’s indev connects to PVM’s mirror’s outdev (via socket), so it receives the packet sent by 1. The filter does not specify an outdev after receiving the packet, so it calls the next filter directly
3. SVM calls filter-rewrite, the direction of this filter is ALL, so the packets to and from SVM will be processed by this filter, if the target is sent, because it is sent to VM so the first direction is TX, COLO will record the various states of this TCP packet
4. Because there is no next filter so it is sent to the qemu network device, and then take the process of sending to the guest
5. From the guest to the qemu network packet, the direction is RX so the filter processing order will be reversed and sent to the rewrite first
6. SVM calls filter-rewrite this time in the direction of RX, so when processing, it will process the tcp packets returned by SVM, compare the input and output of tcp packets through the tcp packets table, and if the processing fails, it will put the packet in the queue and resend it again (note: need to continue deeper analysis), and then the filter- redirector
7. Also in the PVM mirror filter, because there is no subsequent TX filter, the packet is sent directly to the qemu net device and then to the PVM guest.
8. The packets coming out of the PVM guest will be sent to the primary in interface of the colo-compare thing by filter-redirector because it is in the RX direction, so some filters will be performed in the reverse direction
9. SVM will send the return of SVM to the secondary in interface of colo-compare of PVM via redirector’s outdev
   colo-compare receives the packet and starts to do the relevant analysis to decide whether checkpoint is needed
10. The filter-redirector’s indev receives the return from colo-compare after comparison and forwards it to the host net device via outdev

This is the end of a complete packet processing process.

Since colo-compare is responsible for comparing pvm and svm packets, there are some metrics that need to be understood

payload

payload_size = total size - header_size i.e. the size of the whole packet minus the length of the header

packet data offset packet header size after the distance and payload_size comparison is consistent

The following is a summary of what needs to be done here

The logic of colo-compare comparison is organized:

Possible reasons for network packet loss are therefore:

1. colo-compare did not receive the packet correctly
2. svm’s redirector did not successfully forward packets to rewrite
3. mirror did not replicate the packet successfully
4. pvm’s redirector did not successfully send pvm’s return to colo-compare
5. svm’s rewrite did not send/receive packets successfully
6. colo-compare is not sending packets correctly
7. svm’s redirector did not successfully forward packets to colo-compare

Problem processing

The processing of 1 mainly relies on the colo compare mechanism itself, for tcp packets will determine whether there are subsequent packets returned by ack, if there are subsequent packets, it means that the previous is missed

2 If the packet is not successfully sent to rewrite, it will not be processed by svm, so finally colo compare will encounter the situation where pvm has a packet but svm does not have a packet, and the processing is similar to 1

3 If mirror does not successfully copy the package, then there will also be a situation similar to 1, pvm exists package, svm no package

4 if pvm redirector did not successfully send the packet, then it seems from colo compare is pvm lost packets, but the same 1 processing, will wait for the pvm and svm minimum ack is exceeded, that is, both pvm or svm even if packet loss occurs, colo compare will wait for the updated packet to appear before returning the packet otherwise will always card does not reverse the current packet

5 If rewritte’s send-receive fails, this situation will cause the svm to not receive the packet and not return, similar to 1, but if failover occurs at this time, the svm packet is lost

6 this exception will lead to colo send and receive packet exceptions, network anomalies, not very well handled because itself colo compare is the core component

7 similar situation svm seems to have replied or actively sent the packet, but because colo compare did not receive, resulting in the svm within the data that did not reply, the benefit is that if the subsequent failover can occur, rewrite because the packet was recorded, will send the packet again, then it seems to be working again (need to test)

Trigger checkpoint

There are two conditions for triggering from colo-comare, because COLO-FT will establish a notification mechanism when it is established, and colo compare will trigger checkpoint from inside actively through this mechanism

1. Checkpoint will be triggered if the payload of the compare tcp packet is inconsistent
2. Timed to check if there is a certain period of time but has not received the same return packet (i.e., pvm, svm packet chain table content is inconsistent) trigger checkpoint
3. If there is a packet in the pvm list but not in the secondary packet, then it means that the packet reply is late, this situation is handled by 2, if the comparison finds that the non-tcp packet comparison is inconsistent will trigger a checkpoint

KVM虚拟化是一种常用的虚拟化技术，它可以将一台物理服务器划分为多个虚拟机，从而提高服务器的利用率和灵活性。然而，由于虚拟化带来的额外开销，KVM虚拟化的性能问题是一个常见的挑战。为了解决这些问题，我们需要使用一些性能诊断工具来分析和优化KVM虚拟化的性能。

以下是一些常用的KVM虚拟化性能诊断工具：

Perf

Perf是一种Linux性能分析工具，可以用于监视系统性能和调试性能问题。它基于Linux内核提供的性能事件接口，并提供了一个命令行界面，可以用于监视CPU使用率、内存使用情况、磁盘I/O等性能指标。

以下是使用Perf进行KVM虚拟化性能分析的最佳实践：

1. 安装Perf

要安装Perf，请使用以下命令：

1

sudo apt-get install linux-tools-common linux-tools-generic linux-tools-`uname -r`

2. 收集Perf数据

要使用Perf收集性能数据，请使用以下命令：

1

sudo perf record -g -p `pidof qemu-system-x86_64` -F 99

在这个例子中，-g选项表示收集函数调用图（用于生成Flame Graph），-p选项表示监视qemu-system-x86_64进程，-F选项表示使用99Hz的采样频率来收集性能数据。

3. 生成Flame Graph

要生成Flame Graph，请使用以下命令：

1

sudo perf script | stackcollapse-perf.pl | flamegraph.pl > output.svg

在这个例子中，perf script命令将Perf数据转换为脚本输出，stackcollapse-perf.pl命令将脚本输出转换为折叠栈，flamegraph.pl命令将折叠栈转换为Flame Graph。最终的Flame Graph将保存在output.svg文件中。

Sysstat

Sysstat是一个Linux系统性能监控工具，可以用于监视CPU、内存、磁盘I/O等性能指标。在KVM虚拟化中，您可以使用Sysstat来监视虚拟机的性能。以下是使用Sysstat进行KVM虚拟化性能分析的最佳实践：

1. 安装Sysstat

要安装Sysstat，请使用以下命令：

1

sudo apt-get install sysstat

2. 配置Sysstat

要配置Sysstat，请编辑/etc/default/sysstat文件，并更改以下变量：

1
2

HISTORY=7
INTERVAL=60

在这个例子中，Sysstat将每1分钟收集一次性能数据，并将数据保存最近7天。

3. 分析Sysstat数据

Sysstat收集的数据保存在/var/log/sysstat目录下。您可以使用以下命令来查看Sysstat数据：

1
2
3
4

sar -u
sar -r
sar -b
sar -d

这些命令将分别显示CPU使用率、内存使用情况、磁盘I/O等性能数据。

4. 使用Sysstat报告

Sysstat还提供了一个报告生成工具，可以根据Sysstat数据生成报告。要生成报告，请运行以下命令：

1
2

sar -A -o <outfile>
sadf -dh <outfile> > <reportfile>

这将生成一个包含所有性能数据的输出文件，然后使用sadf命令将输出文件转换为HTML格式的报告文件。

希望这些最佳实践可以帮助您更好地使用Sysstat进行KVM虚拟化性能分析。

如果您希望通过Sysstat数据进行趋势分析，可以使用一个名为ksar的工具。

ksar是一个Java应用程序，可以将Sysstat数据转换为图表，从而更方便地进行趋势分析。

要使用ksar，请按照以下步骤操作：

1. 安装Java

ksar是一个Java应用程序，因此您需要安装Java才能运行它。您可以从Oracle官方网站下载Java。

2. 下载和安装ksar

您可以从ksar的官方网站下载最新的版本。下载完成后，将压缩文件解压缩到您选择的目录中。

3. 运行ksar

要运行ksar，请打开终端并导航到ksar目录。然后，运行以下命令：

1

java -jar ksar.jar

4. 加载Sysstat数据文件

在ksar窗口中，单击“File”菜单，然后选择“Open”选项。选择您要加载的Sysstat数据文件。

5. 生成图表

在ksar窗口中，单击“Graphs”菜单，然后选择要生成的图表类型。ksar将生成一个图表，显示Sysstat数据的趋势。

如果您在KVM虚拟化中遇到了网络性能问题，可以使用以下工具来进行诊断：

tcpdump

tcpdump是一种常用的网络抓包工具。在KVM虚拟化中，您可以在宿主机上使用tcpdump来监视虚拟机的网络流量。以下是一个示例命令：

1

sudo tcpdump -i <interface> -w <output-file>

在这个命令中，是您要监视的网络接口，是保存抓包数据的输出文件。运行此命令后，tcpdump将开始监视指定的网络接口上的流量，并将所有数据保存到输出文件中。

Wireshark

Wireshark是一种网络协议分析工具，可以用于分析网络流量。在KVM虚拟化中，您可以在宿主机上使用Wireshark来分析虚拟机的网络流量。以下是一个示例命令：

1

sudo tshark -i <interface> -w <output-file>

virt-top

一个整合KVM虚拟化性能诊断工具的项目是virt-top。virt-top是一个基于ncurses的交互式监视器，可以用于监视KVM虚拟机的性能。以下是使用virt-top进行KVM虚拟化性能分析的最佳实践：

1. 安装virt-top

要安装virt-top，请使用以下命令：

1

sudo apt-get install virt-top

2. 运行virt-top

要运行virt-top，请运行以下命令：

1

sudo virt-top

3. 监视虚拟机性能

在virt-top窗口中，您可以使用上下方向键选择要监视的虚拟机。然后，您可以查看虚拟机的CPU使用率、内存使用情况、磁盘I/O等性能指标。

希望这些最佳实践可以帮助您更好地使用KVM虚拟化性能诊断工具。