最近在 8 张 BW1100 上跑 DeepSeek-V3.2 W8A8，顺手做了一次 Profiler 分析练习。

这篇文章记录低并发 Decode、16K Prefill 和高并发 Decode 下的 kernel 时间分布，以及从这些数据里能看到什么、暂时还不能说明什么。

2026-09-02

XFS 报错但根因不在 XFS：一次 uprobe 越界写排查

Phenomenon

一周内，两台配置相近的 Linux 虚拟机先后出现业务无响应。控制台最后的关键日志来自 XFS：文件系统发现 inode 元数据损坏，随后主动执行 forced shutdown。

XFS console corruption log

从现象上看，这很像 XFS bug、虚拟磁盘写错位置，或者底层存储损坏。但这次真正写坏 XFS 元数据的并不是 XFS。

最终定位到 Linux kernel 的 uprobe 参数抓取路径：第三方安全 Agent 注册的 uprobe 抓取了超过一页大小的参数，store_trace_args() 在边界检查前已经把数据写出 per-CPU buffer。溢出的内容落到相邻内存后，XFS write verifier 成了第一个发现损坏的组件。

换句话说，XFS 是报警的人，不是制造问题的人。

Start from the XFS error

XFS 检测到元数据不符合预期时会主动关闭文件系统，防止错误继续扩大。因此，xfs_do_force_shutdown 说明 XFS 看到了损坏，但不能直接证明损坏由 XFS 产生。

面对这种问题，最开始需要同时保留几条假设：

XFS 自身逻辑错误。
virtio-blk、宿主机或存储后端写错位置。
内存硬件或 DMA 写坏数据。
第三方内核模块破坏内存。
Linux 通用内核路径发生越界写或 use-after-free。

这里最容易犯的错误，是看到 XFS 日志就只沿着文件系统代码向下查。内存损坏问题里，打印错误的函数经常只是受害者。

Find the data signature

真正改变排查方向的是损坏数据的内容。

在异常的 inode metadata buffer 中，可以看到类似路径和 JSON 结束符的数据片段。它们不像 XFS 元数据，也不像随机 bit flip，更像某个用户态程序正在处理的字符串。

这个特征缩小了问题范围：

如果是硬件随机错误，数据通常不会形成可读的 JSON 结构。
如果是普通的 XFS 逻辑错误，也不应该把用户态字符串写进 inode metadata。
更合理的方向，是某条从用户态取字符串的内核路径越过了自己的 buffer 边界。

此时再向前检查控制台日志，可以找到 uprobe 相关 warning，并且触发进程来自第三方安全 Agent。warning、进程和损坏数据的内容开始形成同一条证据链。

The uprobe buffer

uprobe 可以在用户态程序的指定地址放置探针。探针触发时，内核会读取寄存器或用户态内存中的参数，再把结果写入 tracing buffer。

相关路径使用一页大小的 per-CPU 临时 buffer。问题发生时，代码大致遵循下面的顺序：

根据 trace event 和动态参数计算 dsize。
调用 store_trace_args() 把参数写入 ucb->buf。
返回上层后才检查总长度是否超过 PAGE_SIZE。

这类检查虽然存在，但没有保护作用：当检查执行时，越界写已经发生。

上游修复把检查移动到了 store_trace_args() 之前，并在长度超过一页时截断 dsize：

#define MAX_UCB_BUFFER_SIZE PAGE_SIZE

ucb->dsize = tu->tp.size + dsize;
if (WARN_ON_ONCE(ucb->dsize > MAX_UCB_BUFFER_SIZE)) {
    ucb->dsize = MAX_UCB_BUFFER_SIZE;
    dsize = MAX_UCB_BUFFER_SIZE - tu->tp.size;
}

store_trace_args(ucb->buf, &tu->tp, regs, NULL, esize, dsize);

对应的上游提交是 373b9338c972 - uprobe: avoid out-of-bounds memory access of fetching args，该问题也被登记为 CVE-2024-50067。

uprobe out-of-bounds write mechanism

Why XFS was damaged

这次抓取的参数比一页 buffer 多出少量字节。这些字节不会停在 ucb->buf 的结尾，而是继续写入相邻内存。

被覆盖的内存后来作为 XFS inode metadata buffer 使用。XFS 在 write verifier 中检查 magic、字段和结构一致性时发现内容已经损坏，于是执行 forced shutdown。

目前的证据能够确认：

uprobe 参数长度超过一页。
旧代码先写入、后检查，存在确定的越界窗口。
损坏区中出现了与触发进程参数一致的用户态字符串特征。
上游存在与代码路径和现象匹配的修复。

还有一个可以继续补强的物证：从 vmcore 中取出触发 CPU 的 uprobe_cpu_buffer 物理页，再与受损 XFS buffer 的物理页比较，确认两者在故障现场是否直接相邻。它不改变修复方向，但能把最后一段内存落点推断变成可审计的现场证据。

Resolution

短期处理：

升级或停用会触发该路径的第三方 Agent 探针。
在受影响内核上回合上游补丁。
避免继续运行相同版本、相同探针配置的机器。

长期处理：

为生产虚拟机配置可用的 kdump，并验证 vmcore 确实可以写出。
保留串口或控制台日志，不能只依赖重启后仍存在的本地日志。
记录第三方安全、监控 Agent 的版本和变更时间。
将上游 commit、CVE 和内部内核版本建立对应关系。

Debugging notes

这次问题留下几个比较有用的排查原则。

Error reporter is not always the root cause

文件系统、allocator 和 verifier 经常是最先发现内存损坏的组件。看到错误后，需要回答“谁发现了损坏”和“谁制造了损坏”两个不同的问题。

Read before the fatal line

fatal error 往往出现在证据链的末端。向前读取 50 到 200 行，重点寻找 warning、异常进程名、寄存器中的长度值以及第三方模块或 tracing 事件。

Read corrupted bytes as data

不要只把损坏区域当作十六进制。尝试转成 ASCII、路径、JSON 或已知结构体。用户态字符串出现在内核元数据中，本身就是很强的方向性证据。

A bounds check after memcpy is not a bounds check

检查存在不等于检查有效。代码审计时必须确认长度验证是在写入之前还是之后，以及动态字符串、数组的累计长度是否可能超过固定 buffer。

Keep the last unverified link visible

当日志、数据签名、源码和上游补丁已经相互吻合时，也不要把暂未直接验证的物理页相邻关系写成既成事实。把缺口和验证方法留下来，事故结论才方便复查。

2024-02-04

virtualization

Linux guest could not access tsc clocksource

Phenomenon

check TSC disabled from guest

1 2	[ 0.000004] tsc: Detected 1999.999 MHz processor [ 0.251920] tsc: Marking TSC unstable due to TSCs unsynchronized

Steps

lscpu | grep tsc in guest, confirm cpu support tsc
check vm not runs over IBM’s Summit2 by dmidecode
lscpu | grep constant_tsc in both guest and host to confirm this is not in guest
change kernel cmdline, add tsc=reliable if need ( based on cat /proc/cmdline)
check cpu Vendor by lscpu if using Intel cpu
remove acpi from libvirt. xml，but this will let all hot-plug operation fails

change to using other amd cpu，tsc clocksource still could not be found

Cpu that do not has vendor intel runs linux kernel will met this issue.

Resolution

method 1:

change guest os linux cmdline add tsc=reliable

method 2:

remove acpi from libvirt. xml by virsh edit

More information

x86 guest disable constant_tsc in code，see the disscusion: https://lore.kernel.org/qemu-devel/20210426185155.G[A32439@fuller.cnet](mailto:A32439@fuller.cnet)/t/

but linux guest kernel let intel cpu use tsc and as the result intel vs amd cpu has below differences.

go through kernel code

Code snippet 0

/*
 * TSC can be unstable due to cpufreq or due to unsynced TSCs
 */

static int __read_mostly tsc_unstable;`

The code defines a static integer variable tsc_unstable. The static keyword means that this variable is only visible within the file it’s defined in, limiting its scope to tsc.c. This is a common practice when you want to keep the variable private to a particular file and prevent it from being accessed directly from other files.

The __read_mostly attribute is a compiler directive specific to the Linux kernel, which suggests that this variable is expected to be read from often, but seldom written to. This can help the compiler optimize the generated code for this access pattern.

The variable tsc_unstable is used to indicate whether the TSC is unstable. The comment above the variable declaration explains that the TSC can be unstable due to CPU frequency scaling (cpufreq) or due to unsynchronized TSCs across different cores or processors.

In terms of improvements, the code is quite clear and concise. However, it might be helpful to expand the comment to explain what the implications of an unstable TSC are, and how the tsc_unstable variable is used in the system’s logic.

Code snippet 1

static int __init tsc_setup(char *str)
{
    if (!strcmp(str, "reliable"))
        tsc_clocksource_reliable = 1;
    if (!strncmp(str, "noirqtime", 9))
        no_sched_irq_time = 1;
    if (!strcmp(str, "unstable"))
        mark_tsc_unstable("boot parameter");
    if (!strcmp(str, "nowatchdog"))
        no_tsc_watchdog = 1;
    return 1;
}

The __init keyword before the function name is a macro used in the Linux kernel to indicate that this function is used only at initialization time. The memory for this function is freed after the initialization is complete to conserve the kernel’s memory footprint.

Inside the function, there are several if statements that compare the input string str to different string literals using the strcmp and strncmp functions. strcmp compares two strings and returns 0 if they are identical, while strncmp compares up to a specified number of characters from two strings.

If str is “reliable”, it sets the tsc_clocksource_reliable variable to 1. If str starts with “noirqtime”, it sets the no_sched_irq_time variable to 1. If str is “unstable”, it calls the mark_tsc_unstable function with “boot parameter” as an argument. If str is “nowatchdog”, it sets the no_tsc_watchdog variable to 1.

Each of these variables or functions presumably controls a different aspect of the TSC’s behavior. For example, tsc_clocksource_reliable might indicate whether the TSC is a reliable source of time, and no_sched_irq_time might control whether the scheduler uses the TSC for interrupt timing.

In terms of improvements, the function is quite clear and concise. However, adding comments to explain the purpose of each variable and what each string argument represents would improve readability. It would also be beneficial to add error handling to account for the case where str does not match any of the expected values.

And we could use tsc=reliable to kernel cmdline to changes to tsc clock source.

Code snippet 2

static void tsc_cs_mark_unstable(struct clocksource *cs)
{
    if (tsc_unstable)
        return;
 
    tsc_unstable = 1;
    if (using_native_sched_clock())
        clear_sched_clock_stable();
    disable_sched_clock_irqtime();
    pr_info("Marking TSC unstable due to clocksource watchdog\n");
}

The function first checks if the tsc_unstable variable is already set to 1. If it is, the function immediately returns, as the TSC has already been marked as unstable. This is a common pattern in C programming known as a “guard clause”, which is used to exit a function early when certain conditions are met.

If tsc_unstable is not set to 1, the function proceeds to mark the TSC as unstable. It does this by setting tsc_unstable to 1, and then calling two functions: clear_sched_clock_stable and disable_sched_clock_irqtime. These functions presumably perform some cleanup or configuration changes related to the TSC becoming unstable.

Finally, the function logs a message using the pr_info macro, which is a kernel print function that outputs a message to the system log. The message indicates that the TSC has been marked as unstable due to the clocksource watchdog.

In terms of improvements, the function is quite clear and concise. However, adding comments to explain the purpose of the clear_sched_clock_stable and disable_sched_clock_irqtime functions would improve readability. It would also be beneficial to add error handling to account for any potential issues that could occur when these functions are called.

Code snippet 3

/*
 * Make an educated guess if the TSC is trustworthy and synchronized
 * over all CPUs.
 */
int unsynchronized_tsc(void)
{
    if (!boot_cpu_has(X86_FEATURE_TSC) || tsc_unstable)
        return 1;
 
#ifdef CONFIG_SMP
    if (apic_is_clustered_box())
        return 1;
#endif
 
    if (boot_cpu_has(X86_FEATURE_CONSTANT_TSC))
        return 0;
 
    if (tsc_clocksource_reliable)
        return 0;
    /*
     * Intel systems are normally all synchronized.
     * Exceptions must mark TSC as unstable:
     */
    if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL) {
        /* assume multi socket systems are not synchronized: */
        if (num_possible_cpus() > 1)
            return 1;
    }
 
    return 0;
}

The function begins by checking if the boot CPU has the TSC feature and if the TSC is unstable. If either of these conditions is true, the function immediately returns 1, indicating that the TSC is unsynchronized.

Next, if the system is configured for symmetric multiprocessing (SMP), the function checks if the Advanced Programmable Interrupt Controller (APIC) is clustered. If it is, the function returns 1, again indicating that the TSC is unsynchronized.

The function then checks if the boot CPU has the constant TSC feature or if the TSC clocksource is reliable. If either of these conditions is true, the function returns 0, indicating that the TSC is synchronized.

Finally, the function checks if the CPU vendor is not Intel. If it is not, and the system has more than one possible CPU, the function returns 1, indicating that the TSC is unsynchronized. If none of the previous conditions are met, the function returns 0, indicating that the TSC is synchronized.

More practice

SystemTap

Because of above issue, I just spent more time to check the tsc value used by guest and from host cpu do have any different. With systemtap.

observe rdtsc

result

value of tsc clock，average value and stantard deviation has different

and the value from guest os is not stable when compared with host

during live migration, tsc value will be smaller than usual (I think its because live migration has down time, so we need to change tsc to tolerant it)

so just from the small test, its not a good idea to relay on tsc which is not as specific as it on the host

data from my test

The first version, use the script test average value and stantard deviation

in guest:

TSC mean: 2000170717.800000, TSC std dev: 255861.233545
Time mean: 1000101683.030000, Time std dev: 162898.956256
TSC mean: 2000158595.200000, TSC std dev: 340159.343486
Time mean: 1000092746.020000, Time std dev: 170311.019513
TSC mean: 2000116749.600000, TSC std dev: 96417.905701
Time mean: 1000076448.860000, Time std dev: 102460.953979

in guest during live migration:

TSC mean: 1990107194.600000, TSC std dev: 71113321.983521
Time mean: 1000129868.770000, Time std dev: 340417.298586
TSC mean: 1993829457.200000, TSC std dev: 47439246.502752
Time mean: 1001882162.230000, Time std dev: 16929541.16989

Samples from host:

TSC mean: 2000087563.600000, TSC std dev: 16626.290598
Time mean: 1000065114.610000, Time std dev: 8341.142215
TSC mean: 2000084499.400000, TSC std dev: 4760.334824
Time mean: 1000063541.340000, Time std dev: 2447.439965
TSC mean: 2000083391.800000, TSC std dev: 11786.744451
Time mean: 1000062911.800000, Time std dev: 5922.538748

TSC average value will be less that normal during migration.

change the script to check abnormal samples

Sample 54, TSC diff: 1998893560, Time diff: 1000069363 ns
Sample 55, TSC diff: 1998887140, Time diff: 1000065570 ns
Sample 56, TSC diff: 1999007020, Time diff: 1000130480 ns
Sample 57, TSC diff: 1535293100, Time diff: 1000090774 ns
Sample 58, TSC diff: 1998899520, Time diff: 1000073836 ns
Sample 59, TSC diff: 2000588260, Time diff: 1001300447 ns
Sample 60, TSC diff: 1998899540, Time diff: 1000072444 ns

just paste my test code:


#include <stdio.h>
#include <stdint.h>
#include <unistd.h>
#include <time.h>
#include <math.h>
 
#define SAMPLES 100
 
uint64_t rdtsc(){
    unsigned int lo,hi;
    __asm__ __volatile__ ("rdtsc" : "=a" (lo), "=d" (hi));
    return ((uint64_t)hi << 32) | lo;
}
 
double calc_std_dev(uint64_t *data, double mean){
    double sum = 0.0;
    for(int i = 0; i < SAMPLES; i++){
        sum += pow(data[i] - mean, 2);
    }
    return sqrt(sum / SAMPLES);
}
 
int main(){
    struct timespec start, end;
    uint64_t tsc_start, tsc_end;
    uint64_t tsc_diffs[SAMPLES], time_diffs[SAMPLES];
    double tsc_sum = 0.0, time_sum = 0.0;
 
    for(int i = 0; i < SAMPLES; i++){
        clock_gettime(CLOCK_MONOTONIC, &start);
        tsc_start = rdtsc();
 
        sleep(1);
 
        tsc_end = rdtsc();
        clock_gettime(CLOCK_MONOTONIC, &end);
 
        tsc_diffs[i] = tsc_end - tsc_start;
        time_diffs[i] = (end.tv_sec - start.tv_sec) * 1e9 + (end.tv_nsec - start.tv_nsec);
 
        printf("Sample %d, TSC diff: %llu, Time diff: %llu ns\n", i, tsc_diffs[i], time_diffs[i]);
 
        tsc_sum += tsc_diffs[i];
        time_sum += time_diffs[i];
    }
 
    double tsc_mean = tsc_sum / SAMPLES;
    double time_mean = time_sum / SAMPLES;
 
    double tsc_std_dev = calc_std_dev(tsc_diffs, tsc_mean);
    double time_std_dev = calc_std_dev(time_diffs, time_mean);
 
    printf("TSC mean: %f, TSC std dev: %f\n", tsc_mean, tsc_std_dev);
    printf("Time mean: %f, Time std dev: %f\n", time_mean, time_std_dev);
 
    return 0;
}

The disk in the guest OS is unmounted during the kernel startup process.

Phenomenon: When creating a new virtual machine, after the virtual machine enters the “running” state (libvirt reports running, and the qemu process starts), a disk is loaded. During the kernel startup process, the disk (vdb) is recognized, and then qemu receives a device removal event, which is fed back to libvirt. Libvirt updates the XML, causing inconsistency between the disk state recorded in the zstack database and the XML on the host.

The main issue here is that the libvirt loading device interface returns success, and the XML corresponding to the device is also added. However, this device is deleted according to the event feedback from qemu.

Important log information: Here, let’s first analyze the system logs in the guest OS:

Here, we notice the logs related to pciehp because this virtual machine is UEFI-booted, leading to numerous pcie-related logs (due to UEFI boot requiring the q35 machine type, which defaults to pcie devices).

The initially observed logs include an error log from pcieport:

1	pci 0000:00:02.7: BAR 13: failed to assign [io size 0x1000]

Followed by the recognition of the vdb device:

1	_virtio_blk virtio6: [vdb] 104857600 512-byte logical blocks (537 GB/500 GiB)

An external interrupt is sent to the virtual machine:

1	pciehp 0000:00:02.7:pcie004: Slot(0-7): Attention button pressed

Subsequently, through ausearch, it is identified that libvirt received a device deletion event, leading to the removal of the mentioned device:

libvirt received device deleted event, removing the device

Based on these scenarios, we have summarized the steps to reproduce the issue:

During the kernel startup process
Load the data disk.
Check for inconsistencies between the XML and the database.
Through repeated testing of VM boot and data disk loading, the issue can be reproduced.

Regarding the error logs mentioned above, the explanation is as follows:

pci 0000:00:02.7: BAR 13: failed to assign [io size 0x1000]:
- According to https://access.redhat.com/solutions/3144711, this error may occur because in virtualized environments, there might be more PCIe ports than in a real physical environment, leading to this error. However, it does not have any actual impact.
_virtio_blk virtio6: [vdb] 104857600 512-byte logical blocks (537 GB/500 GiB):
- This indicates that the virtio-blk disk has indeed been successfully loaded, as it has been recognized within the virtual machine.
pciehp 0000:00:02.7:pcie004: Slot(0-7): Attention button pressed:
- “Attention button pressed” indicates that when resetting the PCIe slot, QEMU sends the corresponding interrupt. When the host receives this interrupt, the corresponding processing logic prints this log.

As for the key QEMU code, by searching the codebase, it has been confirmed that QEMU sends the corresponding interrupt when resetting the PCIe slot, and the host prints the log as part of the corresponding processing logic.

code from pcie.c

pci_word_test_and_clear_mask(exp_cap + PCI_EXP_SLTSTA,
                             PCI_EXP_SLTSTA_EIS |/* on reset,
                                                    the lock is released */
                             PCI_EXP_SLTSTA_CC |
                             PCI_EXP_SLTSTA_PDC |
                             PCI_EXP_SLTSTA_ABP);

which is used in qdev.c

QLIST_FOREACH(bus, &dev->child_bus, sibling) {
     object_property_set_bool(OBJECT(bus), true, "realized",
                                  &local_err);
     if (local_err != NULL) {
         goto child_realize_fail;
     }
 }
 if (dev->hotplugged) {
     device_reset(dev);
 }
 dev->pending_deleted_event = false;
 
 if (hotplug_ctrl) {
     hotplug_handler_plug(hotplug_ctrl, dev, &local_err);
     if (local_err != NULL) {
         goto child_realize_fail;
     }
}

During the device hotplug process, there will be a reset action.

Kernel-related code:

from pciehp_hpc.c

static int pciehp_poll(void *data)
{
    struct controller *ctrl = data;
 
    schedule_timeout_idle(10 * HZ); /* start with 10 sec delay */
 
    while (!kthread_should_stop()) {
        /* poll for interrupt events or user requests */
        while (pciehp_isr(IRQ_NOTCONNECTED, ctrl) == IRQ_WAKE_THREAD ||
               atomic_read(&ctrl->pending_events))
            pciehp_ist(IRQ_NOTCONNECTED, ctrl);
 
        if (pciehp_poll_time <= 0 || pciehp_poll_time > 60)
            pciehp_poll_time = 2; /* clamp to sane value */
 
        schedule_timeout_idle(pciehp_poll_time * HZ);
    }
 
    return 0;
}
 
    /* Check Attention Button Pressed */
    if (events & PCI_EXP_SLTSTA_ABP) {
        ctrl_info(ctrl, "Slot(%s): Attention button pressed\n",
              slot_name(ctrl));
        pciehp_handle_button_press(ctrl);
    }

This code represents a kernel function for polling PCIe Hot Plug events. It uses a kernel thread (kthread) to continuously poll for interrupt events or user requests related to PCIe Hot Plug. The function includes a timeout mechanism with an initial delay of 10 seconds and then repeats the polling process based on the specified polling time. The function stops when the kernel thread should stop (kthread_should_stop() returns true).

And pciehp_handle_button_press is implemented as following:

void pciehp_handle_button_press(struct controller *ctrl)
{
	mutex_lock(&ctrl->state_lock);
	switch (ctrl->state) {
	case OFF_STATE:
	case ON_STATE:
		if (ctrl->state == ON_STATE) {
			ctrl->state = BLINKINGOFF_STATE;
			ctrl_info(ctrl, "Slot(%s): Powering off due to button press\n",
				  slot_name(ctrl));
		} else {
			ctrl->state = BLINKINGON_STATE;
			ctrl_info(ctrl, "Slot(%s) Powering on due to button press\n",
				  slot_name(ctrl));
		}
		/* blink power indicator and turn off attention */
		pciehp_set_indicators(ctrl, PCI_EXP_SLTCTL_PWR_IND_BLINK,
				      PCI_EXP_SLTCTL_ATTN_IND_OFF);
		schedule_delayed_work(&ctrl->button_work, 5 * HZ);
		break;
	case BLINKINGOFF_STATE:
	case BLINKINGON_STATE:
		/*
		 * Cancel if we are still blinking; this means that we
		 * press the attention again before the 5 sec. limit
		 * expires to cancel hot-add or hot-remove
		 */
		ctrl_info(ctrl, "Slot(%s): Button cancel\n", slot_name(ctrl));
		cancel_delayed_work(&ctrl->button_work);
		if (ctrl->state == BLINKINGOFF_STATE) {
			ctrl->state = ON_STATE;
			pciehp_set_indicators(ctrl, PCI_EXP_SLTCTL_PWR_IND_ON,
					      PCI_EXP_SLTCTL_ATTN_IND_OFF);
		} else {
			ctrl->state = OFF_STATE;
			pciehp_set_indicators(ctrl, PCI_EXP_SLTCTL_PWR_IND_OFF,
					      PCI_EXP_SLTCTL_ATTN_IND_OFF);
		}
		ctrl_info(ctrl, "Slot(%s): Action canceled due to button press\n",
			  slot_name(ctrl));
		break;
	default:
		ctrl_err(ctrl, "Slot(%s): Ignoring invalid state %#x\n",
			 slot_name(ctrl), ctrl->state);
		break;
	}
	mutex_unlock(&ctrl->state_lock);
}

Both OFF_STATE and ON_STATE could changed to each other by same press request.

Based on the test results, we preliminarily conclude that there is a race condition between hot-plug operations and kernel boot, leading to unexpected changes in the PCIe slot’s state from off → on → off. (Note: The crucial point here is that pciehp_handle_button_press(ctrl); simultaneously handles both on and off scenarios.)

With reference to the above keywords, we identified a related Bugzilla entry for QEMU version 4.2 by searching for ‘qemu pci device kernel boot race condition’:

https://bugzilla.kernel.org/show_bug.cgi?id=211691

“The document mentions a virtio-net failover mechanism introduced by QEMU 4.2, addressing the issue of hot-plugging network cards failing during the VM startup phase. This problem arises from a race condition in the QEMU code that sets the PCIe slot’s state. The provided QEMU patch resolves the issue:

QEMU Patch Link

The title of this patch is: ‘pcie: don’t set link state active if the slot is empty.’

Upon reviewing its content, it appears that during PCIe initialization and the hot-plug phase, the ‘reset’ is called, potentially causing inconsistencies in the slot’s state. This patch addresses the problem by preventing the setting of the link state to active if the slot is empty, eliminating the observed issue.”

TIPs:

Search for changes related to the virtual machine process using ausearch:

1	ausearch -m "VIRT_RESOURCE" -p 63259

Libvirt’s XML and QEMU event update mechanism: Details can be found in TIC-1360 - Cloud VM disk does not exist, capacity inconsistency between UI interface and underlying view (Closed).

Quick reference for PCIe events:

From linux pci_regs.h:

#define PCI_EXP_SLTCTL      24  /* Slot Control */
#define  PCI_EXP_SLTCTL_ABPE    0x0001  /* Attention Button Pressed Enable */
#define  PCI_EXP_SLTCTL_PFDE    0x0002  /* Power Fault Detected Enable */
#define  PCI_EXP_SLTCTL_MRLSCE  0x0004  /* MRL Sensor Changed Enable */
#define  PCI_EXP_SLTCTL_PDCE    0x0008  /* Presence Detect Changed Enable */
#define  PCI_EXP_SLTCTL_CCIE    0x0010  /* Command Completed Interrupt Enable */
#define  PCI_EXP_SLTCTL_HPIE    0x0020  /* Hot-Plug Interrupt Enable */
#define  PCI_EXP_SLTCTL_AIC 0x00c0  /* Attention Indicator Control */
#define  PCI_EXP_SLTCTL_ATTN_IND_SHIFT 6      /* Attention Indicator shift */
#define  PCI_EXP_SLTCTL_ATTN_IND_ON    0x0040 /* Attention Indicator on */
#define  PCI_EXP_SLTCTL_ATTN_IND_BLINK 0x0080 /* Attention Indicator blinking */
#define  PCI_EXP_SLTCTL_ATTN_IND_OFF   0x00c0 /* Attention Indicator off */
#define  PCI_EXP_SLTCTL_PIC 0x0300  /* Power Indicator Control */
#define  PCI_EXP_SLTCTL_PWR_IND_ON     0x0100 /* Power Indicator on */
#define  PCI_EXP_SLTCTL_PWR_IND_BLINK  0x0200 /* Power Indicator blinking */
#define  PCI_EXP_SLTCTL_PWR_IND_OFF    0x0300 /* Power Indicator off */
#define  PCI_EXP_SLTCTL_PCC 0x0400  /* Power Controller Control */
#define  PCI_EXP_SLTCTL_PWR_ON         0x0000 /* Power On */
#define  PCI_EXP_SLTCTL_PWR_OFF        0x0400 /* Power Off */
#define  PCI_EXP_SLTCTL_EIC 0x0800  /* Electromechanical Interlock Control */
#define  PCI_EXP_SLTCTL_DLLSCE  0x1000  /* Data Link Layer State Changed Enable */

QEMU systemtap trace, refer to: QEMU Tracing Documentation

1	/usr/bin/qemu-trace-stap run /usr/libexec/qemu-kvm pci_cfg_write

Guest os pcie trace analysis:

[一 12月 4 15:06:48 2023] pciehp 0000:00:02.7:pcie004: pending interrupts 0x0010 from Slot Status
[一 12月 4 15:06:48 2023] pciehp 0000:00:02.7:pcie004: pciehp_green_led_on: SLOTCTRL 6c write cmd 100
[一 12月 4 15:06:48 2023] pciehp 0000:00:02.7:pcie004: pending interrupts 0x0010 from Slot Status
[一 12月 4 15:06:48 2023] pciehp 0000:00:02.7:pcie004: pciehp_set_attention_status: SLOTCTRL 6c write cmd c0
[一 12月 4 15:06:48 2023] pciehp 0000:00:02.7:pcie004: Slot(0-7): Attention button pressed
[一 12月 4 15:06:48 2023] pciehp 0000:00:02.7:pcie004: Slot(0-7): Powering off due to button press
[一 12月 4 15:06:48 2023] pciehp 0000:00:02.7:pcie004: pending interrupts 0x0010 from Slot Status
[一 12月 4 15:06:48 2023] pciehp 0000:00:02.7:pcie004: pciehp_green_led_blink: SLOTCTRL 6c write cmd 200
[一 12月 4 15:06:48 2023] pciehp 0000:00:02.7:pcie004: pending interrupts 0x0010 from Slot Status
[一 12月 4 15:06:48 2023] pciehp 0000:00:02.7:pcie004: pciehp_set_attention_status: SLOTCTRL 6c write cmd c0

[一 12月 4 15:06:53 2023] pciehp 0000:00:02.7:pcie004: pciehp_get_power_status: SLOTCTRL 6c value read 2f1
[一 12月 4 15:06:53 2023] pciehp 0000:00:02.7:pcie004: pciehp_unconfigure_device: domain:bus:dev = 0000:08:00
[一 12月 4 15:06:53 2023] pciehp 0000:00:02.7:pcie004: pending interrupts 0x0010 from Slot Status
[一 12月 4 15:06:53 2023] pciehp 0000:00:02.7:pcie004: pciehp_power_off_slot: SLOTCTRL 6c write cmd 40

[一 12月 4 15:06:54 2023] pciehp 0000:00:02.7:pcie004: pending interrupts 0x0018 from Slot Status
[一 12月 4 15:06:54 2023] pciehp 0000:00:02.7:pcie004: pciehp_green_led_off: SLOTCTRL 6c write cmd 300

[一 12月 4 15:06:48 2023] pciehp 0000:00:02.7:pcie004: pciehp_green_led_on: SLOTCTRL 6c write cmd 100

cmd 0x0100

1	#define PCI_EXP_SLTCTL_PWR_IND_ON 0x0100 /* Power Indicator on */

[一 12月 4 15:06:48 2023] pciehp 0000:00:02.7:pcie004: pciehp_set_attention_status: SLOTCTRL 6c write cmd c0

cmd 0x00c0
0000 0000 1100 0000

1 2	#define PCI_EXP_SLTCTL_ATTN_IND_BLINK 0x0080 /* Attention Indicator blinking / #define PCI_EXP_SLTCTL_ATTN_IND_ON 0x0040 / Attention Indicator on */

[一 12月 4 15:06:48 2023] pciehp 0000:00:02.7:pcie004: pciehp_green_led_blink: SLOTCTRL 6c write cmd 200

cmd 0x0200

1	#define PCI_EXP_SLTCTL_PWR_IND_BLINK 0x0200 /* Power Indicator blinking */

[一 12月 4 15:06:48 2023] pciehp 0000:00:02.7:pcie004: pciehp_set_attention_status: SLOTCTRL 6c write cmd c0

cmd 0x00c0
0000 0000 1100 0000

1 2	#define PCI_EXP_SLTCTL_ATTN_IND_BLINK 0x0080 /* Attention Indicator blinking / #define PCI_EXP_SLTCTL_ATTN_IND_ON 0x0040 / Attention Indicator on */

[一 12月 4 15:06:53 2023] pciehp 0000:00:02.7:pcie004: pciehp_power_off_slot: SLOTCTRL 6c write cmd 400

cmd 0x0400

1	#define PCI_EXP_SLTCTL_PWR_OFF 0x0400 /* Power Off */

[一 12月 4 15:06:54 2023] pciehp 0000:00:02.7:pcie004: pciehp_green_led_off: SLOTCTRL 6c write cmd 300
cmd 0x0300

1	#define PCI_EXP_SLTCTL_PWR_IND_OFF 0x0300 /* Power Indicator off */

2023-05-26

virtualization►cpu

Understanding CPU Topology for Improved Performance

The physical layout of CPU cores in a system is known as CPU topology. Understanding CPU topology can significantly impact the performance of a system, as it determines the effectiveness and efficiency of the cores.

What is CPU Topology?

CPU topology comprises three primary levels:

Socket: A physical connector that holds a CPU. A system can have multiple sockets, each of which can hold multiple cores.
Core: A single processing unit within a CPU that can run multiple threads simultaneously.
Thread: A single flow of execution within a core.

The CPU topology can be described using a tree-like structure, with the socket level at the top and the thread level at the bottom. The cores in a socket are connected to each other via a bus, and the threads in a core are connected to each other by a shared cache.

Importance of CPU Topology

Understanding CPU topology is crucial for improving system performance. The topology can be used to optimize the performance of a system by assigning threads to cores in a way that minimizes the amount of communication between cores. This can enhance the performance of applications that are heavily multithreaded.

Additionally, the CPU topology can be used to troubleshoot performance issues. For example, if an application is running slowly, the CPU topology can be used to identify which cores are being used the most. This information can help identify the source of the performance problem and take appropriate steps to improve it.

Here are some benefits of understanding CPU topology:

It helps to optimize system performance by assigning tasks to the most suitable cores.
It helps to troubleshoot performance issues by identifying heavily used cores.
It helps to understand how the system will scale as more cores are added.

Tools to Display CPU Topology

There are several tools available to display CPU topology, and one of the most commonly used tools is lscpu. Here is an example of using lscpu to display CPU topology:

[root@172-20-1-220 ~]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                80
On-line CPU(s) list:   0-79
Thread(s) per core:    2
Core(s) per socket:    20
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz
Stepping:              7
CPU MHz:               2100.000
BogoMIPS:              4200.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              28160K
NUMA node0 CPU(s):     0-19,40-59
NUMA node1 CPU(s):     20-39,60-79
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp_epp pku ospke avx512_vnni md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities

hwloc-ls

[root@172-20-1-220 ~]# hwloc-ls
Machine (767GB total)
  NUMANode L#0 (P#0 383GB)
    Package L#0 + L3 L#0 (28MB)
      L2 L#0 (1024KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
        PU L#0 (P#0)
        PU L#1 (P#40)
      L2 L#1 (1024KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1
        PU L#2 (P#1)
        PU L#3 (P#41)
      L2 L#2 (1024KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2
        PU L#4 (P#2)
        PU L#5 (P#42)
      L2 L#3 (1024KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3
        PU L#6 (P#3)
        PU L#7 (P#43)
      L2 L#4 (1024KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4
        PU L#8 (P#4)
        PU L#9 (P#44)
      L2 L#5 (1024KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5
        PU L#10 (P#5)
        PU L#11 (P#45)
      L2 L#6 (1024KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6
        PU L#12 (P#6)
        PU L#13 (P#46)
      L2 L#7 (1024KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7
        PU L#14 (P#7)
        PU L#15 (P#47)
      L2 L#8 (1024KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8
        PU L#16 (P#8)
        PU L#17 (P#48)
      L2 L#9 (1024KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9
        PU L#18 (P#9)
        PU L#19 (P#49)
      L2 L#10 (1024KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10
        PU L#20 (P#10)
        PU L#21 (P#50)
      L2 L#11 (1024KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11
        PU L#22 (P#11)
        PU L#23 (P#51)
      L2 L#12 (1024KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12
        PU L#24 (P#12)
        PU L#25 (P#52)
      L2 L#13 (1024KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13
        PU L#26 (P#13)
        PU L#27 (P#53)
      L2 L#14 (1024KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14
        PU L#28 (P#14)
        PU L#29 (P#54)
      L2 L#15 (1024KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15
        PU L#30 (P#15)
        PU L#31 (P#55)
      L2 L#16 (1024KB) + L1d L#16 (32KB) + L1i L#16 (32KB) + Core L#16
        PU L#32 (P#16)
        PU L#33 (P#56)
      L2 L#17 (1024KB) + L1d L#17 (32KB) + L1i L#17 (32KB) + Core L#17
        PU L#34 (P#17)
        PU L#35 (P#57)
      L2 L#18 (1024KB) + L1d L#18 (32KB) + L1i L#18 (32KB) + Core L#18
        PU L#36 (P#18)
        PU L#37 (P#58)
      L2 L#19 (1024KB) + L1d L#19 (32KB) + L1i L#19 (32KB) + Core L#19
        PU L#38 (P#19)
        PU L#39 (P#59)
    HostBridge L#0
      PCI 8086:a1d2
      PCI 8086:a182
      PCIBridge
        PCIBridge
          PCI 1a03:2000
            GPU L#0 "card0"
            GPU L#1 "controlD64"
    HostBridge L#3
      PCIBridge
        PCI 1000:0097
          Block(Disk) L#2 "sda"
  NUMANode L#1 (P#1 384GB)
    Package L#1 + L3 L#1 (28MB)
      L2 L#20 (1024KB) + L1d L#20 (32KB) + L1i L#20 (32KB) + Core L#20
        PU L#40 (P#20)
        PU L#41 (P#60)
      L2 L#21 (1024KB) + L1d L#21 (32KB) + L1i L#21 (32KB) + Core L#21
        PU L#42 (P#21)
        PU L#43 (P#61)
      L2 L#22 (1024KB) + L1d L#22 (32KB) + L1i L#22 (32KB) + Core L#22
        PU L#44 (P#22)
        PU L#45 (P#62)
      L2 L#23 (1024KB) + L1d L#23 (32KB) + L1i L#23 (32KB) + Core L#23
        PU L#46 (P#23)
        PU L#47 (P#63)
      L2 L#24 (1024KB) + L1d L#24 (32KB) + L1i L#24 (32KB) + Core L#24
        PU L#48 (P#24)
        PU L#49 (P#64)
      L2 L#25 (1024KB) + L1d L#25 (32KB) + L1i L#25 (32KB) + Core L#25
        PU L#50 (P#25)
        PU L#51 (P#65)
      L2 L#26 (1024KB) + L1d L#26 (32KB) + L1i L#26 (32KB) + Core L#26
        PU L#52 (P#26)
        PU L#53 (P#66)
      L2 L#27 (1024KB) + L1d L#27 (32KB) + L1i L#27 (32KB) + Core L#27
        PU L#54 (P#27)
        PU L#55 (P#67)
      L2 L#28 (1024KB) + L1d L#28 (32KB) + L1i L#28 (32KB) + Core L#28
        PU L#56 (P#28)
        PU L#57 (P#68)
      L2 L#29 (1024KB) + L1d L#29 (32KB) + L1i L#29 (32KB) + Core L#29
        PU L#58 (P#29)
        PU L#59 (P#69)
      L2 L#30 (1024KB) + L1d L#30 (32KB) + L1i L#30 (32KB) + Core L#30
        PU L#60 (P#30)
        PU L#61 (P#70)
      L2 L#31 (1024KB) + L1d L#31 (32KB) + L1i L#31 (32KB) + Core L#31
        PU L#62 (P#31)
        PU L#63 (P#71)
      L2 L#32 (1024KB) + L1d L#32 (32KB) + L1i L#32 (32KB) + Core L#32
        PU L#64 (P#32)
        PU L#65 (P#72)
      L2 L#33 (1024KB) + L1d L#33 (32KB) + L1i L#33 (32KB) + Core L#33
        PU L#66 (P#33)
        PU L#67 (P#73)
      L2 L#34 (1024KB) + L1d L#34 (32KB) + L1i L#34 (32KB) + Core L#34
        PU L#68 (P#34)
        PU L#69 (P#74)
      L2 L#35 (1024KB) + L1d L#35 (32KB) + L1i L#35 (32KB) + Core L#35
        PU L#70 (P#35)
        PU L#71 (P#75)
      L2 L#36 (1024KB) + L1d L#36 (32KB) + L1i L#36 (32KB) + Core L#36
        PU L#72 (P#36)
        PU L#73 (P#76)
      L2 L#37 (1024KB) + L1d L#37 (32KB) + L1i L#37 (32KB) + Core L#37
        PU L#74 (P#37)
        PU L#75 (P#77)
      L2 L#38 (1024KB) + L1d L#38 (32KB) + L1i L#38 (32KB) + Core L#38
        PU L#76 (P#38)
        PU L#77 (P#78)
      L2 L#39 (1024KB) + L1d L#39 (32KB) + L1i L#39 (32KB) + Core L#39
        PU L#78 (P#39)
        PU L#79 (P#79)
    HostBridge L#5
      PCIBridge
        PCI 8086:1521
          Net L#3 "enp175s0f0"
        PCI 8086:1521
          Net L#4 "enp175s0f1"
        PCI 8086:1521
          Net L#5 "enp175s0f2"
        PCI 8086:1521
          Net L#6 "enp175s0f3"
      PCIBridge
        PCI 8086:10fb
          Net L#7 "enp176s0f0"
        PCI 8086:10fb
          Net L#8 "enp176s0f1"
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)
  Misc(MemoryModule)

Virtual Machines and CPU Topology

Virtual machines (VMs) are software programs that create an isolated environment for running operating systems and applications. VMs are often used to run various operating systems on the same physical machine or to run applications that require more resources than are available on the host machine.

When a VM is created, the hypervisor, which manages the VMs, assigns a single thread to the VM. This is because assigning multiple threads to a VM can lead to performance issues. Threads share the same resources on a core, and multiple threads can compete for resources, leading to contention and slowdowns. Furthermore, threads may interfere with each other, causing further slowdowns.

To optimize VM performance, it’s generally best to assign a single thread to a VM. However, there are exceptions to this rule. For example, if a VM is running an application that is specifically designed to take advantage of multiple threads, it may be beneficial to assign multiple threads to the VM.

To take advantage of multiple threads in a virtual machine, it’s essential to use a hypervisor that supports thread pinning, an operating system that supports thread scheduling, and an application that is designed to take advantage of multiple threads. Multithreaded applications such as web servers, database servers, and media transcoders are good examples of applications that can take advantage of multiple threads.

Why thread of cpu toplogy always 1 or 2

There are two main reasons why the number of threads in a CPU topology is usually limited to 1 or 2:

Physical constraints: A CPU core can only run a single thread at a time due to having a single instruction pointer (IP) and a single set of registers. When two threads run on the same core, they compete for the same resources, leading to performance degradation.
Scheduling overhead: Scheduling threads on different cores can be expensive, as the operating system has to switch between threads and this may cause context switches. Context switches are costly, as they require the operating system to save the state of the current thread and restore the state of the next.

In some cases, having more than two threads per core may be beneficial. For instance, heavily multithreaded applications may take advantage of the extra threads. However, in most cases, the costs of having more than two threads per core outweigh the benefits.

There are a few exceptions to the rule that the number of threads in a CPU topology is usually limited to 1 or 2. For example, some CPUs support hyper-threading, which allows a single core to run two threads simultaneously. However, hyper-threading is not always a good idea, as it can sometimes lead to performance degradation.

Overall, the number of threads in a CPU topology is usually limited to 1 or 2 due to physical constraints and scheduling overhead. While there are exceptions, in most cases, the costs of having more than two threads per core outweigh the benefits.

Sockets and cores with performance

Sockets and cores do have an impact on performance.

Sockets: A socket is a physical connector that holds a CPU. A system can have multiple sockets, each of which can hold multiple cores. The more sockets a system has, the more cores it can have, which can lead to better performance.
Cores: A core is a single processing unit within a CPU. A core can run multiple threads simultaneously. The more cores a system has, the more threads it can run, which can also lead to better performance.

However, it’s important to note that the number of sockets and cores is not the only factor that affects performance. Other factors, such as the clock speed of the CPU, the amount of cache memory, and the type of memory, can also have a significant impact.

In general, systems with more sockets and cores will have better performance than systems with fewer sockets and cores. However, it’s important to choose a system that has the right balance of sockets, cores, clock speed, cache memory, and memory type for your needs.

Here are some examples of how sockets and cores can impact performance:

A system with two sockets and four cores will have better performance than a system with one socket and two cores. This is because the system with two sockets can run more threads simultaneously.
A system with a higher clock speed will have better performance than a system with a lower clock speed. This is because the system with a higher clock speed can execute instructions faster.
A system with more cache memory will have better performance than a system with less cache memory. This is because the system with more cache memory can store more data in memory, which reduces the number of times the CPU has to access slower memory.
A system with faster memory will have better performance than a system with slower memory. This is because the system with faster memory can transfer data to the CPU faster, which reduces the amount of time the CPU has to wait for data.

Why aws only offer single sockets instance?

There are a few reasons why cloud providers like AWS do not offer multi-socket instances.

Cost: Multi-socket instances are more expensive than single-socket instances. This is because they require more hardware, such as more CPUs and more memory.
Complexity: Multi-socket instances are more complex to manage than single-socket instances. This is because they have more components, such as more CPUs, more memory, and more storage.
Performance: Multi-socket instances do not always offer better performance than single-socket instances. This is because the performance of a multi-socket instance can be limited by the speed of the interconnect between the sockets.

For these reasons, cloud providers like AWS choose to offer single-socket instances. Single-socket instances are less expensive, easier to manage, and offer the same or better performance than multi-socket instances.

However, there are some cases where multi-socket instances may be a good choice. For example, if you need a lot of CPU power, or if you need to run applications that are not well-optimized for multi-threading, then a multi-socket instance may be a good option.

If you are considering using a multi-socket instance, it is important to weigh the costs and benefits carefully. You should also make sure that your applications are well-optimized for multi-threading.

Phenomenon

Start from the XFS error

Find the data signature

The uprobe buffer

Why XFS was damaged

Resolution

Debugging notes

Error reporter is not always the root cause

Read before the fatal line

Read corrupted bytes as data

A bounds check after memcpy is not a bounds check

Keep the last unverified link visible

Phenomenon

Steps

Resolution

More information

go through kernel code

Code snippet 0

Code snippet 1

Code snippet 2

Code snippet 3

More practice

SystemTap

observe rdtsc

result

data from my test

more links for reading

TSC

What is CPU Topology?

Importance of CPU Topology

Tools to Display CPU Topology

Virtual Machines and CPU Topology

Why thread of cpu toplogy always 1 or 2

Sockets and cores with performance

Why aws only offer single sockets instance?