application vm are crashing in a cluster with mixed CPU model hardware

[gvaishya]

problem

Problem Description: We have a cluster with mixed hardware and application vm are running with max cpu cycle 2.4GHZ. (example 1 CPU x 2.4 GHz) but recently application has started facing issues that their virtual machines are crashing unexpectedly and they are not able to find any issue at application end.

Application type is VDI.

Is this issue can be related to mix CPU type in a cluster?

Below is the hardware , we are using in a cluster
2 processors, Intel(R) Xeon(R) Gold 6154 CPU (3.00 GHz / 18-core)
2 processors, Intel(R) Xeon(R) Gold 6254 CPU (3.10 GHz / 18-core)
2 processors, Intel(R) Xeon(R) Gold 6148 CPU (2.40 GHz / 20-core)

Hypervisor type: KVM

cat /etc/cloudstack/agent/agent.properties

guest.cpu.mode=host-passthrough
reboot.host.and.alert.management.on.heartbeat.timeout=false
hypervisor.type=kvm
iscsi.session.cleanup.enabled=false
guest.cpu.features=vmx

versions

ACS Version: 4.20.2.0
DB : Maria DB 10.3.39
ACS and KVM host : RHEL 8
Hypervisor type: KVM

The steps to reproduce the bug

...

What to do about it?

Problem Description: We have a cluster with mixed hardware and application vm are running with max cpu cycle 2.4GHZ. (example 1 CPU x 2.4 GHz) but recently application has started facing issues that their virtual machines are crashing unexpectedly and they are not able to find any issue at application end.

Application type is VDI.

Is this issue can be related to mix CPU type in a cluster?

Below is the hardware , we are using in a cluster
2 processors, Intel(R) Xeon(R) Gold 6154 CPU (3.00 GHz / 18-core)
2 processors, Intel(R) Xeon(R) Gold 6254 CPU (3.10 GHz / 18-core)
2 processors, Intel(R) Xeon(R) Gold 6148 CPU (2.40 GHz / 20-core)

Hypervisor type: KVM

cat /etc/cloudstack/agent/agent.properties

guest.cpu.mode=host-passthrough
reboot.host.and.alert.management.on.heartbeat.timeout=false
hypervisor.type=kvm
iscsi.session.cleanup.enabled=false
guest.cpu.features=vmx

[prashanthr2]

@gvaishya

Thanks for the report. This does not looks like a bug, ideally should be a discussions thread. A few things to separate out.

• host-passthrough is unsafe for this cluster. Your hosts mix Skylake-SP (6148, 6154) with Cascade Lake-SP (6254). host-passthrough passes the exact host CPU to the guest, so a VM's visible CPUID changes depending on which host it starts on. Pin to the lowest common denominator, identically on all hosts:

guest.cpu.mode=custom

guest.cpu.model=Skylake-Server

NOTE: Before changing, confirm the exact model string with virsh domcapabilities on a 6148 host (you may need -IBRS or -noTSX-IBRS). Needs an agent restart, and existing VMs need a stop/start (not a guest reboot).

• guest.cpu.features=vmx , is this intentional? This emits , exposing nested virtualisation to every guest on that host. AFAIK for VDI there's usually no reason for this. This is the first thing I'd remove and check

• Also, Check the 1 CPU x 2.4 GHz offering for CPU cap:
  SELECT id, name, cpu, speed, limit_cpu_use FROM cloud.service_offering WHERE uuid = <offering_id_from_UI>;
  If limit_cpu_use = 1, a CFS is applied (virsh dumpxml | grep -A4 cputune). 2.4 GHz is ~77% of a core on a 3.1 GHz host but ~100% on a 2.4 GHz host, so the same offering throttles differently by placement. For VDI that presents as hangs/timeouts users report as crashes.

To identify the actual cause, for one crash with a timestamp ( collect/check below if the above suggested doesn't resolve the issue)

1. Do crashes correlate with one host? If so this is a faulty host, not a mixed-CPU issue --> This is the first thing I will check to isolate the issue further
2. /var/log/libvirt/qemu/i---VM.log , a QEMU abort or reason=crashed points at guest/QEMU; nothing in the log means the host killed the process
3. Host dmesg -T / journalctl -k . OOM killer reaping qemu-kvm, or MCE entries
4. Guest-side: Windows bugcheck code, or Linux kernel panic

Could you paste grep -v '^#' /etc/cloudstack/agent/agent.properties | grep -v '^$' from two hosts (one 6254, one 6148)? That confirms the setting is actually applied and consistent.

Summary: fix the CPU mode regardless, but with no migration occurring I'd rank vmx and CPU-cap throttling as the likelier causes. The libvirt/QEMU log plus guest bugcheck will help isolate it further.