Skip to content

feat(itf): add dual-QEMU ivshmem integration-test environment - #617

Merged
castler merged 2 commits into
eclipse-score:mainfrom
Rahul-Sutariya:rasu_dual_qemu_intergation_test_plugin
Jul 24, 2026
Merged

feat(itf): add dual-QEMU ivshmem integration-test environment#617
castler merged 2 commits into
eclipse-score:mainfrom
Rahul-Sutariya:rasu_dual_qemu_intergation_test_plugin

Conversation

@Rahul-Sutariya

@Rahul-Sutariya Rahul-Sutariya commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Add a dual_qemu ITF/pytest plugin that boots two QNX VMs sharing one QEMU ivshmem-plain region, extending the upstream single-VM qemu plugin.

  • plugin: config.py, ivshmem_qemu.py, dual_qemu_process.py, init.py (target_a/target_b fixtures; sequential boot to avoid KVM race), BUILD, example config, README, example/run_example_manually.sh.
  • integration_testing.bzl: dual_qemu_integration_test macro (QNX-only, flaky).

@Rahul-Sutariya
Rahul-Sutariya force-pushed the rasu_dual_qemu_intergation_test_plugin branch from f56837b to eedfda4 Compare June 30, 2026 11:14
@Rahul-Sutariya
Rahul-Sutariya force-pushed the rasu_dual_qemu_intergation_test_plugin branch 4 times, most recently from 5bd6beb to 38c934b Compare July 13, 2026 07:14
@Rahul-Sutariya
Rahul-Sutariya marked this pull request as ready for review July 13, 2026 10:00

@castler castler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's first align on the test infrastructure - then we can check out the example :)

Comment thread quality/visibility_guard/public_targets.golden Outdated
Comment thread quality/integration_testing/integration_testing.bzl Outdated
Comment on lines +198 to +201
# Driving two real QNX guests under KVM has rare, environment-induced boot
# nondeterminism (e.g. a guest occasionally wedging during device bring-up).
# The fixtures already stagger boots and wait for stable SSH; mark the test
# flaky so bazel transparently retries such infrastructure hiccups.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ouch - is this possible to be fixed? Should we add a ticket with a todo?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a KVM + QNX -kernel boot path timing issue — the second guest occasionally wedges during SMP AP startup when both initialize under the same KVM instance. The plugin already mitigates by booting VMs sequentially and using single-core guests, which makes it very rare but not fully eliminable from our side.

A proper fix would require either:

  1. QNX fixing the boot-path race in their -kernel loader (upstream QNX issue), or
  2. Using the -bios/UEFI boot path instead of -kernel (different QNX IFS packaging, not trivial).

Comment thread quality/integration_testing/environments/dual_qemu/README.md Outdated
Comment thread quality/integration_testing/environments/dual_qemu/README.md Outdated
Comment thread quality/integration_testing/environments/dual_qemu/ivshmem_qemu.py Outdated
Comment thread quality/integration_testing/environments/dual_qemu/ivshmem_qemu.py Outdated
Comment thread quality/integration_testing/environments/dual_qemu/dual_qemu_process.py Outdated
Comment thread quality/integration_testing/environments/dual_qemu/__init__.py Outdated
@Rahul-Sutariya
Rahul-Sutariya force-pushed the rasu_dual_qemu_intergation_test_plugin branch 2 times, most recently from 6c98422 to 01bea3d Compare July 14, 2026 13:59
@Rahul-Sutariya
Rahul-Sutariya requested a review from castler July 14, 2026 13:59
Comment thread quality/integration_testing/environments/dual_qemu/__init__.py Outdated
@Rahul-Sutariya
Rahul-Sutariya force-pushed the rasu_dual_qemu_intergation_test_plugin branch 2 times, most recently from b783e72 to 535aa46 Compare July 17, 2026 13:04
Comment thread quality/integration_testing/environments/dual_qemu/example_xvm/README.md Outdated
Comment thread quality/integration_testing/environments/dual_qemu/example_xvm/BUILD Outdated
@Rahul-Sutariya
Rahul-Sutariya force-pushed the rasu_dual_qemu_intergation_test_plugin branch from 535aa46 to 1b86c08 Compare July 21, 2026 06:38
@anmittag

Copy link
Copy Markdown
Member

@Rahul-Sutariya some checks are failing in your PR. Please adapt.

@anmittag anmittag moved this from Backlog to Ready in COM - Communication FT Jul 21, 2026
@castler

castler commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

@Rahul-Sutariya some checks are failing in your PR. Please adapt.

@anmittag as already stated in #617 (comment), this is right now blocked.

Add a `dual_qemu` ITF/pytest plugin that boots two QNX VMs sharing one
QEMU ivshmem-plain region, extending the upstream single-VM `qemu` plugin.

- plugin: config.py, ivshmem_qemu.py, dual_qemu_process.py, __init__.py
  (target_a/target_b fixtures; sequential boot to avoid KVM race), BUILD,
  example config, README, example/run_example_manually.sh.
- integration_testing.bzl: dual_qemu_integration_test macro (QNX-only, flaky).
- example: offset_list.h + simple_ivshmem.cpp map the ivshmem BAR via
  pci-server and exchange a self-relative offset-pointer list, proving
  cross-VM pointer resolution; simple_ivshmem_test.py verifies both VMs.
note: Cross-VM shared memory forces the region onto the ivshmem PCI BAR (device/SHMCTL_PHYS memory),
but the production lib-memory ControlBlock puts a process-shared pthread_mutex there and locks it during init.

QNX refuses to operate a process-shared mutex on device-mapped BAR memory (returns EINVAL even with write-back caching),
so the producer aborts — making the umbrella score::memory::shared unusable over ivshmem, unlike the mutex-free demo.
@Rahul-Sutariya
Rahul-Sutariya force-pushed the rasu_dual_qemu_intergation_test_plugin branch from 1b86c08 to ac16d8b Compare July 24, 2026 04:34
@Rahul-Sutariya
Rahul-Sutariya requested a review from castler July 24, 2026 04:58
@castler
castler added this pull request to the merge queue Jul 24, 2026
Merged via the queue into eclipse-score:main with commit 576d4de Jul 24, 2026
8 checks passed
@github-project-automation github-project-automation Bot moved this from Ready to Done in COM - Communication FT Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants