feat(kvp): add diagnostics layer over KvpPoolStore

[Peyton Robertson (peytonr18)]

Summary

This PR adds a DiagnosticsKvp layer that centralizes azure-init's KVP telemetry behavior, including event key formatting, message chunking, and message reassembly. This provides a single tested implementation that can be reused by the follow-up wiring PR and simplifies the existing logic in kvp.rs and logging.rs.

It also extends the existing dump command with a --parse-diagnostics mode for decoding and viewing KVP telemetry as readable events, making provisioning diagnostics easier to inspect and troubleshoot.

Specifically, this PR adds:

• DiagnosticsKvp, a typed view over KvpPoolStore that implements telemetry-specific behavior while keeping the underlying store format-agnostic:

  • Event keys formatted and parsed as <prefix>|<vm_id>|<level>|<name>|<event_id>.
  • Message chunking at UTF-8 boundaries for values larger than a single KVP record, written via append_multiple under a single lock. Each chunk gets a unique |<subevent_index>-suffixed key so the Hyper-V host (which keeps one record per key) retains every fragment, and the chunks are reassembled when read.
  • Classification of records as diagnostic events, raw records such as PROVISIONING_REPORT, or malformed event keys.

• Diagnostics folded into the existing raw KVP commands via flags rather than a separate command tree:

  • dump --parse-diagnostics reassembles chunked events and decodes each record. Raw output remains the default when the flag is absent.
  • --include-raw also prints raw (non-event) records; --level, --name, and -n/--tail narrow the output to a filtered, events-only view.
  • The filter flags (--level, --name, -n/--tail) require --parse-diagnostics, and --include-raw is mutually exclusive with them — it applies only to the unfiltered view, where raw records can appear.
  • clear --diagnostics removes every diagnostic key (events and malformed event keys) while leaving raw records such as PROVISIONING_REPORT intact.

Example

A long event is stored as multiple records, each under a unique |<subevent_index> key. The raw view shows the fragments; --parse-diagnostics decodes and reassembles them into a single event:

$ libazureinit-kvp dump
azure-init-1.0.0|3f25…|INFO|user:create_user|8f3e…|0=Time: … | Event: creating
azure-init-1.0.0|3f25…|INFO|user:create_user|8f3e…|1= azureuser (uid 1000) …
PROVISIONING_REPORT=result=success|…

$ libazureinit-kvp dump --parse-diagnostics --name user:create_user
event level=INFO name=user:create_user event_id=8f3e… message=Time: … | Event: creating azureuser (uid 1000) …

$ libazureinit-kvp dump --parse-diagnostics --include-raw
event level=INFO name=user:create_user event_id=8f3e… chunks=2 message=Time: … | Event: creating azureuser (uid 1000) …
raw key=PROVISIONING_REPORT value=result=success|…

$ libazureinit-kvp dump --parse-diagnostics --include-raw --json
[{"chunks":2,"event_id":"8f3e…","kind":"event","level":"INFO","message":"…","name":"user:create_user"},{"key":"PROVISIONING_REPORT","kind":"raw","value":"result=success|…"}]

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.04%. Comparing base (02fc780) to head (0d011f2).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #313      +/-   ##
==========================================
+ Coverage   95.70%   96.04%   +0.33%     
==========================================
  Files          23       24       +1     
  Lines        7705     8360     +655     
==========================================
+ Hits         7374     8029     +655     
  Misses        331      331              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR introduces a DiagnosticsKvp layer in libazureinit-kvp to provide a typed, reusable view over KvpPoolStore for diagnostic telemetry (event key formatting/parsing, UTF-8-safe chunking, and chunk reassembly), and adds new diag CLI subcommands to inspect this decoded event view.

Changes:

• Added DiagnosticsKvp, DiagnosticEvent, and DiagnosticRecord with chunking/reassembly and record classification logic.
• Added diag CLI subcommands (dump, events, tail, clear) for decoding and filtering diagnostic events (including JSON output support).
• Added integration and CLI tests covering round-trips, chunking, classification, and clear scoping; enabled uuid v4 generation via crate features.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
libazureinit-kvp/tests/diagnostics.rs	New integration tests for diagnostics emit/read, chunking, classification, scoped clear, and concurrency behavior.
libazureinit-kvp/tests/cli.rs	New CLI tests validating diag command behaviors for text/JSON output, filtering, tailing, and clear confirmation.
libazureinit-kvp/src/lib.rs	Exposes the new diagnostics module and re-exports its public types/constants.
libazureinit-kvp/src/error.rs	Adds a dedicated error for rejecting `
libazureinit-kvp/src/diagnostics.rs	Implements the diagnostics layer: event key format/parse, UTF-8 chunking, reassembly, classification, and scoped clearing.
libazureinit-kvp/src/cli.rs	Adds diag subcommands and renders diagnostics records/events in text and JSON.
libazureinit-kvp/Cargo.toml	Enables uuid v4 feature required for Uuid::new_v4().

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Chris Patterson (cjp256)]

yes is overkill i think. It's volatile data and likely past the window of collection if someone is using it. Would suggest removing it.

[Peyton Robertson (peytonr18)]

Removed this

[Chris Patterson (cjp256)]

this seems more like "delete" than "clear" to me.

I would expect clear to nuke all diagnostics-based keys.

[Peyton Robertson (peytonr18)]

I've adjusted this so clear gets rid of all diagnostic keys!

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

libazureinit-kvp/src/diagnostics.rs:455

• reassemble() currently concatenates any consecutive records whose base_event_key() matches, which also merges adjacent raw records that happen to share the same key (e.g., two consecutive PROVISIONING_REPORT records appended via KvpPoolStore::append). That loses record boundaries and contradicts the doc comment that grouping is for event-chunk reassembly.

        while dumped
            .peek()
            .is_some_and(|(next, _)| base_event_key(next) == base)
        {

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

libazureinit-kvp/src/diagnostics.rs:490

• records()/reassemble() assumes that all chunks for a given event remain consecutive in on-disk order. While emit() writes chunks contiguously under one lock, other store operations (notably KvpPoolStore::delete_multiple, which swaps deletions with the tail and does not preserve order) can break that contiguity and lead to incorrect reassembly for surviving multi-chunk events. Consider either documenting this invariant explicitly or making reassembly robust to record reordering.

    /// Records are returned in on-disk order. Consecutive records that
    /// share an event key — ignoring the `|<subevent_index>` suffix — are
    /// one event; because [`emit`](Self::emit) writes an event's chunks
    /// contiguously under a single lock, reassembly is correct even under
    /// concurrent writers.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

libazureinit-kvp/src/cli.rs:698

• diagnostics_event_json moves String fields (name, event_id, message) out of a borrowed &DiagnosticEvent, which won’t compile. Serialize these fields by reference (or clone) instead.

fn diagnostics_event_json(event: &DiagnosticEvent) -> serde_json::Value {
    json!({
        "kind": "event",
        "level": event.level.to_string(),
        "name": event.name,
        "event_id": event.event_id,
        "message": event.message,
    })

libazureinit-kvp/src/diagnostics.rs:596

• reassemble() compares *next == base where next is a borrowed &String from peek(). Dereferencing attempts to move the String out of the peeked reference and won’t compile. Compare by reference instead.

    while let Some((base, index, value)) = parsed.next() {
        let mut indexed = vec![(index, value)];
        while parsed.peek().is_some_and(|(next, _, _)| *next == base) {
            let (_, next_index, next_value) =
                parsed.next().expect("peeked value exists");

[Chris Patterson (cjp256)]

I tried testing on older cloud-init versions using canonical:0001-com-ubuntu-server-jammy:22_04-lts:22.04.202205280 and got an error:

cloud-init-event type=diagnostic name=diagnostic message uuid=b9c3606d-3752-4f26-97ff-26f4f04acda4 ts=2026-08-06T20:20:13.477830Z incarnation=1786047606 chunks=1 message=Dumping last 56746 bytes of cloud-init.log file to KVP starting from index: 24961
malformed key=CLOUD_INIT|1786047606|compressed|cloud-init.log|b7a822ba-4eea-46c0-b559-e84396101132 reason=invalid cloud-init JSON value: EOF while parsing a string at line 1 column 1016 value={"name":"cloud-init.log","type":"compressed","ts":"2026-08-06T20:20:13.479078Z"

[Chris Patterson (cjp256)]

Is there a way to write diagnostics via CLI?