Skip to content

feat(governance): track-event telemetry sink for App Insights customEvents#135

Open
viswa-uipath wants to merge 2 commits into
feat/governance-policy-fetch-hoistfrom
feat/traces-metrics
Open

feat(governance): track-event telemetry sink for App Insights customEvents#135
viswa-uipath wants to merge 2 commits into
feat/governance-policy-fetch-hoistfrom
feat/traces-metrics

Conversation

@viswa-uipath

Copy link
Copy Markdown

Summary

Stacked on top of #133 (feat/governance-policy-fetch-hoist). Adds a new platform-mandated audit sink that emits governance evaluation telemetry to App Insights customEvents via the UiPathPlatformGovernanceProvider.track_event callable shipped in uipath-python PR #1745 (uipath-platform 0.1.74).

The runtime stays decoupled from uipath-platform — the sink takes a Callable[..., None]; the host adapts provider.track_event to it at wiring time. No new dep added to uipath-runtime.

Event vocabulary

Event When Per-hook count
governance.rule.denied A rule matched and its configured action ≠ allow one per matched-restrictive rule
governance.hook.summary Hook end, with at least one rule evaluated or skipped exactly one

Volume-controlled: a 50-rule pack on a chatty agent doesn't multiply per-step telemetry by 50. Passed + skipped + matched-allow rules roll into the hook summary; only denials get individual events.

Payload contract

Every event carries:

Field Source
execution_engine GovernanceRuntimeMetadata (default uipath_native_governance_checker, host-overridable for future engines like AGT)
agent_type host-supplied (e.g. uipath_coded, uipath_lowcode, servicenow)
agent_framework host-supplied (e.g. langchain)
runtime_version auto-resolved via importlib.metadata.version(\"uipath-runtime\")
operation_id (HTTP header) AuditEvent.trace_id → App Insights operation_Id for cross-event correlation
agent_name / hook / timestamp from the audit event

Per-rule events add: pack, clause, rule_name, mode, evaluator_result (DENY/HITL), action_applied (mode-adjusted), duration_ms, mapped_to_uipath, detail.

Per-hook summary adds: mode, passed_count, denied_count, skipped_count, skipped_policy_names, guardrail_dispatched_count (UiPath-mapped guardrails that fell back to /runtime/govern), duration_ms, final_action.

Filter rules (accepts() + emit() defense-in-depth)

Case track_event fires?
Any event with mode = DISABLED
RULE_EVALUATION, matched = False ❌ (rolls into hook summary's passed_count)
RULE_EVALUATION, matched = True, action = allow ❌ (positive informational match, not a denial)
RULE_EVALUATION, matched = True, action ∈ {deny, escalate, audit}
HOOK_END, total_rules = 0 AND skipped_count = 0 ❌ (empty hook, zero operator value)
HOOK_END, total > 0 OR skipped > 0

Wiring

The audit manager auto-registers the sink at construction (mirrors _register_traces_sink shape, wrapped in a broad try/except so a misconfigured wiring layer never crashes the agent):

```python
manager = AuditManager(
track_event=provider.track_event, # from PR #1745
runtime_metadata=GovernanceRuntimeMetadata(
agent_type="uipath_coded",
agent_framework="langchain",
),
)
```

If the host forgets to wire track_event, the registration logs a warning and the sink is skipped — telemetry off, runtime continues.

Count semantics

Field Meaning
matched_rules Raw "any check matched" count — unchanged for backward compat with the legacy traces sink
denied_count (new) Matched AND action ≠ allow — the spec-meaning denial count
passed_count total_rules - denied_count — includes both unmatched and matched-allow rules

Test plan

  • uv run ruff check src/uipath/runtime/governance tests — clean
  • uv run mypy src/uipath/runtime/governance — clean (12 source files)
  • uv run pytest --no-cov374 passed, 1 skipped (was 326 → +49 new tests, +1 skipped pre-existing)

New test files:

  • tests/test_governance_metadata.py (5) — defaults, overrides, frozen behavior, PackageNotFoundError fallback
  • tests/test_track_events_sink.py (30) — filter (accepts + emit defense-in-depth) × event types × DISABLED × empty-hook × matched-allow, payload shape, mode handling, operation_id correlation
  • tests/test_audit_manager_track_event_wiring.py (7) — happy path, auto-registration alongside traces sink, missing-callable warning, synthetic registration-failure recovery, opt-out
  • tests/test_evaluator_telemetry.py (7) — per-rule duration, per-hook duration, skipped tracking, denied/passed counts, matched-allow contributes to passed, guardrail dispatched count

Files touched

File Change
src/uipath/runtime/governance/_audit/metadata.py (new) GovernanceRuntimeMetadata dataclass
src/uipath/runtime/governance/_audit/track_events.py (new) TrackEventAuditSink
src/uipath/runtime/governance/_audit/base.py Auto-register the sink; emit_* signature extensions (backward-compatible)
src/uipath/runtime/governance/native/evaluator.py Per-rule + per-hook timing; skipped-rule tracking; denied/passed split

Net: 8 files, +1524 / -11.

Out of scope

  • Per-rule visibility into UiPath-mapped guardrails (would be a governance.guardrail.dispatched event from the compensator). Today the per-hook guardrail_dispatched_count supports the mapped-vs-native ratio query.
  • A real TelemetryProvider protocol in uipath-core — the callable indirection is sufficient for current needs.

🤖 Generated with Claude Code

…vents

Adds a new platform-mandated audit sink that emits two event types
through a caller-supplied ``track_event`` callable (wired by the host
to ``UiPathPlatformGovernanceProvider.track_event`` from
uipath-platform 0.1.74 / PR #1745):

  - ``governance.rule.denied`` — one per matched-and-restrictive rule
  - ``governance.hook.summary`` — one per hook end, carries
    passed/denied/skipped/guardrail-dispatched counts +
    skipped_policy_names

Volume control by design: 50-rule packs no longer multiply per-step
telemetry calls by 50. Passed + skipped roll into the hook summary;
only denials get individual events.

Per-runtime metadata
- New ``GovernanceRuntimeMetadata`` frozen dataclass stamps every
  payload with execution_engine (default
  ``uipath_native_governance_checker``), agent_type, agent_framework,
  and ``runtime_version`` (auto-resolved via importlib.metadata).
  Host supplies agent type / framework; engine defaults are
  override-ready for future engines (e.g. AGT).

Trace correlation
- AuditEvent.trace_id flows from the runtime ctor through the evaluator
  into the sink as ``operation_id`` (App Insights operation_Id header)
  so every event in one agent run correlates in KQL.

Pack + clause split, mapped-vs-unmapped insight
- ``pack`` and ``clause`` are separate fields for KQL aggregation.
- Per-rule ``mapped_to_uipath`` is always False for events the sink
  emits (UiPath-mapped guardrails are server-traced via
  /runtime/govern). The hook summary's ``guardrail_dispatched_count``
  carries the per-hook count for the mapped-vs-native ratio.

Per-rule + per-hook timing
- ``time.monotonic()`` instrumentation in the evaluator measures each
  rule individually and the hook in total; both ``duration_ms`` values
  flow through ``emit_rule_evaluation`` / ``emit_hook_summary``.

Wiring: AuditManager auto-registers the sink (mirrors
_register_traces_sink shape, wrapped in try/except so a misconfigured
wiring layer never crashes the agent). Missing track_event surfaces as
a warning and the sink is skipped — telemetry off, agent continues.

Filters (accepts() + emit() defense-in-depth):
- DISABLED enforcement mode → zero events of any type
- matched=False rule → suppressed (rolls into hook summary)
- matched=True with action=allow → suppressed (positive informational
  match isn't a denial)
- Empty hook summary (total=0 AND skipped=0) → suppressed; one with
  skipped>0 still fires because skipped_policy_names is
  operator-relevant

Count semantics
- ``matched_rules`` keeps the historical "any check matched" sense for
  backward compat with the legacy traces sink.
- ``denied_count`` (new) = matched AND action != allow.
- ``passed_count`` = total - denied_count (matched-allow counts as
  passed now).

Net: +42 tests covering metadata defaults, sink filters, payload
shape, mode handling, wiring auto-registration, missing-callable
warning, registration-failure recovery, evaluator timing, count
splits, and DISABLED-mode/empty-hook suppression.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GovernanceRuntimeMetadata is imported at module top — metadata.py has
no back-dependency on base.py (only stdlib), so no cycle.
TrackEventCallable is replaced with its underlying ``Callable[...,
None]`` shape directly in the two annotation sites; the alias would
have created a real cycle (track_events.py imports AuditEvent /
AuditSink / EventType from base.py).

ruff + mypy clean, 374 passed + 1 skipped.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant