feat(governance): track-event telemetry sink for App Insights customEvents#135
Open
viswa-uipath wants to merge 2 commits into
Open
feat(governance): track-event telemetry sink for App Insights customEvents#135viswa-uipath wants to merge 2 commits into
viswa-uipath wants to merge 2 commits into
Conversation
…vents
Adds a new platform-mandated audit sink that emits two event types
through a caller-supplied ``track_event`` callable (wired by the host
to ``UiPathPlatformGovernanceProvider.track_event`` from
uipath-platform 0.1.74 / PR #1745):
- ``governance.rule.denied`` — one per matched-and-restrictive rule
- ``governance.hook.summary`` — one per hook end, carries
passed/denied/skipped/guardrail-dispatched counts +
skipped_policy_names
Volume control by design: 50-rule packs no longer multiply per-step
telemetry calls by 50. Passed + skipped roll into the hook summary;
only denials get individual events.
Per-runtime metadata
- New ``GovernanceRuntimeMetadata`` frozen dataclass stamps every
payload with execution_engine (default
``uipath_native_governance_checker``), agent_type, agent_framework,
and ``runtime_version`` (auto-resolved via importlib.metadata).
Host supplies agent type / framework; engine defaults are
override-ready for future engines (e.g. AGT).
Trace correlation
- AuditEvent.trace_id flows from the runtime ctor through the evaluator
into the sink as ``operation_id`` (App Insights operation_Id header)
so every event in one agent run correlates in KQL.
Pack + clause split, mapped-vs-unmapped insight
- ``pack`` and ``clause`` are separate fields for KQL aggregation.
- Per-rule ``mapped_to_uipath`` is always False for events the sink
emits (UiPath-mapped guardrails are server-traced via
/runtime/govern). The hook summary's ``guardrail_dispatched_count``
carries the per-hook count for the mapped-vs-native ratio.
Per-rule + per-hook timing
- ``time.monotonic()`` instrumentation in the evaluator measures each
rule individually and the hook in total; both ``duration_ms`` values
flow through ``emit_rule_evaluation`` / ``emit_hook_summary``.
Wiring: AuditManager auto-registers the sink (mirrors
_register_traces_sink shape, wrapped in try/except so a misconfigured
wiring layer never crashes the agent). Missing track_event surfaces as
a warning and the sink is skipped — telemetry off, agent continues.
Filters (accepts() + emit() defense-in-depth):
- DISABLED enforcement mode → zero events of any type
- matched=False rule → suppressed (rolls into hook summary)
- matched=True with action=allow → suppressed (positive informational
match isn't a denial)
- Empty hook summary (total=0 AND skipped=0) → suppressed; one with
skipped>0 still fires because skipped_policy_names is
operator-relevant
Count semantics
- ``matched_rules`` keeps the historical "any check matched" sense for
backward compat with the legacy traces sink.
- ``denied_count`` (new) = matched AND action != allow.
- ``passed_count`` = total - denied_count (matched-allow counts as
passed now).
Net: +42 tests covering metadata defaults, sink filters, payload
shape, mode handling, wiring auto-registration, missing-callable
warning, registration-failure recovery, evaluator timing, count
splits, and DISABLED-mode/empty-hook suppression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GovernanceRuntimeMetadata is imported at module top — metadata.py has no back-dependency on base.py (only stdlib), so no cycle. TrackEventCallable is replaced with its underlying ``Callable[..., None]`` shape directly in the two annotation sites; the alias would have created a real cycle (track_events.py imports AuditEvent / AuditSink / EventType from base.py). ruff + mypy clean, 374 passed + 1 skipped. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on top of #133 (
feat/governance-policy-fetch-hoist). Adds a new platform-mandated audit sink that emits governance evaluation telemetry to App InsightscustomEventsvia theUiPathPlatformGovernanceProvider.track_eventcallable shipped in uipath-python PR #1745 (uipath-platform 0.1.74).The runtime stays decoupled from
uipath-platform— the sink takes aCallable[..., None]; the host adaptsprovider.track_eventto it at wiring time. No new dep added touipath-runtime.Event vocabulary
governance.rule.deniedallowgovernance.hook.summaryVolume-controlled: a 50-rule pack on a chatty agent doesn't multiply per-step telemetry by 50. Passed + skipped + matched-allow rules roll into the hook summary; only denials get individual events.
Payload contract
Every event carries:
execution_engineGovernanceRuntimeMetadata(defaultuipath_native_governance_checker, host-overridable for future engines like AGT)agent_typeuipath_coded,uipath_lowcode,servicenow)agent_frameworklangchain)runtime_versionimportlib.metadata.version(\"uipath-runtime\")operation_id(HTTP header)AuditEvent.trace_id→ App Insightsoperation_Idfor cross-event correlationagent_name/hook/timestampPer-rule events add:
pack,clause,rule_name,mode,evaluator_result(DENY/HITL),action_applied(mode-adjusted),duration_ms,mapped_to_uipath,detail.Per-hook summary adds:
mode,passed_count,denied_count,skipped_count,skipped_policy_names,guardrail_dispatched_count(UiPath-mapped guardrails that fell back to/runtime/govern),duration_ms,final_action.Filter rules (
accepts()+emit()defense-in-depth)track_eventfires?mode = DISABLEDmatched = Falsepassed_count)matched = True,action = allowmatched = True, action ∈ {deny, escalate, audit}total_rules = 0ANDskipped_count = 0total > 0ORskipped > 0Wiring
The audit manager auto-registers the sink at construction (mirrors
_register_traces_sinkshape, wrapped in a broadtry/exceptso a misconfigured wiring layer never crashes the agent):```python
manager = AuditManager(
track_event=provider.track_event, # from PR #1745
runtime_metadata=GovernanceRuntimeMetadata(
agent_type="uipath_coded",
agent_framework="langchain",
),
)
```
If the host forgets to wire
track_event, the registration logs a warning and the sink is skipped — telemetry off, runtime continues.Count semantics
matched_rulesdenied_count(new)passed_counttotal_rules - denied_count— includes both unmatched and matched-allow rulesTest plan
uv run ruff check src/uipath/runtime/governance tests— cleanuv run mypy src/uipath/runtime/governance— clean (12 source files)uv run pytest --no-cov— 374 passed, 1 skipped (was 326 → +49 new tests, +1 skipped pre-existing)New test files:
tests/test_governance_metadata.py(5) — defaults, overrides, frozen behavior,PackageNotFoundErrorfallbacktests/test_track_events_sink.py(30) — filter (accepts + emit defense-in-depth) × event types × DISABLED × empty-hook × matched-allow, payload shape, mode handling, operation_id correlationtests/test_audit_manager_track_event_wiring.py(7) — happy path, auto-registration alongside traces sink, missing-callable warning, synthetic registration-failure recovery, opt-outtests/test_evaluator_telemetry.py(7) — per-rule duration, per-hook duration, skipped tracking, denied/passed counts, matched-allow contributes to passed, guardrail dispatched countFiles touched
src/uipath/runtime/governance/_audit/metadata.py(new)GovernanceRuntimeMetadatadataclasssrc/uipath/runtime/governance/_audit/track_events.py(new)TrackEventAuditSinksrc/uipath/runtime/governance/_audit/base.pyemit_*signature extensions (backward-compatible)src/uipath/runtime/governance/native/evaluator.pyNet: 8 files, +1524 / -11.
Out of scope
governance.guardrail.dispatchedevent from the compensator). Today the per-hookguardrail_dispatched_countsupports the mapped-vs-native ratio query.TelemetryProviderprotocol inuipath-core— the callable indirection is sufficient for current needs.🤖 Generated with Claude Code