feat: Add LLM token usage tracking to Application Insights by Prachig-Microsoft · Pull Request #586 · microsoft/content-processing-solution-accelerator

Prachig-Microsoft · 2026-05-12T11:29:51Z

Implement comprehensive token usage tracking across all LLM call sites in ContentProcessor and ContentProcessorWorkflow, following the MACAE psl-token-usage branch pattern.

Changes:

Add token_usage_utils.py with extract/emit helpers for both projects
Instrument MapHandler (ContentProcessor) with per-call token events
Instrument RAI, Summarize, GapAnalysis executors (Workflow)
Add summary and per-model token events in save_handler
Add 18 KQL queries for Azure Workbook visualization
Add unit tests (18 tests) for token_usage_utils

Events emitted to Application Insights:

LLM_Agent_Token_Usage: per agent/step with model, process_id
LLM_Model_Token_Usage: per model deployment aggregation
LLM_Token_Usage_Summary: per document totals

Does this introduce a breaking change?

Yes
No

Golden Path Validation

[ x] I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

[ x] I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid

...

Other Information

github-actions · 2026-05-12T11:31:24Z

Coverage Report •

File	Stmts	Miss	Cover	Missing
TOTAL	1217	161	86%

report-only-changed-files is enabled. No files were changed during this commit :)

Tests	Skipped	Failures	Errors	Time
244	0 💤	0 ❌	0 🔥	4.955s ⏱️

- Pass file metadata in TokenUsageScope for MapHandler, Summarize, GapAnalysis, and RAI executors so the workbook file-type query works - Fix dedup in model cost KQL query (workbook + queries file) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add SHA-256 based event_id (event_name|process_id|agent_name|model) to each emitted event, enabling KQL dedup without accelerator-specific max() patterns. This makes the workbook generic across accelerators. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Change emit_all() logging from INFO to DEBUG to reduce production log volume - Optional token-detail fields default to None when absent instead of 0 - Remove redundant extract_usage_from_stream_chunk() fallback in TokenUsageScope.add() - Pin azure-monitor-events-extension to ==0.1.0 in both projects - Revert enableMonitoring to false in default parameters Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ntProcessor) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 15 out of 17 changed files in this pull request and generated 3 comments.

…acking # Conflicts: # infra/main.json

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The devcontainers/python image on MCR is returning 403 Forbidden. Use the official Python slim image instead, which is lighter and appropriate for production containers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated 5 comments.

…c file_types - Pass agent_name and model_deployment_name to emit_summary for unique event_id - Sort file_types set before joining for deterministic telemetry dimensions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated 4 comments.

Prachig-Microsoft temporarily deployed to production May 12, 2026 11:29 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 13, 2026 15:52 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 13, 2026 16:22 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 14, 2026 07:43 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 14, 2026 07:57 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 14, 2026 08:34 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 14, 2026 11:07 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 14, 2026 11:08 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 14, 2026 11:41 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 15, 2026 10:36 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 15, 2026 10:43 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 15, 2026 12:19 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 15, 2026 12:20 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 15, 2026 14:04 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 15, 2026 16:04 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 15, 2026 17:24 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 18, 2026 09:46 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production May 18, 2026 11:02 — with GitHub Actions Inactive

Prachig-Microsoft had a problem deploying to production May 18, 2026 11:03 — with GitHub Actions Failure

Prachig-Microsoft temporarily deployed to production May 18, 2026 12:23 — with GitHub Actions Inactive

Prachig-Microsoft marked this pull request as ready for review May 25, 2026 10:13

Copilot AI review requested due to automatic review settings May 25, 2026 10:13

Prachig-Microsoft requested review from Avijit-Microsoft, Prajwal-Microsoft, Roopan-Microsoft, Vinay-Microsoft, aniaroramsft, nchandhi and toherman-msft as code owners May 25, 2026 10:13

Prachig-Microsoft and others added 3 commits June 2, 2026 15:58

Prachig-Microsoft force-pushed the feature/token-usage-tracking branch from 321eca7 to f4ee50e Compare June 2, 2026 10:28

Prachig-Microsoft temporarily deployed to production June 2, 2026 10:28 — with GitHub Actions Inactive

fix: exclude telemetry module from Workflow coverage (tested in Conte…

efebd81

…ntProcessor) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 2, 2026 10:33

Prachig-Microsoft temporarily deployed to production June 2, 2026 10:33 — with GitHub Actions Inactive

Copilot started reviewing on behalf of Prachig-Microsoft June 2, 2026 10:33 View session

Copilot AI reviewed Jun 2, 2026

View reviewed changes

Comment thread src/ContentProcessorWorkflow/src/steps/summarize/executor/summarize_executor.py Outdated

Comment thread src/ContentProcessorWorkflow/src/steps/rai/executor/rai_executor.py Outdated

Comment thread src/ContentProcessorWorkflow/src/steps/gap_analysis/executor/gap_executor.py Outdated

Merge remote-tracking branch 'origin/dev' into feature/token-usage-tr…

eae9ecd

…acking # Conflicts: # infra/main.json

Prachig-Microsoft had a problem deploying to production June 11, 2026 09:27 — with GitHub Actions Failure

chore: remove token usage workbook files from branch

ab495c8

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 11, 2026 09:29

Prachig-Microsoft temporarily deployed to production June 11, 2026 09:29 — with GitHub Actions Inactive

Copilot started reviewing on behalf of Prachig-Microsoft June 11, 2026 09:29 View session

Prachig-Microsoft temporarily deployed to production June 11, 2026 09:30 — with GitHub Actions Inactive

Copilot AI reviewed Jun 11, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings June 11, 2026 09:43

Prachig-Microsoft temporarily deployed to production June 11, 2026 09:43 — with GitHub Actions Inactive

Copilot started reviewing on behalf of Prachig-Microsoft June 11, 2026 09:43 View session

Copilot AI reviewed Jun 11, 2026

View reviewed changes

Comment thread infra/main.json

Comment thread infra/main.json

Comment thread src/ContentProcessor/src/libs/llm_token_telemetry.py

Comment thread src/ContentProcessorWorkflow/src/libs/llm_token_telemetry.py

Prachig-Microsoft temporarily deployed to production June 11, 2026 16:31 — with GitHub Actions Inactive

Prachig-Microsoft temporarily deployed to production June 11, 2026 16:38 — with GitHub Actions Inactive

Prachig-Microsoft had a problem deploying to production June 11, 2026 16:40 — with GitHub Actions Failure

Prachig-Microsoft had a problem deploying to production June 11, 2026 16:47 — with GitHub Actions Failure

Prachig-Microsoft had a problem deploying to production June 11, 2026 17:14 — with GitHub Actions Failure

Prachig-Microsoft had a problem deploying to production June 11, 2026 17:21 — with GitHub Actions Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add LLM token usage tracking to Application Insights#586

feat: Add LLM token usage tracking to Application Insights#586
Prachig-Microsoft wants to merge 27 commits into
devfrom
feature/token-usage-tracking

Prachig-Microsoft commented May 12, 2026

Uh oh!

github-actions Bot commented May 12, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Prachig-Microsoft commented May 12, 2026

Does this introduce a breaking change?

Golden Path Validation

Deployment Validation

What to Check

Other Information

Uh oh!

github-actions Bot commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented May 12, 2026 •

edited

Loading