Release v0.42.0 - #6207
Conversation
Release-Triggered-By: samuv
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6207 +/- ##
==========================================
+ Coverage 72.46% 72.51% +0.05%
==========================================
Files 739 739
Lines 76719 76719
==========================================
+ Hits 55594 55633 +39
+ Misses 17160 17104 -56
- Partials 3965 3982 +17 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
📝 Generated release notes for
|
| Config kind | Workload kinds tracked | Spec paths |
|---|---|---|
MCPOIDCConfig |
MCPServer, MCPRemoteProxy, VirtualMCPServer | spec.oidcConfigRef.name; spec.incomingAuth.oidcConfigRef.name (vMCP) |
MCPAuthzConfig |
MCPServer, MCPRemoteProxy, VirtualMCPServer | spec.authzConfigRef.name; spec.incomingAuth.authzConfigRef.name (vMCP) |
MCPTelemetryConfig |
MCPServer, MCPRemoteProxy, VirtualMCPServer | spec.telemetryConfigRef.name |
MCPExternalAuthConfig |
MCPServer, MCPRemoteProxy | spec.externalAuthConfigRef.name, or spec.authServerRef.name when spec.authServerRef.kind == "MCPExternalAuthConfig" |
MCPToolConfig |
MCPServer | spec.toolConfigRef.name |
MCPWebhookConfig |
MCPServer | spec.webhookConfigRef.name |
Note kubectl --field-selector will not work for these paths — the operator's indexes are controller-runtime cache indexes, not API-server field selectors. Use -o json | jq or -o custom-columns.
Migration steps
- While still on v0.41.x, snapshot anything you may need:
kubectl get mcpoidcconfigs,mcpauthzconfigs,mcpexternalauthconfigs,mcptoolconfigs,mcpwebhookconfigs,mcptelemetryconfigs -A -o json > /tmp/thv-config-refs-pre-0.42.json - Grep your automation for
referenceCount,referencingWorkloads, and theReferences/REFERENCEScolumn — shell scripts,kubectl wait --for=jsonpath=, Chainsaw/kuttl assertions, Argo CD/Flux health checks, kube-state-metrics configs, Grafana panels, Kyverno/Gatekeeper rules. - Rewrite each hit with the query for that config kind from the table above. For "is this config still in use?" checks, prefer the condition:
kubectl -n NS get mcpoidcconfig my-oidc -o jsonpath='{.status.conditions[?(@.type=="DeletionBlocked")].message}' helm upgradetheoperator-crdschart, then theoperatorchart. No pre/post hooks needed.- Verify:
kubectl -n toolhive-system get mcpoidcconfigshowsNAME SOURCE VALID AGE, and deletion of a referenced config still leaves it withDeletionBlocked=True. - Go consumers: drop
.Status.ReferencingWorkloads/.Status.ReferenceCountreads. TheWorkloadReferencetype (Kind,Name) is still exported if you want to keep your own list shape.
Migration guide: Cedar policy now sees the post-mutation request
Who is affected: only workloads configured with at least one mutating webhook and either Cedar authorization or any consumer of audit / telemetry / usage metrics. Both are shipped, supported, non-mutually-exclusive configurations — thv run --webhook-config <file with a mutating: entry> --authz-config <file>, or MCPWebhookConfig.spec.mutating in the operator. Workloads with no mutating webhook see zero change; the republish is gated on the body actually having changed.
What was wrong: ParsingMiddleware parses the request body once and refuses to parse again. The mutating webhook replaced r.Body but passed the request through unchanged, so Cedar evaluated policy against the tool name and arguments that arrived while the backend executed the ones that ran. The audit half was reachable in the default configuration: the event type and target.name resolve through the parsed-request holder regardless of includeRequestData (which defaults to false), so the audit trail named a request that never executed. Telemetry and usage metrics drifted the same way.
Security framing, stated precisely: before v0.42.0, a client could reach a tool or argument set Cedar would have denied by sending a permitted request shape that the webhook rewrote into a forbidden one. A second bug narrowed this in practice: r.ContentLength was not refreshed alongside r.Body, so a mutation that shrank the body failed at the reverse proxy and one that grew it was truncated into invalid JSON. The bypass was live for length-preserving rewrites — which is exactly case/format normalization, and a webhook can pad JSON whitespace to hold length constant. That stale Content-Length is also fixed here.
Before
client request ──► ParsingMiddleware ──► parse cached ──► mutating webhook rewrites body
│ │
Cedar reads ◄────────────────┘ (pre-mutation)
audit reads backend runs post-mutation
After
client request ──► ParsingMiddleware ──► parse cached ──► mutating webhook rewrites body
│
RepublishParsedMCPRequest (body changed)
│
Cedar reads ◄────────────────┘ (post-mutation)
audit reads backend runs post-mutation
Migration steps
- Check whether you set
--webhook-configwith amutating:entry (orMCPWebhookConfig.spec.mutating). If not, stop — no action needed. - Read each mutating webhook's patch and enumerate what it rewrites: the JSON-RPC
method,params.name, and/orparams.arguments. - Re-check your Cedar policies against the post-mutation shape — resource names (
MCP::Tool::"<name>") and everywhen { context.arg_* }clause. Policies that were passing only because they never saw the rewrite will now deny, and vice versa. - Update SIEM rules, dashboards, and saved queries keyed on audit
typeortarget.name— for mutated requests those values change on upgrade. - Expect two new fail-closed responses replacing what previously reached the backend: 400 if a webhook rewrites a single request into a JSON-RPC batch, and 500 if a webhook emits a body that is not a valid JSON-RPC request.
Gaps this deliberately does not close, all documented rather than fixed:
- With
includeRequestData: true, the recorded request payload is still the pre-mutation body (audit readsr.Bodybefore the webhook), so event type/target name are post-mutation while the payload is not. - After a webhook renames a tool, the
Mcp-Method/Mcp-Nameheaders forwarded to the backend still name the original tool. A conformant Modern backend rejects the mismatch, so it fails closed — but a mutating webhook should not rename tools on the Modern path. - The tool-call filter and rate limiter run outside
ParsingMiddlewareand still decide against the request as received, so--toolsfiltering remains bypassable by a webhook rename. Tracked in #6134.
Migration guide: Recovered panics are no longer logged
This is an unintended regression, not a design decision. It is called out here because it costs you diagnostics silently, and a one-line fix is expected in a patch release.
Who is affected: any operator who relies on ToolHive's logs to diagnose a recovered HTTP panic — including log-based alerts, log-derived metrics, and support bundles. Everyone running without Sentry configured (the default) is affected most.
What changed: pkg/recovery became a thin shim over toolhive-core/recovery. Core's Middleware recovers panics silently unless a logger is injected via WithLogger, and ToolHive's shim passes only WithPanicHandler. The OTel span error recording and Sentry issue reporting are genuinely preserved — same span status (codes.Error, "panic recovered"), same sanitization, same raw value to Sentry, same ordering — but the slog.Error line and its stack trace are gone, and no other middleware picks them up.
Before (v0.41.0)
time=... level=ERROR msg="Panic recovered: runtime error: index out of range [3] with length 2
Stack trace:
goroutine 42 [running]:
runtime/debug.Stack()
..."
After (v0.42.0)
(no log output — the client receives 500 Internal Server Error and nothing is recorded locally)
Migration steps
- If you have alerts or log-based metrics matching
Panic recovered, they will stop firing. Do not interpret the silence as "no panics" — re-point them at the 500-response rate or at Sentry until the log line returns. - Configure Sentry if you have not already;
ReportPanicstill sends the raw panic value, so panics remain visible as Sentry Issues with full context. - Traces are unaffected — the request span still carries
RecordErrorplus an error status, so OTel-based panic detection keeps working. - When the fix lands, the restored line will be structured (
msg="panic recovered"withpanic,method,path, andstackattributes) rather than the old single formatted string, so write any new log parser against that shape.
PR: #6145
Migration guide: Go API changes
Who is affected: only out-of-tree Go code importing ToolHive packages. No CLI, REST API, or CRD surface changes here, and no in-tree caller is affected.
pkg/telemetry/providers was deleted (#6146)
The package and its /otlp and /prometheus subpackages were removed and consumed from toolhive-core instead. The graduation is verbatim — every non-test file is byte-identical apart from two self-referential import paths — so all 12 options (WithServiceName, WithServiceVersion, WithOTLPEndpoint, WithHeaders, WithInsecure, WithCACertPath, WithTracingEnabled, WithMetricsEnabled, WithSamplingRate, WithEnablePrometheusMetricsPath, WithCustomAttributes, WithExtraSpanProcessors) plus NewCompositeProvider, ProviderOption, and CompositeProvider keep identical names and signatures. Nothing about emitted telemetry changes — resource attributes, service-name defaulting, OTLP exporter/TLS config, and Prometheus exporter registration all behave as before.
Before
import "github.com/stacklok/toolhive/pkg/telemetry/providers"After
import "github.com/stacklok/toolhive-core/telemetry/providers"Two optimizerdec constants were removed (#6175)
pkg/vmcp/session/optimizerdec no longer exports CallToolArgToolName or CallToolArgParameters. Both have been part of the published API since v0.15.0. They existed to read the call_tool target out of a raw arguments map, a pattern that is now known-unsafe: encoding/json falls back to case-insensitive field matching, so a map index and a struct decode resolve different key sets.
Before
toolName, _ := args[optimizerdec.CallToolArgToolName].(string)
params, _ := args[optimizerdec.CallToolArgParameters].(map[string]any)After
// Decode with the same call both dispatch sites use, so key matching cannot diverge.
in, err := schema.Translate[optimizer.CallToolInput](args)registry.Provider gained three methods (#6135)
ListAvailablePlugins(), GetPlugin(namespace, name), and SearchPlugins(query) were added to the interface. Implementations that embed registry.BaseProvider pick up no-op defaults and need no change; anything satisfying the old method set directly will no longer compile.
Migration: embed registry.BaseProvider in your provider struct, or implement the three methods.
🔄 Deprecations
pkg/audit's MCP event constants,LevelAudit, andNewAuditLoggerare now transitional aliases forgithub.com/stacklok/toolhive-core/auditand will be removed once the migration's cleanup wave rewrites imports per subtree — prefer thetoolhive-core/auditsymbols in new code (#6148)
🆕 New Features
- Manage plugins for AI coding tools with the new
thv ai-plugincommand group —build,validate,push,install,list,info,uninstall, plus local build management viabuildsandbuilds remove— targeting Claude Code and Codex (#5782) - The same plugin surface is available over REST at
/api/v1beta/plugins(10 endpoints) with a matching Go HTTP client inpkg/plugins/client, so the CLI, API, and external tooling share one contract (#5782) thv ai-plugin install <name>now resolves a plain name against the configured registry instead of failing with a 404 hint, and new catalog routes let you browse and search plugins in a registry (#6135)- Project-scoped skill installs now verify Sigstore signatures before anything is extracted or recorded, recording the signer identity as lock-file
provenance:on first use and rejecting unsigned artifacts unless you pass--allow-unsigned(#6129) thv skill syncre-verifies each managed skill's stored Sigstore bundle offline against the lock file's recorded identity, treating a failed re-verification as drift so a CI gate catches signature changes exactly like content changes (#6131)thv skill upgraderefuses to move a skill to an artifact signed by a different identity — or to an unsigned one — reportingsigner-change-blockedunless you explicitly rotate trust with--allow-signer-change(#6132)- Git-installed skills get full gitsign commit-signature verification, with the chain of trust checked against embedded Fulcio roots and no network access (#6121, #6091)
The skills signing features above are all behind the experimental TOOLHIVE_SKILLS_LOCK_ENABLED gate and apply only to project-scoped installs. With the gate unset, thv skill install behaves exactly as in v0.41.0. Note that git (gitsign) provenance is recorded as provisional: true because the embedded Rekor transparency-log proof is not yet validated — signing time is checked only against the Fulcio certificate's own ~10-minute validity window. OCI provenance is not provisional.
🐛 Bug Fixes
- Multiple MCP clients can now connect to a single stdio MCP server through ToolHive — the first handshake is cached and replayed instead of every client after the first getting
duplicate "initialize" received, which also unblocks vMCP aggregating stdio backends (#6153) - A client that retries
initializeon a live connection behind the transparent proxy now receives a fresh session instead of a hard failure, because the proxy no longer forwards a session ID oninitialize(#6152) - vMCP gateways aggregating a dual-era backend such as
github-mcp-serverv1.6.0 no longer oscillate between the Modern and Legacy revisions and fail roughly half their health checks — a Modern promotion must now win a confirmingserver/discoverprobe rather than trusting the negotiated version alone (#6158) - A vMCP backend redeployed from a hint-lying Legacy server to a genuinely Modern one now corrects its reported MCP revision within ~5 minutes instead of staying Legacy until the pod restarts (#6185)
- vMCP now relays backend log and progress notifications from Modern (2026-07-28) backends to the downstream client, opting in through the per-request
io.modelcontextprotocol/logLevel_metakey that replaced the removedlogging/setLevelRPC (#6140) - vMCP clients on the Legacy revision now receive non-reserved backend
_meta(trace ids, custom fields) onresources/readresults, matching what the Modern path already delivered (#6180) - A vMCP pod with the optimizer enabled no longer permanently loses its tool index while continuing to report itself healthy — the in-memory SQLite database is now pinned alive by a dedicated connection, so one cancelled request can't destroy it for the life of the process (#6157)
find_tool'stool_keywordsinput now actually affects results instead of being decoded and dropped, and it drives the lexical BM25 arm whiletool_descriptiondrives semantic matching (#6124)call_toolnow accepts the common LLM malformation wheretool_nameis nested insideparameters, and a genuinely missingtool_nameproduces an error that states the expected shape and lists the parameter names received (#6150)- On Windows, the discovery directory and
server.jsonunder%LOCALAPPDATA%are now protected with an explicit DACL granting only the ToolHive user and SYSTEM, and are ownership-validated before being trusted — POSIX mode bits are advisory on NTFS, so any local account with Modify could previously rewrite thenpipe://discovery URL and redirect the next MCP client (#5951) - The authorization middleware now resolves a
call_tooltarget through the same decoder dispatch uses, closing three case-sensitivity divergences that could skip a policy check or drop arguments (#6175)
🧹 Misc
pkg/telemetry/providers(~2,900 LOC) is deleted in favour of the verbatim graduation intoolhive-core, with no change to emitted telemetry (#6146)pkg/recoverybecomes a thin shim overtoolhive-core/recovery, keeping ToolHive's OTel and Sentry wiring through a panic-handler hook (#6145)- MCP histogram buckets are sourced from
toolhive-core's semconv preset instead of a local literal; the boundaries are unchanged (#6144) - Audit event constants,
LevelAudit, andNewAuditLoggerbecome aliases overtoolhive-core/auditwith byte-identical values, so the audit wire format is untouched (#6148) - Pinned a regression test for vMCP elicitation failing fast when the client advertised the capability but holds no standalone SSE stream, and documented both delivery constraints (#6182)
- Fixed a port-selection TOCTOU race that flaked e2e tests under sharded CI by having the OIDC and LLM gateway mocks hold their own listener from construction (#6142)
- Pinned the
ida-pro-mcpe2e image by digest after an upstream rebuild pulled in the breaking mcp Python SDK 2.0.0, and addedtest/e2e/images/**to the lifecycle suite's trigger filter so an image change can no longer skip the tests that consume it (#6159) - Pinned the
mcp-server-timee2e image by digest for the same upstream breakage, unblocking the proxy suites (#6160) - Fixed the operator integration suites'
timeout waiting for process kube-apiserver to stopflake — 32 of the job's last 51 failures — by awaiting manager shutdown before tearing down envtest (#6179)
📦 Dependencies
| Module | Version |
|---|---|
github.com/stacklok/toolhive-core |
v0.0.35 → v0.0.38 |
github.com/stacklok/toolhive-catalog |
v0.20260804.0 |
github.com/tailscale/hujson |
b80ff77 |
coverallsapp/github-action |
8d6379e |
github/codeql-action |
f205ea1 |
anthropics/claude-code-action |
v1.0.183 |
toolhive-core was bumped across #6144, #6146, and #6180 rather than by a dependency PR; v0.0.38 also carries transitive bumps to aws-sdk-go-v2, go-containerregistry, moby/client, prometheus, and otel.
👋 Welcome to our newest contributor: @Tanguille 🎉
Full commit log
What's Changed
- Add plugin REST API, thv ai-plugin CLI, and HTTP client (Phase 4) by @JAORMX in Add plugin REST API, thv ai-plugin CLI, and HTTP client (Phase 4) #5782
- Remove referencingWorkloads/referenceCount from config CRD statuses by @ChrisJBurns in Remove referencingWorkloads/referenceCount from config CRD statuses #5631
- Wire tool_keywords into the BM25 search arm by @aponcedeleonch in Wire tool_keywords into the BM25 search arm #6124
- Refresh the cached MCP parse after webhook mutation by @jhrozek in Refresh the cached MCP parse after webhook mutation #6136
- Add plugin registry catalog surface and name-based install by @JAORMX in Add plugin registry catalog surface and name-based install #6135
- Fix port-selection TOCTOU race in e2e tests by @jhrozek in Fix port-selection TOCTOU race in e2e tests #6142
- Adopt core MCP semconv bucket preset in telemetry by @JAORMX in Adopt core MCP semconv bucket preset in telemetry #6144
- Delegate recovery middleware to toolhive-core with observability hooks by @JAORMX in Delegate recovery middleware to toolhive-core with observability hooks #6145
- Alias audit event constants and logger to toolhive-core by @JAORMX in Alias audit event constants and logger to toolhive-core #6148
- Pin ida-pro-mcp e2e image by digest by @aponcedeleonch in Pin ida-pro-mcp e2e image by digest #6159
- Pin mcp-server-time e2e image by digest by @amirejaz in Pin mcp-server-time e2e image by digest #6160
- Confirm Modern promotion with a discover probe by @amirejaz in Confirm Modern promotion with a discover probe #6158
- Strip session ID from proxied initialize requests by @amirejaz in Strip session ID from proxied initialize requests #6152
- Update github.com/tailscale/hujson digest to b80ff77 by @renovate[bot] in Update github.com/tailscale/hujson digest to b80ff77 #6166
- Update coverallsapp/github-action digest to 8d6379e by @renovate[bot] in Update coverallsapp/github-action digest to 8d6379e #6165
- Update github/codeql-action digest to f205ea1 by @renovate[bot] in Update github/codeql-action digest to f205ea1 #6167
- Authorize and accept call_tool with nested tool_name by @Tanguille in Authorize and accept call_tool with nested tool_name #6150
- Keep optimizer store's in-memory database alive by @aponcedeleonch in Keep optimizer store's in-memory database alive #6157
- Decode the call_tool target as dispatch does by @aponcedeleonch in Decode the call_tool target as dispatch does #6175
- Serve repeat initialize from a cached result by @amirejaz in Serve repeat initialize from a cached result #6153
- Await manager shutdown before stopping envtest by @aponcedeleonch in Await manager shutdown before stopping envtest #6179
- Add skills verifier wrapping toolhive-core Sigstore exports by @samuv in Add skills verifier wrapping toolhive-core Sigstore exports #6091
- Add gitsign commit-signature verification by @samuv in Add gitsign commit-signature verification #6121
- Verify skill signatures at install time by @samuv in Verify skill signatures at install time #6129
- Adopt telemetry providers from toolhive-core by @JAORMX in Adopt telemetry providers from toolhive-core #6146
- Opt Modern backends into log notifications via logLevel _meta by @JAORMX in Opt Modern backends into log notifications via logLevel _meta #6140
- Re-verify stored signatures offline during sync by @samuv in Re-verify stored signatures offline during sync #6131
- Update anthropics/claude-code-action action to v1.0.183 by @renovate[bot] in Update anthropics/claude-code-action action to v1.0.183 #6173
- Block upgrades that change the signer identity by @samuv in Block upgrades that change the signer identity #6132
- Set Windows DACL on discovery directory under LOCALAPPDATA by @stantheman0128 in Set Windows DACL on discovery directory under LOCALAPPDATA #5951
- Pin elicitation fail-fast without a standalone stream by @JAORMX in Pin elicitation fail-fast without a standalone stream #6182
- Expire the refuted-Modern-hint memory after a TTL by @JAORMX in Expire the refuted-Modern-hint memory after a TTL #6185
- Forward resource-read _meta in vMCP and the transport bridge by @JAORMX in Forward resource-read _meta in vMCP and the transport bridge #6180
- Update module github.com/stacklok/toolhive-catalog to v0.20260804.0 by @renovate[bot] in Update module github.com/stacklok/toolhive-catalog to v0.20260804.0 #6193
- Release v0.42.0 by @toolhive-release-app[bot] in Release v0.42.0 #6207
New Contributors
- @Tanguille made their first contribution in Authorize and accept call_tool with nested tool_name #6150
Full Changelog: v0.41.0...v0.42.0
🔗 Full changelog: v0.41.0...v0.42.0
Release v0.42.0
Version Bump
minor release
Files Updated
VERSIONdeploy/charts/operator-crds/Chart.yaml(path:version)deploy/charts/operator-crds/Chart.yaml(path:appVersion)deploy/charts/operator/Chart.yaml(path:version)deploy/charts/operator/Chart.yaml(path:appVersion)deploy/charts/operator/values.yaml(path:operator.image)deploy/charts/operator/values.yaml(path:operator.toolhiveRunnerImage)deploy/charts/operator/values.yaml(path:operator.vmcpImage)Next Steps
Checklist