[FEAT] HTTP Headers Capture - #1163
Conversation
0cf3bbb to
1c3b17e
Compare
1c3b17e to
f9f867a
Compare
f9f867a to
eb092fd
Compare
|
be7468a to
9590fd3
Compare
There was a problem hiding this comment.
Pull request overview
Adds opt-in HTTP request/response header capture to RUM resource events (via XHR interception), including configuration compilation, security filtering, and size limiting before forwarding captured headers as _dd.request_headers / _dd.response_headers attributes on stopResource.
Changes:
- Introduces
headerCaptureRulesRUM config (defaults shortcut + composable rules), compiles it once at init, and plumbs it through resource tracking. - Captures request headers from
setRequestHeader, parses response headers fromgetAllResponseHeaders(), applies sensitive/tracing header exclusion, then enforces payload size limits. - Adds extensive unit/integration tests plus a feature README; exposes rule types from the package entrypoint.
Reviewed changes
Copilot reviewed 35 out of 36 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/core/src/sdk/DatadogProvider/tests/initialization.test.tsx | Updates provider init snapshot (adds headerCaptureRules, changes RN version handling). |
| packages/core/src/rum/instrumentation/resourceTracking/requestProxy/XHRProxy/XHRProxy.ts | Implements per-request header capture + size limiting in XHR proxy pipeline. |
| packages/core/src/rum/instrumentation/resourceTracking/requestProxy/XHRProxy/DatadogRumResource/ResourceReporter.ts | Adds _dd.request_headers / _dd.response_headers to stop context when present. |
| packages/core/src/rum/instrumentation/resourceTracking/requestProxy/XHRProxy/DatadogRumResource/tests/ResourceReporter.test.ts | Tests header attributes presence/absence in stop context. |
| packages/core/src/rum/instrumentation/resourceTracking/requestProxy/XHRProxy/tests/XHRProxy.test.ts | Adds XHRProxy-level tests for header capture, security filtering, and integration into stopResource. |
| packages/core/src/rum/instrumentation/resourceTracking/requestProxy/interfaces/RumResource.ts | Extends resource interface with optional captured headers. |
| packages/core/src/rum/instrumentation/resourceTracking/requestProxy/interfaces/RequestProxy.ts | Extends tracking context with compiled header-capture config. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/types.ts | Defines compiled header-capture config/rule runtime types. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tracingHeaderExclusion.ts | Implements explicit tracing-header exclusion set. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/sensitiveHeaderBlocklist.ts | Implements sensitive-header regex blocklist. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/README.md | Documents configuration, behavior, security model, and constraints. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/parseResponseHeaders.ts | Adds safe parsing for getAllResponseHeaders() output. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/isHeaderAllowed.ts | Composes sensitive + tracing exclusion into a single check. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/enforceSizeLimits.ts | Adds truncation + count + total-budget limiting for captured headers. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/compileHeaderConfig.ts | Compiles user config into URL regex + sets/maps for O(1) matching. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/captureHeaders.ts | Core capture logic: accumulate request headers, capture/merge/filter response headers, apply casing rules. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/tracingHeaderExclusion.test.ts | Tests tracing header exclusion list behavior. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/sensitiveHeaderBlocklist.test.ts | Tests sensitive header regex behavior. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/parseResponseHeaders.test.ts | Tests response header parsing edge cases. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/isHeaderAllowed.test.ts | Tests composed allow/deny decision. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/enforceSizeLimits.test.ts | Unit tests for truncation/count/budget behavior. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/enforceSizeLimits.integration.test.ts | End-to-end test ensuring limits apply before reaching native stopResource. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/compileHeaderConfig.test.ts | Tests compilation behavior for rule variants and forURLs. |
| packages/core/src/rum/instrumentation/resourceTracking/headerCapture/tests/captureHeaders.test.ts | Tests request accumulation, response capture, casing, union/precedence behavior. |
| packages/core/src/rum/instrumentation/resourceTracking/distributedTracing/firstPartyHosts.ts | Exports escapeRegExp for reuse in header URL matching. |
| packages/core/src/rum/instrumentation/resourceTracking/DdRumResourceTracking.tsx | Plumbs headerCaptureRules into tracking start via compilation. |
| packages/core/src/rum/instrumentation/resourceTracking/tests/utils/XMLHttpRequestMock.ts | Adjusts mock getAllResponseHeaders default return value. |
| packages/core/src/index.tsx | Exports header-capture rule types publicly. |
| packages/core/src/DdSdkReactNative.tsx | Wires headerCaptureRules through initialization; warns if set without trackResources. |
| packages/core/src/config/features/RumConfiguration.type.ts | Adds headerCaptureRules option + rule type definitions. |
| packages/core/src/config/features/RumConfiguration.ts | Adds headerCaptureRules to defaults and configuration instance. |
| packages/core/src/config/async/AutoInstrumentationConfiguration.ts | Adds headerCaptureRules to auto-instrumentation config shapes/defaulting. |
| packages/core/src/config/tests/FileBasedConfiguration.test.ts | Updates snapshots for new rum config field. |
| packages/core/src/tests/DdSdkReactNativeConfiguration.test.ts | Updates snapshots for new rum config field. |
| packages/core/src/tests/DdSdkReactNative.test.tsx | Updates expectations to include headerCaptureRules. |
| .gitignore | Ignores .planning directory/file. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
3390b99 to
89808ed
Compare
d8d9645 to
0fac228
Compare
cdn34dd
left a comment
There was a problem hiding this comment.
Overall very looks good, Great job! Although I think we should align the option name before merging.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
0dfa9a8 to
0fa9a4b
Compare
0fa9a4b to
0e3fbe4
Compare
0e3fbe4 to
433cef2
Compare
| const escapedHost = escapeRegExp(hostname); | ||
| // If a path prefix is given, escape it and match as prefix. | ||
| // If hostname-only, accept /, ?, # or end-of-string as valid URL terminators. | ||
| const pathSuffix = pathPrefix ? escapeRegExp(pathPrefix) : '(/|\\?|#|$)'; |
| // Remove any existing entry with different casing for same header | ||
| // (last-value-wins: both value and casing from latest call) | ||
| for (const existing of Object.keys(accumulator)) { | ||
| if (existing.toLowerCase() === lowered) { | ||
| delete accumulator[existing]; | ||
| break; | ||
| } | ||
| } |
| // Accumulate for header capture (only user-set, non-Datadog headers) | ||
| if (this._datadog_xhr?.capturedRequestHeaders !== undefined) { | ||
| accumulateRequestHeader( | ||
| this._datadog_xhr.capturedRequestHeaders, | ||
| header, | ||
| value | ||
| ); | ||
| } | ||
| return result; |
| * Silent filtering — no debug log for blocked headers (these are expected | ||
| * normal behavior, not warnings). |
| }); | ||
| }); | ||
|
|
||
| /** Helper: compute total bytes for all headers */ |
cdn34dd
left a comment
There was a problem hiding this comment.
What was the reasoning for adding 2 separate properties ? What scenario does it improve ?
From my point of view this makes things a bit strange as the 2 properties are silently coupled. If we have something like this:
{
trackResourceHeaders: false,
headerCaptureRules: [{ ..... }]
}where the user is setting the headerCaptureRules property but it has no effect because trackResourceHeaders is set to false, leading to nothing happening. An even worse scenario is this:
{
headerCaptureRules: [{ ..... }]
}Which would also not work, since the default value is false for trackResourceHeaders.
I think a better approach would be to do something like this:
trackResourceHeaders?: boolean | HeaderCaptureRule[]Which allows us to keep the internal HeaderCaptureRule type, but have a public facing API that's on par with how the other SDKs are doing things.
It provides a simple boolean property for enabling & disabling HTTP headers capture, it keeps platforms aligned and extra customization can be performed using the I don't like the suggested approach ( I think the current approach is pretty straightforward, and it is already documented to avoid misinterpretations: dd-sdk-reactnative/packages/core/src/config/features/RumConfiguration.type.ts Lines 178 to 179 in 433cef2 |
What does this PR do?
Adds out-of-the-box HTTP request and response header capture to RUM resource events. When
trackResources: trueis set, users can opt in to capturing headers by setting the newtrackResourceHeadersoption totruein their RUM configuration. Once enabled, theheaderCaptureRulesoption (defaults to'defaults') controls which headers are captured.Three modes are supported:
'defaults'shortcut — captures a predefined set of 10 response headers (cache-control,etag,age,expires,content-type,content-encoding,content-length,vary,server-timing,x-cache) and 2 request headers (cache-control,content-type) across all URLs. This is the default value ofheaderCaptureRulesoncetrackResourceHeaders: trueis set.HeaderCaptureRuleobjects, each with a discriminatedtypefield ('defaults','matchHeaders','matchRequestHeaders','matchResponseHeaders'). Rules can be scoped to specific URLs via aforURLsfield (same format asfirstPartyHosts). Multiple matching rules are merged additively.trackResourceHeadersdefaults tofalse, so no headers are captured and no overhead is added, regardless ofheaderCaptureRules.Captured headers appear as
_dd.request_headersand_dd.response_headerscontext attributes onstopResourceevents. These attributes are omitted entirely when there is nothing to report.Security model
Two layers of non-overridable filtering are always applied, regardless of configuration:
token,cookie,secret,authorization,password,credential,bearer, API/access key variants, forwarding and IP headers) are blocked at capture time and never stored in memory.Implementation pipeline
The feature is structured as a pure pipeline applied per-request inside the XHR proxy:
compileHeaderConfig.ts— converts the user-facingheaderCaptureRulesconfig into an array of pre-builtRegExp+Set<string>objects at SDK initialization (run-once cost, zero per-request config traversal).captureHeaders.ts— accumulates request headers atsetRequestHeadertime (after security filtering), then merges matching rules and filters by URL at request completion.parseResponseHeaders.ts— parses the rawgetAllResponseHeaders()string into aRecord<string, string>at response time.enforceSizeLimits.ts— caps individual header values and total header payload size before the attributes are forwarded to the native layer, preventing runaway memory and event size growth.Configuration is snapshotted per-request at
open()time, so in-flight requests are never affected by config changes.Motivation
Caching and content negotiation issues are among the most common and hardest-to-diagnose categories of network problems in mobile apps. Headers like
Cache-Control,ETag,Age,Server-Timing, andVarycarry the information needed to understand whether resources are being served from cache, how long they live, and whether CDN or server configuration is correct — but none of this was previously visible in RUM resource events without custom instrumentation.This feature makes that class of debugging available out-of-the-box, while keeping security as a first-class constraint: sensitive headers are hard-blocked at the SDK level and cannot be accidentally enabled through misconfiguration.
Additional Notes
trackResourceHeaders: trueto enable it; existing integrations are unaffected.fetchAPI in React Native usesXMLHttpRequestunder the hood, so all fetch requests go through the same interception pipeline automatically.trackResourceHeadersisfalse(default), the capture path is not entered at all, regardless ofheaderCaptureRules— zero overhead on the XHR hot path.headerCaptureRulesrequirestrackResourceHeaders: trueto take effect; it defaults to'defaults'.packages/core/src/rum/instrumentation/resourceTracking/headerCapture/README.mdReview checklist (to be filled by reviewers)