feat(go): native Go policy evaluator for Kubernetes SSI targeting

[iamluc]

Description

Adds a self-contained, dependency-free Go policy evaluator to this repo
(go/policies), mirroring the semantics of the C engine: tri-state
TRUE/FALSE/ABSTAIN over an AND/OR/NOT tree of leaf evaluators, plus a dd-wls
JSON decoder (the JSON projection of the FlatBuffers policy schema).

This is the Go evaluator the Cluster Agent will use to target Kubernetes
Single Step Instrumentation (SSI) workloads natively (no CGO at runtime).

To make "shares the same test suite" real, this PR also adds a cross-engine
conformance suite: a single go test runs a shared corpus through both the
Go engine and the real C engine (libpolicies, linked via cgo) in one
process and asserts they agree. The K8s string evaluators are added to the
schema (and to the C engine's public enum) so policies can target
pods/namespaces on both sides.

Scope is deliberately limited to the engine + schema + conformance. The
target -> policy compiler and the Cluster Agent wiring (RC subscription,
mutator) stay in datadog-agent and land in follow-up PRs.

What's in here

Go engine (go/policies)

• model + node constructors, tri-state evaluator with C-parity semantics
  (boolean folding, signed/unsigned numeric leaves, key=value/key= label
  leaves, depth limit mirroring PLCS_MAX_EVAL_DEPTH);
• dd-wls JSON decoder (ParsePolicies);
• Evaluate (rule tree -> tri-state Result) and Decide (folds the outcomes
  of all matching policies in document order, matching the C engine's
  no-first-match-short-circuit behavior — not first-match-wins).

Cross-engine conformance

• go/policies/testdata/conformance/vectors.json: shared corpus (boolean
  composition, string comparators, labels, numeric, ABSTAIN propagation).
• conformance_test.go: portable Go-only runner (no cgo, runs everywhere).
• conformance_cgo.go + conformance_cross_test.go: cgo bridge to the real C
  engine plus host evaluators that mirror what the Go engine bakes in (constant
  ALWAYS_* leaves, the key=value/key= label evaluator, and numeric
  evaluators that abstain on an unset source). Each vector is serialized to a
  FlatBuffers NodeTypeWrapper buffer and run through libpolicies; the result
  must equal both the Go engine's result and the expected value. Gated behind
  the conformance_cgo build tag, so the default CGO_ENABLED=0 builds are
  unaffected. Run with make -C go conformance-cross.

Schema + C engine parity

• fbs-schema/evaluator_ids.fbs: NAMESPACE_NAME, NAMESPACE_LABEL,
  POD_LABEL, POD_ANNOTATION (append-only); regenerated Go schema.
• c/include/dd/policies/evaluator_types.h: extend the public
  string-evaluator enum with the same 4 K8s ids so the engine reaches schema
  parity (the _Static_assert against the schema count now matches);
  regenerate evaluator_ids_reader.h.

CI

• .github/workflows/dev.yml: new conformance-cross-engine job running the
  cross-engine suite in the build container on Linux amd64 + arm64.

Related Issue

Type of Change

• Bug fix
• New feature
• Breaking change
• Refactor / cleanup
• CI / tooling
• Documentation

Checklist

• Documentation updated if needed (package doc)
• Tests added or updated
• No unintended breaking changes (schema additions are append-only)

How Has This Been Tested?

In the build container:

• CGO_ENABLED=0 go vet ./policies/... and the Go-only conformance test are
  green; the cgo-gated file is correctly excluded from default builds.
• make -C go conformance-cross: all corpus vectors pass through both the
  Go engine and the real C engine in one process (semantic parity with
  libpolicies proven on the same inputs, not just ported expectations).
• make -C c test: the full C unit-test suite still passes after the public
  enum extension.
• TestDecoderIdentifiersMatchSchema pins every identifier the decoder accepts
  to the .fbs source shared with the C engine, so the two cannot drift apart.

Additional Notes

• Numeric "unset" semantics: the default C numeric evaluator compares a
  missing fact against the NOT_SET sentinel (FALSE), whereas the Go model
  abstains. Rather than change the engine (and break existing C tests), the
  conformance host registers numeric evaluators that abstain on NOT_SET,
  reproducing the Go semantics at the host level — where the generic C engine
  expects such decisions to live.
• JSON schema diff: policy.schema.json was regenerated with
  flatc 25.2.10; the committed file lagged the .fbs, so the regen also
  resyncs it in addition to the 4 new K8s ids.
• Follow-ups (in datadog-agent): target -> policy compiler + golden
  tests; Cluster Agent wiring (facts, mutator, RC apm-wls subscription) with a
  differential test vs the current TargetMutator.

[datadog-datadog-prod-us1]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 21.71%

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 37fc60f | Docs | Datadog PR Page | Give us feedback!}

[knusbaum]

Looks like this never runs TestConformanceCorpus and won't run any other tests under conformance if they're added.

Is that intentional? Should we just do:

Suggested change

	go test -tags conformance_cgo ./policies/conformance/ -run TestConformanceCrossEngine -v
	go test -tags conformance_cgo ./policies/conformance/ -run . -v

and run everything under the conformance directory?

[knusbaum]

There are tests in policies that aren't exercised, it looks like. We probably want those to run in CI, yes?

[knusbaum]

strings.Cut's found bool is discarded here, so a malformed ENABLE_SDK value missing =version (e.g. "java" instead of "java=1.2.3") is silently accepted as TracerVersions["java"] = "". decodeLabel (below, for POD_LABEL/NAMESPACE_LABEL values) hard-errors on the identical missing-= shape — this should too.

Regression test (fails today, run against origin/luc/wls-go-policy-evaluator):

func TestDecodeActionsEnableSDKRejectsMissingEquals(t *testing.T) {
	raw := `{
      "policies": [{
        "description": "malformed enable_sdk value",
        "rules": {"node_type": "EvaluatorNode", "node": {"eval_type": "StrEvaluator",
          "eval": {"id": "ALWAYS_TRUE"}}},
        "actions": [{"action": "ENABLE_SDK", "values": ["java"]}]
      }]
    }`
	if _, err := ParsePolicies([]byte(raw)); err == nil {
		t.Fatalf("expected ParsePolicies to reject malformed ENABLE_SDK value %q, got nil error", "java")
	}
}

--- FAIL: TestDecodeActionsEnableSDKRejectsMissingEquals (0.00s)
    dd_wls_test.go:200: expected ParsePolicies to reject malformed ENABLE_SDK value "java", got nil error

[knusbaum]

This switch has no default case, so any action id not in {INJECT_ALLOW, INJECT_DENY, ENABLE_SDK, ENABLE_PROFILER} — e.g. SET_ENVAR or REEXEC, both of which exist in the generated ActionId schema (go/schema/dd/wls/ActionId.go) — is silently dropped, indistinguishable from a policy with no actions at all.

Regression test (fails today):

func TestDecodeActionsRejectsUnhandledActionIDs(t *testing.T) {
	raw := `{
      "policies": [{
        "description": "set_envar action",
        "rules": {"node_type": "EvaluatorNode", "node": {"eval_type": "StrEvaluator",
          "eval": {"id": "ALWAYS_TRUE"}}},
        "actions": [{"action": "SET_ENVAR", "values": ["DD_FOO=bar"]}]
      }]
    }`
	if _, err := ParsePolicies([]byte(raw)); err == nil {
		t.Fatalf("expected ParsePolicies to reject unhandled action id %q, got nil error", "SET_ENVAR")
	}
}

--- FAIL: TestDecodeActionsRejectsUnhandledActionIDs (0.00s)
    dd_wls_test.go:218: expected ParsePolicies to reject unhandled action id "SET_ENVAR", got nil error

[maattdd]

For forward compatibility, we do want to ignore unknown action (like on the C side)

[knusbaum]

wls.EnumValuesCmpTypeSTR[se.Cmp] (and the analogous EnumValuesStringEvaluators/EnumValuesNumericEvaluators/EnumValuesCmpTypeNUM lookups on this line and lines 355/362) is a plain map index with no , ok check. An unknown/typo'd string (e.g. "CMP_BOGUS" in testdata/vectors.json) silently resolves to the map's zero value (CmpTypeSTRCMP_STR_UNKNOWN == 0) and gets built into the buffer instead of failing loudly. The Go engine's own decoder (policies.ParsePolicies → decodeStrCmp) hard-errors on the identical shape — see the "unknown string cmp" case in dd_wls_test.go's TestParseErrors. buildRulesBuffer should do the same, otherwise a typo in the shared corpus could silently mask a real cross-engine divergence rather than failing the conformance test with a clear error.

Regression test (fails today, requires the cgo/C toolchain — make -C go conformance-cross build environment):

func TestBuildRulesBufferRejectsUnknownComparator(t *testing.T) {
	rules := []byte(`{
      "node_type": "EvaluatorNode",
      "node": {
        "eval_type": "StrEvaluator",
        "eval": {"id": "NAMESPACE_NAME", "cmp": "CMP_BOGUS", "value": "x"}
      }
    }`)
	if _, err := buildRulesBuffer(rules); err == nil {
		t.Fatalf("expected buildRulesBuffer to reject unknown comparator %q, got nil error", "CMP_BOGUS")
	}
}

--- FAIL: TestBuildRulesBufferRejectsUnknownComparator (0.00s)
    buildrules_bug_test.go:30: expected buildRulesBuffer to reject unknown comparator "CMP_BOGUS", got nil error

Verified by building libpolicies.a + flatcc from source in this repo and running it with CGO_ENABLED=1 against the real static lib.