ci: add a Twilio Buildkite pipeline for CDN release#1382
Open
abueide wants to merge 6 commits into
Open
Conversation
Publishing built bundles to the CDN means writing to S3/CloudFront infrastructure in Segment's own AWS account. Twilio's GitHub Actions acceptable-use policy is explicit that infra deploys never run on GHA, so this can't live next to npm publish (release.yml) - it needs its own home on Buildkite, scoped to just this one step. Auth follows the pattern confirmed working on segmentio/ajs-renderer's own Twilio Buildkite migration: an ambient Twilio-account identity (SEGMENT_CONTEXTS: aws-credentials) assumes a role that lives in Segment's account via a plain `aws sts assume-role` call. This replaces the old release.sh, which sourced a Segment-Buildkite-only helper (SEGMENT_LIB_PATH/run-with-role) that has no equivalent on Twilio agents - that's the concrete reason CDN release stopped working partway through the original migration attempt. Several pieces are flagged inline as unconfirmed rather than guessed at silently: whether AJS_PRIVATE_ASSETS_UPLOAD's trust policy already permits assumption from this identity, and the actual bucket/OAI/role values, which need to be provisioned as a Buildkite cluster secret before this can run for real. The pipeline object and its GitHub webhook also still need to be created and connected - this PR is the code side only.
|
✅ TVS Validation PassedShow detailsAll validation checks have successfully completed. |
Codecov Report✅ All modified and coverable lines are covered by tests. Please upload reports for the commit 74246b5 to get more accurate results. Additional details and impacted files@@ Coverage Diff @@
## master #1382 +/- ##
=======================================
Coverage 91.59% 91.59%
=======================================
Files 127 127
Lines 4142 4142
Branches 1033 1033
=======================================
Hits 3794 3794
Misses 348 348
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
didiergarcia
previously approved these changes
Jul 17, 2026
Previously these were two independently-triggered systems that happened to share a gate condition (same "Version Packages" commit-message check) - but nothing actually coupled them. release.yml's workflow_dispatch bypasses that message check entirely for manual runs, and the Buildkite pipeline had no equivalent manual-override path, so a manual npm release would never fire a CDN release. There should never be a situation where one happens without the other. Removes the GitHub webhook driving Buildkite off generic pushes, and adds a "Trigger CDN release build" step to release.yml's publish job that calls the Buildkite REST API directly right after npm publish and GitHub release creation succeed. release.yml is now the only thing that ever creates a build on cdp-analytics-next - the pipeline no longer builds off push events at all, only explicit API-triggered builds (from release.yml, or a human manually creating one to retry). Needs a new repo secret, BUILDKITE_API_TOKEN, scoped to write_builds on cdp-analytics-next - I can't provision this myself. If the trigger step itself fails, npm has already published but the CDN release was never kicked off - that's a real, visible failure mode this doesn't eliminate (two independent systems can't be made atomic from one side), but it does make it loud: the publish job goes red, and a human needs to either re-run this step or manually trigger a build on cdp-analytics-next rather than the two silently drifting apart.
Recovered arn:aws:iam::812113486725:role/ajs-private-assets-upload from the last real, successful legacy Buildkite build of this exact step (segment/analytics-next build #51166) - not guessed. It's a different Segment account (812113486725) than the 528451384384 account ajs-renderer's own buildkite-agent role lives in, which is why guessing based on that role alone wouldn't have found it. Also confirmed PROD_BUCKET's real value (segment-ajs-renderer-compiled- production, via chamber read analytics-next prod_bucket, cross-checked against the same build log) - documented in the header alongside the still-unconfirmed keys. Still unconfirmed: whether this role's trust policy actually permits assumption from Twilio Buildkite's ambient identity. Two human roles in 528451384384 (dev-write, ops-write) were directly tested and confirmed NOT able to assume it - it was only ever used by Segment's own legacy Buildkite v1 CI identity historically. This needs an actual on-agent test run to know either way.
Public repo - detailed rationale, account IDs, and role names belong in internal docs, not committed comments. Pulls the role ARN back out of the pipeline file too (sourced from the Buildkite secret instead).
abueide
changed the title
abueide
enabled auto-merge (squash)
July 17, 2026 22:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a minimal Buildkite pipeline scoped to CDN release only. CI/test and npm publish stay on GitHub Actions where they already work; this piece specifically needs to live on Buildkite per internal policy on where infra deploys are allowed to run.
release.yml'spublishjob (GitHub Actions) is the only thing that triggers a build on this pipeline, right after a successful npm publish - there's no separate webhook, so npm publish and the CDN release can't happen independently of each other.Still needed before this is live
BUILDKITE_API_TOKENrepo secret for the trigger stepTest plan