Skip to content

Add UTF-8-safe base64 encoding for code blocks#2381

Merged
MarceloRGonc merged 1 commit into
mainfrom
mg/OPS-4569
Jun 26, 2026
Merged

Add UTF-8-safe base64 encoding for code blocks#2381
MarceloRGonc merged 1 commit into
mainfrom
mg/OPS-4569

Conversation

@MarceloRGonc

@MarceloRGonc MarceloRGonc commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Part of OPS-4569.

Additional Notes

Code blocks running in the V8 isolate get a pure-JS base64 polyfill (btoa/atob). Those built-ins are Latin1-only and throw the string to be encoded contains characters outside of the Latin1 range whenever the input contains non-Latin1 characters (accented text, currency symbols, CJK, emoji), so any block that base64-encodes user data fails on such input.

This PR:

  • Adds UTF-8-safe btoaUtf8/atobUtf8 helpers to the base64 polyfill that is bootstrapped into every code-block isolate. They are additive and behind typeof guards — existing btoa/atob behavior is unchanged.
  • Ships ambient type declarations for the new helpers. Code blocks are type-checked with tsc before they run in the isolate, so without a declaration tsc rejects any block referencing the helpers with TS2304: Cannot find name 'btoaUtf8'. The declarations live next to the runtime polyfill (BASE64_POLYFILL_TYPES) and are written as openops-globals.d.ts into each block's compile directory by the code builder, keeping runtime and types in sync.

Testing Checklist

Check all that apply:

  • I tested the feature thoroughly, including edge cases

  • I verified all affected areas still work as expected

  • Automated tests were added/updated if necessary

  • Changes are backwards compatible with any existing data, otherwise a migration script is provided

Testing notes

  • Added isolate round-trip tests in v8-isolate-code-sandbox.test.ts covering btoaUtf8/atobUtf8 encode, decode, a unicode round-trip (Coût € — 日本語 🚀), and a JSON.parse(atobUtf8(...)) decode of an encoded payload.
  • Verified the compile-time fix with a standalone tsc --build: a block calling btoaUtf8 compiles with openops-globals.d.ts present and fails with TS2304 without it.
  • npx nx lint engine and npx nx test engine pass.

Copilot AI review requested due to automatic review settings June 25, 2026 18:14
@linear

linear Bot commented Jun 25, 2026

Copy link
Copy Markdown

OPS-4569

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds UTF-8-safe Base64 helpers to the engine’s V8 isolate bootstrap so code blocks can Base64-encode/decode arbitrary Unicode strings (not just Latin1), and ensures code blocks type-check cleanly by emitting matching ambient type declarations during compilation.

Changes:

  • Extend the isolate base64 polyfill with btoaUtf8 / atobUtf8 (guarded, additive).
  • Emit openops-globals.d.ts into each code-block compile directory so tsc recognizes the new globals.
  • Add isolate-level tests validating UTF-8 encode/decode and round-trips for Unicode data.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
packages/engine/test/core/code/v8-isolate-code-sandbox.test.ts Adds coverage for UTF-8 base64 helpers in the isolate sandbox.
packages/engine/src/lib/core/code/polyfills/base64-polyfill.ts Introduces btoaUtf8/atobUtf8 runtime polyfills and keeps their ambient type declarations alongside the polyfill.
packages/engine/src/lib/code-block/code-builder.ts Writes openops-globals.d.ts during code-block compilation so tsc accepts the new globals.

@MarceloRGonc MarceloRGonc changed the title Add UTF-8-safe base64 encoding for code block Add UTF-8-safe base64 encoding for code blocks Jun 25, 2026
@MarceloRGonc MarceloRGonc enabled auto-merge (squash) June 25, 2026 18:21
@MarceloRGonc MarceloRGonc merged commit f15c0ba into main Jun 26, 2026
22 checks passed
@MarceloRGonc MarceloRGonc deleted the mg/OPS-4569 branch June 26, 2026 05:56
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants