diff --git a/projects/hathor-wallet-lib/token_amount_v2/0000-htr-lib-nodejs-bindings.md b/projects/hathor-wallet-lib/token_amount_v2/0000-htr-lib-nodejs-bindings.md new file mode 100644 index 0000000..5c1b318 --- /dev/null +++ b/projects/hathor-wallet-lib/token_amount_v2/0000-htr-lib-nodejs-bindings.md @@ -0,0 +1,423 @@ +- Feature Name: htr-lib-nodejs-bindings +- Start Date: 2026-06-10 +- Author: André Carneiro + +# Summary +[summary]: #summary + +Expose the pure-Rust `htr-lib` domain types (`TokenAmount`, `TokenBalance`, +`TokenAmountVersion`) to TypeScript/JavaScript through a +[napi-rs](https://napi.rs) binding, published to npm as `@hathor/htr-lib`. The +binding mirrors the role `htr-lib-py` plays for Python: it is the single, +canonical implementation of token-amount arithmetic and the V1↔V2 normalization +logic, shared across languages. A WebAssembly artifact provides a universal +fallback for environments without a prebuilt native binary (browsers, Deno, Bun, +edge runtimes, unsupported architectures). An initial implementation is under +development in [hathor-core PR #1725](https://github.com/HathorNetwork/hathor-core/pull/1725). + +# Motivation +[motivation]: #motivation + +Hathor is introducing a new "Token Amount V2" representation: token output +values move from the legacy 2-decimal interpretation (where `123` means `1.23`) +to an 18-decimal representation (where `1000000000000000000` means `1` unit). +This affects every place an amount is serialized, normalized, compared, or +arithmetically combined, across the full node, the wallet libraries, and any +external integration. + +If each language re-implements the normalization factor, the truncation rules, +the overflow guards, and the V1/V2 conversion semantics independently, the +implementations *will* drift. A subtle disagreement about, say, when a V2→V1 +conversion is allowed to truncate is a consensus-relevant bug. The Rust crate +`htr-lib` is the agreed source of truth; `htr-lib-py` already binds it for +Python. This RFC defines the equivalent binding for the Node.js/TypeScript +ecosystem so that the wallet-lib and other JS consumers compute amounts with +exactly the same code the full node uses. + +The expected outcome is a published npm package, `@hathor/htr-lib`, whose +TypeScript types are generated from the Rust source, so the JS surface cannot +silently fall out of sync with the Rust domain logic. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +`@hathor/htr-lib` exposes two value types and one enum: + +- `TokenAmount`: an **unsigned** token amount. Internally normalized to the V2 + (18-decimal) representation regardless of which version produced it. +- `TokenBalance`: a **signed** balance (can be negative), used for balance + accounting where inputs and outputs net out. +- `TokenAmountVersion`: an enum `{ V1 = 1, V2 = 2 }` identifying the + interpretation an amount was created with. + +The most important thing to internalize is that **JavaScript has no operator +overloading**. In Python, `htr-lib-py` can expose `a + b`, `a == b`, and +`a < b` directly via dunder methods (`__add__`, `__eq__`, `__lt__`). JavaScript +cannot do the equivalent for two custom objects. So arithmetic and equality are +exposed as **named methods**, and only the four relational operators get native +"sugar". + +## Numbers cross the boundary as `bigint` + +Rust's `BigUint`/`BigInt` map to JavaScript's native `bigint`. There is no +precision loss and no `number`/float involved: + +```typescript +import { TokenAmount } from '@hathor/htr-lib'; + +const oneTokenV2 = TokenAmount.fromV2(1_000_000_000_000_000_000n); // 1 unit, 18 decimals +const oneTokenV1 = TokenAmount.fromV1(100n); // "1.00" in legacy form + +oneTokenV1.normalized(); // => 1_000_000_000_000_000_000n (always V2-normalized) +oneTokenV1.raw(); // => 100n (the value as stored for its version) +oneTokenV1.eq(oneTokenV2); // => true (same normalized value, different versions) +``` + +## Arithmetic and equality are methods + +```typescript +const a = TokenAmount.fromV2(10n); +const b = TokenAmount.fromV2(3n); + +const c = a.add(b); // TokenAmount(13) (NOT a + b) +const d = a.sub(b); // TokenAmount(7) +a.eq(b); // false (NOT a === b) +a.ne(b); // true +``` + +`a + b` and `a === b` are deliberately **not** supported on `TokenAmount`. `+` +would coerce both operands to raw `bigint` and silently drop the wrapper type; +`===` between two objects compares reference identity and would always be +`false`. Using the methods is the only correct path. + +## Relational operators work natively + +The four relational operators (`<`, `>`, `<=`, `>=`) *do* work, because +JavaScript coerces operands through `[Symbol.toPrimitive]` before comparing: + +```typescript +const a = TokenAmount.fromV2(5n); +const b = TokenAmount.fromV2(8n); + +a < b; // true (native, via Symbol.toPrimitive) +a.lt(b); // true (explicit method, equivalent) +a.compare(b); // -1 (-1 | 0 | 1) +``` + +Both forms exist. The methods (`lt`/`le`/`gt`/`ge`/`compare`) are the +recommended, type-safe form; the native operators are ergonomic sugar with one +caveat (see [cross-type safety](#semantics--error-mapping)). + +## Versions and conversions + +```typescript +import { TokenAmount, TokenAmountVersion } from '@hathor/htr-lib'; + +const amount = TokenAmount.fromV1(150n); // "1.50" legacy +amount.isV1(); // true +amount.isV2(); // false + +const asV2 = amount.toV2(); // always succeeds (V1 ⊂ V2) +const backToV1 = amount.toV1(); // succeeds (150 fits the 2-decimal grid) + +// A V2 value with sub-V1 precision cannot become V1 without truncating: +const fine = TokenAmount.fromV2(1_500_000_000_000_000_001n); +fine.toV1(); // throws Error: "cannot denormalize value, would truncate (...)" +fine.maybeToV1(); // returns null instead of throwing +``` + +## Balances are signed + +`TokenBalance` is the type to reach for when accounting can go negative +(e.g. a wallet's net position while summing inputs and outputs): + +```typescript +import { TokenBalance } from '@hathor/htr-lib'; + +const bal = new TokenBalance(5n); +const neg = bal.sub(new TokenBalance(8n)); // TokenBalance(-3) +neg.asBool(); // true (raw != 0) +neg.isZero(); // false +neg.toAmount(); // throws Error: "cannot convert negative TokenBalance to TokenAmount (...)" +``` + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +## Toolchain + +- **napi-rs v3** (chosen over v2 specifically for its WASM target support) plus + `@napi-rs/cli`. +- The CLI **auto-generates** `index.js` and `index.d.ts` from `#[napi]` + annotations. TypeScript types are generated, never hand-written, which is what + keeps the JS surface locked to the Rust source. +- Naming convention: Rust `snake_case` → JS `camelCase` (napi-rs default). +- Numbers: `BigUint`/`BigInt` ↔ JS `bigint`. + +## Crate and file layout + +A new workspace crate `crates/htr-lib-napi`, compiled as a `cdylib` native addon +and (via emnapi) to a `wasm32-wasi` artifact. The layout mirrors `htr-lib-py`: + +``` +crates/htr-lib-napi/ + Cargo.toml # cdylib, napi + napi-derive deps, build-dep napi-build + build.rs # napi_build::setup() + package.json # name @hathor/htr-lib, @napi-rs/cli scripts, targets, optionalDependencies + src/ + lib.rs # module wiring + token_amount.rs # NapiTokenAmount + token_balance.rs # NapiTokenBalance + __tests__/ # ava integration tests + npm/ # generated per-target packages (binaries added at publish time) + linux-x64-gnu/ linux-arm64-gnu/ linux-x64-musl/ linux-arm64-musl/ + darwin-x64/ darwin-arm64/ win32-x64-msvc/ wasm32-wasi/ + index.js # generated by @napi-rs/cli + index.d.ts # generated by @napi-rs/cli + README.md # usage + manual build/publish steps +``` + +`index.js` / `index.d.ts` are generated artifacts. Whether they are committed or +git-ignored follows the napi-rs default for the chosen template and the +surrounding repo convention; this RFC does not mandate one. + +## TypeScript API surface + +```typescript +export const enum TokenAmountVersion { V1 = 1, V2 = 2 } + +export class TokenAmount { + static setNormalizationFactor(v1DecimalPlaces: number, v2DecimalPlaces: number): void + static getNormalizationFactor(): bigint + static fromV1(amount: bigint): TokenAmount + static fromV2(amount: bigint): TokenAmount + static fromVersion(amount: bigint, version: TokenAmountVersion): TokenAmount + static zero(): TokenAmount + + isV1(): boolean + isV2(): boolean + normalized(): bigint + raw(): bigint + asBool(): boolean // raw != 0 (JS truthiness can't be overridden) + isZero(): boolean // raw == 0 (convenience inverse of asBool) + toBalance(): TokenBalance + toV1(): TokenAmount // throws Error if it would truncate + maybeToV1(): TokenAmount | null + toV2(): TokenAmount + toVersion(version: TokenAmountVersion): TokenAmount // throws Error if truncate + + add(other: TokenAmount): TokenAmount + sub(other: TokenAmount): TokenAmount + + eq(other: TokenAmount): boolean + ne(other: TokenAmount): boolean + lt(other: TokenAmount): boolean + le(other: TokenAmount): boolean + gt(other: TokenAmount): boolean + ge(other: TokenAmount): boolean + compare(other: TokenAmount): number // -1 | 0 | 1 + + toString(): string // mirrors Python __repr__ (Rust Debug) + [Symbol.toPrimitive](hint: string): bigint // returns normalized; enables native <, >, <=, >= +} + +export class TokenBalance { + constructor(balance?: bigint) // default 0n + raw(): bigint + asBool(): boolean // raw != 0 + isZero(): boolean // raw == 0 + toBalance(): TokenBalance // identity + toAmount(): TokenAmount // throws Error if negative + + add(other: TokenBalance): TokenBalance + sub(other: TokenBalance): TokenBalance + neg(): TokenBalance + + eq(other: TokenBalance): boolean + ne(other: TokenBalance): boolean + lt(other: TokenBalance): boolean + le(other: TokenBalance): boolean + gt(other: TokenBalance): boolean + ge(other: TokenBalance): boolean + compare(other: TokenBalance): number // -1 | 0 | 1 + + toString(): string + [Symbol.toPrimitive](hint: string): bigint // returns raw; enables native <, >, <=, >= +} +``` + +## Design decisions + +### Methods + relational sugar + +The table below is the core constraint that drives the whole API shape: + +| Operator | Possible on two custom objects? | Mechanism / reason | +| ----------------- | ------------------------------- | --------------------------------------------------------------------------- | +| `<` `>` `<=` `>=` | Yes | Relational operators coerce operands via `[Symbol.toPrimitive]`. | +| `+` `-` | Harmful | Coerce too, but yield a raw `bigint`, dropping the wrapper type. Not used. | +| `==` `!=` | No (object vs object) | No coercion between two objects; falls back to reference identity. | +| `===` `!==` | No | Strict equality never coerces. | + +Hence: arithmetic (`add`/`sub`/`neg`) and equality (`eq`/`ne`) are methods; the +four relational operators are *additionally* supported natively via +`[Symbol.toPrimitive]`. + +### `TokenAmountVersion` as an enum + +Exported as a napi enum `{ V1 = 1, V2 = 2 }`, not a raw number. This is +type-safe and eliminates the "unknown version" runtime path Python has to carry. + +### Comparison value source + +- `TokenAmount` orders and compares on `normalized()`, so its + `[Symbol.toPrimitive]` returns the normalized value, keeping native `<`/`>` + consistent with `compare()`/`lt()`. +- `TokenBalance` orders and compares on `raw` (signed), so its + `[Symbol.toPrimitive]` returns `raw`. + +## Semantics & error mapping + +- **Cross-type safety:** the `eq`/`ne`/`lt`/… methods take a typed + `TokenAmount`/`TokenBalance`; napi-rs runtime-validates the argument and throws + a `TypeError` on a foreign type, matching Python `__richcmp__`'s guard *on the + methods*. The native `<`/`>` sugar via `Symbol.toPrimitive` does **not** carry + that guard; coercion produces two `bigint`s and JS compares them with no type + check. This is a known, accepted tradeoff of enabling native relational ops. +- **Fallible conversions:** `toV1`, `toVersion` (to V1), and `toAmount` return + `Option::None` in Rust on failure (truncation / negative); the binding throws + an `Error` carrying the same descriptive message text Python uses (e.g. + `"cannot denormalize value, would truncate (...)"`, + `"cannot convert negative TokenBalance to TokenAmount (...)"`). `maybeToV1` + returns `null` instead of throwing. +- **Arithmetic underflow:** `TokenAmount.sub` underflow panics in Rust (a + `BigUint` cannot be negative); napi-rs catches the panic and surfaces it as a + thrown JS error. `TokenBalance` is signed and does not underflow; use it when + a negative result is legitimate. +- **Global factor:** `setNormalizationFactor` is idempotent for an equal factor + and panics (→ thrown error) on a conflicting re-set, unchanged from `htr-lib`. + The factor is backed by a process-wide `OnceLock`. + +## Packaging + +For external consumers across multiple architectures: + +- **Main package** `@hathor/htr-lib`: JS loader + `index.d.ts`, listing each + platform package as an `optionalDependency`. The loader prefers a matching + native binary and falls back to the `wasm32-wasi` package when none matches. +- **Per-platform packages** under `npm/`, one `.node` (or WASM) binary each, with + `os`/`cpu`/`libc` fields so npm installs only the matching one: + `linux-x64-gnu`, `linux-arm64-gnu`, `linux-x64-musl`, `linux-arm64-musl`, + `darwin-x64`, `darwin-arm64`, `win32-x64-msvc`, and `wasm32-wasi` (the + universal fallback: browsers, Deno, Bun, edge runtimes, unsupported arches). + +### Publishing + +**No CI / automated publishing is added in this PR.** The package structure is +set up to be publishable, and the README documents the manual steps: building +each target with `napi build --release --target `, populating the `npm/` +directories, and publishing with `npm publish` (and `napi prepublish` as +appropriate). Building all targets locally requires the relevant +cross-compilation toolchains; the README notes this. Publishing is done manually +for now. + +## Testing + +- **ava integration tests** (`__tests__/`) exercise the real JS-facing surface: + `bigint` round-trips, every method, native `<`/`>`/`<=`/`>=` via + `Symbol.toPrimitive`, thrown errors (truncation, negative `toAmount`, + cross-type comparison), cross-version equality/ordering, and the `toV1` + truncation cases. ava is the test runner napi-rs scaffolds by default. +- **Rust `#[cfg(test)]` unit tests** cover only pure helper logic not naturally + reachable from JS (e.g. `TokenAmountVersion` mapping, error-message + construction). +- **Global-factor caveat:** unlike Rust's nextest (process-per-test isolation), + the ava test process shares one `OnceLock`-backed normalization factor. Tests + set a single agreed factor once for the process rather than per-test. + +# Drawbacks +[drawbacks]: #drawbacks + +- **No ergonomic arithmetic.** `c = a.add(b)` instead of `c = a + b` is verbose + and unfamiliar. Consumers who want operator ergonomics must use native + `bigint` and lose the wrapper type; this is the central tension the companion + RFC (`0001`) analyzes for the wallet-lib. +- **Two ways to compare.** Both `a.lt(b)` and `a < b` work, but only the method + form is type-guarded. Mixing the two invites subtle bugs where a native + comparison silently coerces a foreign object. +- **A native dependency.** Shipping prebuilt `.node` binaries for eight targets + plus a WASM fallback is operational overhead (cross-compilation toolchains, + per-platform packages, manual publishing for now). A pure-JS implementation + would avoid this, at the cost of the cross-language drift this RFC exists to + prevent. +- **Process-global normalization factor.** The `OnceLock` factor is convenient + but makes the binding stateful in a way that complicates testing and forbids + two different factors in one process. +- **`Symbol.toPrimitive` foot-gun.** Native relational operators bypass the + cross-type guard. A future maintainer may not realize `tokenAmount < someBalance` + coerces both sides and compares unrelated magnitudes without complaint. + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +- **Why a Rust binding at all?** The normalization factor, truncation rules, and + conversion semantics are consensus-relevant. A second hand-written JS + implementation would drift from the full node. Binding the canonical Rust crate + makes drift structurally impossible: the TS types are *generated* from the Rust + source. +- **Why napi-rs v3 over v2?** v3 adds first-class WASM target support, which is + required for the universal fallback (browsers, Deno, Bun, edge). v2 would force + a separate WASM toolchain. +- **Why methods + relational sugar, not pure methods?** Native `<`/`>` are free + (JS coerces via `Symbol.toPrimitive` anyway) and dramatically improve + readability for the common case of ordering checks. The accepted cost is that + the native form skips the type guard. +- **Why not expose `+`/`-` via `Symbol.toPrimitive`?** They would coerce to raw + `bigint` and silently return the wrong type (a `bigint`, not a `TokenAmount`), + hiding bugs. Comparisons return `boolean` either way, so there is no type to + lose, which is exactly why relational sugar is safe but arithmetic sugar is + not. +- **Impact of not doing this:** the wallet-lib (and every JS integration) must + re-implement V1/V2 normalization by hand, accepting permanent drift risk + against the full node. + +# Prior art +[prior-art]: #prior-art + +- **`htr-lib-py`** is the direct sibling. It binds the same `htr-lib` crate via + PyO3 and uses Python dunder methods for full operator overloading. This RFC is + the JS analogue; the API is intentionally a 1:1 mapping where the language + allows, and a documented divergence (methods instead of operators) where it + does not. +- **napi-rs** is the established standard for Rust↔Node native modules; major + projects (SWC, Prisma's query engine, Parcel's transformer) ship multi-target + prebuilt binaries with a WASM fallback in exactly this shape. +- **Ethereum's `ethers.js`/`viem`** model amounts as native `bigint` with helper + functions (`parseUnits`/`formatUnits`) rather than a wrapper class, evidence + that the JS ecosystem leans toward `bigint`-plus-helpers over wrapper types, + which is relevant to the companion RFC's "keep bigint" option. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- Should `index.js` / `index.d.ts` be committed or git-ignored? Deferred to the + napi-rs template default and repo convention. +- The exact normalization factor (V1 = 2 decimals, V2 = 18 decimals → factor + `10^16`) is set by the V2 decimal-precision decision tracked separately; this + binding only consumes it via `setNormalizationFactor`. +- Whether the wallet-lib adopts `TokenAmount` as a first-class type or keeps + native `bigint` and uses this binding only at boundaries, the subject of + RFC `0001`. + +# Future possibilities +[future-possibilities]: #future-possibilities + +- **CI / automated multi-target publishing** via GitHub Actions (build matrix per + triple, `napi prepublish`), replacing the manual process documented in the + README. +- **Additional domain types.** As `htr-lib` grows (e.g. richer + transaction-amount helpers, fee math), the same binding pattern extends to new + `#[napi]` types with no change to the packaging or publishing story. +- **Browser-first ergonomics.** Now that a WASM artifact exists, a thin + higher-level wrapper could expose `parseUnits`/`formatUnits`-style helpers + familiar to web3 developers, layered on top of the generated bindings. diff --git a/projects/hathor-wallet-lib/token_amount_v2/0001-wallet-lib-token-amount-v2.md b/projects/hathor-wallet-lib/token_amount_v2/0001-wallet-lib-token-amount-v2.md new file mode 100644 index 0000000..488dfa4 --- /dev/null +++ b/projects/hathor-wallet-lib/token_amount_v2/0001-wallet-lib-token-amount-v2.md @@ -0,0 +1,331 @@ +- Feature Name: wallet-lib-token-amount-v2 +- Start Date: 2026-06-10 +- Author: André Carneiro + +# Summary +[summary]: #summary + +Hathor is introducing **Token Amount V2**: token output values move from the +legacy 2-decimal interpretation (V1, where `123` means `1.23`) to an 18-decimal +interpretation (V2, where `1000000000000000000` means `1` whole token). The +canonical V1↔V2 logic lives in the Rust crate `htr-lib`, exposed to JavaScript +through the `@hathor/htr-lib` Node.js binding (see [bindind design](./0000-htr-lib-nodejs-bindings.md)). + +This RFC compares **two ways the wallet-lib could adopt V2**, and estimates the +effort of each: + +- **Approach A: Normalize at the boundary.** Keep the wallet-lib's internal + amount type as the native `bigint` it already uses; convert any incoming V1 + amount to its V2 form at the edges (network, storage, user input) and work in + V2 everywhere inside. The binding is used only for conversion/validation. +- **Approach B: `TokenAmount` everywhere.** Replace the internal `bigint` amount + type with the binding's `TokenAmount`/`TokenBalance` wrapper classes across the + whole library, so all arithmetic flows through the Rust implementation. + +A fact that frames both options: because the wallet-lib **already uses `bigint`** +for every amount (migrated in Dec 2024), Approach A is a focused boundary change, +while Approach B is a library-wide rewrite plus a breaking major release for +every downstream consumer. Approach A is not entirely free of downstream impact, +though: adopting V2 as the default changes how integer amount values are +*interpreted* across the public API (integer `1` shifts from `0.01` to +`0.000000000000000001`), so consumers still need minimal changes to their display +and validation logic, even though the `bigint` type itself is unchanged. This RFC +lays out the benefits, drawbacks, and effort of each approach so the team can +decide; it does not advocate for either. + +# Motivation +[motivation]: #motivation + +The wallet-lib must correctly handle V2 amounts: parse them, validate them, +display them, compute balances with them, and build transactions that spend +them, while remaining compatible with the V1 amounts already on-chain. The open +question is *not whether* to support V2, but *how the library should represent an +amount internally*, because that choice determines: + +- how much code changes and how risky the change is, +- whether downstream consumers (wallet-headless, wallet-mobile, wallet-service, + the web wallet) face a breaking upgrade, +- runtime performance of hot paths like balance computation, +- and how tightly the wallet stays aligned with the full node's canonical logic. + +This document lays the two options side by side with explicit trade-offs and +effort estimates so the team can make an informed call. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +## The starting point: the wallet-lib already speaks `bigint` + +A critical fact frames this entire decision. The wallet-lib **already migrated +every token amount from `number` to `bigint`** (the `feat: implement bigint +support` change, Dec 2024). There is a single canonical type alias: + +```typescript +// src/types.ts +export type OutputValueType = bigint; +``` + +used in ~225 places across ~29 source files. Amounts are `bigint` end to end: +in transaction outputs, balances, UTXO selection, serialization, and every +public method signature. Decimal places are *already* runtime-configurable +(`Storage.getDecimalPlaces()` reads `decimal_places` from the full node's version +API), and the display helper `numberUtils.prettyValue(value, decimalPlaces)` is +already decimal-agnostic. + +So this is **not** a "replace numbers with big integers" project; that work is +done. V2 is about **decimal semantics, value range, conversion at boundaries, +and display**, not the numeric type. + +This is what makes the two approaches so different in cost. + +## Approach A: Normalize at the boundary + +Internally, the wallet-lib continues to use native `bigint`, but every amount it +holds is **always a V2-normalized value**. The library establishes a rule: + +> Inside the wallet-lib, an amount is always V2. V1 only exists at the edges. + +Whenever a V1 amount enters (parsing an old transaction from the network, an +API response, or stored data), it is immediately converted to V2 using the +`@hathor/htr-lib` binding. Whenever an amount leaves toward something that still +expects V1, it is converted back (and the binding tells us if that conversion +would truncate). Everything between the edges is untouched: `balance += value`, +`change = inputs - outputs - fee`, `value > 0n` all stay exactly as they are, +because they were already correct `bigint` operations. + +```typescript +// At the network/storage boundary: convert V1 in, work in V2: +const v2 = TokenAmount.fromV1(rawV1Value).normalized(); // bigint, V2-normalized +// ... everything downstream is ordinary bigint arithmetic, unchanged ... +balance += v2; +``` + +The binding is the **source of truth for the conversion math only**. The rest of +the library's "inner workings are kept the same." + +## Approach B: `TokenAmount` everywhere + +The wallet-lib replaces `OutputValueType = bigint` with the binding's wrapper +classes: `TokenAmount` for unsigned amounts, `TokenBalance` for signed balances. +Every amount the library handles becomes a wrapper object, and every operation on +it goes through the Rust implementation: + +```typescript +// Every arithmetic site changes shape: +balance = balance.add(value); // was: balance += value +const change = inputs.sub(outputs).sub(fee); // was: inputs - outputs - fee +if (amount.gt(TokenAmount.zero())) {} // was: if (amount > 0n) +if (a.eq(b)) {} // was: if (a === b) +``` + +The appeal is **type safety and zero drift**: a V1 amount added to a V2 amount is +reconciled by Rust, and any future change to `htr-lib`'s semantics propagates the +moment the binding is upgraded. The cost is that **JavaScript has no operator +overloading** (see [bindind design](./0000-htr-lib-nodejs-bindings.md)), so every `+`, `-`, and `===` on an amount must +become a method call, across the entire library, its public API, its tests, and +every consumer. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +## Work shared by both approaches (the baseline) + +Regardless of which approach is chosen, V2 support requires the following. These +are listed first so the comparison that follows isolates only the *difference* +between A and B. + +| Shared work item | What it involves | Key files | +| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | +| Integrate `@hathor/htr-lib` | Add the native binding + ensure the WASM fallback actually loads in React Native (wallet-mobile) and the browser (electron wallets, browser, ), not just Node. **Highest-risk item.** | new dependency; loader config | +| Per-token decimals | `getDecimalPlaces()` is per-*network* today; V2 needs decimals tracked per *token* (V1 vs V2 tokens coexist). Token-metadata plumbing. | `src/storage/storage.ts`, token metadata | +| Value range / wire format | `MAX_OUTPUT_VALUE = 2^63` caps a single output at ~9.2 whole tokens at 18 decimals (a hard blocker). Encode/decode of V2 values. *Gated on the core's encoding decision (see new-decimal-precision RFC).* | `src/constants.ts`, `src/utils/buffer.ts` | +| Deposit / melt precision | `getDepositAmount`/`getWithdrawAmount` round-trip through `Number()` with a float multiply, unsafe at 18 decimals; must move to integer math. | `src/utils/tokens.ts` | +| Display & input parsing | Formatting (`prettyValue`, already param-driven) and parsing user-entered decimal strings into V2 base units. | `src/utils/numbers.ts` | + +**Estimated shared baseline: ~16–24 dev-days**, with the binding's +cross-platform integration carrying most of the schedule risk. + +## Approach A: additional work + +On top of the baseline, Approach A adds only conversion and validation at the +boundaries; internal arithmetic is untouched. + +| A-specific work item | What it involves | +| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Boundary normalization | Convert V1→V2 on ingest (network parse, API schemas, stored data) and V2→V1 on egress where a V1 consumer remains; surface truncation errors from the binding. | +| Schema validation | The ~36 `bigIntCoercibleSchema` (zod) sites validate/normalize at the JSON boundary; extend to normalize V1 values. | +| Public API value reinterpretation | Adopting V2 as the default changes the *meaning* of amount integers emitted/accepted by the public API (integer `1`: `0.01` → `0.000000000000000001`). The `bigint` type signature is unchanged, but consumers formatting/validating with 2 decimals must be updated. A real, if minimal, downstream change across the consuming projects. | +| Targeted tests | Test conversion, validation, range, and display. **Existing arithmetic tests and fixtures stay valid** because the internal representation does not change. | + +**Internal arithmetic (~200–250 sites), public type *signatures*, JSON +serialization, and the ~74 test files with ~2,400 hardcoded amount literals are +untouched. The *values* those public APIs carry are reinterpreted V1→V2, +however, so consuming projects need minimal display/validation updates (see the +"Public API value reinterpretation" row above).** + +**Approach A delta: ~6–10 dev-days.** + +## Approach B: additional work + +On top of the baseline, Approach B replaces the internal type, which cascades +across the whole library and every consumer. + +| B-specific work item | What it involves | Magnitude (from code audit) | +| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | +| Rewrite arithmetic | Every `+=`/`-=` accumulation → `.add`/`.sub`; every `===`/`!==` → `.eq`/`.ne`. (Relational `<`,`>` can survive via `Symbol.toPrimitive`, but mixing typed and untyped comparisons is a hazard.) | ~135 accumulation lines across 22 files; ~53 comparison sites | +| Change public API types | `OutputValueType` is in nearly every exported interface (`IBalance`, `IUtxo`, `getBalance()`, `sendTransaction()`, …). Swapping `bigint`→`TokenAmount` is a **breaking major** for all consumers. | ~225 references; ~120 exported symbols | +| Serialization / storage | `IStore` is implemented by consumers (mobile, desktop persist to disk). `bigint` serializes via the existing `JSONBigInt`; `TokenAmount` needs custom (de)serialization in **every** consumer store. | `JSONBigInt`, all `IStore` implementations | +| Schema coercion | The ~36 zod coercion sites must produce/accept `TokenAmount`, not `bigint`. | ~36 sites | +| Test & fixture rewrite | Fixtures use raw `bigint` literals (`value: 1n`); assertions use `===`/`toEqual(100n)`. All must wrap/unwrap. | ~74 test files, ~2,400 literal lines | +| Performance validation | Every arithmetic op now crosses the JS↔native (FFI/WASM) boundary. Summing thousands of UTXOs for a balance becomes thousands of boundary crossings, which must be benchmarked on the hot paths and on the WASM fallback (mobile/web). | balance & UTXO-selection loops | +| Downstream migration | wallet-headless, wallet-mobile, wallet-service, and the web wallet each consume these types and must migrate in lockstep with the breaking release. | 4+ external repos | + +**Approach B delta: ~32–56 dev-days in the wallet-lib alone**, plus +separate migration projects in each downstream repo. + +## Trade-off comparison + +| Dimension | Approach A (Normalize at boundary) | Approach B (`TokenAmount` everywhere) | +| -------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------- | +| **Internal amount type** | Native `bigint` (unchanged) | `TokenAmount` / `TokenBalance` wrappers | +| **Code churn** | Boundaries only (~edges of ~5 files) | Library-wide (~200–250 arithmetic sites + types + tests) | +| **Public API impact** | Type-compatible (still `bigint`), but amount *values* are reinterpreted V1→V2; consumers need minimal display/validation updates | **Breaking major** (every consumer must migrate) | +| **Type safety on amounts** | Convention-enforced ("always V2"); a stray V1 value is a logic bug | Compiler-enforced; V1/V2 mixing reconciled by Rust | +| **Drift from full node** | Conversion math is canonical (binding); arithmetic is local | Zero drift (all amount logic is the Rust crate) | +| **Operator ergonomics** | Natural `a + b`, `a > b`, `a === b` retained | `a.add(b)`, `a.eq(b)`; only `<`/`>` stay native | +| **Runtime performance** | No per-op overhead (native bigint) | FFI/WASM crossing per op; matters on balance/UTXO loops | +| **Serialization/storage** | Unchanged (`JSONBigInt`) | Custom (de)serialization in every consumer store | +| **Binding criticality** | Boundary-only; a binding bug has limited blast radius | On every hot path; a binding/WASM bug affects everything | +| **Test/fixture impact** | Mostly reused | ~74 files / ~2,400 literals rewritten | +| **Blast radius if wrong** | Contained, reversible | Large, hard to reverse (breaking release shipped) | + +## Effort & timeline estimation + +Assumptions: figures are **dev-days** (1 engineer-week = 4 dev-days) for focused +work by an engineer familiar with the wallet-lib; calendar time assumes 1 +engineer and includes +review/QA. Ranges express genuine uncertainty; the dominant schedule risk in both +is validating the binding (native + WASM fallback) across Node, React Native, and +the browser. The wire-format item is **gated on the core's encoding decision** +and may shift if that lands later. + +| | Approach A | Approach B | +| --------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------ | +| Shared baseline | 16–24 dev-days | 16–24 dev-days | +| Approach-specific delta | 6–10 dev-days | 32–56 dev-days | +| **Total (wallet-lib only)** | **~22–34 dev-days** | **~48–80 dev-days** | +| Calendar (1 engineer) | ~1.5–2 months | ~3–5 months | +| Downstream consumer work | Minimal but non-zero: amount values reinterpreted V1→V2, consumers update decimal handling | **Separate migration in 4+ repos** (headless, mobile, service, web), likely adding 1–3 months org-wide | +| Confidence | Medium-High | Medium-Low (rewrite + cross-repo coordination widen the variance) | + +**Reading the numbers:** in the wallet-lib alone, Approach A is roughly **2–3× +cheaper** than Approach B. Approach B also carries mandatory downstream +migrations (a breaking release across 4+ repos) that Approach A does not, and a +larger estimate variance. These figures are inputs to the decision, not a verdict; +the choice also weighs the type-safety, drift, and performance dimensions in +the trade-off table above. + +# Drawbacks +[drawbacks]: #drawbacks + +**Approach A drawbacks** + +- Type safety is by *convention* ("internally always V2"), not by the compiler. + A V1 value that slips past a boundary is an ordinary `bigint` and will compute + silently-wrong results. Mitigation: centralize conversion at the edges, lint + for raw amount construction, and test boundaries hard. +- Conversion logic in the wallet-lib (deciding *where* to normalize) is local + code, not the Rust crate, so the *placement* of conversions can drift even + though the conversion *math* cannot. +- Not fully non-breaking at the public API. The amount integers emitted/accepted + by public methods change meaning when V2 becomes the default (integer `1`: + `0.01` → `0.000000000000000001`). The `bigint` type is unchanged, so it + compiles, but every consumer that formats or validates amounts with 2 decimals + must be updated. The change is minimal but must be coordinated across the + consuming projects. + +**Approach B drawbacks** + +- A breaking major release forces every downstream consumer to migrate in + lockstep, a large coordination cost with real release-management risk. +- Per-operation FFI/WASM overhead on hot paths (balance aggregation, UTXO + selection) is a genuine performance concern, especially via the WASM fallback + on mobile and web. +- Serialization becomes a cross-repo problem: every `IStore` implementation must + (de)serialize the wrapper type instead of relying on the existing `JSONBigInt`. +- The large surface area (~200–250 sites, ~74 test files) raises the chance of a + subtle arithmetic-translation bug surviving into a shipped breaking release. + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +This section frames *how to weigh* the two approaches without picking one; the +decision belongs to the team reading this document. + +- **The case for Approach A.** The wallet-lib already standardized on `bigint`, + and V2 is largely a *decimal-semantics and boundary* problem rather than a + *numeric-type* problem. Approach A addresses that with a contained, reversible + change, while still using the canonical binding for the part that must not + drift: the conversion math. It is the smaller, lower-variance, cheaper path, + and its downstream impact is limited to a minimal value-reinterpretation update + (the `bigint` type signatures do not change). Its costs are giving up + compiler-enforced V1/V2 type safety in favor of a convention, and that the + public-API value shift, though small, still has to be coordinated with + consuming projects. +- **The case for Approach B.** It buys compiler-enforced V1/V2 type safety and + zero drift on *all* amount logic, not just conversion, valuable if amount + bugs are considered high-severity and worth strong static guarantees. The cost + is a library-wide rewrite, a breaking release across four-plus repos, a + per-operation performance tax on hot paths, and a cross-repo serialization + change. +- **A hybrid path.** Start with Approach A and *optionally* introduce + `TokenAmount` later in narrow, high-value spots (e.g. a public conversion API + surface) without rewriting internal arithmetic. This captures part of B's + type-safety benefit at the boundary while keeping A's internals; a door A + leaves open that B does not. Listed as an option, not a recommendation. +- **Impact of doing nothing.** The wallet-lib cannot represent, validate, or + display V2 amounts, blocking V2 adoption for every wallet that depends on it. + +# Prior art +[prior-art]: #prior-art + +- **`htr-lib-py` / full node.** The Rust crate normalizes internally to V2 and + exposes conversions, exactly the model Approach A mirrors in JS (work in the + normalized form, convert at the edges). +- **Ethereum JS ecosystem (`ethers.js`, `viem`).** The dominant pattern is + native `bigint` for amounts plus helper functions (`parseUnits`/`formatUnits`) + for decimal conversion at the boundary, i.e. Approach A's shape. Wrapper-class + amount types (closer to Approach B) exist but did not become the ecosystem + norm, partly because of the operator-overloading gap this RFC describes. +- **The wallet-lib's own `number`→`bigint` migration (Dec 2024).** A prior + library-wide amount-type change; its scope is a useful real-world calibration + for how invasive Approach B's type swap would be. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- The wire-format encoding for V2 values (and the new maximum output value) is + decided by the core's decimal-precision work; the wallet-lib's serialization + estimate assumes that lands and may shift if it is delayed. +- Exactly *where* per-token decimals are sourced (full node metadata vs. + token-creation data) needs to be pinned down for the per-token decimals item. +- For Approach A: which boundaries still legitimately emit V1 (and therefore need + V2→V1 egress conversion with truncation handling) vs. boundaries that can be + V2-only. +- Whether the `@hathor/htr-lib` WASM fallback meets performance and bundle-size + needs on React Native and the browser, a prerequisite for **both** approaches, + but a far more load-bearing one for Approach B. + +# Future possibilities +[future-possibilities]: #future-possibilities + +- **Incremental `TokenAmount` adoption.** Having shipped Approach A, narrow + high-value surfaces (public conversion APIs, a typed amount in new features) + can adopt `TokenAmount` opportunistically, without a big-bang rewrite, keeping + the door to B's type-safety benefits open at low cost. +- **Shared validation across the stack.** As more of the stack (wallet-service, + headless) consumes `@hathor/htr-lib`, V1/V2 validation and conversion converge + on one implementation, reducing the surface for cross-service inconsistencies. +- **Web3-style helpers.** A thin `parseUnits`/`formatUnits`-style layer over the + binding would give wallet developers a familiar, ergonomic API for V2 amounts + regardless of the internal representation chosen here. diff --git a/projects/hathor-wallet-lib/token_amount_v2/0002-token-amount-v2-on-ledger.md b/projects/hathor-wallet-lib/token_amount_v2/0002-token-amount-v2-on-ledger.md new file mode 100644 index 0000000..2394e84 --- /dev/null +++ b/projects/hathor-wallet-lib/token_amount_v2/0002-token-amount-v2-on-ledger.md @@ -0,0 +1,61 @@ +# Effort Estimation: Token Amount v2 Serialization in the Hathor Ledger App + +## TL;DR + +**Estimated effort: ~7–8 developer-days (≈1.5 weeks for one developer)**, plus Ledger's +external app-review cycle (calendar weeks, low effort on our side). The wire-format change +itself is small and well-localized, but one consequence dominates the estimate: **v2 amounts +can be up to 15 bytes (117 bits), so the app's `uint64_t` value representation and all +decimal formatting must be rewritten for 128-bit arithmetic with 18 decimal places.** +Dropping v1 support entirely (as planned) keeps this from being worse — there's no +dual-path code. + +## The v2 protocol (from hathor-core PR #1706 / RFC #113) + +The encoding is a length-prefixed big-endian integer: + +| Field | Size | Rules | +| ------- | -------------- | ----------------------------------------------------------------- | +| length | 1 byte | 1–15 (`0x0F` max); 0 only allowed in non-strict mode (zero value) | +| payload | `length` bytes | Big-endian, canonical (first byte must not be `0x00`), value > 0 | + +- **Max value:** `2^63 × 10^16` = 92,233,720,368,547,758,080,000,000,000,000,000 (15 bytes / 117 bits) +- **Decimals:** 18 places instead of v1's 2 (same max *token* amount as v1, 16 extra digits of precision) +- Decoder must reject: length > 15, leading-zero payloads (non-canonical), values above max, truncated payloads, and (strict) zero values. + +## Current state of the ledger app + +The value handling is tightly localized — `output.value` is read in exactly one place and consumed in exactly one place: + +- **Parse:** `parse_output_value()` in `src/transaction/deserialize.c:41-64` decodes the v1 4-or-8-byte sign-bit format into a `uint64_t`. +- **Display:** `format_value()` in `src/common/format.c:147-176` formats that `uint64_t` with hardcoded 2 decimals into a 30-char buffer (`display.c:331`). +- The signing flow (`sign_tx.c`) only needs the parser to consume the right number of bytes; the signature is over the raw stream, so it's unaffected beyond the decode call. +- Tests: Python integration client (`tests/app_client/transaction.py` mirrors the v1 encoding), CMocka unit tests, and a libFuzzer tx-parser harness (`fuzzing/fuzz_tx_parser.cc`) all encode v1 amounts and need updating. +- Targets: Nano S, Nano S+, Nano X (`ledger_app.toml`). + +## Key technical findings (cost drivers) + +1. **`uint64_t` is no longer sufficient.** `tx_output_t.value` must become a 128-bit representation (two `uint64_t`s or a 16-byte big-endian array — the Ledger SDK has no native `__int128` guarantee across targets). The decode side is easy; the cost is downstream. +2. **Decimal formatting becomes the hardest task.** Rendering a 117-bit integer as a decimal string requires big-number div/mod-by-10 (or by 10^9 in limbs) — `format_value()` and `format_fpu64()` are both 64-bit-only. Add 18 decimal places, thousands separators, and the display string grows to ~45+ chars: the `g_amount[30]` buffer must grow, and on-screen pagination behavior on Nano S needs verification. A product decision is needed on trailing-zero trimming (showing `1.000000000000000000 HTR` is hostile UX). +3. **Strictness should mirror the fullnode.** The canonical-encoding check matters on a signer: two byte-encodings of the same value would produce different tx hashes, so the device should reject non-canonical input rather than normalize it. +4. **Memory impact is minor.** +8 bytes × 10 stored outputs ≈ 80 B RAM; v2 encoding is actually *smaller* on the wire for typical amounts, so the 300-byte chunk buffer is fine. Nano S (the tightest target, and deprecated in newer Ledger SDKs) should still fit. + +## Task breakdown + +| # | Task | Scope | Estimate | +|---| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | +| 1 | **Spec & design** | Confirm with the fullnode team how v2 is signaled on the wire (tx version / decimal-version field — defined in the earlier PRs in this series and RFC #113); decide zero-value and authority-output handling; decide 18-decimal display policy (trim trailing zeros?) | 0.5 d | +| 2 | **Value type migration** | Replace `uint64_t value` in `tx_output_t` with a 128-bit type; ripple through `deserialize.c`, `sign_tx.c`, `display.c`; remove v1 parse path | 1 d | +| 3 | **v2 decoder** | Rewrite `parse_output_value()`: length prefix, bounds (1–15), canonical check, max-value check, partial-chunk handling (value can now span APDU chunk boundaries at any length 2–16 bytes) | 1 d | +| 4 | **Display & formatting** | 128-bit → decimal string conversion; 18-decimal formatting with separators; resize `g_amount`; verify pagination/rendering on all 3 device targets in Speculos | 2 d | +| 5 | **Tests** | CMocka unit tests for decoder edge cases and the new formatter; update fuzz harness + seed corpus; rewrite Python client `serialize_value()`/`from_bytes()` to v2; update all integration tests and `qa.py`; regenerate any golden screens | 2–3 d | +| 6 | **Docs & release prep** | `doc/TRANSACTION.md` output format, `COMMANDS.md` if version signaling changes, CHANGELOG, app version bump (this is a breaking change — major bump) | 0.5 d | +| 7 | **Ledger review & release** | Submission to Ledger for security review and app-store deployment | low effort, **weeks of calendar lead time** | + +**Total: ~7–8 dev-days.** Tasks 2–3 and 4 can be parallelized across two developers, compressing wall-clock dev time to under a week. + +## Risks & dependencies + +- **Ledger review lead time** is out of our control. +- **\[Open question\] version signaling:** The device-side dispatch ("this tx uses v2 amounts") depends on how the version/decimal-version field is serialized, defined in the earlier parts. This must be pinned down in task 1 before task 3 starts. +- **128-bit formatting correctness** is the main defect risk, it's hand-rolled arithmetic on a signing device. Budget for exhaustive unit tests and fuzzing (already counted in task 5).