Skip to content

feat: typed StandardLineage app builder for cross-connection lineage - #1010

Draft
mitshah-atlan wants to merge 1 commit into
mainfrom
feat/standard-lineage-app-builder
Draft

feat: typed StandardLineage app builder for cross-connection lineage#1010
mitshah-atlan wants to merge 1 commit into
mainfrom
feat/standard-lineage-app-builder

Conversation

@mitshah-atlan

Copy link
Copy Markdown

✨ Description

Adds a typed StandardLineage builder for the Standard Lineage (cross-connection lineage) app,
which had none — callers had to hand-build the raw inputs dict for client.app.create/update.

Standard Lineage builds lineage across several connections of one connector, from query history
their own miners already extracted. One workflow owns a set of connections, so the defining
operation is re-scoping an existing workflow — adding a connection as it is onboarded. That is
what this PR makes a one-liner:

StandardLineage(client).add_connections(slug, ["default/bigquery/1700000002"])

Why hand-written rather than generated (added to the generator's _HAND_WRITTEN set) — two
things a UI configmap cannot express:

  1. cross_connection_qualified_names is declared str in the app's input contract but means a
    list of connection qualified names. A native list fails validateInputsAgainstContract
    server-side; it has to be json.dumps'd, and Heracles parses it back into a list for the
    manifest placeholder. connections() takes a List[str] and encodes it.
  2. Re-scoping is an update against a slug. No generated builder does updates — AppBuilder
    only has create()/run().

The re-scope path reads before it writes, and that is load-bearing rather than a convenience.
client.app.update is a full replace, and the workflow's own connection entity is republished by
the DAG's create-connection node on every run. So a rebuilt or partial connection does not merely
lose fields in the payload — the next run writes it over the real connection in Atlan, stripping its
name, admin users, admin roles, category and row limit, with no error at any point.
set_connections therefore carries the persisted connection and run_role over verbatim, read
from client.app.get(slug).dag (AppSummary tolerates unmodelled fields, so the DAG arrives as an
extra).

Surface

StandardLineage(client).connection(name=...).connections([...]).run()   # create
StandardLineage(client).add_connections(slug, [...])                    # onboard a connection
StandardLineage(client).remove_connections(slug, [...])                 # hand one back
StandardLineage(client).set_connections(slug, [...])                    # replace the scope
StandardLineage(client).get_connections(slug)                           # read the scope

add_connections / remove_connections are idempotent and return None without publishing a
version when nothing would change, so an onboarding portal can replay safely.

Validation is client-side, where the error is actionable: the app requires a non-empty,
same-connector scope, and passing the workflow's own standard-lineage connection as its scope
— a natural mistake, since both are "connections" — is rejected with a message saying which was
wanted.

Note the two distinct connectors: the workflow's own connection is minted under standard-lineage
(_CONNECTOR_NAME), while the connector input names the connector of the connections in scope and
is derived from them.

Independent of the app-side fix

Worth knowing for sequencing, but not a blocker for this PR: on tenants where the lineage app does
not yet serve GET /workflows/v1/input-contract, Heracles fails closed and client.app.update
returns 1003. That is true today with or without this PR, so the builder is safe to merge on its
own — it changes nothing for existing callers and fails identically to the hand-built payload it
replaces. The app-side change is tracked separately.

Jira link: TBD — no ticket raised yet; happy to attach one if BLDX prefers a BLDX-xxxx reference.


🧩 Type of change

  • 🚀 New feature (non-breaking change that adds functionality)
  • 🐛 Bug fix
  • 🔄 Refactor
  • 🧹 Maintenance
  • 💥 Breaking change
  • 📦 Dependency upgrade/downgrade
  • 📚 Documentation updates

Nothing existing changes behaviour: one new module, one new export, one entry added to the
generator's _HAND_WRITTEN set and to the corresponding test exclusion.


✅ How has this been tested?

Unit — 22 new tests (tests/unit/apps/test_standard_lineage.py), covering the JSON encoding,
connector derivation, all four validation refusals, both persisted-scope shapes (native list and
JSON string), idempotent no-ops, the empty-scope refusal, and — most importantly — that
set_connections sends the persisted connection verbatim.

  • app suite: 535 passed, 4 skipped
  • full unit suite: 7110 passed, 6 skipped
  • ruff check + ruff format clean

Live tenant. Exercised against a real Standard Lineage workflow with 3 BigQuery connections in
scope:

Step Result
get_connections(slug) 3 connections
remove_connections(slug, [qn]) published a new version, scope → 2
add_connections(slug, [qn]) published a new version, scope → 3
add_connections(slug, [qn]) again returned None, no version published
remove_connections(slug, [not-in-scope]) returned None, no version published
set_connections(slug, []) refused client-side

After the re-scope the re-rendered workflow was compared against the original: all 14 connection
attributes preserved
, the create-connection and publish nodes byte-identical, and the Temporal
workflow_type unchanged. That comparison is the reason set_connections reads first.

create() is implemented but deliberately not exercised on that tenant — it would mint a new
workflow and a new connection. It shares AppBuilder._create with every other builder, and its
payload is covered offline via preview().


📋 Checklist

  • My code follows the project's style guidelines
  • I've performed a self-review of my code
  • I've added comments in tricky or complex areas
  • I've updated the documentation as needed
  • There are no new warnings from my changes
  • I've added tests to cover my changes
  • All new and existing tests pass locally

Standard Lineage (cross-connection lineage) had no typed builder, so callers had
to hand-build the raw `inputs` dict for `client.app.create/update`. Two details
make that unreasonable to ask of a caller, and neither is expressible from a UI
configmap — hence hand-written, and added to the generator's _HAND_WRITTEN set:

1. `cross_connection_qualified_names` is declared `str` in the app's input
   contract but means a LIST of connection qualified names. Sending a native list
   fails validateInputsAgainstContract server-side; it has to be json.dumps'd, and
   Heracles parses it back into a list for the manifest placeholder. `connections()`
   takes a List[str] and encodes it.

2. The defining operation is re-scoping an EXISTING workflow — adding a connection
   as it is onboarded — which is an update against a slug. No generated builder
   does updates; AppBuilder only has create()/run().

The re-scope path reads before it writes, and that is load-bearing rather than a
convenience: `client.app.update` is a full replace, and the workflow's own
connection entity is republished by the DAG's create-connection node on every run,
so sending a rebuilt or partial connection would overwrite the real one in Atlan
and strip its name and admins. `set_connections` therefore carries the persisted
`connection` and `run_role` over verbatim, read from `client.app.get(slug).dag`
(AppSummary tolerates unmodelled fields, so the DAG arrives as an extra).

Surface:

    StandardLineage(client).connection(name=...).connections([...]).run()   # create
    StandardLineage(client).add_connections(slug, [...])                    # onboard
    StandardLineage(client).remove_connections(slug, [...])                 # hand back
    StandardLineage(client).set_connections(slug, [...])                    # replace
    StandardLineage(client).get_connections(slug)                           # read

add_connections/remove_connections are idempotent and return None without
publishing a version when nothing would change, so an onboarding portal can replay
safely. Validation is client-side where the error is actionable: the app requires a
non-empty, same-connector scope, and passing the workflow's OWN standard-lineage
connection as its scope — a natural mistake, since both are "connections" — is
rejected with a message that says which is wanted.

Note the two distinct connectors: the workflow's own connection is minted under
`standard-lineage` (_CONNECTOR_NAME), while the `connector` input names the
connector of the connections in scope and is derived from them.

Verified end-to-end against a live tenant (create path deliberately not exercised
there — it would mint a workflow and a connection): the read, add, remove,
idempotent no-op and empty-scope refusal all behave as specified, and the
re-rendered DAG came back with all 14 connection attributes intact, both
downstream nodes identical, and the Temporal workflow type unchanged.

22 new tests; app suite 535 passed, full unit suite 7110 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: mitshah-atlan <mit.shah@atlan.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant