feat: typed StandardLineage app builder for cross-connection lineage - #1010
Draft
mitshah-atlan wants to merge 1 commit into
Draft
feat: typed StandardLineage app builder for cross-connection lineage#1010mitshah-atlan wants to merge 1 commit into
mitshah-atlan wants to merge 1 commit into
Conversation
Standard Lineage (cross-connection lineage) had no typed builder, so callers had
to hand-build the raw `inputs` dict for `client.app.create/update`. Two details
make that unreasonable to ask of a caller, and neither is expressible from a UI
configmap — hence hand-written, and added to the generator's _HAND_WRITTEN set:
1. `cross_connection_qualified_names` is declared `str` in the app's input
contract but means a LIST of connection qualified names. Sending a native list
fails validateInputsAgainstContract server-side; it has to be json.dumps'd, and
Heracles parses it back into a list for the manifest placeholder. `connections()`
takes a List[str] and encodes it.
2. The defining operation is re-scoping an EXISTING workflow — adding a connection
as it is onboarded — which is an update against a slug. No generated builder
does updates; AppBuilder only has create()/run().
The re-scope path reads before it writes, and that is load-bearing rather than a
convenience: `client.app.update` is a full replace, and the workflow's own
connection entity is republished by the DAG's create-connection node on every run,
so sending a rebuilt or partial connection would overwrite the real one in Atlan
and strip its name and admins. `set_connections` therefore carries the persisted
`connection` and `run_role` over verbatim, read from `client.app.get(slug).dag`
(AppSummary tolerates unmodelled fields, so the DAG arrives as an extra).
Surface:
StandardLineage(client).connection(name=...).connections([...]).run() # create
StandardLineage(client).add_connections(slug, [...]) # onboard
StandardLineage(client).remove_connections(slug, [...]) # hand back
StandardLineage(client).set_connections(slug, [...]) # replace
StandardLineage(client).get_connections(slug) # read
add_connections/remove_connections are idempotent and return None without
publishing a version when nothing would change, so an onboarding portal can replay
safely. Validation is client-side where the error is actionable: the app requires a
non-empty, same-connector scope, and passing the workflow's OWN standard-lineage
connection as its scope — a natural mistake, since both are "connections" — is
rejected with a message that says which is wanted.
Note the two distinct connectors: the workflow's own connection is minted under
`standard-lineage` (_CONNECTOR_NAME), while the `connector` input names the
connector of the connections in scope and is derived from them.
Verified end-to-end against a live tenant (create path deliberately not exercised
there — it would mint a workflow and a connection): the read, add, remove,
idempotent no-op and empty-scope refusal all behave as specified, and the
re-rendered DAG came back with all 14 connection attributes intact, both
downstream nodes identical, and the Temporal workflow type unchanged.
22 new tests; app suite 535 passed, full unit suite 7110 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: mitshah-atlan <mit.shah@atlan.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
✨ Description
Adds a typed
StandardLineagebuilder for the Standard Lineage (cross-connection lineage) app,which had none — callers had to hand-build the raw
inputsdict forclient.app.create/update.Standard Lineage builds lineage across several connections of one connector, from query history
their own miners already extracted. One workflow owns a set of connections, so the defining
operation is re-scoping an existing workflow — adding a connection as it is onboarded. That is
what this PR makes a one-liner:
Why hand-written rather than generated (added to the generator's
_HAND_WRITTENset) — twothings a UI configmap cannot express:
cross_connection_qualified_namesis declaredstrin the app's input contract but means alist of connection qualified names. A native list fails
validateInputsAgainstContractserver-side; it has to be
json.dumps'd, and Heracles parses it back into a list for themanifest placeholder.
connections()takes aList[str]and encodes it.AppBuilderonly has
create()/run().The re-scope path reads before it writes, and that is load-bearing rather than a convenience.
client.app.updateis a full replace, and the workflow's own connection entity is republished bythe DAG's
create-connectionnode on every run. So a rebuilt or partial connection does not merelylose fields in the payload — the next run writes it over the real connection in Atlan, stripping its
name, admin users, admin roles, category and row limit, with no error at any point.
set_connectionstherefore carries the persistedconnectionandrun_roleover verbatim, readfrom
client.app.get(slug).dag(AppSummarytolerates unmodelled fields, so the DAG arrives as anextra).
Surface
add_connections/remove_connectionsare idempotent and returnNonewithout publishing aversion when nothing would change, so an onboarding portal can replay safely.
Validation is client-side, where the error is actionable: the app requires a non-empty,
same-connector scope, and passing the workflow's own
standard-lineageconnection as its scope— a natural mistake, since both are "connections" — is rejected with a message saying which was
wanted.
Note the two distinct connectors: the workflow's own connection is minted under
standard-lineage(
_CONNECTOR_NAME), while theconnectorinput names the connector of the connections in scope andis derived from them.
Independent of the app-side fix
Worth knowing for sequencing, but not a blocker for this PR: on tenants where the lineage app does
not yet serve
GET /workflows/v1/input-contract, Heracles fails closed andclient.app.updatereturns
1003. That is true today with or without this PR, so the builder is safe to merge on itsown — it changes nothing for existing callers and fails identically to the hand-built payload it
replaces. The app-side change is tracked separately.
Jira link: TBD — no ticket raised yet; happy to attach one if BLDX prefers a BLDX-xxxx reference.
🧩 Type of change
Nothing existing changes behaviour: one new module, one new export, one entry added to the
generator's
_HAND_WRITTENset and to the corresponding test exclusion.✅ How has this been tested?
Unit — 22 new tests (
tests/unit/apps/test_standard_lineage.py), covering the JSON encoding,connector derivation, all four validation refusals, both persisted-scope shapes (native list and
JSON string), idempotent no-ops, the empty-scope refusal, and — most importantly — that
set_connectionssends the persisted connection verbatim.535 passed, 4 skipped7110 passed, 6 skippedruff check+ruff formatcleanLive tenant. Exercised against a real Standard Lineage workflow with 3 BigQuery connections in
scope:
get_connections(slug)remove_connections(slug, [qn])add_connections(slug, [qn])add_connections(slug, [qn])againNone, no version publishedremove_connections(slug, [not-in-scope])None, no version publishedset_connections(slug, [])After the re-scope the re-rendered workflow was compared against the original: all 14 connection
attributes preserved, the
create-connectionandpublishnodes byte-identical, and the Temporalworkflow_typeunchanged. That comparison is the reasonset_connectionsreads first.create()is implemented but deliberately not exercised on that tenant — it would mint a newworkflow and a new connection. It shares
AppBuilder._createwith every other builder, and itspayload is covered offline via
preview().📋 Checklist