M kovalsky/sm to osi - #329
Open
m-kovalsky wants to merge 46 commits into
Open
Conversation
…-valid
The initial `converters/microsoft` drop could not be imported, installed or
validated. This fixes the three blocking issues and adds the tests that catch
them:
1. Broken public API. `__init__.py` re-exported `semantic_model_to_ossie`
while the module defines `convert_semantic_model_to_ossie`, so importing
the package raised ImportError.
2. Not a package. The sources sat directly in `src/` and both
`pyproject.toml` and `README.md` were empty, so the converter could not
be installed, run or tested. Sources now live in `src/ossie_microsoft/`
(matching the other Python converters), with a populated pyproject, an
`ossie-microsoft` CLI, a README documenting the mapping and limitations,
ASF license headers and a CI workflow.
3. Output did not validate. The converter emits a `DAX` dialect that was not
part of the spec, so every generated document failed schema validation:
[Schema] ... -> dialect: 'DAX' is not one of ['ANSI_SQL', 'SNOWFLAKE',
'MDX', 'TABLEAU', 'DATABRICKS', 'MAQL', 'BIGQUERY']
`DAX` is added to the Dialect enum in osi-schema.json, spec.yaml, spec.md
and OSIDialect, and to validate.py's skip list because sqlglot cannot parse
DAX. `POWER_BI` is registered as a vendor name for custom extensions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The TMSL -> Apache Ossie importer previously dropped everything that had no Apache Ossie counterpart, and did so silently. Rework it around a new shared module so a conversion is auditable and, where possible, reversible: * Add `ossie_microsoft/_common.py` with the constants, the bidirectional data type mapping and the `custom_extensions` stash protocol shared by both conversion directions, mirroring `converters/databricks`. * Route every lossy step through `warn()` so callers can escalate warnings to errors and get a hard guarantee that a conversion was lossless. * Preserve unrepresentable TMSL properties (annotations, partitions, hierarchies, roles, perspectives, cultures, format strings, display folders, KPIs, cross-filter behaviour) plus excluded tables and skipped relationships in a versioned POWER_BI `custom_extensions` blob. * Correct the data type mapping. TMSL has no date-only, time-only or timezone-aware member, so the previous 'date'/'time' entries could never fire; date-only intent is now detected from the format string instead. `double` maps to Float rather than Decimal, `binary`/`variant` map to Opaque with a warning, and `automatic`/`unknown` omit the type instead of inventing one. * Record a measure's home table and a flipped relationship's original orientation so the export direction can rebuild the model faithfully. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The converter was import-only. Add `ossie_to_semantic_model` so an Apache Ossie model can be materialized as a Power BI `model.bim`, completing the round trip, and expose it through the package and an `export` subcommand. Expressions are never rewritten between languages. A metric is emitted as a measure only when it carries a DAX expression; a field becomes a `sourceColumn` only when its expression is a plain column reference. A SQL aggregate mechanically rewritten into DAX ignores filter context, so it would yield a measure that is wrong rather than one that is missing -- those cases warn and are skipped instead. Constructs Power BI cannot express are reported rather than approximated: composite keys and composite relationships have no equivalent, `Opaque` has no data type, and `Time`/`DateTimeTz` collapse onto `dateTime`. A dataset with no preserved partition gets a placeholder that raises an M `error` naming the missing source, so a refresh fails with an actionable message instead of loading nothing. A `model.bim` converted to Apache Ossie and back reproduces the original model, including the tables, relationships and properties the import left out of the vendor-neutral document. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extend the suite from 26 to 82 tests, adding the export direction and the behaviour introduced with it: * A round-trip test asserting a `model.bim` converted to Apache Ossie and back is structurally the same model, and a second test asserting a further pass is a fixed point, so the pipeline cannot drift. * Explicit tests for each construct the converter refuses to guess at: computed SQL expressions, metrics with no DAX, composite keys, composite relationships and relationships with a missing endpoint. * A test that escalating warnings to errors passes for a model with no Power BI specifics, which is the guarantee callers rely on to prove a conversion was lossless. * Full data type coverage, including date-only detection from a format string and the quoted-literal case where 'h' is display text rather than an hour token. The fixture gains column and table annotations, a format string, a display folder and a summarizeBy so the passthrough paths are exercised on real data. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The README described an import-only converter. Rewrite it around what the package now does and, more importantly, why it refuses to do certain things: * State the design rule -- expressions are carried across in the dialect they were authored in and never machine-translated -- and explain the failure mode it avoids, namely a model that loads and returns the wrong number rather than one that visibly lacks a measure. * Document the losslessness contract: what is preserved in the POWER_BI stash, and how to escalate warnings to errors to prove a conversion lost nothing. * Add the data type table with the caveats that actually bite -- TMSL has no date-only, time-only or timezone-aware type, date-only intent travels in the format string, 'm' means month in a VBA-style format unless it follows an hour token, and 'double' is approximate so it maps to Float not Decimal. * Tabulate the constructs that have no equivalent in either direction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A named format such as `Short Date` is a whole-string name, not a sequence of tokens, so tokenizing it misreads the `h` in "Short" as an hour token and the `n` in "Long" as a minute token. Both were classified as having a time part, so a date-only column round-tripped as DateTime. Match the named formats before tokenizing, and cover them plus the numeric formats that must not be mistaken for dates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two ways the losslessness guarantee leaked, both found in review: * The stash used per-level allow-lists, so any TMSL property not named in them was dropped without a warning -- keepUniqueRows, alignment, displayOrdinal and sourceProviderType among them, and any property a future TMSL version adds. Invert them into deny-lists keyed on what the Apache Ossie mapping actually consumes, so an unrecognized property is preserved by default rather than lost by default. * Relationship cardinalities were recorded only when the import had to flip a one-to-many relationship. A one-to-one relationship therefore exported without cardinalities and picked up the TMSL many-to-one defaults, silently widening it. Record cardinalities whenever the source states them. Also preserve the two things the import legitimately sets aside: rowNumber columns, and a dataType with no portable equivalent (binary, variant, automatic, unknown) so the export restores the original TMSL type instead of guessing one back from the portable model. The test fixture now round-trips structurally identical through bim -> Apache Ossie -> bim. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two more ways the stash could produce a wrong model, both found in review: * The replay loops assigned stashed values over properties the mapping had already derived. Importing a `type: data` column, editing its Ossie expression to DAX and exporting produced a column carrying both `type: data` and an `expression`, which is contradictory TMSL. Replay with setdefault throughout, so the core document is the source of truth whenever the two disagree and preserved values only fill gaps. * TMSL allows a description on both the document and the model, but Apache Ossie has one. A document-level description was moved onto the model, and when both were present the document one was dropped. Record which one the Apache Ossie description came from and keep the other verbatim, so both return to where they were authored. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Its test suite validates converter output against core-spec/osi-schema.json, so a schema change can break it without touching converters/microsoft. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A measure's result type is inferred by the engine from its DAX expression; TMSL has no writable dataType on a measure, so emitting one risks Analysis Services rejecting the model. The export derived it from the Apache Ossie metric datatype, and the test fixture carried the property, which made the round-trip tests treat it as legitimate. The export now warns instead of emitting, and the import preserves a dataType verbatim in the stash if a source model happens to carry one, rather than reconstructing it from the portable type. The measure keeps its Apache Ossie datatype for consumers that want the hint; it is simply never pushed back into TMSL. The fixture drops the property. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The export skipped the data type mapping entirely whenever the stash held a TMSL type, so a preserved `binary`, `variant`, `automatic` or `unknown` was replayed even after someone had edited the Apache Ossie datatype to something else. That is the one remaining place the stash outranked the core document. Replay the stashed type only while the portable type still agrees with it, which is exactly the condition under which it was stashed in the first place. Once the core datatype changes, it is authoritative. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A TMSL sourceColumn names a column in the table's source query, which may be spelled with spaces, a hyphen or a leading digit -- "Order Date" is ordinary in a Power BI model. The export only accepted a bare SQL identifier, so any such column was treated as a computed expression and dropped, taking a valid column out of the model on a plain round trip. Preserve a sourceColumn that is not a bare identifier and replay it while the field expression still matches it. An edited expression continues to take precedence, consistent with the rest of the stash. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Power BI stash carries a format version, but it was discarded without being read. A payload written by a later converter would have been replayed under this converter's assumptions, which is precisely the silent-wrong-answer outcome the design rule exists to prevent. Reject a version this converter does not understand, and a non-integer version, with a message that says which version was found and what to do. An absent version is still read as the current format, so documents written before this change keep working. Also record an Analysis Services deployment smoke test in the roadmap: these tests check the shape of the TMSL, not that the engine accepts it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Round-trip losslessness was already guaranteed by the Power BI stash, and that guarantee hid something: an Apache Ossie document is meant to be read by tools that are not Power BI, and to those tools a preserved-but-unmapped construct is simply absent. Row-level security roles, perspectives, translations, hierarchies, KPIs, calculation groups, sort-by-columns and date table variations all crossed over in silence. Report them, at the level where they occur, saying which model has no counterpart and what happened to it. The import says "preserved for round trip but not represented"; the export says "dropped", because a TMSL document has nowhere to keep an Ossie construct -- ai_context and label were being discarded with no signal at all. Purely presentational properties (isHidden, displayFolder, formatString) stay unreported on purpose: a warning on every cosmetic property would bury the ones that matter. Each report now goes to both a UserWarning and the ossie_microsoft logger. The two answer different questions -- a warning filter gives a caller a hard programmatic guarantee of losslessness, while a log serves an application that never installs one. warn() also gained stacklevel=2 so a -W error traceback names the conversion call site instead of the helper. The CLI grows --strict, which exits non-zero if anything could not be carried across faithfully, and -q/--quiet. Both are accepted before or after the subcommand, because that is where people type them; argparse needed SUPPRESS defaults on the subcommand copies to stop them overwriting an already-parsed value. The CLI silences the warning channel so each message prints once. The fixture gains all of these constructs, which proves both halves at once: every one is reported, and the model still round-trips structurally identical. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The converter's contract is that every lossy branch reports itself, which is only worth as much as the tests that exercise it. Nothing enforced either the lint rules the rest of the repo uses or the coverage that makes the contract credible. Adopt ruff with the rule set converters/gooddata and converters/orionbelt already use, and enforce branch coverage at 95% (currently 97%). The bar is deliberately high: an unexercised branch is an unproven warning. CI now lints, measures coverage, round-trips the fixture through the installed console script -- which is the only step that proves the entry point is wired up -- and validates the result against the core spec. pytest's default warning filter is set to ignore, so the suite runs clean with a bare `pytest`. This does not weaken anything: the tests that assert a lossless conversion is silent turn warnings into errors themselves, locally and explicitly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ossie into m-kovalsky/sm_to_osi
The converter had been flattened from src/ossie_microsoft/*.py to loose
modules directly under src/. That is off-convention here (all nine Python
converters use src/<package>/) and it broke the distribution in ways the
unit tests could not see, because they import through pythonpath = ["src"]
and so pass under either layout:
* the wheel shipped _common.py, cli.py, ossie_to_semantic_model.py and
semantic_model_to_ossie.py as top-level modules, squatting on generic
names in every environment that installed it, while omitting
__init__.py entirely, so `import ossie_microsoft` failed after install;
* the console script pointed at "cli:main", which no longer resolved;
* coverage measured flat module names that no longer existed.
Move the modules back under src/ossie_microsoft/, restore relative sibling
imports so the package does not depend on src/ being on sys.path, and
point the console script, wheel target and coverage source at the package.
Add tests/test_packaging.py to keep this from regressing silently. Beyond
checking the declarations, it builds the wheel and inspects it, since the
declarations can be correct while the built artifact is not. All fourteen
of its assertions fail on the flattened layout.
Also credit the individual contributors via [project] maintainers, leaving
authors as the ASF to match the sibling converters.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Conflicts: # converters/README.md
The importer had been rewritten in a way that dropped the vendor stash
entirely: it no longer called write_stash, so every TMSL construct without
an Apache Ossie counterpart was silently discarded instead of preserved.
That contradicted the README, the exporter and 54 tests. The exporter was
never changed to match and still reads the stash at six sites, so the two
directions had simply come apart.
Restore the stash-aware importer and carry the newer work forward onto it
rather than the other way around, since the newer file was a strict subset:
* calculated tables are excluded from the model, and the reason a table
was excluded is now reported specifically ("calculated tables are not
converted to Apache Ossie") instead of listing every rule that might
have applied;
* relationship cardinalities are compared case-insensitively.
Roles, perspectives, cultures, shared expressions, query groups, excluded
tables and per-column, per-measure and per-relationship properties are
preserved again, and sales_model.bim now round trips with all six tables
and every model-level construct intact.
Two problems found while restoring this:
Expression annotations were written unconditionally, including for DAX.
DAX goes straight into the TMSL expression property, so the annotation
only duplicated it and made a model that had merely round tripped differ
from the original. Annotate only when the authored dialect is not DAX,
which is the case where the text would otherwise be lost.
A delimited table reference such as "my schema"."my table" was rejected by
the bare-identifier rule and fell through to the query branch, which
emitted Sql.Database(..., [Query="""my schema"".""my table"""]) - not
valid SQL, and exactly the kind of plausible-looking expression the
converter is not supposed to invent. Hold only undelimited parts to the
identifier rule.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Power BI evaluates a measure or calculated column only as DAX; TMSL has no property that can carry an expression in another dialect. The exporter handled that by writing BLANK() into `expression` and parking the authored text in OssieExpression / OssieExpressionDialect annotations. That produces a model which deploys and refreshes without error and then answers every query involving the object with a wrong number. Nothing in the Power BI experience surfaces the annotations, so the failure is silent. An absent measure, by contrast, is something a modeller notices immediately. A non-DAX expression is now reported through `warn` and the object is skipped, matching how the converter already handles every other construct it cannot represent. The authored expression is untouched in the Apache Ossie source, so re-running the conversion after adding a DAX expression picks it up. The annotation pair existed only to accompany the stand-in and nothing read it back, so it is removed along with it. `OssieAIContext` is unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Power BI evaluates a measure only as DAX, so a metric authored in SQL was
skipped outright. That is safe but unhelpful: the most common metrics in an
Apache Ossie model are a single aggregate over one column, and those have an
unambiguous DAX equivalent.
Adds a translator covering exactly that tier -- SUM/MIN/MAX/COUNT/AVG/MEDIAN,
the sample and population STDDEV/VARIANCE forms, COUNT(DISTINCT x) and
COUNT(*) -- reading ANSI_SQL, Snowflake, Databricks and BigQuery via sqlglot,
which is already a runtime dependency of the dbt and gsf converters.
The mapping is not a passthrough: DAX renames AVG to AVERAGE, STDDEV to
STDEV.S, VARIANCE to VAR.S and COUNT(DISTINCT x) to DISTINCTCOUNT, so emitting
the SQL spelling unchanged would be silently wrong. The mappings follow the
table in core-spec/expression_language.md.
DAX also has no bare column reference, so a metric only translates when its
column resolves to exactly one field in exactly one dataset. The SQL names the
physical column, which TMSL carries as sourceColumn, while DAX addresses the
column by its model name -- so the translation maps between the two. A name
found in two datasets is refused rather than guessed at.
Everything outside the curated set is reported and skipped as before:
arithmetic between aggregates, aggregates over expressions, CASE, windowed and
filtered aggregates, qualified references, and percentiles, whose DAX spelling
depends on an interpolation the SQL does not state. Calculated fields are still
never translated, because they evaluate in row context where SUM('T'[X]) would
return the whole-column total on every row.
Tests weight refusal over breadth, since a missed translation is a nuisance
while a wrong one deploys cleanly and reports bad numbers. Every emitted form,
including bracket and quote escaping, was checked to parse as valid DAX.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Review of the previous commit found three defects in the SQL-to-DAX mappings, two of which produced silently wrong numbers -- exactly what the converter's design rule exists to prevent. SQL COUNT(DISTINCT x) excludes NULL, but DAX DISTINCTCOUNT counts BLANK as a distinct value, so the measure was off by one on any nullable column. Now DISTINCTCOUNTNOBLANK. SQL COUNT(x) counts non-NULL values of any type, but DAX COUNT documents TRUE/FALSE columns as unsupported, so a boolean column would fail rather than count. COUNTA counts non-blank values of any type and is the faithful equivalent. Now COUNTA. Both mappings therefore depart from the summary table in core-spec/expression_language.md, which does not account for NULL and BLANK differing. The deviation is documented where the mapping is declared. Snowflake and Databricks also accept COUNT(a, b), counting rows where every argument is non-NULL. sqlglot exposes the first argument as `this` and the rest as `expressions`, so reading only `this` quietly dropped the others and overcounted. Any aggregate carrying extra arguments is now refused. Two further divergences are left in place and documented, because both fail visibly rather than returning a plausible wrong figure: DAX counts return BLANK over an empty set where SQL returns 0, and the DAX deviation and variance functions error below two non-blank rows where SQL yields NULL or 0. Also adds the ASF licence header, which the new module was missing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Ossie -> Power BI converter work surfaced several mappings in the DAX column of the cross-reference tables that are not equivalent to the SQL they claim to map, in ways that produce a model which loads cleanly and then reports wrong numbers. Aggregation table: * COUNT(x) mapped to DAX COUNT. Both count non-blank values, but DAX COUNT documents TRUE/FALSE columns as unsupported, whereas SQL COUNT(x) counts non-NULL values of any type. COUNTA is the faithful equivalent. * COUNT(DISTINCT x) mapped to DISTINCTCOUNT. SQL excludes NULL from the distinct set, but DISTINCTCOUNT counts BLANK as a distinct value, so the measure is off by one on any nullable column. DISTINCTCOUNTNOBLANK is the equivalent. Date table: * DATEADD(day, n, d) mapped to `DATE(d) + n` or `DATEADD(d, n, DAY)`. Both are wrong. DAX DATE takes three arguments (year, month, day), so DATE(d) is not a valid call. DAX DATEADD is a time-intelligence function that returns a table of shifted dates for the current filter context and requires a marked date table with a contiguous date range -- it is not a scalar row-level add. The scalar equivalents are `d + n` for days and EDATE(d, n) for months. Also records two differences that have no faithful DAX equivalent, so that implementers expect them rather than working around them: the DAX count functions return BLANK over an empty set where SQL returns 0, and the deviation and variance functions error below two non-blank rows where SQL yields NULL or 0. And notes that the DAX column's `x` stands for a qualified `Table[Column]` reference, since DAX has no bare column reference at all. Every corrected DAX form was checked to parse against the Power BI DAX grammar; the semantics follow the Microsoft DAX function reference. The Power BI converter's own documentation is updated in the same change, since it previously called out these two mappings as departures from this table. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The optional TOM job was pinned to windows-latest because Linux could not be verified locally. It can. The Microsoft.AnalysisServices assemblies target net8.0, and the package's only native payload is the MSAL authentication broker, which offline validation never loads. Verified in a clean ubuntu:24.04 container, not just by inspection: scripts/restore_tom.py performed a live NuGet restore on Linux, the CI step 'pytest tests/test_tom_integration.py' passed 3/3, the full suite passed 274 with coverage at 96.24%, and the sales fixture, the CLI round trip and the TPC-DS export all validated clean. A negative control confirmed the validator still rejects a reverted compatibility level, a removed hierarchy and a dangling relationship column. Note for the runner image: .NET requires libicu, which the GitHub ubuntu-latest image already provides. Also aligns the uv install step with the convention already used by the build job in this workflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ruff's import-sort rule (I001) requires a blank line between the standard library and third-party groups, so 'ruff check' failed on the lint step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Offline TOM checks TMSL structure and object references, but it does not parse DAX at all: a measure referencing a missing column, calling an unknown function, or left with an unbalanced parenthesis loads clean. Only the engine compiles DAX. validate_with_engine deploys a model to a Fabric workspace, refreshes it (which is what compiles the DAX), evaluates every table and measure, and deletes it again. Each partition is rewritten as an inline M literal of generated sample rows, so the refresh needs no gateway, lakehouse or stored credential while tables, columns, relationships and measures stay untouched -- what the engine compiles is the DAX the converter produced. A measure whose DAX fails to compile is dropped from the deployed model, so asking for it by name returns no rows rather than an error. The expression is re-evaluated inline to recover the real diagnostic; a silent empty result is the failure mode this package exists to prevent. Running this against the test fixture found three defects that both the existing tests and offline TOM accepted, because the tests only compare dictionaries and never load the model into an engine: - relationship e5f6a7b8 declared a From end cardinality of One, which the engine rejects unless the relationship is one-to-one; - the Calendar[Date] date variation named a relationship defined on another table, and its target table lacked showAsVariationsOnly; - the LocalDateTable hierarchy levels had no ordinals. The one-to-many relationship the fixture used to exercise the import flip is now authored in a dedicated test model instead, so that coverage is kept while the shared fixture stays deployable. Live validation is the only validation here that leaves the local machine: it creates real items and consumes capacity, so it is opt-in, outside the default test run and deliberately not wired into CI. Its tests stub the transport and touch no network. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Align the DAX column of the expression mapping tables with DAX semantics
There was a problem hiding this comment.
Pull request overview
This PR adds a new Microsoft Power BI / Fabric semantic model (TMSL/TMDL) ↔ Ossie converter, expands the core spec to recognize DAX as an expression dialect, and wires up CI plus extensive tests/fixtures/documentation for the new converter.
Changes:
- Add
DAXas a first-class dialect in the spec/schema/Python models and skip SQL validation for DAX in the validator. - Introduce the
converters/microsoftPython package with import/export logic, optional TOM-based offline validation/TMDL serialization, and optional live engine-backed validation. - Add a dedicated GitHub Actions workflow to lint/test/coverage the converter and validate output against the core spec.
Reviewed changes
Copilot reviewed 32 out of 34 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
validation/validate.py |
Adds DAX to the dialect map and SQL-validation skip list. |
ROADMAP.md |
Adds Microsoft converter entry to the roadmap list. |
python/src/ossie/models.py |
Adds DAX dialect and introduces a Power BI vendor token. |
core-spec/spec.yaml |
Adds DAX to dialects and updates vendor examples. |
core-spec/spec.md |
Documents DAX dialect and adds vendor example for Power BI extensions. |
core-spec/osi-schema.json |
Adds DAX to dialect enum and extends vendor examples. |
core-spec/expression_language.md |
Updates DAX function mappings and adds explanatory notes. |
converters/README.md |
Adds Power BI/Fabric to the supported vendor extensions list. |
converters/microsoft/tests/test.ipynb |
Adds an exploratory notebook demonstrating conversions. |
converters/microsoft/tests/test_tom.py |
Adds unit tests for TOM helper module behavior. |
converters/microsoft/tests/test_tom_integration.py |
Adds integration tests gated on optional TOM runtime/assemblies. |
converters/microsoft/tests/test_sql_to_dax.py |
Adds tests for curated SQL→DAX translation behavior. |
converters/microsoft/tests/test_semantic_model_to_ossie.py |
Adds extensive tests for Power BI→Ossie conversion and stashing. |
converters/microsoft/tests/test_packaging.py |
Adds packaging/layout tests to prevent wheel/module regressions. |
converters/microsoft/tests/test_engine.py |
Adds offline tests for engine-backed validation orchestration and transport. |
converters/microsoft/tests/test_edge_cases.py |
Adds defensive/error-path tests for both converter directions. |
converters/microsoft/tests/fixtures/sales_model.tmdl |
Adds a TMDL fixture for round-trip and TOM parsing coverage. |
converters/microsoft/tests/fixtures/sales_model.bim |
Adds a TMSL fixture to exercise conversion/stash behavior. |
converters/microsoft/tests/conftest.py |
Adds shared fixtures for the Microsoft converter test suite. |
converters/microsoft/src/ossie_microsoft/tom.py |
Implements optional TOM loading, offline validation, and TMDL (de)serialization. |
converters/microsoft/src/ossie_microsoft/semantic_model_to_ossie.py |
Implements Power BI semantic model → Ossie conversion and stashing. |
converters/microsoft/src/ossie_microsoft/ossie_to_semantic_model.py |
Implements Ossie → Power BI semantic model conversion, including Direct Lake defaults. |
converters/microsoft/src/ossie_microsoft/engine.py |
Implements optional live engine-backed model validation via Fabric/Power BI APIs. |
converters/microsoft/src/ossie_microsoft/cli.py |
Adds ossie-microsoft CLI with strict/quiet modes and reporting. |
converters/microsoft/src/ossie_microsoft/_sql_to_dax.py |
Adds curated, refusal-first SQL→DAX translation utilities. |
converters/microsoft/src/ossie_microsoft/_common.py |
Adds shared constants, type mappings, warning/reporting, and stash protocol. |
converters/microsoft/src/ossie_microsoft/__init__.py |
Exposes the public API surface for the converter package. |
converters/microsoft/scripts/restore_tom.py |
Adds a script to reproducibly restore TOM assemblies from NuGet. |
converters/microsoft/README.md |
Documents installation, optional features, mapping rules, and usage. |
converters/microsoft/pyproject.toml |
Adds packaging metadata, deps, extras, and coverage/lint/test config. |
.gitignore |
Ignores coverage.xml and .tom/ restored assemblies output. |
.github/workflows/converter-microsoft-ci.yml |
Adds CI workflow for lint/tests/coverage/round-trip/spec validation and optional TOM tests. |
Suppressed comments (1)
converters/microsoft/src/ossie_microsoft/ossie_to_semantic_model.py:114
- This function signature violates PEP8/Ruff style (missing spaces around
=, overly long line). As-is, CI linting is likely to fail.
def convert_ossie_to_semantic_model(ossie_yaml_str, source: dict=None, output_format: Literal["TMSL", "TMDL"]="TMSL") -> dict | str:
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| | `GOODDATA` | GoodData-specific attributes | | ||
| | `HONEYDEW` | Honeydew-specific attributes | | ||
| | `WISDOM` | WisdomAI-specific attributes | | ||
| | `POWER_BI` | Power BI / Fabric semantic model-specific attributes | |
Comment on lines
+54
to
+56
| # Vendor id used for the `custom_extensions` stash. | ||
| VENDOR = "POWER_BI" | ||
|
|
Comment on lines
63
to
75
| class OSIVendor(str, Enum): | ||
| """Well-known vendor names for custom extensions.""" | ||
|
|
||
| COMMON = "COMMON" | ||
| SNOWFLAKE = "SNOWFLAKE" | ||
| SALESFORCE = "SALESFORCE" | ||
| DBT = "DBT" | ||
| DATABRICKS = "DATABRICKS" | ||
| GOODDATA = "GOODDATA" | ||
| SEMANTIDO = "SEMANTIDO" | ||
| WISDOM = "WISDOM" | ||
| POWER_BI = "POWER_BI" | ||
|
|
Comment on lines
+39
to
+43
| import json | ||
| import re | ||
| from typing import Literal | ||
| import yaml | ||
|
|
Comment on lines
+369
to
+372
| A calculated *field* whose expression is not already DAX is likewise skipped, even when | ||
| the same expression would translate as a metric: a calculated column evaluates in row | ||
| context, where `SUM('T'[X])` returns the whole-column total on every row instead of the | ||
| row's own value. |
Comment on lines
+235
to
+236
| assert total["description"] == "Sum of sales amount" | ||
| assert total["description"] == "Sum of sales amount" |
Comment on lines
+84
to
+86
| "workspace_id = \"49bd15b9-0a7e-4e41-94ae-ee5d8e8d1990\" # ID of workspace\n", | ||
| "item_id = \"6703e922-40a2-4fb1-86a2-f4c02784dd96\" # ID of Lakehouse\n", | ||
| "output_format = \"TMDL\" # Can be \"TMSL\" or \"TMDL\"\n", |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR contains converters to and from Microsoft Power BI Semantic Models and Ossie models.
Checklist
Specification
Validation
Examples
examples/are added or updated for any new spec constructs or converter supportTests