Add two-ONNX comparison to winml eval --mode compare#1139
Draft
xieofxie wants to merge 4 commits into
Draft
Conversation
Introduce a --reference option so --mode compare can diff two ONNX files directly instead of only comparing an ONNX candidate against its HF PyTorch reference. Both models run as raw ORT sessions on identical RandomDataset inputs, so --model-id and --task are not required and no torch/transformers reference is loaded. - config: add reference_path field (to_dict/from_dict) - cli: --reference option, _resolve_reference validation/normalization, allow bare ONNX candidate without --model-id, require --mode compare - evaluate: skip task resolution and return no model for the two-ONNX path; print reference in config summary - evaluator: _ONNXSessionModel wrapper; build both raw sessions when reference_path is set, reusing the existing similarity metrics - docs + unit tests
added 2 commits
July 20, 2026 14:23
Support comparing a candidate and reference on real input tensors from a .npz archive instead of randomly generated ones, mirroring winml perf's --input-data. The whole archive is treated as a single sample and its keys are validated/cast against the candidate's inputs. Only valid with --mode compare (accuracy modes need labels). - datasets/input_data.py: shared load_input_data (moved from perf) plus a single-sample InputDataDataset wrapper - perf: load_input_data now delegates to the shared module (lazy import, same public symbol) - config: add input_data field (to_dict/from_dict) - cli: --input-data option, compare-mode guard, example; report header shows Reference / Input data / Samples: 1 - evaluator: prepare_data builds InputDataDataset when input_data is set - evaluate: print the input-data path in the config summary - docs + unit tests
xieofxie
marked this pull request as draft
July 20, 2026 06:40
Treat the leading axis of each .npz array as the sample axis so a single archive can hold N samples. All inputs must share the same leading length N; scalars and empty arrays are rejected with a clear error. The compare loop runs each sample independently (sliced to a batch of 1), so the similarity metrics table reflects a real distribution (mean/std/min/max) across N inputs. The effective sample count is propagated to the report header and JSON output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends
winml eval --mode compareso the reference can be a second ONNX file instead of only the HuggingFace PyTorch model derived from--model-id.A new
--reference <path>option points at a reference.onnx. When set, both the candidate (-m) and the reference run as raw ORT sessions (WinMLSession) on identicalRandomDatasetinputs, and the existing tensor-similarity metrics (cosine, SQNR, PSNR, MSE, max-abs-diff) are computed over their aligned outputs. Because both are raw ONNX graphs, no HF config/task is loaded, so--model-idand--taskare not required in this path. Both sessions honor--device/--ep.Usage
winml eval -m candidate.onnx --mode compare --reference reference.onnxChanges
eval/config.py: addreference_pathfield (cli_name="reference") withto_dict/from_dictsupport.commands/eval.py: add--referenceoption;_resolve_reference()normalizes hub refs and validates.onnxexistence;allow_missing_model_idlets a bare ONNX candidate skip the--model-idrequirement; guard so--referenceonly works with--mode compare; print the reference path.eval/evaluate.py: skip task resolution and load no torch model for the two-ONNX path; guardNonemodel_id/task and print the reference in the config summary.eval/tensor_similarity_evaluator.py: add_ONNXSessionModelwrapper (exposesonnx_path,io_config,__call__returning torch tensors); build both raw sessions whenreference_pathis set, reusing the existing similarity metrics.docs/commands/eval.md: document--reference, clarify--mode, add a two-ONNX example._resolve_referencevalidation, the compare-mode guard, bare-ONNX candidate handling, the two-ONNX evaluate flow, and the_ONNXSessionModelwrapper.Scope
Phase 1 supports an ONNX-file reference only. HF-local-folder and composite-model references are deferred.