recipe(koelectra): add NSMC text classification recipe

[kujin66]

Summary

This recipe-only L0 contribution adds verified CPU recipes for daekeun-ml/koelectra-small-v3-nsmc, a Korean NSMC binary sentiment classifier, at fp32 and fp16. At exact PR head a7ed837aa1d20a946eb67fd5fe0660e6710513c3, both required tuples reach Goal L3 PASS: fresh build, concrete CPU perf, PyTorch parity, and pinned NSMC accuracy all pass. Current main already builds, perfs, and evaluates the checkpoint recipe-free; the shipped value is reproducible, checkpoint-specific fp32/fp16 recipe coverage with a pinned evaluation contract rather than a new architecture implementation.

Contributor history is preserved: @kujin66 authored the original contribution and first commit; @ssss141414 supplied the subsequent scope/evidence repairs. The PR intentionally makes no README publication or w8a16 support claim.

Model metadata

What the model does

This Korean text encoder classifies one movie-review document as negative or positive sentiment. Token IDs, an attention mask, and optional token-type IDs pass through a 12-layer ELECTRA encoder; a sequence-classification head reads the first-token representation and emits two logits.

• Evidence: pinned checkpoint model card, pinned config.json (ElectraForSequenceClassification, single-label classification, two labels), and Transformers 4.57.6 ElectraForSequenceClassification source.
• Confidence: verified.

Primary user stories

• A user supplies a Korean movie-review document to obtain a negative-or-positive sentiment prediction for review analysis or moderation.
  • Evidence: the pinned model-card examples provide Korean positive/negative reviews and Neg/Pos outputs; checkpoint metadata declares dataset:nsmc and pipeline_tag:text-classification.
  • Confidence: verified.

Supported tasks

Task	Support surfaces	Evidence	Confidence
text-classification	checkpoint, Transformers, Optimum ONNX, WinML	Pinned pipeline tag/config; ElectraForSequenceClassification; current-main Optimum and WinML inspection	verified

The current-main Optimum probe reports VENDOR-ONLY: vendor and post-WinML task sets both contain feature extraction, fill-mask, multiple choice, question answering, text classification, and token classification; WinML adds no ELECTRA task registration.

Model architecture

ElectraForSequenceClassification
├── ElectraModel
│   ├── Word (35000×128) + position (512×128) + token-type (2×128) embeddings
│   ├── Embedding projection (128 → 256)
│   └── ElectraEncoder layer × 12
│       ├── Self-attention (4 heads; hidden width 256)
│       ├── Feed-forward (256 → 1024 → 256, GELU)
│       └── Residual connections + LayerNorm
└── ElectraClassificationHead
    ├── First-token state → dense (256 → 256) + GELU
    └── Output projection (256 → 2 logits)

• Source/confidence: pinned checkpoint dimensions and Transformers 4.57.6 ELECTRA source (verified).

Validation and support evidence

Baseline

The superseding current-main baseline used commit 5deebd422e95f28fe8fd912ee50ce874710187b3 and winml, version 0.2.0.

Baseline surface	Current-main result
Optimum coverage	VENDOR-ONLY; WinML adds no ELECTRA task registration
Recipe-free build	PASS, exit 0, 36.5 s
Artifact	Opset 17; 428 nodes; three INT32 [1,512] inputs; FLOAT [1,2] logits
CPU fp32 perf	mean 58.056 ms; p50 58.084 ms; 17.22 samples/s; RSS total delta 93.98 MB
Pinned NSMC eval	accuracy 0.9 on 100 unshuffled streaming test samples
Auto-config	Builds successfully; generated recipe has no /eval contract and /quant is null

The executable evaluation source is e9t/nsmc revision 153263700c285ac0997b3e2a2d80b826aaf672d6, the pinned official parquet conversion of source revision e8ac4f539e5604fec0b3c67ed040728e0862b5e1. The direct source revision is script-based and was rejected by datasets; the pinned parquet revision preserves the authoritative NSMC schema in executable form. Explicit mapping aligns checkpoint labels 0=Neg, 1=Pos with dataset classes negative=0, positive=1.

Goal

• Effort: L0 — two recipe files only; no source or test changes.
• Goal ceiling: L3.
• Outcome: L0.
• Success definition: fresh CPU fp32 and fp16 passes at L0–L3 with exact pinned NSMC evaluation and explicit label mapping.
• Ceiling change: the stale L1 body is superseded; charter revision 2 set L3 after the current-main baseline itself reached the L3 floor.

Outcome

• Highest reached: L3 PASS.
• Coverage: full for the frozen CPU/cpu precision plan.
• Deferred tuples: none.
• Shipped paths: examples/recipes/daekeun-ml_koelectra-small-v3-nsmc/cpu/cpu/text-classification_fp32_config.json and examples/recipes/daekeun-ml_koelectra-small-v3-nsmc/cpu/cpu/text-classification_fp16_config.json.
• Learner handoff: no model-knowledge or methodology append was required. The L0 run introduced no durable ELECTRA-family workaround and no unresolved blocker; existing _meta-016, _meta-031, _meta-037, _meta-065, _meta-071, and _meta-083 already cover the observed host/provenance methodology.
• Publication scope: examples/recipes/README.md is untouched. The former flat fp16/w8a16 files are not shipped, and this PR claims no w8a16 coverage.

Per-EP/device/precision results

Goal ladder

Tier	Verdict	Evidence
L0	PASS	Both mandatory CPU precisions freshly built and structurally validated
L1	PASS	Concrete 100-iteration CPU perf for both precisions
L2	PASS	Both ONNX artifacts match pinned-revision PyTorch logits and argmax
L3	PASS	Both precisions produce accuracy 0.9 on the exact pinned NSMC contract

Build and perf

EP / device	Precision	L0 build	Graph / precision proof	Mean	p50	Throughput	RAM Δ	VRAM Δ
CPUExecutionProvider / cpu	fp32	PASS, 28.262 s	428 nodes; 208 FLOAT initializers; 56,640,268 bytes	45.279 ms	45.564 ms	22.09 samples/s	95.23 MB	0 MB
CPUExecutionProvider / cpu	fp16	PASS, 28.289 s	429 nodes; 208 FLOAT16 initializers; 28,395,382 bytes; resolved quant.mode=fp16	57.451 ms	57.504 ms	17.41 samples/s	100.09 MB	0 MB

The fp16 artifact is distinct and 0.5013285× the fp32 size. On this CPU host, fp16 is smaller but slower; no acceleration claim is made.

L2 PyTorch parity

Pinned checkpoint revision: d1c0233fc013ae8bd879d7b61f2bec69aa3af4c9.

Precision	Cosine similarity	Max absolute error	Mean absolute error	Argmax
fp32	0.999999999999998	9.5367431640625e-07	7.152557373046875e-07	match
fp16	0.9999999998935075	0.00012969970703125	7.426738739013672e-05	match

L3 pinned NSMC accuracy

EP / device	Precision	Dataset / revision	Split and sample policy	Input / label mapping	Metric	Eval throughput / latency
CPUExecutionProvider / cpu	fp32	e9t/nsmc / 153263700c285ac0997b3e2a2d80b826aaf672d6	test; 100; unshuffled; streaming	document; negative=0, positive=1	accuracy 0.9	19.489896 samples/s; 0.051308637 s
CPUExecutionProvider / cpu	fp16	e9t/nsmc / 153263700c285ac0997b3e2a2d80b826aaf672d6	test; 100; unshuffled; streaming	document; negative=0, positive=1	accuracy 0.9	16.142403 samples/s; 0.061948645 s

Mapping basis: checkpoint Neg/Pos id2label indices 0/1 align with authoritative NSMC ClassLabel negative/positive indices 0/1.

Delta

Both files use the required nested CPU path. Relative to fresh current-main auto-config, loader/export/optimization/compilation settings are unchanged.

fp32

• /eval: absent →

{
  "task": "text-classification",
  "dataset": {
    "path": "e9t/nsmc",
    "split": "test",
    "samples": 100,
    "shuffle": false,
    "streaming": true,
    "revision": "153263700c285ac0997b3e2a2d80b826aaf672d6",
    "columns_mapping": { "input_column": "document" },
    "label_mapping": { "negative": 0, "positive": 1 }
  }
}

fp16

• /quant: null →

{
  "mode": "fp16",
  "samples": 10,
  "calibration_method": "minmax",
  "weight_type": "uint8",
  "activation_type": "uint8",
  "per_channel": false,
  "symmetric": false,
  "weight_symmetric": null,
  "activation_symmetric": null,
  "save_calibration": false,
  "distribution": "uniform",
  "seed": null,
  "calibration_load_path": null,
  "calibration_save_path": null,
  "op_types_to_quantize": null,
  "nodes_to_exclude": null,
  "fp16_keep_io_types": true,
  "fp16_op_block_list": null
}

• /eval: absent → the exact fp32 evaluation object above.

The evaluation choice and label semantics are checkpoint-specific human intent, not derivable from ELECTRA architecture metadata, so a recipe is the consistent fix class. Recipe-free acceptance independently passes in 55.492 s and confirms there is no class-wide code defect. No code paths change, and the production recipe README remains untouched.

Analyze summary — component level and op level

Static rule analysis completed for both artifacts; this is compatibility analysis, not runtime execution. The fp16 analyzer returned exit 1 intentionally for partial operator support while still producing all 11 complete EP/device rows with has_errors=false; this is not an analysis crash.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32	embeddings; projection; 12× encoder attention/FFN; classifier	423 mapped, 5 optimized helper nodes unmapped; partial confidence	QNN NPU/GPU partial: Add, Div, Erf, Mul; none unsupported
fp16	embeddings; projection; 12× encoder attention/FFN; classifier	423 mapped, 6 optimized helper/output-cast nodes unmapped; partial confidence	QNN NPU/GPU partial: Add, Div, Erf, Mul; none unsupported

Op-level summary

Artifact	Graph	Dominant ops	EP roll-up
fp32	428 ops / 14 types	Add 124; MatMul 97; Reshape 48; Transpose 48; Mul 27; LayerNormalization 25	TensorRT GPU and OpenVINO NPU/GPU/CPU fully supported by static rules; QNN partial as above
fp16	429 ops / 14 types	Add 124; MatMul 97; Reshape 48; Transpose 48; Mul 27; LayerNormalization 25	TensorRT GPU and OpenVINO NPU/GPU/CPU fully supported by static rules; QNN partial as above

Rule-less CUDA, MIGraphX, DML, CPU, and VitisAI rows classify all operator types as unknown. These static rows do not claim runtime execution on those EPs.

Quality caveats

• The test host's globally installed OpenVINO plugin could crash during normal CLI provider teardown after complete CPU output. Tester-owned isolated CPU-registry reruns preserved the unchanged Click command and ORT CPUExecutionProvider execution and exited 0 for fp32/fp16 perf and eval. This is host/plugin evidence, not a model verdict downgrade.
• Focused quality passed: 309 tests, package mypy over 409 source files, lock check, and diff check.
• This PR changes no Python files, so scoped Ruff is not applicable. Full-repository Ruff exits 1 at both the exact PR head and exact current-main baseline because of identical pre-existing findings; the PR introduces no full-Ruff exit regression.
• L2 parity used a tester-owned comparison script because WinML has no single public command that computes PyTorch-vs-ONNX parity. The measured values above are retained exactly; the portable commands below reproduce the public build/perf/eval/analyze surfaces.

Reproduce commands

PowerShell, from a fresh checkout. The first block pins the exact reviewed head and recreates the tested frozen environment; the remaining commands are the tester's portable public command set.

git clone https://github.com/microsoft/winml-cli.git
Set-Location winml-cli
git fetch origin pull/1144/head:pr-1144
git checkout --detach a7ed837aa1d20a946eb67fd5fe0660e6710513c3
if ((git rev-parse HEAD).Trim() -ne 'a7ed837aa1d20a946eb67fd5fe0660e6710513c3') { throw 'Exact PR head was not checked out' }
uv sync --frozen --all-extras --all-groups
& ./.venv/Scripts/Activate.ps1

$OUT='temp/koelectra-nsmc-repro'
$MODEL='daekeun-ml/koelectra-small-v3-nsmc'
$DATASET_REVISION='153263700c285ac0997b3e2a2d80b826aaf672d6'
New-Item -ItemType Directory -Force $OUT | Out-Null
$MAPPING=Join-Path $OUT 'nsmc-label-mapping.json'
'{"negative":0,"positive":1}' | Set-Content -Encoding utf8 $MAPPING

winml build -c examples/recipes/daekeun-ml_koelectra-small-v3-nsmc/cpu/cpu/text-classification_fp32_config.json -m $MODEL -o (Join-Path $OUT 'fp32') --ep cpu --device cpu --precision fp32 --no-analyze --no-compile --rebuild --no-color
winml perf -m (Join-Path $OUT 'fp32/model.onnx') --ep cpu --device cpu --precision fp32 --iterations 100 --warmup 10 --memory --output (Join-Path $OUT 'perf-fp32.json') --overwrite --format json --no-color
winml eval --model (Join-Path $OUT 'fp32/model.onnx') --model-id $MODEL --dataset e9t/nsmc --dataset-revision $DATASET_REVISION --task text-classification --device cpu --ep cpu --samples 100 --split test --no-shuffle --streaming --column input_column=document --label-mapping $MAPPING --output (Join-Path $OUT 'eval-fp32.json') --overwrite --format json --no-color
winml analyze --model (Join-Path $OUT 'fp32/model.onnx') --ep all --device all --output (Join-Path $OUT 'analyze-fp32.json') --overwrite --format json --no-color

winml build -c examples/recipes/daekeun-ml_koelectra-small-v3-nsmc/cpu/cpu/text-classification_fp16_config.json -m $MODEL -o (Join-Path $OUT 'fp16') --ep cpu --device cpu --precision fp16 --no-analyze --no-compile --rebuild --no-color
winml perf -m (Join-Path $OUT 'fp16/model.onnx') --ep cpu --device cpu --precision fp16 --iterations 100 --warmup 10 --memory --output (Join-Path $OUT 'perf-fp16.json') --overwrite --format json --no-color
winml eval --model (Join-Path $OUT 'fp16/model.onnx') --model-id $MODEL --dataset e9t/nsmc --dataset-revision $DATASET_REVISION --task text-classification --device cpu --ep cpu --samples 100 --split test --no-shuffle --streaming --column input_column=document --label-mapping $MAPPING --output (Join-Path $OUT 'eval-fp16.json') --overwrite --format json --no-color
winml analyze --model (Join-Path $OUT 'fp16/model.onnx') --ep all --device all --output (Join-Path $OUT 'analyze-fp16.json') --overwrite --format json --no-color

[kujin66]

Reviewer verdict: APPROVE fallback comment / Lane B model PR is ready for maintainer review.

Checked items:

• PR body contains the 9-item adding-model-support report and no longer has stale gh-auth or rebase blocker text.
• Diff scope is L0 recipe-only: README plus two KOElectra recipe JSON files.
• Branch is based on current origin/main e0751f6dedfb45e2a1a177a7fff74a111b880514; git rev-list --count e0751f6dedfb45e2a1a177a7fff74a111b880514..origin/main returned 0.
• CI is green: Analyze, CodeQL, CLA, lint, and all test jobs passed.
• Goal L1 evidence is present for CPU: fp16 perf and w8a16 perf are reported, and the w8a16 artifact has nonzero QDQ nodes (QuantizeLinear=426, DequantizeLinear=634).
• PR is mergeable according to GitHub. GitHub-side approval metadata could not be set because the same account cannot approve its own pull request.

Open process note:

• Methodology friction was observed around quantized-recipe validation requiring artifact-level QDQ checks. It is captured in local Skill/model knowledge as model_knowledge/electra.json / electra-001 and in local Skill methodology knowledge as skill_meta/findings.json / _meta-057, with paired agents/reviewer.md checklist text.
• Skill Lane A files are intentionally not included in this Lane B winml-cli PR so the model diff remains recipe-only.

I attempted gh pr review --approve; GitHub rejected it with Review Can not approve your own pull request, so this structured APPROVE comment is the _meta-035 fallback verdict.

[ssss141414]

Supersedes the earlier fallback comment at #1144 (comment) for the repaired exact head a7ed837aa1d20a946eb67fd5fe0660e6710513c3.

The live PR body now reflects the frozen charter/producer/tester/learner handoffs: nested CPU fp32/fp16 recipes only, Goal L3 evidence, current-main baseline, pinned NSMC mapping/accuracy, and component/op analysis. The earlier README, flat-path, w8a16, L1-ceiling, WINML-ONLY, and knowledge-append statements describe a superseded head and are not current claims.

This is an explainer correction, not an approval or reviewer verdict. The original @kujin66 contribution and commit history remain credited. Per current shipment policy, the PR is Draft and retains model-scale-by-skill.