recipe(google/owlvit-base-patch32): add verified CPU fp32/fp16 recipes

[codykk]

Add verified CPU zero-shot-object-detection recipes (fp32 and fp16) for google/owlvit-base-patch32. Effort L0 (recipe-only, no source code changes). Goal L2 reached — all outputs achieve cosine similarity ≥ 0.99999 against PyTorch. L3 blocked: winml eval does not support zero-shot-object-detection.

Model metadata

What the model does: OWL-ViT (Vision Transformer for Open-World Localization) performs open-vocabulary object detection — given an image and a set of free-text category queries, it predicts bounding boxes and class logits without requiring training on those specific categories.

Primary user stories:

• A developer supplies an image and natural-language queries to obtain bounding-box coordinates and confidence scores for detecting arbitrary object categories without fine-tuning.

Supported tasks:

Task	Support surfaces	Confidence
zero-shot-object-detection	checkpoint, transformers, optimum-onnx, winml	verified

Model architecture:

OwlViTForObjectDetection
├── OwlViTModel
│   ├── OwlViTTextTransformer
│   │   ├── Embeddings (vocab 49408, dim 512)
│   │   └── Encoder x 12 (self-attention 8 heads, FFN 512→2048→512, QuickGELU)
│   └── OwlViTVisionTransformer (patch_size=32, image 768×768)
│       ├── Embeddings (patch 32×32 → dim 768, + CLS + 576 position)
│       └── Encoder x 12 (self-attention 12 heads, FFN 768→3072→768, QuickGELU)
├── class_head (LayerNorm + Linear 512→512 → logits per query)
├── box_head (Linear 768→512 + Linear 512→4 → pred_boxes)
├── visual_projection (Linear 768→512 → image_embeds)
└── text_projection (Linear 512→512 → text_embeds)

• Source/confidence: OwlViTForObjectDetection from transformers pinned checkpoint google/owlvit-base-patch32 (verified).

Validation and support evidence

Baseline

• origin/main: ace130f24cbe5a2935333c7c7deacadf8d8ba65e
• winml --version: 0.2.0
• Optimum probe: zero-shot-object-detection in vendor-only (Optimum native); winml adds no extra tasks. Verdict: VENDOR-ONLY.
• Baseline build (no recipe, auto-config): PASS (22.2s, 583.4 MB)
• Baseline perf: PASS (p50 169.03ms, 6.33 samples/sec, +611 MB RAM)
• Goal floor: L1 (build + perf inherited from main)
• Starting recipe: winml config -m google/owlvit-base-patch32 auto-generated; input order pixel_values → input_ids → attention_mask matches forward() signature (input-reorder fix landed in recipe(owlvit): add verified CPU fp32 and fp16 support #1155).

Goal

• Effort: L0 (recipe-only, no source code)
• Goal ceiling: L2 (PyTorch numeric parity)
• Outcome: L0 (recipe JSON + structured report)
• L3 unreachable: winml eval does not support zero-shot-object-detection (CLI feature gap)

Outcome

Goal L2 reached on both precisions. Coverage: full (all target tuples PASS on this host). No deferred tuples.

Shipped recipes:

• examples/recipes/google_owlvit-base-patch32/cpu/cpu/zero-shot-object-detection_fp32_config.json
• examples/recipes/google_owlvit-base-patch32/cpu/cpu/zero-shot-object-detection_fp16_config.json

Per-EP/device/precision results

Tier	EP / Device	Precision	Verdict	p50	Throughput	RAM Δ
L0	CPUExecutionProvider / cpu	fp32	PASS	—	—	—
L0	CPUExecutionProvider / cpu	fp16	PASS	—	—	—
L1	CPUExecutionProvider / cpu	fp32	PASS	178.75 ms	5.74 samples/s	+610.6 MB
L1	CPUExecutionProvider / cpu	fp16	PASS	194.18 ms	5.20 samples/s	+770.2 MB

L2 — Numeric parity (PyTorch vs ONNX):

Precision	Output	Cosine similarity	Max abs diff
fp32	logits	1.00000000	9.82e-05
fp32	pred_boxes	1.00000000	2.04e-05
fp32	text_embeds	1.00000000	8.94e-08
fp32	image_embeds	1.00000000	5.05e-05
fp16	logits	0.99999943	4.74e-02
fp16	pred_boxes	0.99999849	1.03e-02
fp16	text_embeds	0.99999976	8.91e-05
fp16	image_embeds	0.99999948	4.54e-02

L3: CLI-BLOCKED — winml eval does not support task zero-shot-object-detection.

Delta

The fp32 recipe is identical to winml config auto-generated output (filed for verified cpu/cpu coverage). The fp16 recipe adds the quant block with mode: "fp16" for fp16 conversion. No source code changes. examples/recipes/README.md untouched.

Analyze summary

CLI-BLOCKED: winml analyze requires runtime rule parquet files not present in this environment. Static rule analysis unavailable; does not affect build/perf/parity results.

Reproduce commands

# fp32 build + perf
winml build -c examples/recipes/google_owlvit-base-patch32/cpu/cpu/zero-shot-object-detection_fp32_config.json `
  -m google/owlvit-base-patch32 -o $OUT/fp32 --ep cpu --device cpu
winml perf -m $OUT/fp32/model.onnx --ep cpu --device cpu

# fp16 build + perf
winml build -c examples/recipes/google_owlvit-base-patch32/cpu/cpu/zero-shot-object-detection_fp16_config.json `
  -m google/owlvit-base-patch32 -o $OUT/fp16 --ep cpu --device cpu --precision fp16
winml perf -m $OUT/fp16/model.onnx --ep cpu --device cpu

[ssss141414]

<html>
<body>
<!--StartFragment--><h3 dir="auto" style="box-sizing: border-box; margin-top: 24px; margin-bottom: 16px; font-size: 1.25em; font-weight: 600; line-height: 1.25; color: rgb(240, 246, 252); font-family: &quot;Mona Sans VF&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;Noto Sans&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(13, 17, 23); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">5. Delta</h3><p dir="auto" style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: rgb(240, 246, 252); font-family: &quot;Mona Sans VF&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;Noto Sans&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(13, 17, 23); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong style="box-sizing: border-box; font-weight: 600;">Recipe-vs-<code class="notranslate" style="box-sizing: border-box; font-family: &quot;Monaspace Neon&quot;, ui-monospace, SFMono-Regular, &quot;SF Mono&quot;, Menlo, Consolas, &quot;Liberation Mono&quot;, monospace; font-size: 11.9px; tab-size: 4; white-space: break-spaces; background-color: rgba(101, 108, 118, 0.2); border-radius: 6px; margin: 0px; padding: 0.2em 0.4em;">winml config</code><span> </span>diff</strong>:</p><markdown-accessiblity-table data-catalyst="" style="box-sizing: border-box; display: block; color: rgb(240, 246, 252); font-family: &quot;Mona Sans VF&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;Noto Sans&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(13, 17, 23); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">
Field (JSON pointer) | Auto-config value | Shipped recipe value | Reason
-- | -- | -- | --
export.input_tensors order | [pixel_values, input_ids, attention_mask] | [input_ids, pixel_values, attention_mask] | Match forward() signature order per owlv2-003 finding

</markdown-accessiblity-table><!--EndFragment-->
</body>
</html>

This is strange that recipe is different from winml config produces in input tensor orders... Could you sync latest skill and then run again with these question?

[codykk]

<html>
<body>
<!--StartFragment--><h3 dir="auto" style="box-sizing: border-box; margin-top: 24px; margin-bottom: 16px; font-size: 1.25em; font-weight: 600; line-height: 1.25; color: rgb(240, 246, 252); font-family: &quot;Mona Sans VF&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;Noto Sans&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(13, 17, 23); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">5. Delta</h3><p dir="auto" style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: rgb(240, 246, 252); font-family: &quot;Mona Sans VF&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;Noto Sans&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(13, 17, 23); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong style="box-sizing: border-box; font-weight: 600;">Recipe-vs-<code class="notranslate" style="box-sizing: border-box; font-family: &quot;Monaspace Neon&quot;, ui-monospace, SFMono-Regular, &quot;SF Mono&quot;, Menlo, Consolas, &quot;Liberation Mono&quot;, monospace; font-size: 11.9px; tab-size: 4; white-space: break-spaces; background-color: rgba(101, 108, 118, 0.2); border-radius: 6px; margin: 0px; padding: 0.2em 0.4em;">winml config</code><span> </span>diff</strong>:</p><markdown-accessiblity-table data-catalyst="" style="box-sizing: border-box; display: block; color: rgb(240, 246, 252); font-family: &quot;Mona Sans VF&quot;, -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, &quot;Noto Sans&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(13, 17, 23); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">
Field (JSON pointer) | Auto-config value | Shipped recipe value | Reason
-- | -- | -- | --
export.input_tensors order | [pixel_values, input_ids, attention_mask] | [input_ids, pixel_values, attention_mask] | Match forward() signature order per owlv2-003 finding

</markdown-accessiblity-table><!--EndFragment-->
</body>
</html>

This is strange that recipe is different from winml config produces in input tensor orders... Could you sync latest skill and then run again with these question?

I ran it with the latest skill and it looks fine; I only added the recipe.