-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathreport.html
More file actions
294 lines (275 loc) · 13 KB
/
Copy pathreport.html
File metadata and controls
294 lines (275 loc) · 13 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="AMAM benchmark report: why this dataset is a strong benchmark for metallography segmentation.">
<title>AMAM Benchmark Report</title>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Space+Grotesk:wght@400;500;600;700&family=Source+Sans+3:wght@400;600;700&display=swap" rel="stylesheet">
<link rel="stylesheet" href="assets/css/styles.css">
<link rel="stylesheet" href="assets/css/report.css">
</head>
<body>
<header class="site-header">
<div class="header-shell">
<a href="index.html#top" class="brand" aria-label="AMAM Home">
<span class="brand-mark">AM</span>
<span class="brand-text">AMAM Benchmark Dataset</span>
</a>
<nav class="top-nav" aria-label="Primary">
<a href="index.html#overview">Overview</a>
<a href="index.html#downloads">Downloads</a>
<a href="index.html#explorer">Explorer</a>
<a href="#model-results">Model Results</a>
<a href="report.html">Benchmark Report</a>
</nav>
</div>
</header>
<main id="top" class="report-main">
<section class="report-hero panel">
<div class="report-hero-main">
<p class="kicker">AMAM-128 No-Split Campaign</p>
<h1 id="reportHeadline">AMAM Benchmark Report</h1>
<p id="reportOverview" class="hero-overview">Loading campaign profile...</p>
<div class="hero-actions">
<a class="btn btn-primary" href="index.html#explorer">Open Dataset Explorer</a>
<a class="btn btn-ghost" href="assets/data/amam-dataset.json" download>Download Benchmark Metadata</a>
</div>
</div>
<aside class="report-hero-side">
<h2>Core Claim</h2>
<p>
The AMAM-128 campaign evaluates 45 methods on the same 128 paired images with common
subset-macro aggregation and disclosed track-specific label handling.
</p>
<ul class="report-list">
<li><code>fullset_no_holdout</code>: every model is evaluated on all 128 paired images</li>
<li>10 classical, 29 supervised deep, 6 foundation/edge add-on models</li>
<li>Per-model provenance manifest and published-artifact consistency audit</li>
<li>The 29 deep configurations have five-run mIoU summaries; classical and
foundation/edge results remain seed-17 point estimates</li>
</ul>
</aside>
</section>
<section class="panel">
<div class="panel-head">
<h2>Why Benchmarking This Problem Is Hard</h2>
<p>Metallography segmentation is not one task; it is a family of related tasks with shifting appearance, scale, and class semantics.</p>
</div>
<div class="challenge-grid">
<article class="challenge-card">
<h3>Semantic Drift Across Materials</h3>
<p>Class definitions vary by subset (for example ferrite/pearlite versus dendrite/interdendritic regions), so one global label map is often invalid.</p>
</article>
<article class="challenge-card">
<h3>Scale and Morphology Shift</h3>
<p>Low and high magnification expose different structures. A model that works on one scale can fail sharply on another.</p>
</article>
<article class="challenge-card">
<h3>Artifact Sensitivity</h3>
<p>Scratches, pores, glare, and blurry interfaces are common failure drivers and can be confused with true phases if not benchmarked explicitly.</p>
</article>
<article class="challenge-card">
<h3>Monolithic Scoring Can Mislead</h3>
<p>Single aggregate scores can hide minority-subset failures. Benchmark design must preserve subset-level reporting and OOD evaluation.</p>
</article>
</div>
</section>
<section class="panel">
<div class="panel-head">
<h2>Verified Dataset Evidence</h2>
<p>All values below are computed directly from the local release metadata and pair mappings.</p>
</div>
<div class="report-metrics-grid" id="reportMetrics"></div>
<div class="report-chart-grid">
<article class="chart-card">
<h3>Pairs by Material Family</h3>
<div id="familyChart" class="bar-chart" aria-live="polite"></div>
</article>
<article class="chart-card">
<h3>Pairs by Magnification Regime</h3>
<div id="magChart" class="bar-chart" aria-live="polite"></div>
</article>
</div>
<article class="subset-table-card">
<h3>Subset-Level Coverage</h3>
<p>Coverage = matched original-label pairs divided by local originals in the subset.</p>
<div class="table-wrap">
<table class="subset-table">
<thead>
<tr>
<th>Subset</th>
<th>Family</th>
<th>Condition</th>
<th>Magnification</th>
<th>Pairs</th>
<th>Coverage</th>
</tr>
</thead>
<tbody id="subsetTableBody"></tbody>
</table>
</div>
</article>
</section>
<section class="panel" id="model-results">
<div class="panel-head">
<h2>Model Benchmark Results (All 45 Runs)</h2>
<p>Complete campaign table across 10 classical baselines, 29 supervised deep models, and 6 foundation/edge inference add-ons under one no-split protocol.</p>
</div>
<div class="results-summary" id="resultsSummary"></div>
<div class="results-tools">
<div class="field-group">
<label for="resultsGroupFilter">Model group</label>
<select id="resultsGroupFilter">
<option value="all">All groups</option>
<option value="classical">Classical baselines</option>
<option value="deep_general">Deep supervised (general)</option>
<option value="deep_metallography">Deep supervised (metallography)</option>
<option value="foundation_edge">Foundation/edge add-ons</option>
</select>
</div>
<div class="field-group">
<label for="resultsSortBy">Sort by</label>
<select id="resultsSortBy">
<option value="miou">mIoU</option>
<option value="dice">Dice</option>
<option value="pixelAcc">Pixel Accuracy</option>
</select>
</div>
<div class="field-group">
<label for="resultsSearch">Search model/category</label>
<input id="resultsSearch" type="search" placeholder="e.g. segformer, edge, clahe..." autocomplete="off">
</div>
</div>
<div class="results-downloads">
<a class="btn btn-ghost" href="assets/data/results/benchmark_summary.csv" download>Classical Seed-17 CSV</a>
<a class="btn btn-ghost" href="assets/data/results/deep_macro_over_subsets.csv" download>Deep Seed-17 Details</a>
<a class="btn btn-ghost" href="assets/data/results/deep_survey_multiseed_summary.csv" download>Deep Five-Run Summary</a>
<a class="btn btn-ghost" href="assets/data/results/deep_survey_multiseed_runs.csv" download>Deep Five-Run Values</a>
<a class="btn btn-ghost" href="assets/data/results/foundation_edge_summary.csv" download>Foundation/Edge Seed-17 CSV</a>
<a class="btn btn-ghost" href="repro/results/model_provenance_manifest.csv" download>Download Provenance Manifest</a>
<a class="btn btn-ghost" href="repro/results/reproducibility_audit_45_models.json" download>Download Consistency Audit</a>
</div>
<p class="results-status" id="resultsStatus">Loading benchmark results...</p>
<div class="table-wrap">
<table class="results-table">
<thead>
<tr>
<th>Displayed position</th>
<th>Model</th>
<th>Group</th>
<th>Category</th>
<th>mIoU</th>
<th>Dice</th>
<th>Pixel Acc</th>
<th>Run variability</th>
<th>Deep rank range</th>
</tr>
</thead>
<tbody id="resultsTableBody"></tbody>
</table>
</div>
<p class="results-note">
Deep mIoU is the five-run mean over clean end-to-end runs using seeds
17–21. Run variability reports the sample standard deviation and
<em>n</em>=5; it includes both model-side stochasticity and seeded label
decoding. The deep rank range is the best-to-worst position among the
29 supervised deep models in each seed-specific run, not a rank among
all 45 methods. The displayed-position column reflects the currently
selected sorting and filtering and is descriptive. Deep Dice and Pixel
Accuracy remain seed-17 values. The 10 classical and 6 foundation/edge
entries are also seed-17 point estimates, so no cross-seed uncertainty
is claimed for those tracks. See
<code>repro/benchmark/README.md</code>, “Scope of
reproducibility”.
</p>
</section>
<section class="panel">
<div class="panel-head">
<h2>Annotation and Curation Workflow</h2>
<p>The workflow follows the local AMAM documentation: expert partial labels, model-assisted completion, iterative correction, and final dense labels.</p>
</div>
<div class="workflow-layout">
<figure class="workflow-card">
<img src="assets/images/amam-workflow.png" alt="AMAM annotation and refinement workflow">
<figcaption>AMAM annotation and refinement workflow.</figcaption>
</figure>
<div>
<ol class="workflow-steps" id="workflowSteps"></ol>
<div class="quality-box">
<h3>Documented Edge Cases</h3>
<ul class="report-list">
<li>Pores/scratches can be confused with true phase regions</li>
<li>Scale bar or mounting media can be mistakenly segmented</li>
<li>Blurry phase boundaries can drive local under/over-labeling</li>
<li>Reflective precipitates can invert local contrast appearance</li>
</ul>
</div>
</div>
</div>
</section>
<section class="panel">
<div class="panel-head">
<h2>Representative Labeled Tuples</h2>
<p>One original-mask pair preview from each included subset, for quick visual inspection of task diversity.</p>
</div>
<div id="samplePairsGrid" class="sample-grid"></div>
</section>
<section class="panel">
<div class="panel-head">
<h2>Campaign Protocol (Executed)</h2>
<p>The AMAM-128 campaign runs one fixed benchmark protocol so all model families are directly comparable.</p>
</div>
<div class="protocol-grid">
<article class="protocol-card">
<h3>Coverage Rule</h3>
<p><code>fullset_no_holdout</code> mode: each benchmarked model is evaluated on all 128 paired images.</p>
</article>
<article class="protocol-card">
<h3>Model Scope</h3>
<p>45 total models: 10 classical, 29 supervised deep, and 6 foundation/edge add-ons.</p>
</article>
<article class="protocol-card">
<h3>Metrics</h3>
<p>Subset-aware macro mIoU, Dice, and Pixel Accuracy with per-subset and per-image outputs.</p>
</article>
<article class="protocol-card">
<h3>Auditability</h3>
<p>Published checkpoint/source provenance plus a 45-method consistency audit that checks
agreement within the published artifact bundle. It is not an independent rerun test.</p>
</article>
</div>
</section>
<section class="panel">
<div class="panel-head">
<h2>Post-Campaign Next Steps</h2>
</div>
<div class="limits-grid">
<article class="limits-card">
<h3>Already Strong</h3>
<ul class="report-list">
<li>One 45-method result collection with synchronized reporting</li>
<li>Paired original-mask tuples with transparent subset definitions</li>
<li>Per-model provenance and consistency artifacts for the published bundle</li>
<li>Deep-model values and spread reported across five seeds</li>
</ul>
</article>
<article class="limits-card">
<h3>To Reach Full Benchmark Governance</h3>
<ul class="report-list" id="nextStepsList">
<li>Add hidden-test governance and submission protocol</li>
<li>Define official OOD tracks with compatible target-label mappings</li>
<li>Publish annotation reliability and inter-rater audit stats</li>
</ul>
</article>
</div>
</section>
</main>
<footer class="site-footer">
<p>AMAM Benchmark Report • Built from local AMAM release artifacts and verified dataset metadata.</p>
</footer>
<script src="assets/js/report.js" defer></script>
</body>
</html>