Accelerate DNAscope Hybrid selection, annotation, and population transfer

[iamh2o]

Summary

This keeps the public sentieon-cli dnascope-hybrid interface and native Sentieon algorithms unchanged while removing avoidable Python/object, process-launch, compression, and DAG overhead around the core Hybrid stages.

• hybrid_select.py now reads indexed raw VCF records directly and emits the final clipped BED, replacing vcflib object construction plus bcftools view -> query -> bedtools slop
• hybrid_anno.py uses bounded ordered raw-record processing instead of materializing full vcflib records for the hot path
• population transfer becomes one scheduler job with bounded ordered workers, in-process ALT trimming, and one final ordered BGZF/index publication instead of 493 independently compressed/indexed DAG jobs plus concat
• selector, annotation, population transfer, and the existing three-process final normalizer now claim their real scheduler thread budgets
• the existing final bcftools view -a -> bcftools norm -f -> sentieon util vcfconvert path is intentionally unchanged

There are no new CLI options and no changes to output names, modes, sample handling, defaults, native Sentieon algorithms, intervals, or stage/shard boundaries.

Matched direct-CLI benchmark

HG003 inputs were prepared once from slim-data 5x Illumina plus 5x ONT FASTQs. The baseline and candidate reused those frozen SR/LR CRAMs and ran as direct CLI commands with 128 CPUs on the same i128nvme node class. Raw preparation, Slurm pending time, QC, and workflow wrappers were excluded.

Lane	Exact source	Slurm job	Direct CLI wall time
Baseline	Sentieon/sentieon-cli v1.7.0 / 1bf377d	2467	58:05.27
Candidate	focused runtime implementation	2468	41:50.35

The candidate was 974.92 seconds faster: a 27.97% reduction and 1.388x throughput. User plus system CPU time fell 6.48%; peak RSS increased 0.67%.

Output comparison

The comparison streamed every decompressed output record and exercised the published indexes:

• main hard VCF: 5,661,729 records, exact record-body SHA-256 equality
• LongReadSV VCF: 15,847 records, exact record-body SHA-256 equality
• CNV VCF: 2,942 records, exact record-body SHA-256 equality
• sample/contig order, tabix contig lists, and sampled tabix queries: exact
• final normalizer: 5,661,729 total and 14,010 realigned records in both lanes
• header differences: limited to expected execution timestamps and job-specific scratch paths in ##SentieonCommandLine.*

This benchmark covers the native hard-VCF mode on one HG003 5x/5x dataset. It does not claim a completed gVCF or multi-sample performance matrix.

Validation

• pytest -q: 153 passed, 1 local-only legacy-bedtools oracle skip
• pytest --doctest-modules -q: 153 passed, 1 skip
• the exact legacy selector oracle passed in the deployed environment with bedtools 2.31.1
• repository CI Black command: pass
• Flake8: pass
• mypy: pass
• live direct CLI commands: exit 0 with valid BGZF/tabix outputs

A fork release containing the same retained runtime code and full execution ledger is available at iamh2o/sentieon-cli 1.7.1i.

[iamh2o]

Existing HG003 5x Illumina + 5x ONT evidence was found and reused before
launching any new compute. I filed the construction-path question separately as
issue #29.

For upstream CLI v1.7.0 (1bf377d) with Sentieon Genomics 202503.03, the
same completed CLI gVCF was evaluated directly and converted with:

sentieon driver --reference REF --thread_count 32 --temp_dir TMP \
  --algo GVCFtyper -d DBSNP --vcf INPUT.g.vcf.gz \
  --emit_mode variant hard.raw.vcf.gz

All three lanes used identical HG003 GIAB v4.2.1 truth/BED/SDF and
rtg vcfeval --decompose --squash-ploidy --ref-overlap:

Callset	TP call	FP	FN	Precision	Sensitivity	F1
Raw CLI gVCF	3,604,748	17,651	227,142	0.9951	0.9407	0.9672
GVCFtyper hard from that gVCF	3,604,748	17,651	227,142	0.9951	0.9407	0.9672
Native CLI hard	3,603,895	17,202	227,997	0.9952	0.9405	0.9671

The raw-gVCF and GVCFtyper-hard summary.txt files are byte-identical
(74c800d876398824ea989cb5278e79b4c86f2f48826224fc64da6e9cbf6a28e2).
The native hard path differs. After bcftools norm -f REF -m -both, the
native-hard versus GVCFtyper-hard comparison has 953,551 native-only, 5,018
derived-only, and 4,773,983 shared records. Of the native-only records,
925,199 are MLrejected.

This existing result establishes the CLI 1.7.0 native-hard versus
gVCF-derived semantic distinction. It does not claim that this PR's gVCF
output has already been proven identical to upstream 1.7.0; that exact
upstream-versus-candidate gVCF A/B remains a separate validation boundary.