[OMNIML-5562] Add FAR3D ONNX PTQ and accuracy evaluation example

.pre-commit-config.yaml

@@ -126,6 +126,7 @@ repos:
               examples/llm_eval/lm_eval_hf.py|
               examples/llm_eval/mmlu.py|
               examples/llm_eval/modeling.py|
+              examples/onnx_ptq/far3d/evaluate.py|
               examples/llm_qat/train.py|
               examples/llm_sparsity/weight_sparsity/finetune.py|
               examples/specdec_bench/specdec_bench/models/specbench_medusa.py|

CHANGELOG.rst

@@ -41,6 +41,7 @@ Experimental
   - **Multi-Latent Attention (MLA)**, such as DeepSeek — MLA latent ranks are not pruned.
   - **Latent MoE**, such as Nemotron-3-Super — ``hidden_size`` pruning resizes the latent projections while the MoE latent dim itself is not pruned.
 - Add Minitron pruning support for the language model part of vision-language models (e.g. Qwen3.5-VL, Gemma3-VL) via ``examples/megatron_bridge/prune_minitron.py``. The language model is pruned while the vision tower is left intact and the full VLM is saved back; ``hidden_size`` is not pruned if it is shared with the vision projector. Pruning importance is estimated from image-text calibration (the full VLM forward over vision-conditioned activations) by default, or from a text dataset for text-only ablations.
+- Add an end-to-end FAR3D ONNX PTQ example with calibration data generation, INT8 and FP8 quantization, TensorRT engine building, and Argoverse 2 accuracy evaluation. See `examples/onnx_ptq/far3d/README.md <https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/onnx_ptq/far3d>`_ for details.
 - Add PTQ support for the language model part of vision-language models (e.g. Qwen3.5-VL, Gemma3-VL) via ``examples/megatron_bridge/quantize.py``. Only the language model is quantized (vision tower + projector left in full precision) and the full VLM is saved as a Megatron checkpoint. The calibration modality is inferred from ``--calib_dataset_name``: an image-text dataset drives the full VLM forward (vision-conditioned activations), while a text dataset runs text-only calibration of the language model. Image-text calibration shards across data-parallel ranks (context parallelism is supported only for text-only calibration). HuggingFace unified export of a quantized VLM is not yet supported.
 - Add Megatron-Bridge distillation and QAD support for the language model part of vision-language models (e.g. Qwen3.5-VL, Gemma3-VL) via ``examples/megatron_bridge/distill.py``.
 - Add NVFP4 Four-Over-Six (4/6) weight quantization (``mtq.NVFP4_FOUR_OVER_SIX_CFG``): MSE weight calibration picks, per block, between an M=6 and an M=4 dynamic range (the choice is folded into the FP8 per-block scales), with the ``four_over_six: true`` flag normalizing those scales by 256 (vs 448) for M=4 headroom. Supported via ``mtq.quantize`` and HF / Megatron export only -- **not** ``mtq.compress``, which does not preserve the per-block M=4/M=6 choice

LICENSE

@@ -223,6 +223,7 @@ the following copyright holders, licensed under the Apache License, Version 2.0
   Copyright 2023 Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li
   Copyright (c) 2024 Heming Xia
   Copyright 2025 The Qwen team, Alibaba Group and the HuggingFace Inc. team
+  Copyright (c) OpenMMLab. All rights reserved.
 
 Licensed under the Apache License, Version 2.0 (the "License"); you may not
 use these files except in compliance with the License. You may obtain a copy

examples/onnx_ptq/README.md

@@ -129,6 +129,10 @@ The top5 accuracy of the model is <accuracy score between 0-100%>
 Inference latency of the model is <X> ms
 ```
 
+### FAR3D 3D object detection
+
+The [FAR3D example](./far3d/) demonstrates an end-to-end workflow that exports and quantizes the FAR3D ONNX image encoder, builds TensorRT engines, and evaluates 3D object detection mAP on the Argoverse 2 validation set.
+
 ## Advanced Features
 
 ### Per node calibration of ONNX models

examples/onnx_ptq/far3d/Dockerfile

@@ -0,0 +1,54 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# TensorRT 11.1 from the base image builds and runs the FAR3D engines.
+FROM nvcr.io/nvidia/pytorch:26.07-py3
+
+ENV LD_LIBRARY_PATH=/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
+
+RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+        libgl1 \
+        libglib2.0-0 && \
+    rm -rf /var/lib/apt/lists/*
+
+ENV UV_PYTHON_INSTALL_DIR=/opt/python
+
+RUN python -m pip install --no-cache-dir uv && \
+    uv python install 3.8 && \
+    uv venv --seed --python 3.8 /opt/far3d
+
+COPY examples/onnx_ptq/far3d/requirements*.txt /tmp/far3d-requirements/
+RUN env -u PIP_CONSTRAINT /opt/far3d/bin/python -m pip install --no-cache-dir \
+        -r /tmp/far3d-requirements/requirements-torch.txt && \
+    env -u PIP_CONSTRAINT /opt/far3d/bin/python -m pip install --no-cache-dir \
+        -r /tmp/far3d-requirements/requirements.txt && \
+    env -u PIP_CONSTRAINT /opt/far3d/bin/python -m pip install --no-cache-dir \
+        --no-build-isolation \
+        -r /tmp/far3d-requirements/requirements-mmdet3d.txt && \
+    mkdir -p /opt/far3d/lib/python3.8/site-packages/tensorrt && \
+    cp /opt/far3d/lib/python3.8/site-packages/tensorrt_bindings/__init__.py \
+        /opt/far3d/lib/python3.8/site-packages/tensorrt/__init__.py && \
+    cp /opt/far3d/lib/python3.8/site-packages/tensorrt_bindings/tensorrt.so \
+        /opt/far3d/lib/python3.8/site-packages/tensorrt/tensorrt.so
+
+COPY . /opt/Model-Optimizer
+RUN cd /opt/Model-Optimizer && \
+    env -u PIP_CONSTRAINT python -m pip install --no-cache-dir \
+        -e ".[onnx]" \
+        "onnxruntime-gpu[cuda,cudnn]~=1.24.2" \
+        "tensorrt-cu12-libs==10.11.0.33"
+
+# The TensorRT EP in ONNX Runtime 1.24 requires TensorRT 10 during decoder quantization.
+ENV ORT_TRT10_LIB_PATH=/usr/local/lib/python3.12/dist-packages/tensorrt_libs

examples/onnx_ptq/far3d/README.md

@@ -0,0 +1,163 @@
+# FAR3D ONNX PTQ and Argoverse 2 evaluation
+
+This example quantizes the FAR3D image encoder and decoder to INT8 or FP8 with Model Optimizer and evaluates the complete pipeline on the Argoverse 2 validation set. It follows the [NVIDIA DL4AGX FAR3D workflow](https://github.com/NVIDIA/DL4AGX/tree/master/AV-Solutions/far3d-trt).
+
+FAR3D uses a legacy PyTorch/MMCV environment that is incompatible with the current Model Optimizer Python dependencies. The provided image uses `nvcr.io/nvidia/pytorch:26.07-py3` with TensorRT 11.1 for engine build and evaluation, and isolates the legacy FAR3D packages in a Python 3.8 virtual environment. The TensorRT EP in ONNX Runtime 1.24 requires CUDA 12 and TensorRT 10.11 compatibility libraries during decoder quantization; these libraries are not used to build or run the TensorRT 11.1 engines.
+
+## 1. Prepare FAR3D and Argoverse 2
+
+Clone DL4AGX, initialize its submodules, and apply its FAR3D patch:
+
+```bash
+git clone https://github.com/NVIDIA/DL4AGX.git
+cd DL4AGX
+git submodule update --init --recursive
+cd AV-Solutions/far3d-trt/dependencies/Far3D
+git apply ../../patch/far3d.patch
+git apply /path/to/Model-Optimizer/examples/onnx_ptq/far3d/far3d_optional_flash_attn.patch
+cd ../..
+```
+
+The second patch makes the unused CUDA 11-only FlashAttention implementation optional; the reference configuration uses MMCV `MultiheadAttention`.
+
+Download the [Argoverse 2 sensor validation set](https://www.argoverse.org/av2.html), the [reference FAR3D checkpoint](https://github.com/NVIDIA/DL4AGX/tree/master/AV-Solutions/far3d-trt#pytorch-model-to-onnx), and its configuration. The remaining commands assume:
+
+```text
+far3d-trt/
+├── data/av2/val/
+├── dependencies/Far3D/projects/configs/far3d.py
+└── weights/iter_82548.pth
+```
+
+Build the example image from the Model Optimizer checkout:
+
+```bash
+docker build \
+  -f /path/to/Model-Optimizer/examples/onnx_ptq/far3d/Dockerfile \
+  -t far3d-modelopt \
+  /path/to/Model-Optimizer
+```
+
+Start the image and mount the FAR3D checkout:
+
+```bash
+docker run --rm -it --network=host --gpus=all --shm-size=80G --privileged \
+  -v /data/av2:/data/av2 \
+  -v /path/to/far3d-trt:/workspace/far3d-trt \
+  far3d-modelopt
+```
+
+Use `/opt/far3d/bin/python` for data preparation, export, and evaluation. It selects the isolated legacy FAR3D environment:
+
+```bash
+export PYTHONPATH=/workspace/far3d-trt/dependencies/Far3D
+cd /workspace/far3d-trt
+/opt/far3d/bin/python /opt/Model-Optimizer/examples/onnx_ptq/far3d/prepare_metadata.py data/av2
+```
+
+## 2. Export the ONNX models
+
+```bash
+/opt/far3d/bin/python tools/export_onnx.py \
+  dependencies/Far3D/projects/configs/far3d.py \
+  weights/iter_82548.pth
+```
+
+This produces `far3d.encoder.onnx` and `far3d.decoder.onnx`.
+
+## 3. Prepare calibration batches
+
+Build temporary engines from the exported models. They run the reference pipeline while collecting representative encoder and decoder inputs:
+
+```bash
+trtexec \
+  --onnx=far3d.encoder.onnx \
+  --saveEngine=far3d.encoder.fp16.engine \
+  --fp16 \
+  --skipInference
+trtexec \
+  --onnx=far3d.decoder.onnx \
+  --saveEngine=far3d.decoder.fp16.engine \
+  --stronglyTyped \
+  --skipInference
+```
+
+Extract 512 batches sampled every 20 frames from the Argoverse 2 validation loader:
+
+```bash
+/opt/far3d/bin/python /opt/Model-Optimizer/examples/onnx_ptq/far3d/prepare_calibration.py \
+  dependencies/Far3D/projects/configs/far3d.py \
+  data/far3d_calibration \
+  --encoder-engine far3d.encoder.fp16.engine \
+  --decoder-engine far3d.decoder.fp16.engine \
+  --num-samples 512 \
+  --sample-skip-interval 20
+```
+
+The calibration directory contains separate `encoder/` and `decoder/` batches. Decoder batches include the image features, camera geometry, and temporal state seen by the reference decoder.
+
+## 4. Quantize the models
+
+Use the base Python environment for Model Optimizer:
+
+```bash
+LD_LIBRARY_PATH="${ORT_TRT10_LIB_PATH}:${LD_LIBRARY_PATH}" \
+python /opt/Model-Optimizer/examples/onnx_ptq/far3d/quantize.py \
+  --encoder-onnx far3d.encoder.onnx \
+  --decoder-onnx far3d.decoder.onnx \
+  --calibration-dir data/far3d_calibration
+```
+
+Both models use max calibration. INT8 is the default; use `--quantization-mode fp8` to produce `far3d.encoder.fp8.onnx` and `far3d.decoder.fp8.onnx` instead. FP8 deployment requires an FP8-capable GPU.
+
+The quantizer preserves the accuracy-sensitive exclusions used by the DL4AGX reference: the `OSA4_5` block and nodes downstream of `lateral_convs` remain in high precision.
+
+To keep the decoder in its original mixed FP16/FP32 precision, add `--fp16-decoder`; decoder calibration batches are not required in that mode. This flag can be combined with either quantization mode.
+
+Build both engines in the same container. Serialized TensorRT engines are not portable across TensorRT versions or GPU architectures.
+
+Set the precision to the quantization mode used above:
+
+```bash
+precision=int8  # Use fp8 for FP8 models.
+trtexec \
+  --onnx=far3d.encoder.${precision}.onnx \
+  --saveEngine=far3d.encoder.${precision}.engine \
+  --stronglyTyped \
+  --skipInference
+trtexec \
+  --onnx=far3d.decoder.${precision}.onnx \
+  --saveEngine=far3d.decoder.${precision}.engine \
+  --stronglyTyped \
+  --skipInference
+```
+
+When using `--fp16-decoder`, build `far3d.decoder.onnx` as `far3d.decoder.fp16.engine` instead.
+
+## 5. Evaluate accuracy
+
+```bash
+precision=int8  # Use fp8 for FP8 models.
+/opt/far3d/bin/python /opt/Model-Optimizer/examples/onnx_ptq/far3d/evaluate.py \
+  dependencies/Far3D/projects/configs/far3d.py \
+  far3d.encoder.${precision}.engine \
+  far3d.decoder.${precision}.engine
+```
+
+Use `--max-samples N` for an inference smoke test. Dataset metrics are skipped when only part of the validation set is processed.
+
+## Results on Argoverse 2 validation set
+
+The following historical results use TensorRT 10.11.0.33 on an NVIDIA RTX 6000 Ada Generation GPU. Model quantization uses PyTorch 2.8.0a0 from the 25.06 PyTorch container, while the FAR3D export and evaluation environment uses PyTorch 1.13.1. Accuracy is measured over all 23,522 validation frames after calibration with 512 batches sampled every 20 frames. These numbers are not directly reproducible with the current 26.07/TensorRT 11.1 image; rerun the workflow to measure the current toolchain.
+
+| Encoder precision | Decoder precision | Framework | GPU compute time (ms) | Accuracy (mAP) |
+| --- | --- | --- | ---: | ---: |
+| FP32 | FP32 | TensorRT 10.11 | 92.5 | 0.241 |
+| FP16 | FP32 | TensorRT 10.11 | 47.8 | 0.241 |
+| FP16 | FP16 | TensorRT 10.11 | 45.0 | 0.241 |
+| INT8 | FP16 | TensorRT 10.11 | 24.6 | 0.236 |
+| FP8 | FP16 | TensorRT 10.11 | 31.5 | 0.241 |
+
+Quantizing the decoder to INT8 or FP8 produced severe accuracy degradation in this evaluation and is not recommended. Keep the decoder in its original mixed FP16/FP32 precision.
+
+GPU compute time is the sum of the encoder and decoder median times reported by `trtexec`, with host-to-device and device-to-host transfers disabled. Results depend on the TensorRT version and GPU architecture and are not directly comparable with the DRIVE Orin-X measurements in the [DL4AGX reference](https://github.com/NVIDIA/DL4AGX/tree/master/AV-Solutions/far3d-trt#results-on-argoverse2-validation-set).