[None][feat] Add duration-based execution to benchmark

[weikuo0506]

Description

This PR adds a duration-based execution feature to the TensorRT-LLM Python benchmark suite (trtllm-bench). Currently, benchmarks rely on a fixed num_requests, which can lead to excessively long run times for scenarios with high Input Sequence Length (ISL) and Output Sequence Length (OSL).

The new --duration option (in seconds) allows users to limit the benchmark run time. The implementation uses a wall-clock check in the worker loop to stop pulling new requests after the duration has elapsed, while allowing in-flight requests to drain to ensure valid statistics.

This affects both throughput and latency commands as they share the same execution logic.

Fixes #13487

Potential Impacts

• Performance: This change only affects the benchmarking tool and has no impact on inference performance.
• Functional: It adds a new optional CLI parameter and changes the stop condition of the benchmark loop when used.

Test Coverage

• Added a unit test test_bench_async.py to verify LlmManager duration logic using mocks.
• Local verification was skipped due to missing dependencies (torch, pytest) in the host environment, but the test is designed to be run in the standard container.

PR Checklist

[✓] PR description clearly explains what and why.
[✓] PR Follows TRT-LLM CODING GUIDELINES.
[✓] Test cases are provided for new code paths.

[weikuo0506]

/bot run

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR adds an optional duration parameter to benchmark execution settings and CLI options, enabling runtime-limited benchmarks. The parameter threads through the benchmark stack from CLI arguments to the async execution manager, where it enforces a time-based execution cap by monitoring elapsed time and stopping request processing when reached.

Changes

Cohort / File(s)	Summary
Settings & Configuration tensorrt_llm/bench/benchmark/__init__.py, tensorrt_llm/bench/benchmark/low_latency.py, tensorrt_llm/bench/benchmark/throughput.py	Add duration field to GeneralExecSettings and introduce --duration CLI parameter in latency and throughput benchmark commands, forwarding it to async execution.
Core Implementation tensorrt_llm/bench/benchmark/utils/asynchronous.py	Implement duration-based execution control: add duration parameter to async_benchmark and LlmManager, track request start time, check elapsed time in worker loop, drain inbox when duration limit reached, and modify shutdown logic to conditionally cancel tasks.
Testing tests/unittest/llmapi/test_bench_async.py	Add unit test for LlmManager duration enforcement, verifying request processing stops when time limit is exceeded.

Sequence Diagram

sequenceDiagram
    actor CLI
    participant async_benchmark
    participant LlmManager
    participant worker as Worker Loop
    participant inbox as Inbox Queue
    
    CLI->>async_benchmark: Call with duration=N seconds
    async_benchmark->>LlmManager: Create with duration=N
    
    Note over CLI,LlmManager: Execution Phase
    activate worker
    worker->>worker: Initialize start_time=None
    
    loop For each request
        inbox->>worker: Receive request
        worker->>worker: Set start_time on first request
        worker->>worker: Calculate elapsed = now() - start_time
        alt elapsed < duration
            worker->>worker: Process request
        else elapsed >= duration
            worker->>worker: Log duration reached
            worker->>inbox: Drain remaining requests
            worker->>worker: Break loop
        end
    end
    
    deactivate worker
    
    Note over worker: Shutdown Phase
    alt stop event is set
        worker->>worker: Cancel remaining tasks
    else stop event not set
        worker->>worker: Wait for in-flight tasks
    end

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: adding duration-based execution capability to the benchmark system.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	PR description provides clear explanation of the feature, changes, impacts, test coverage, and checklist verification.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

tests/unittest/llmapi/test_bench_async.py (1)

17-17: Remove unused import.

The time module is imported but never used in this file.

🧹 Proposed fix

 import asyncio
-import time
 from unittest.mock import MagicMock

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/unittest/llmapi/test_bench_async.py` at line 17, Remove the unused
top-level import "time" from tests/unittest/llmapi/test_bench_async.py; locate
the import statement at the top of the file and delete the line "import time" so
the file no longer imports an unused module.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@tests/unittest/llmapi/test_bench_async.py`:
- Line 17: Remove the unused top-level import "time" from
tests/unittest/llmapi/test_bench_async.py; locate the import statement at the
top of the file and delete the line "import time" so the file no longer imports
an unused module.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 0a86d31f-8f27-450a-8a37-83b52e329f83

📥 Commits

Reviewing files that changed from the base of the PR and between b3a3b94 and abb477e.

📒 Files selected for processing (5)

• tensorrt_llm/bench/benchmark/__init__.py
• tensorrt_llm/bench/benchmark/low_latency.py
• tensorrt_llm/bench/benchmark/throughput.py
• tensorrt_llm/bench/benchmark/utils/asynchronous.py
• tests/unittest/llmapi/test_bench_async.py

[weikuo0506]

/bot run

[dc3671]

@FrankD412 Can you help review this?

[FrankD412]

@FrankD412 Can you help review this?

Can do -- will handle it asap 🙂

[weikuo0506]

Hi @dc3671 , can you help to review? Thanks

[karljang]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #53742 [ run ] triggered by Bot. Commit: a18ac06 Link to invocation

[tensorrt-cicd]

PR_Github #53742 [ run ] completed with state FAILURE. Commit: a18ac06
/LLM/main/L0_MergeRequest_PR pipeline #42867 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[karljang]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #54496 [ run ] triggered by Bot. Commit: a18ac06 Link to invocation

[karljang]

/bot kill

[karljang]

@weikuo0506,
It appears there are pre-commit failures. Could you please address them before we proceed?

[tensorrt-cicd]

PR_Github #54501 [ kill ] triggered by Bot. Commit: a18ac06 Link to invocation

[tensorrt-cicd]

PR_Github #54496 [ run ] completed with state ABORTED. Commit: a18ac06

Link to invocation

[tensorrt-cicd]

PR_Github #54501 [ kill ] completed with state SUCCESS. Commit: a18ac06
Successfully killed previous jobs for commit a18ac06

Link to invocation

[weikuo0506]

@karljang @YihuiLu512 — pushed fixes for both of YihuiLu512's findings, plus a third defect that running the tests surfaced (_tensorrt_engine was removed in #15918 and reached this branch via a main merge, breaking collection). Verified locally against tensorrt-llm==1.3.0rc21: 4 failed → 4 passed, stable across 3 runs.

One change worth flagging: this test file was not in any test list, so CI has never collected it — that's how the defects survived. I registered it in l0_cpu_x86.yml / l0_cpu_arm.yml (both system_gpu_count: 0, tests are fully mocked). Please let me know if a different stage suits better. Ready for CI whenever convenient.

[YihuiLu512]

LGTM

Please address the remaining corner cases as appropriate.

[weikuo0506]

Thanks @YihuiLu512 — both corner cases are pushed: the failure path now cancels in-flight requests (with a regression test), and --duration is click.IntRange(min=1) on both commands.

On your earlier point about the test not being in the CI list, I corrected it rather than removing it and registered it in l0_cpu_x86.yml / l0_cpu_arm.yml. Happy to move it if the normalization effort you mentioned prefers otherwise.

[karljang]

Marked all conversations as resolved, since all approved the PR.
I will trigger CI~

[karljang]

/bot run

[karljang]

The latest /bot run comment has not received a bot acknowledgement. Could an authorized maintainer please trigger CI for the current head (6d74e9bc)?

[tensorrt-cicd]

PR_Github #62592 [ run ] triggered by Bot. Commit: 6d74e9b Link to invocation

[tensorrt-cicd]

PR_Github #62592 [ run ] completed with state SUCCESS. Commit: 6d74e9b
/LLM/main/L0_MergeRequest_PR pipeline #50733 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[karljang]

/bot run

[tensorrt-cicd]

PR_Github #62615 [ run ] triggered by Bot. Commit: eea4060 Link to invocation

[BowenFu]

The default path is clean and I checked it carefully, since the finally restructure runs on every benchmark, not just --duration ones: with duration=None, _duration_exceeded() is always False, the worker early-exit never fires, drain_in_flight stays False so every task is still cancelled and awaited, and assert finished_requests == len(requests) still runs. The pending = set(self._tasks) snapshot is safe — only the worker adds to _tasks and the done callback only discards — and it incidentally fixes the old await asyncio.wait(self._tasks) racing the callback that mutates that set. Good change.

Two things worth fixing before it lands:

1. --duration is a no-op on the most common throughput invocation. throughput defaults concurrency to -1, i.e. unlimited (throughput.py:254-258), and the enforcement point is after the concurrency slot is acquired. So trtllm-bench throughput ... --duration 60 dispatches the whole dataset immediately, every request is already past the semaphore, and the run goes to completion — the user gets a warning and the full dataset. That contradicts the flag's own help text ("Maximum run time in seconds. Benchmark stops at whichever limit is hit first"). latency is fine because it defaults to concurrency 1 (low_latency.py:179-183).

A warning isn't enough for a flag whose entire purpose is bounding runtime on long ISL/OSL runs — that's precisely the scenario in your description. I'd make it a validation error when duration is set and concurrency is unlimited, or have --duration imply an enforceable concurrency.

2. The exact-count assertions will flake in pre-merge CI. Both list entries are stage: pre_merge (l0_cpu_x86.yml:12, l0_cpu_arm.yml:12), so these run on every PR, on x86 and ARM. test_llm_manager_duration asserts outbox.qsize() == 2 from 0.6s sleeps against a 1s deadline: request 2 acquires its slot at t≈0.6 and only just makes the deadline. A ~0.4s scheduling stall on a loaded runner pushes it past 1.0s, one request completes, and the test fails. test_async_benchmark_duration (assert len(stats.requests) == 2) has the same shape.

The elapsed-time bounds elsewhere are generously slack and fine; it's the exact equalities on either side of the deadline that are the problem. Relaxing them to >= 1 and < 3 would still catch "duration had no effect" without pinning the boundary.

Minor, non-blocking: the timer starts when the first request coroutine runs, not when the benchmark starts, so enqueue time is excluded from "maximum run time"; a multi-turn request truncated mid-conversation is emitted as an ordinary completed perf item, which mixes partial and complete conversations in duration-mode stats; and the worker logs "finishing" twice on normal exit.

Heads-up on a collision, not a review point: #16498 deletes both l0_cpu_x86.yml and l0_cpu_arm.yml and replaces them with a unified l0_cpu.yml that lists only the three existing tests. Whichever of the two lands second will hit a modify/delete conflict on those files, and the easy resolution silently drops unittest/llmapi/test_bench_async.py from CI entirely. Worth coordinating with @tongyuantongyu so the entry is carried into l0_cpu.yml.

[tensorrt-cicd]

PR_Github #62615 [ run ] completed with state SUCCESS. Commit: eea4060
/LLM/main/L0_MergeRequest_PR pipeline #50757 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[fredricz-20070104]

Review summary - Approve

Reviewed the full diff; no blocking or major issues found.

Minor, non-blocking notes:

• tensorrt_llm/bench/benchmark/utils/asynchronous.py: --duration silently ineffective when concurrency is unbounded (common default)
• tests/unittest/llmapi/test_bench_async.py: Exact-count assertion is timing-sensitive

Automated review by NVCortex Lite, run by @fredricz-20070104.