[Pipe] Fairly limit concurrent TsFile parsers

[Caideyipi]

Description

This is the parser admission and fairness part of the Pipe resilience work.

Global and per-Pipe-region limits

• Add pipe_tsfile_parser_in_flight_max_num, defaulting to half of the available processors with a minimum of 1.
• Add pipe_tsfile_parser_in_flight_max_num_per_pipe_region, defaulting to 1.
• Enforce both limits even when Pipe memory accounting is disabled. Since every admitted parser reserves pipe_tsfile_parser_memory, the per-Pipe-region limit also provides an approximate parser memory quota.
• Scope the local limit by (pipeName, creationTime, dataRegionId), so different regions of one Pipe can parse concurrently while TsFiles from the same Pipe region remain bounded.
• Document both properties in iotdb-system.properties.template and support hot reload. Increasing either limit wakes the next eligible queued reservation immediately.

Hierarchical fair admission

PipeMemoryManager schedules waiting requests in three levels:

1. Round-robin across Pipes.
2. Round-robin across waiting regions within the selected Pipe.
3. FIFO across TsFiles within the selected Pipe region.

After a Pipe obtains one parser, its remaining requests move behind other waiting Pipes. After a region obtains one parser, its remaining requests move behind other waiting regions of the same Pipe. A multi-region Pipe therefore keeps region parallelism without receiving extra weight in the cross-Pipe fairness queue.

Each TsFile event owns a stable reservation key and carries its stable DataRegion identity through reservation, cancellation, release, and resource finalization. Timeout, explicit close, and resource finalization cancel pending reservations so an abandoned request cannot block a queue head.

Waiting events block on their reservation keys instead of polling every 10 ms. Admission, parser release, request cancellation, memory release, and parser-limit hot reload signal only the next eligible reservation. The signal is retained until consumed, avoiding a lost wake-up between the failed reservation attempt and the wait.

When only hard-threshold memory headroom remains, the selector skips Pipes that already own a parser and admits a waiting Pipe with no parser first.

Diagnostics

An unmatched TsFile parser memory release is treated as an invariant violation and emits a WARN containing the Pipe identity and DataRegion.

Tests

• Verify global and per-Pipe-region limits together.
• Verify different regions of one Pipe can hold parsers concurrently while the same region remains limited.
• Verify round-robin admission across Pipes and across regions within one Pipe.
• Verify a Pipe with many waiting regions does not receive more cross-Pipe scheduling weight.
• Verify hard-threshold headroom goes to a Pipe that has no active parser.
• Start nine concurrent TsFile reservations from three Pipes against one parser slot; wait on directed notifications (without polling), verify every TsFile is eventually admitted, and verify the first three admissions come from different Pipes.
• Verify both parser limits take effect through DataNode hot reload.

---

This PR has:

• been self-reviewed.
  • concurrent read and write
• added comments explaining non-obvious scheduling decisions.
• added unit tests for the new admission paths.

---

Key changed/added classes

• PipeMemoryManager
• PipeTsFileInsertionEvent
• PipeMemoryManagerTest
• PropertiesTest

[jt2594838]

The default per-pipe limit may need further discussion.
For single-pipe-multi-region scenario, the performance downgrade may be considerable.

[jt2594838]

I wonder if it would be better to let the caller wait on reservationKey when allocation fails.
And this method notify the next-to-wait reservationKey to reduce cases where the callers call this method only to find that the caller does not have the priority.

[Caideyipi]

Applied in 043f798. Each waiting event now blocks on a dedicated sticky reservation key instead of polling every 10 ms. The manager signals only the next eligible key after admission, parser release, cancellation, memory release, or a hot-reloaded limit increase. The concurrent fairness test now waits on these directed notifications without polling.

[jt2594838]

Is this normal? If not, may print a warn log to help debugging.

[Caideyipi]

Agreed, this is not expected. Applied in 043f798: an unmatched release now emits a WARN with the Pipe name, creation time, and DataRegion. The message was added to both English and Chinese locale sources.

[jt2594838]

Add to the template. Better to support hot-reload, in case that a single pipe is too slow due to the limitation.

[Caideyipi]

Applied in 043f798. Both properties are now documented in iotdb-system.properties.template with effectiveMode: hot_reload. The existing Pipe hot-reload path updates the values, and it now also wakes the next eligible queued reservation so an increased limit takes effect immediately.

[Caideyipi]

Regarding the single-Pipe/multi-region concern: addressed in f75b6c6. The local limit is now scoped by (pipeName, creationTime, dataRegionId) rather than the whole Pipe, so different DataRegions of one Pipe can parse concurrently. Region scheduling remains round-robin within the Pipe, while the Pipe still has one weight in cross-Pipe fairness. The limit is also hot-reloadable now in 043f798.