[ConfigNode][DataNode] Fence Region creation across database deletion

[CRZbulabula]

Description

Region creation spans procedure execution, persisted maintenance tasks, and DataNode-local initialization. Without a lifecycle fence, delayed work can race with DROP DATABASE, recreate Regions after their RegionGroups have been removed, or survive leader changes and snapshots. Large retry queues can also amplify RPC and direct-memory pressure, while partial DataNode failures can leave local resources behind.

This PR keeps RemoveRegionGroupProcedure as the deletion mechanism and closes these lifecycle gaps without adding database generations or changing historical procedure and snapshot framing.

Database lifecycle admission fence

• Add a database-level procedure lock shared by RegionGroup creation, database creation, and database deletion admission.
• Allocate the RegionGroup plan and submit CreateRegionGroupsProcedure in one critical section, so deletion cannot be admitted between those operations.
• Check unfinished create/delete procedures while admitting CREATE DATABASE under the same lock, preventing a same-name database incarnation from overlapping old lifecycle work.
• Revalidate database ownership before every CreateRegionGroupsProcedure state.
• Preserve the existing serialization formats of CreateRegionGroupsPlan, CreateRegionGroupsProcedure, and PartitionInfo snapshots. No database generation field or custom snapshot envelope is introduced.

Explicit cleanup ownership

CreateRegionGroupsProcedure handles fencing according to the durable progress boundary:

1. Before any create RPC is sent, a fenced procedure exits without compensation because no remote state was created.
2. After create RPCs may have been sent but before RegionGroups are persisted, it submits idempotent RemoveRegionGroupProcedures for every originally planned replica set, covering successful, failed, and indeterminate RPC outcomes.
3. After RegionGroups are persisted, the create procedure does not compensate. DROP DATABASE owns cleanup through PartitionInfo and RemoveRegionGroupProcedure.

This keeps compensation limited to the only window where DataNode state may exist without durable PartitionInfo ownership.

Persisted maintenance-task cancellation

• Add an idempotent, database-scoped BatchRemoveRegionCreateTasksPlan that removes all matching queued tasks through ConfigNode Consensus and snapshots.
• Apply the cancellation only while the database is currently pre-deleted, so a delayed plan cannot affect a later same-name database.
• Make DeleteDatabaseProcedure pre-delete the database, prevent new maintenance selection, durably cancel all queued create tasks, wait for already-issued batches, submit RemoveRegionGroupProcedure, and then delete the database partition table.
• Revalidate each task immediately before dispatch: the database must exist and not be pre-deleted, the Region must still belong to it, the replica set must still contain the target DataNode, and the target replica must still be missing. Invalid tasks are removed durably.
• Repair orphaned maintenance tasks when a ConfigNode becomes leader.

Retry and resource control

• Limit each maintenance batch per DataNode to 32 SchemaRegions and 64 DataRegions.
• Attempt a task once per scheduler cycle instead of performing six immediate retries.
• Add exponential backoff with jitter, per-node OOM cooldown, and bounded in-flight create RPCs.

Transactional DataNode creation

• Commit a newly created DataRegion only after local Region and WAL/Consensus initialization both succeed.
• Roll back newly allocated Region, Consensus/WAL state, metrics, executors, and direct-memory budget on failure.
• Apply equivalent rollback behavior to SchemaRegion creation.
• Distinguish state created by the current request from pre-existing state so idempotent retries never remove a healthy Region.

Relationship to #18273

This PR incorporates and supersedes the useful race fixes from #18273: atomic missing/pre-delete validation, database procedure synchronization, and cleanup of pre-persistence RPC outcomes. It uses lifecycle admission fencing instead of database generations, avoiding compatibility changes to Consensus plans, procedures, and snapshots.

Verification

• ConfigNode focused tests: 136 passed.
• DataNodeRegionManagerTest: 5 passed.
• Full English reactor test-compile: 50/50 modules passed.
• Full Chinese-locale reactor test-compile: 50/50 modules passed.
• Spotless, Checkstyle, and git diff --check passed.

Focused coverage includes lifecycle admission synchronization, unfinished-procedure detection, historical snapshot framing, all three compensation windows, persisted batch cancellation, multiple failed replicas for one Region, concurrent maintenance and DROP, leader/snapshot recovery, same-name database recreation, and DataNode rollback after WAL/direct-memory or Ratis failures.

This PR has:

• been self-reviewed.
  • concurrent write
• added comments explaining non-obvious concurrency and recovery behavior.
• added or updated unit tests for the new code paths.
• added integration tests.
• been tested in a fault-injected 3C3D cluster.

[Copilot]

Pull request overview

This PR strengthens database/RegionGroup lifecycle safety across ConfigNode procedures, persisted maintenance tasks, and DataNode-local Region initialization to prevent late/delayed Region creation from racing with DROP DATABASE and reintroducing Regions after their metadata ownership has been removed.

Changes:

• Add database-scoped lifecycle locking/fencing in ConfigNode procedures and admission paths (region-group creation, database create, database delete) to prevent overlapping incarnations and to revalidate lifecycle ownership during procedure progress.
• Add durable cancellation and validation for persisted RegionCreate maintenance tasks (including snapshot-safe removal) plus batching/backoff controls to reduce retry/RPC pressure.
• Make DataNode Region creation more transactional with explicit rollback of newly-created local state on consensus/WAL/direct-memory failures, and add unit tests for rollback/idempotency behavior.

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeRegionManagerTest.java	New tests covering DataNode region-creation rollback/idempotent retry behaviors and deletion fencing.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/StorageEngine.java	Introduce createDataRegionIfAbsent to distinguish “created here” vs “already existed”.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/DataRegion.java	Ensure constructor/mark-delete cleanups release resources (metrics, thread pools, direct memory) on failures and deletions.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/service/RegionMigrateService.java	Fence delayed create RPCs when deletion tasks (negative task ids) run, preventing post-drop recreation.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/SchemaEngine.java	Add createSchemaRegionIfAbsent to support transactional creation/rollback flows.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeRegionManager.java	Add striped creation locks + deleted-region fencing + rollback logic for schema/data region creation.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/consensus/SchemaRegionConsensusImpl.java	Add setInstance hook for testability.
iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/procedure/impl/schema/DeleteDatabaseProcedureTest.java	Update serialization test to reflect new procedure state captured data (target replica sets).
iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/procedure/impl/CreateRegionGroupsProcedureTest.java	Add tests for fencing/compensation windows and database-lock mutual exclusion.
iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/persistence/PartitionInfoTest.java	Add tests for atomic validation, snapshot framing preservation, and durable maintenance-task cancellation semantics.
iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/manager/ProcedureManagerTest.java	Add unfinished lifecycle procedure detection test coverage.
iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/manager/ClusterSchemaManagerTest.java	Test admission lock behavior before checking unfinished lifecycle procedures.
iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/consensus/request/ConfigPhysicalPlanSerDeTest.java	Add SerDe coverage for new/updated plans (including batch task removal).
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/state/schema/DeleteDatabaseState.java	Append new state while preserving legacy ordinals for compatibility.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/scheduler/LockQueue.java	Add isIdle() to support safe lock-queue map cleanup.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/schema/DeleteDatabaseProcedure.java	Add pre-delete capture of target replica sets + durable task cancellation state + backward-compatible serialization.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/region/CreateRegionGroupsProcedure.java	Add per-state lifecycle validation and pre-persistence compensation via RemoveRegionGroupProcedure.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/AbstractDatabaseProcedure.java	New base procedure to hold exclusive database lifecycle locks for its target databases.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/env/ConfigNodeProcedureEnv.java	Implement multi-database lifecycle locking plus task-cancellation and plan-validation helpers.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/persistence/partition/PartitionInfo.java	Add atomic create-plan validation, durable create-task cancellation, and snapshot consistency improvements.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/persistence/executor/ConfigPlanExecutor.java	Wire new BatchRemoveRegionCreateTasks plan execution into the executor.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/schema/ClusterSchemaManager.java	Serialize database creation admission with lifecycle admission lock + unfinished-procedure checks.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ProcedureManager.java	Add lifecycle admission lock, new submit/wait split for CreateRegionGroupsProcedure, and unfinished lifecycle detection.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/partition/PartitionManager.java	Add durable task cancellation, head-of-queue validation, batching limits, and backoff/jitter resource controls for region maintenance.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/consensus/request/write/region/BatchRemoveRegionCreateTasksPlan.java	New idempotent plan to remove queued create tasks for a pre-deleted database.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/consensus/request/ConfigPhysicalPlanType.java	Register new plan type id for BatchRemoveRegionCreateTasks.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/consensus/request/ConfigPhysicalPlan.java	Add factory construction support for BatchRemoveRegionCreateTasksPlan.
iotdb-core/confignode/src/main/i18n/zh/org/apache/iotdb/confignode/i18n/ManagerMessages.java	Add i18n messages for lifecycle admission rejection cases (zh).
iotdb-core/confignode/src/main/i18n/zh/org/apache/iotdb/confignode/i18n/ConfigNodeMessages.java	Add i18n messages for CreateRegionGroupsPlan validation rejection (zh).
iotdb-core/confignode/src/main/i18n/en/org/apache/iotdb/confignode/i18n/ManagerMessages.java	Add i18n messages for lifecycle admission rejection cases (en).
iotdb-core/confignode/src/main/i18n/en/org/apache/iotdb/confignode/i18n/ConfigNodeMessages.java	Add i18n messages for CreateRegionGroupsPlan validation rejection (en).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.