support the high availability of the device template related procedure#18265
support the high availability of the device template related procedure#18265alpass163gmail wants to merge 6 commits into
Conversation
| .setStatus(RpcUtils.SUCCESS_STATUS) | ||
| .setTableInfo(clusterSchemaManager.getAllTableInfoForDataNodeActivation()); | ||
| .setTableInfo(clusterSchemaManager.getAllTableInfoForDataNodeActivation()) | ||
| .setTemplateInfo(clusterSchemaManager.getAllTemplateSetInfo()); |
There was a problem hiding this comment.
reloadCacheAfterLeaseRecovery() should not return SUCCESS when the template snapshot cannot be read.
ClusterSchemaManager#getAllTemplateSetInfo() catches ConsensusException and returns new byte[0]. The recovery RPC then returns SUCCESS with that empty value. On the DataNode side, the field is considered set, so the template cache is cleared and the metadata lease returns to NORMAL.
The added regression tests reproduce both steps: an injected ConfigRegion read failure produces a zero-length template snapshot, and that snapshot still produces a SUCCESS (code 200) lease-recovery response.
This can unfence a DataNode with all template metadata missing. Please propagate the snapshot read failure through the RPC status so the DataNode remains fenced.
| req.setTemplateInfo(getInvalidateTemplateSetInfo()); | ||
|
|
||
| final boolean proceed = | ||
| new ClusterCachePropagator(SchemaUtils.filterFencedDataNode(env.getConfigManager())) |
There was a problem hiding this comment.
The rollback path is not HA-safe after allowing the forward invalidation to skip fenced DataNodes.
If UNSET reaches the activation check and finds that the template is still in use, rollback calls executeRollbackInvalidateCache() before rollbackPreUnsetSchemaTemplate(). The former still broadcasts to every registered DataNode and fails when one DataNode is offline. Because of that exception, the ConfigNode PRE_UNSET state is never restored.
The added regression test injects the DataNode rollback failure and confirms that rollbackPreUnsetSchemaTemplate() is called zero times.
Please restore the authoritative ConfigNode state independently of DataNode acknowledgements, and then use fence-aware propagation to repair the caches of reachable DataNodes.
| public void updateTemplateInfo(Template template) { | ||
| readWriteLock.writeLock().lock(); | ||
| try { | ||
| failIfMetadataLeaseFenced(); |
There was a problem hiding this comment.
Template propagation can fail during the asynchronous lease-recovery window.
The heartbeat schedules cache recovery asynchronously, so the ConfigNode may already consider a DataNode reachable while its local metadata state is still fenced. A template update then throws from failIfMetadataLeaseFenced().
The added regression test reproduces the actual RPC path: the Thrift application exception is converted by DataNodeTSStatusRPCHandler to EXECUTE_STATEMENT_ERROR (301), after which ClusterCachePropagator returns FAIL instead of WAIT.
This causes SET/UNSET/ALTER to fail transiently while the DataNode is recovering. Please preserve/classify the fenced condition as retryable and retry until cache recovery completes.
2 participants
support the high availability of the device template related procedure