[AMORO-4235] Fix the issue of optimizing tasks get stuck when stale ack arrives after task reset by OptimizerKeeper

[zhangwl9]

Why are the changes needed?

Close #4235.

Root Cause

A race condition between two threads:

• Thread OptimizerKeeper
  Detects optimizer expired → 'OptimizerKeeper#retryTask()' → 'OptimizingQueue#reset()' sets token=null , status back to PLANNED

• Optimizer Thread
  Sends 'OptimizingQueue#ackTask()' request with old token → 'TaskRuntime#validThread()' sees token==null → throws TaskRuntimeException

The 'TaskRuntime#ack()' directly called 'TaskRuntime#validThread()' which throws an unrecoverable exception when token==null .

This exception causes the 'PersistentBase.doAsTransaction()' transaction to roll back, leaving the task in an inconsistent state and blocking the entire optimizing process for the table.

Brief change log

Replace the validThread(thread) call in ack() with a two-step guard:

1. If token == null → the task has been reset by OptimizerKeeper, this is a stale ack → log a warning and return gracefully
2. If thread does not match → throw TaskRuntimeException (still a real error)

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before making a pull request

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 57.14286% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 30.14%. Comparing base (99fcc08) to head (9f09d75).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
...rg/apache/amoro/server/optimizing/TaskRuntime.java	57.14%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #4239      +/-   ##
============================================
+ Coverage     23.09%   30.14%   +7.04%     
- Complexity     2706     4382    +1676     
============================================
  Files           463      680     +217     
  Lines         42826    55272   +12446     
  Branches       6044     7092    +1048     
============================================
+ Hits           9891    16662    +6771     
- Misses        32076    37352    +5276     
- Partials        859     1258     +399     

Flag	Coverage Δ
core	30.14% <57.14%> (?)
trino	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.