Surface the real cause of internal PIT failures instead of an opaque message

[ahkcs]

Description

Internal Point-in-Time (PIT) creation/deletion failures were surfaced to callers as the opaque, cause-less message Error occurred while creating PIT for internal plugin operation, and a permission denial was only recognized when the OpenSearchSecurityException was the direct cause of the ExecutionException.

On a multi-node domain the CreatePit shard action runs remotely, so a denial comes back wrapped (e.g. RemoteTransportException -> OpenSearchSecurityException). The single-level instanceof check missed that wrapped case and flattened the 403 into the generic string — hiding both the missing permission and any non-permission failure (e.g. PIT-context exhaustion, circuit breaker) behind one indistinguishable message. This made a customer-reported failure on the Statistics/PPL path effectively undiagnosable from the error alone.

This is a diagnostics/robustness fix, not a behavior change to PIT itself:

• Walk the full cause + suppressed chain with ExceptionsHelper.unwrapCausesAndSuppressed. If an OpenSearchSecurityException is anywhere in the chain, rethrow it as-is so the caller still receives a clean 403.
• Otherwise include the root cause (ExceptionsHelper.unwrapCause) in the message, so a non-permission failure is diagnostic instead of opaque.
• Applied to both execution paths that shared this latent bug:
  • Calcite path — OpenSearchNodeClient
  • Legacy path — PointInTimeHandlerImpl, whose catch (OpenSearchSecurityException) was dead code (Future.get() always wraps in ExecutionException, so the security exception was never the top-level type caught).
• Re-set the interrupt flag on InterruptedException.

Why "always use PIT" isn't touched: rather than change production routing, the new IT lowers index.max_result_window to 1 on a 2-doc index, which forces any ordinary query down the PIT path. This lets us exercise the PIT permission failure on a tiny dataset instead of the fidelity-scale volume that triggers it in production.

Scope / honesty note

This change makes the error legible; it is not itself the cure for the reported incident. If the customer's failure is a wrapped permission denial, this surfaces the clean 403 (and the remediation is granting indices:data/read/point_in_time/{create,delete}). If it is instead a resource failure at peak, this now puts the real root cause in the message so it can be diagnosed. Determining which requires the raw stack trace from the affected domain.

Testing

Unit (OpenSearchNodeClientTest, PointInTimeHandlerImplTest):

• direct OpenSearchSecurityException → rethrown as 403
• transport-wrapped OpenSearchSecurityException (the multi-node case) → still rethrown as 403 (previously flattened)
• non-security failure → root cause preserved in the message

Integration (PPLPermissionsIT, security cluster):

• testForcedPitQueryWithoutPitPermissionReturnsCleanForbidden — query forced through PIT; user without PIT permission gets a clean 403 naming the permission, and not the opaque generic message.
• testForcedPitQueryWithPitPermissionSucceeds — granting the two PIT permissions lets the same query succeed (the remediation).

All new and existing tests in the touched classes pass; spotlessJavaCheck is clean on :opensearch, :legacy, :integ-test.

Known follow-ups (out of scope)

• OpenSearchRestClient.createPit has the same single-level unwrap pattern and should get the same treatment.
• BackgroundSearchScanner shares the pattern but its direct-cause handling is load-bearing for the afterKey == EMPTY case — left untouched here.

Check List

• New functionality includes testing.
• New functionality has javadoc added.
• New functionality has been documented. (error-message/diagnostics change; no user-facing doc)
• Commits are signed per the DCO using --signoff or -s.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ No major issues detected

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
General	Log thread interruption before throwing The InterruptedException handler correctly sets the interrupt flag but then throws an exception, which may prevent proper thread cleanup. Consider logging the interruption before throwing, or ensure the calling code handles interrupted threads appropriately. legacy/src/main/java/org/opensearch/sql/legacy/pit/PointInTimeHandlerImpl.java [88-93] } catch (InterruptedException e) { Thread.currentThread().interrupt(); + LOG.warn("Thread interrupted while creating PIT", e); throw pitFailure("creating", e); } catch (ExecutionException e) { throw pitFailure("creating", e); } Suggestion importance[1-10]: 4 __ Why: Adding logging for InterruptedException is a minor improvement for debugging, but the current implementation already correctly sets the interrupt flag and propagates the exception. The suggestion doesn't address a critical issue.	Low