Skip to content

WorkerFactory.awaitTermination races with command-worker cleanup #2951

Description

@eamsden

Description

During shutdown, WorkerFactory asynchronously clears workerCommandWorker:

workerCommandWorker = null;

Meanwhile, awaitTermination checks the field and later dereferences it inside a lambda:

if (workerCommandWorker != null) {
  long t = timeoutMillis;
  ShutdownManager.runAndGetRemainingTimeoutMs(
      t, () -> workerCommandWorker.awaitTermination(t, TimeUnit.MILLISECONDS));
}

The field can become null between the check and lambda execution, producing:

java.lang.NullPointerException: Cannot invoke
"io.temporal.internal.worker.SuspendableWorker.awaitTermination(long, java.util.concurrent.TimeUnit)"
because "this.workerCommandWorker" is null
    at io.temporal.worker.WorkerFactory.lambda$awaitTermination$12(WorkerFactory.java:519)

This was observed during normal AWS Lambda worker shutdown using SDK 1.37.0.

Expected behavior

awaitTermination should complete without an exception and consistently track the command worker’s termination.

Suggested resolution

A local capture prevents the immediate NPE but leaves timing-dependent lifecycle behavior.

The likely fix is to retain workerCommandWorker for the lifetime of WorkerFactory, as is done for ordinary workers, and remove the asynchronous null assignment. This also prevents isTerminated and awaitTermination from prematurely ignoring a command worker whose shutdown future completed before every underlying executor terminated.

If clearing the reference is intentional, it should use an explicit thread-safe lifecycle transition and stable termination primitive.

Environment

  • Temporal Java SDK 1.37.0
  • DefaultLambdaWorkerRuntime
  • AWS Lambda
  • Temporal Cloud

Introduced in sdk-java PR #2917.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions