ENT-14140: psql_wrapper.sh: retry psql commands on transient failures

[larsewi]

Observed a race condition in CI where bundle agent superhub_schema interacts with postgres shortly after service restart.

03:12:04 systemd: Stopping CFEngine Enterprise PostgreSQL Database...
03:12:04 systemd: Started CFEngine Enterprise PostgreSQL Database.
03:12:04 cf-agent: Executing ... psql_wrapper.sh cfdb select superhub_schema(...)
03:12:05 cf-agent: returned code '2' defined as promise failed

Fixed by gating superhub_schema, ensure_feeders, and imported_data on a persistent class set by the cf-postgres restart.

Ticket: ENT-14140

Backported to:

• ENT-14140: psql_wrapper.sh: retry psql commands on transient failures (3.27.x) #3167
• ENT-14140: psql_wrapper.sh: retry psql commands on transient failures (3.24.x) #3168

[craigcomstock]

This doesn't feel quite right to me. It would seem we need more of a sequence of actions and not a class/gate situation. We need the restart to finish and then run superhub_schema(). With this solution superhub_schema() would be run at next agent interval, which is OK but maybe not ideal. Could we instead of gating on recent restart gate on postgresql up and ready in hopes that superhub_schema() could run in the same agent run?

[larsewi]

2 if the connection to the server went bad and the session was not interactive

• https://www.postgresql.org/docs/current/app-psql.html

@craigcomstock, @nickanderson what if we have the psql_wrapper.sh retry in case of return code 2 ?

[nickanderson]

2 if the connection to the server went bad and the session was not interactive

• https://www.postgresql.org/docs/current/app-psql.html

@craigcomstock, @nickanderson what if we have the psql_wrapper.sh retry in case of return code 2 ?

yeah I think that would be better.

[larsewi]

2 if the connection to the server went bad and the session was not interactive

• https://www.postgresql.org/docs/current/app-psql.html

@craigcomstock, @nickanderson what if we have the psql_wrapper.sh retry in case of return code 2 ?

yeah I think that would be better.

The only thing @nickanderson is that it will cause the agent to hang while it bootstraps. Or should it perhaps run these commands in the background?

[nickanderson]

The only thing @nickanderson is that it will cause the agent to hang while it bootstraps. Or should it perhaps run these commands in the background?

Hang permanently, or just be slow? Permanent hang needs to be avoided. Just reading the code there it looks like it might hang for up to 30 seconds while re-trying. Also, this hang would be limited to the hub bootstrapping to itself, is that right?

[larsewi]

The only thing @nickanderson is that it will cause the agent to hang while it bootstraps. Or should it perhaps run these commands in the background?

Hang permanently, or just be slow? Permanent hang needs to be avoided. Just reading the code there it looks like it might hang for up to 30 seconds while re-trying. Also, this hang would be limited to the hub bootstrapping to itself, is that right?

No, every agent run on the super hub is affected. Feeders are not as far as I can see (due to am_superhub::). But there is no delay on successful runs. The delay should only be there when postgres is not responding.

[craigcomstock]

better. yes.

[larsewi]

@cf-bottom Jenkins please :)

[cf-bottom]

Alright, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/13880/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-13880/