Skip to content

LCORE-2916: Slack notification for providers e2e tests #2199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jrobertboos wants to merge 1 commit into
base: main
Choose a base branch
from
Open

LCORE-2916: Slack notification for providers e2e tests #2199

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 95 additions & 1 deletion .github/workflows/e2e_tests_providers.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -352,7 +352,8 @@ jobs:
uv sync

echo "Running comprehensive e2e test suite..."
make test-e2e
set -o pipefail
make test-e2e 2>&1 | tee e2e-test-output.log

- name: Show logs on failure
if: failure()
Expand All @@ -369,3 +370,96 @@ jobs:
echo "=== lightspeed-stack (library mode) logs ==="
docker compose -f docker-compose-library.yaml logs lightspeed-stack
fi

- name: Record result for Slack summary
if: always()
env:
JOB_STATUS: ${{ job.status }}
run: |
mkdir -p ./e2e-result

# Extract the "Failing scenarios:" block that behave prints on failure.
FAILING_SCENARIOS=""
if [ -f e2e-test-output.log ]; then
FAILING_SCENARIOS=$(sed 's/\x1b\[[0-9;]*m//g' e2e-test-output.log \
| awk '/^Failing scenarios:/{flag=1; next} /^$/{flag=0} flag' \
| sed 's/^[[:space:]]*//')
fi

if [ -z "${FAILING_SCENARIOS}" ]; then
if [ "${JOB_STATUS}" == "success" ]; then
FAILING_SCENARIOS="None"
else
FAILING_SCENARIOS="Job ${JOB_STATUS} - no failing scenarios captured (see run logs)"
fi
fi

jq -n \
--arg mode "${{ matrix.mode }}" \
--arg environment "${{ matrix.environment }}" \
--arg status "${JOB_STATUS}" \
--arg failing_scenarios "${FAILING_SCENARIOS}" \
'{mode: $mode, environment: $environment, status: $status, failing_scenarios: $failing_scenarios}' \
> ./e2e-result/result.json
Comment on lines +376 to +403

@coderabbitai coderabbitai Bot Jul 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Use environment variables for matrix context to prevent shell injection risks.

While the current matrix values are statically defined and safe, it is a security best practice to pass GitHub Actions expressions (like ${{ matrix... }}) into bash scripts via environment variables rather than inline template expansion. This addresses static analysis warnings and prevents potential shell injection vulnerabilities if the matrix is ever updated to use dynamic or external inputs.

♻️ Proposed refactor 
         env:
           JOB_STATUS: ${{ job.status }}
+          MATRIX_MODE: ${{ matrix.mode }}
+          MATRIX_ENVIRONMENT: ${{ matrix.environment }}
         run: |
           mkdir -p ./e2e-result
 
           # Extract the "Failing scenarios:" block that behave prints on failure.
           FAILING_SCENARIOS=""
           if [ -f e2e-test-output.log ]; then
             FAILING_SCENARIOS=$(sed 's/\x1b\[[0-9;]*m//g' e2e-test-output.log \
               | awk '/^Failing scenarios:/{flag=1; next} /^$/{flag=0} flag' \
               | sed 's/^[[:space:]]*//')
           fi
 
           if [ -z "${FAILING_SCENARIOS}" ]; then
             if [ "${JOB_STATUS}" == "success" ]; then
               FAILING_SCENARIOS="None"
             else
               FAILING_SCENARIOS="Job ${JOB_STATUS} - no failing scenarios captured (see run logs)"
             fi
           fi
 
           jq -n \
-            --arg mode "${{ matrix.mode }}" \
-            --arg environment "${{ matrix.environment }}" \
+            --arg mode "${MATRIX_MODE}" \
+            --arg environment "${MATRIX_ENVIRONMENT}" \
             --arg status "${JOB_STATUS}" \
             --arg failing_scenarios "${FAILING_SCENARIOS}" \
             '{mode: $mode, environment: $environment, status: $status, failing_scenarios: $failing_scenarios}' \
             > ./e2e-result/result.json
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
env:
JOB_STATUS: ${{ job.status }}
run: |
mkdir -p ./e2e-result
# Extract the "Failing scenarios:" block that behave prints on failure.
FAILING_SCENARIOS=""
if [ -f e2e-test-output.log ]; then
FAILING_SCENARIOS=$(sed 's/\x1b\[[0-9;]*m//g' e2e-test-output.log \
| awk '/^Failing scenarios:/{flag=1; next} /^$/{flag=0} flag' \
| sed 's/^[[:space:]]*//')
fi
if [ -z "${FAILING_SCENARIOS}" ]; then
if [ "${JOB_STATUS}" == "success" ]; then
FAILING_SCENARIOS="None"
else
FAILING_SCENARIOS="Job ${JOB_STATUS} - no failing scenarios captured (see run logs)"
fi
fi
jq -n \
--arg mode "${{ matrix.mode }}" \
--arg environment "${{ matrix.environment }}" \
--arg status "${JOB_STATUS}" \
--arg failing_scenarios "${FAILING_SCENARIOS}" \
'{mode: $mode, environment: $environment, status: $status, failing_scenarios: $failing_scenarios}' \
> ./e2e-result/result.json
env:
JOB_STATUS: ${{ job.status }}
MATRIX_MODE: ${{ matrix.mode }}
MATRIX_ENVIRONMENT: ${{ matrix.environment }}
run: |
mkdir -p ./e2e-result
# Extract the "Failing scenarios:" block that behave prints on failure.
FAILING_SCENARIOS=""
if [ -f e2e-test-output.log ]; then
FAILING_SCENARIOS=$(sed 's/\x1b\[[0-9;]*m//g' e2e-test-output.log \
| awk '/^Failing scenarios:/{flag=1; next} /^$/{flag=0} flag' \
| sed 's/^[[:space:]]*//')
fi
if [ -z "${FAILING_SCENARIOS}" ]; then
if [ "${JOB_STATUS}" == "success" ]; then
FAILING_SCENARIOS="None"
else
FAILING_SCENARIOS="Job ${JOB_STATUS} - no failing scenarios captured (see run logs)"
fi
fi
jq -n \
--arg mode "${MATRIX_MODE}" \
--arg environment "${MATRIX_ENVIRONMENT}" \
--arg status "${JOB_STATUS}" \
--arg failing_scenarios "${FAILING_SCENARIOS}" \
'{mode: $mode, environment: $environment, status: $status, failing_scenarios: $failing_scenarios}' \
> ./e2e-result/result.json
🧰 Tools
🪛 zizmor (1.26.1)

[warning] 398-398: code injection via template expansion (template-injection): may expand into attacker-controllable code

(template-injection)

[warning] 399-399: code injection via template expansion (template-injection): may expand into attacker-controllable code

(template-injection)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e_tests_providers.yaml around lines 376 - 403, Update
the workflow step to expose matrix.mode and matrix.environment through
environment variables alongside JOB_STATUS, then pass those variables to jq via
the existing --arg options instead of interpolating GitHub Actions expressions
inside the run script. Preserve the current result.json fields and values.

Source: Linters/SAST tools


- name: Upload result artifact
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-result-${{ matrix.mode }}-${{ matrix.environment }}
path: ./e2e-result/result.json
retention-days: 1

notify:
name: Notify Slack
needs: [e2e_tests]
if: always()
runs-on: ubuntu-latest
steps:
- name: Download all result artifacts
continue-on-error: true
uses: actions/download-artifact@v4
with:
pattern: e2e-result-*
path: ./results

- name: Build and send consolidated Slack report
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
WORKFLOW_RESULT: ${{ needs.e2e_tests.result }}
run: |
if [ -z "${SLACK_WEBHOOK_URL}" ]; then
echo "SLACK_WEBHOOK_URL is not set, skipping Slack notification"
exit 0
fi

ALL_RESULTS=$(find ./results -name 'result.json' 2>/dev/null -exec cat {} \; | jq -s '.' 2>/dev/null || echo '[]')

echo "Workflow result: ${WORKFLOW_RESULT}"

# Base payload guarantees every mode/environment key is present even if a
# matrix job never uploaded a result artifact (e.g. it was skipped/cancelled).
BASE_FIELDS='{
"server_azure": "N/A",
"server_vertexai": "N/A",
"server_watsonx": "N/A",
"server_bedrock": "N/A",
"library_azure": "N/A",
"library_vertexai": "N/A",
"library_watsonx": "N/A",
"library_bedrock": "N/A"
}'

# Build one field per "<mode>_<environment>" key holding that job's failing scenarios.
payload=$(echo "${ALL_RESULTS}" | jq \
--argjson fields "${BASE_FIELDS}" \
--arg run "${RUN_URL}" \
'{run_url: $run} + (reduce .[] as $r ($fields; . + {(($r.mode + "_" + $r.environment)): $r.failing_scenarios}))')

echo "Payload: ${payload}"
Comment on lines +455 to +460

@coderabbitai coderabbitai Bot Jul 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Use compact JSON output for the payload.

jq pretty-prints JSON by default, which introduces internal newlines into the payload bash variable. While curl generally handles multiline data properly, generating a compact JSON string using the -c flag is safer for bash variables and produces cleaner, more readable log output on the subsequent echo command.

♻️ Proposed refactor 
           # Build one field per "<mode>_<environment>" key holding that job's failing scenarios.
-          payload=$(echo "${ALL_RESULTS}" | jq \
+          payload=$(echo "${ALL_RESULTS}" | jq -c \
             --argjson fields "${BASE_FIELDS}" \
             --arg run "${RUN_URL}" \
             '{run_url: $run} + (reduce .[] as $r ($fields; . + {(($r.mode + "_" + $r.environment)): $r.failing_scenarios}))')
 
           echo "Payload: ${payload}"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
payload=$(echo "${ALL_RESULTS}" | jq \
--argjson fields "${BASE_FIELDS}" \
--arg run "${RUN_URL}" \
'{run_url: $run} + (reduce .[] as $r ($fields; . + {(($r.mode + "_" + $r.environment)): $r.failing_scenarios}))')
echo "Payload: ${payload}"
payload=$(echo "${ALL_RESULTS}" | jq -c \
--argjson fields "${BASE_FIELDS}" \
--arg run "${RUN_URL}" \
'{run_url: $run} + (reduce .[] as $r ($fields; . + {(($r.mode + "_" + $r.environment)): $r.failing_scenarios}))')
echo "Payload: ${payload}"
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e_tests_providers.yaml around lines 455 - 460, Update
the jq invocation in the payload construction to request compact JSON output
with the -c option. Keep the existing --argjson, --arg, and transformation logic
unchanged so payload contains a single-line JSON string before the subsequent
echo.


if ! curl -fsSL -X POST -H 'Content-type: application/json' \
--data "${payload}" "${SLACK_WEBHOOK_URL}"; then
echo "::warning::Failed to post Slack notification (workflow result unchanged)"
fi
Loading