feat: Telegram downtime & recovery notifications#5
Open
nebjak wants to merge 2 commits into
Open
Conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds optional Telegram Bot API notifications to the backend so operators receive DOWN and recovery alerts for monitored services, with restart-safe seeding from DB history.
Changes:
- Introduces a
Notifierabstraction for TelegramsendMessagefan-out with per-request timeouts and message formatting tests. - Extends the worker check loop with a per-service notification state machine (failure threshold, down/recovery transitions) and consolidates DB insert behavior via
CheckOutcome. - Adds
[telegram]configuration support, wiring, and DB helper to seed “already down” state across restarts.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tickers.toml | Documents the new optional [telegram] config section and defaults. |
| backend/src/worker.rs | Adds notification state tracking and integrates notifier calls into the check loop. |
| backend/src/notifier.rs | New Telegram notifier implementation + message formatting/duration unit tests. |
| backend/src/main.rs | Wires the new notifier module. |
| backend/src/db.rs | Adds get_last_is_up() to seed notification state from persisted history. |
| backend/src/config.rs | Adds TelegramConfig, enablement logic, and partial-config warning. |
| backend/Cargo.toml | Enables reqwest JSON support for Telegram requests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+6
to
+7
| /// Cap on a single Telegram API call so a slow/hung send can't stall a check loop. | ||
| const SEND_TIMEOUT: Duration = Duration::from_secs(10); |
Comment on lines
+189
to
+197
| notifier | ||
| .notify_down(service, outcome.error_message.as_deref()) | ||
| .await; | ||
| state.alerted_down = true; | ||
| info!( | ||
| service_id = %service.id, | ||
| failures = state.consecutive_failures, | ||
| "Sent downtime notification" | ||
| ); |
Comment on lines
+176
to
+180
| let down_for = state.down_since.map(|t| t.elapsed()); | ||
| notifier.notify_recovery(service, down_for).await; | ||
| state.alerted_down = false; | ||
| state.down_since = None; | ||
| info!(service_id = %service.id, "Sent recovery notification"); |
| let detail = resp.text().await.unwrap_or_default(); | ||
| warn!(%chat_id, %status, detail = %detail, "Telegram API returned an error"); | ||
| } | ||
| Err(e) => error!(%chat_id, error = %e, "Failed to send Telegram notification"), |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds outbound Telegram notifications so you get pinged when a monitored service goes down — and again when it recovers — instead of having to watch the page.
One bot per tickers instance; every alert fans out to all configured chats.
Config
New optional global
[telegram]section (omit it to disable — zero overhead, app behaves exactly as before):Behavior
failure_threshold(default 3) consecutive failed checks — filters transient blips.parse_mode) → no Markdown/HTML escaping pitfalls.Example messages:
Changes
notifier.rs(new) —Notifierover the TelegramsendMessageAPI; reuses the worker'sreqwest::Client.from_configreturnsNonewhen disabled. + unit tests.worker.rs—perform_checkreturnsCheckOutcome(two DB inserts consolidated into one); per-serviceNotifyStatetransition state machine drives the alerts.config.rs—[telegram]section +is_enabled()+ partial-config warning.db.rs—get_last_is_up()to seed state across restarts.main.rs/Cargo.toml/tickers.toml— module wiring, reqwestjsonfeature, example config.Tests
cargo build,cargo clippy --all-targets,cargo testall clean (5 new unit tests for message formatting + duration humanizing).Manual end-to-end with a real bot is the remaining check (needs a live @Botfather token + chat_id).
🤖 Generated with Claude Code