Skip to content

feat: Telegram downtime & recovery notifications#5

Open
nebjak wants to merge 2 commits into
mainfrom
feat/telegram-notifications
Open

feat: Telegram downtime & recovery notifications#5
nebjak wants to merge 2 commits into
mainfrom
feat/telegram-notifications

Conversation

@nebjak

@nebjak nebjak commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

What

Adds outbound Telegram notifications so you get pinged when a monitored service goes down — and again when it recovers — instead of having to watch the page.

One bot per tickers instance; every alert fans out to all configured chats.

Config

New optional global [telegram] section (omit it to disable — zero overhead, app behaves exactly as before):

[telegram]
bot_token = "123456:ABC-DEF..."            # from @BotFather
chat_ids = ["123456789", "-1001234567890"] # users, groups, or @channel
failure_threshold = 3                       # consecutive fails before a DOWN alert (default 3)

Behavior

  • DOWN alert fires after failure_threshold (default 3) consecutive failed checks — filters transient blips.
  • Recovery fires on the first success after an alert, including outage duration.
  • State is seeded from the DB on startup, so a restart mid-outage doesn't re-alert an already-down service (a later recovery still fires).
  • Messages are plain text (no parse_mode) → no Markdown/HTML escaping pitfalls.
  • Send failures are logged and never break the check loop; 10s per-send timeout.

Example messages:

🔴 Example Service is DOWN
https://api.example.com/health
Timeout after 10000ms
2026-06-03 14:32 UTC

✅ Example Service recovered
was down for 4m 12s
2026-06-03 14:36 UTC

Changes

  • notifier.rs (new) — Notifier over the Telegram sendMessage API; reuses the worker's reqwest::Client. from_config returns None when disabled. + unit tests.
  • worker.rsperform_check returns CheckOutcome (two DB inserts consolidated into one); per-service NotifyState transition state machine drives the alerts.
  • config.rs[telegram] section + is_enabled() + partial-config warning.
  • db.rsget_last_is_up() to seed state across restarts.
  • main.rs / Cargo.toml / tickers.toml — module wiring, reqwest json feature, example config.

Tests

cargo build, cargo clippy --all-targets, cargo test all clean (5 new unit tests for message formatting + duration humanizing).

Manual end-to-end with a real bot is the remaining check (needs a live @Botfather token + chat_id).

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds optional Telegram Bot API notifications to the backend so operators receive DOWN and recovery alerts for monitored services, with restart-safe seeding from DB history.

Changes:

  • Introduces a Notifier abstraction for Telegram sendMessage fan-out with per-request timeouts and message formatting tests.
  • Extends the worker check loop with a per-service notification state machine (failure threshold, down/recovery transitions) and consolidates DB insert behavior via CheckOutcome.
  • Adds [telegram] configuration support, wiring, and DB helper to seed “already down” state across restarts.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
tickers.toml Documents the new optional [telegram] config section and defaults.
backend/src/worker.rs Adds notification state tracking and integrates notifier calls into the check loop.
backend/src/notifier.rs New Telegram notifier implementation + message formatting/duration unit tests.
backend/src/main.rs Wires the new notifier module.
backend/src/db.rs Adds get_last_is_up() to seed notification state from persisted history.
backend/src/config.rs Adds TelegramConfig, enablement logic, and partial-config warning.
backend/Cargo.toml Enables reqwest JSON support for Telegram requests.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread backend/src/notifier.rs
Comment on lines +6 to +7
/// Cap on a single Telegram API call so a slow/hung send can't stall a check loop.
const SEND_TIMEOUT: Duration = Duration::from_secs(10);
Comment thread backend/src/worker.rs
Comment on lines +189 to +197
notifier
.notify_down(service, outcome.error_message.as_deref())
.await;
state.alerted_down = true;
info!(
service_id = %service.id,
failures = state.consecutive_failures,
"Sent downtime notification"
);
Comment thread backend/src/worker.rs
Comment on lines +176 to +180
let down_for = state.down_since.map(|t| t.elapsed());
notifier.notify_recovery(service, down_for).await;
state.alerted_down = false;
state.down_since = None;
info!(service_id = %service.id, "Sent recovery notification");
Comment thread backend/src/notifier.rs
let detail = resp.text().await.unwrap_or_default();
warn!(%chat_id, %status, detail = %detail, "Telegram API returned an error");
}
Err(e) => error!(%chat_id, error = %e, "Failed to send Telegram notification"),
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants