Skip to content

Improve scraper resilience and patch dependabot security alerts#16

Merged
valuecodes merged 4 commits into
mainfrom
fix-errors
May 23, 2026
Merged

Improve scraper resilience and patch dependabot security alerts#16
valuecodes merged 4 commits into
mainfrom
fix-errors

Conversation

@valuecodes
Copy link
Copy Markdown
Owner

What

How to test

pnpm typecheck
pnpm lint
pnpm test

Security review

Closes 15 Dependabot alerts (hono, esbuild, postcss, brace-expansion); 3 vite alerts intentionally unresolved.

Copilot AI review requested due to automatic review settings May 23, 2026 11:19
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves scraping reliability in the operator by truncating oversized HTTP responses (instead of failing) and improves browser-scraper error signaling by distinguishing “browser unavailable” outages (503) from other render failures. It also updates dependencies (notably hono) and adds pnpm overrides to address Dependabot security alerts.

Changes:

  • Truncate native fetch response bodies at 2MB and propagate a truncated flag through scrapeUrl.
  • Add browser error classification to return HTTP 503 for likely Cloudflare browser-worker outages.
  • Patch dependency alerts via hono bump and pnpm.overrides (plus lockfile updates).

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
apps/operator/src/services/scrape.ts Implements body-size truncation (2MB) and propagates truncated into scrapeUrl results.
apps/operator/src/services/scrape.test.ts Updates tests to assert truncation behavior instead of failure on >2MB bodies.
apps/browser-scraper/src/services/playwright.ts Adds classifyBrowserError to separate “browser unavailable” from internal failures.
apps/browser-scraper/src/services/playwright.test.ts Adds unit tests for classifyBrowserError.
apps/browser-scraper/src/index.ts Returns 503 for “browser unavailable” errors; keeps 500 for other unexpected failures.
package.json Adds pnpm.overrides for vulnerable transitive dependencies.
apps/operator/package.json Bumps hono from 4.12.12 to 4.12.18.
pnpm-lock.yaml Captures dependency bumps and override-resolved versions.
.claude/settings.json Adjusts the gh api allowlist pattern syntax.
Files not reviewed (1)
  • pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread apps/operator/src/services/scrape.ts Outdated
Comment thread package.json Outdated
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1f55b8f328

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/operator/src/services/scrape.ts Outdated
@valuecodes valuecodes merged commit 8d40364 into main May 23, 2026
7 checks passed
@valuecodes valuecodes deleted the fix-errors branch May 23, 2026 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants