Skip to content

feat: add scheduler jobs and content filters#931

Open
bg6jxd wants to merge 17 commits into
NanmiCoder:mainfrom
bg6jxd:codex/scheduler-multi-instance
Open

feat: add scheduler jobs and content filters#931
bg6jxd wants to merge 17 commits into
NanmiCoder:mainfrom
bg6jxd:codex/scheduler-multi-instance

Conversation

@bg6jxd

@bg6jxd bg6jxd commented Jul 2, 2026

Copy link
Copy Markdown

Summary

  • Add a multi-instance scheduler with isolated job configuration, browser profile, run history, logs, artifacts, and a scheduler WebUI.
  • Add content filters for publish time and platform metrics, wired into search, detail, and creator flows before storage/comment/media work.
  • Add scheduler artifact summaries with works list, comment word cloud, artifact actions, and docs for the new workflows.

Compatibility with latest main

  • Rebased on latest origin/main before opening this PR.
  • Kept upstream WebUI/build output changes intact; this PR adds scheduler_webui separately and does not remove existing webui assets.
  • Resolved Zhihu creator-flow changes by preserving the current url_token callback API and applying filters inside the callback.

Responsible-use notes

  • Does not increase default crawl scale.
  • Filters only reduce saved/comment/media targets.
  • Scheduler keeps one active run per job and preserves the project non-commercial learning-use boundary.

Tests

  • UV_CACHE_DIR=/private/tmp/mediacrawler-uv-cache uv run pytest tests
  • node --check scheduler_webui/app.js
  • git diff --check origin/main...HEAD

@bg6jxd bg6jxd requested a review from NanmiCoder as a code owner July 2, 2026 00:25
@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. enhancement New feature or request labels Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant