Skip to content

feat(xhs): add "collect" crawler type to fetch collected/收藏 notes#933

Open
dake6767 wants to merge 2 commits into
NanmiCoder:mainfrom
dake6767:feat/xhs-collected-notes
Open

feat(xhs): add "collect" crawler type to fetch collected/收藏 notes#933
dake6767 wants to merge 2 commits into
NanmiCoder:mainfrom
dake6767:feat/xhs-collected-notes

Conversation

@dake6767

@dake6767 dake6767 commented Jul 4, 2026

Copy link
Copy Markdown

What

Adds a new CRAWLER_TYPE="collect" for Xiaohongshu that crawls the
logged-in user's collected (收藏) notes. Previously the xhs crawler only
supported search / detail / creator, so there was no way to export
your own 收藏.

How

  • client.py: get_self_user_id() resolves the numeric user_id via
    /api/sns/web/v2/user/me (note: selfinfo only returns red_id, not the
    numeric user_id the collect API needs), plus get_notes_by_collect() /
    get_all_notes_by_collect() paging /api/sns/web/v2/note/collect/page.
  • core.py: new dispatch branch + get_collected_notes(), reusing the
    existing note-detail / media / comment pipeline — so ENABLE_GET_COMMENTS,
    CRAWLER_MAX_NOTES_COUNT, and all storage backends work unchanged.
  • config: XHS_COLLECT_USER_ID_LIST (empty = own collection; or list
    bare user_ids / full profile URLs with xsec_token for others' public
    collections) and CrawlerTypeEnum.COLLECT in cmd_arg (otherwise the
    config value silently falls back to search).

Usage

# config/base_config.py
PLATFORM = "xhs"
CRAWLER_TYPE = "collect"
# config/xhs_config.py
XHS_COLLECT_USER_ID_LIST = []   # empty = the logged-in user's own collection
python main.py            # or: python main.py --platform xhs --type collect

Output goes to the usual data/xhs/... sink (jsonl/csv/db/...), written as
collect_contents_* / collect_comments_*.

Testing

Verified end-to-end against a live logged-in account: resolves the user_id,
pages the collection, fetches note details + comments, and writes the output
files. Only xhs implements collect; other platforms treat the type as a
no-op via the existing else branch, so nothing else is affected.

Adds a new CRAWLER_TYPE="collect" for Xiaohongshu that crawls the
logged-in user's collected (收藏) notes. Previously only search, detail
and creator modes were supported, so there was no way to export one's
own collection.

- client: get_self_user_id() via /api/sns/web/v2/user/me, plus
  get_notes_by_collect() / get_all_notes_by_collect() hitting
  /api/sns/web/v2/note/collect/page
- core: dispatch branch + get_collected_notes(), reusing the existing
  note-detail + comment pipeline
- config: XHS_COLLECT_USER_ID_LIST (empty = own collection; may also
  list bare user_ids or full profile URLs for others' public collections)
- cmd_arg: add CrawlerTypeEnum.COLLECT
@dake6767 dake6767 requested a review from NanmiCoder as a code owner July 4, 2026 00:29
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels Jul 4, 2026
- tests/test_xhs_collect.py: `--type collect` is accepted (not downgraded
  to search), empty XHS_COLLECT_USER_ID_LIST resolves the logged-in user's
  own id, and a configured profile URL is parsed into user_id + xsec_token
- README: document `--type collect` and XHS_COLLECT_USER_ID_LIST usage
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant