TTLabTweetCorpus

Project website: https://texttechnologylab.github.io/TTLabTweetCorpus/

TTLabTweetCorpus is a research project developed in the context of the Text Technology Lab at Goethe University Frankfurt. It connects public X posts and their media data with parliamentary records from GerParCor and supports downstream processing with DUUI.

TTLabTweetCrawler is the download component of TTLabTweetCorpus. It collects public X posts with metadata and attached images or videos. Media can be embedded as Base64 and extracted as separate files.

The resulting datasets can be connected with parliamentary records from the German Parliamentary Corpus (GerParCor) and passed to Docker Unified UIMA Interface (DUUI) pipelines for downstream processing. The Python package includes commands for downloading GerParCor XMI documents and matching complete tweets against individual parliamentary sentences with cosine similarity.

Project structure

• python/ contains the installable ttlab_tweet_crawler package and all crawling and media-processing commands.
• docs/ contains the static project website.
• A future Java-based DUUI pipeline can live in a separate java/ subproject.

To preview it locally from the repository root, run:

python3 -m http.server 8000 --directory docs

Then open http://localhost:8000/. Every push that changes docs/ on main deploys the site through GitHub Pages; the workflow can also be started manually.

Research analysis

In the associated study, we used DUUI to annotate all 33,921 non-empty tweets with nine text-based models: three sentiment models, three topic models, and three emotion models. To examine the relationships among their outputs, we trained one classifier for each model, using the annotations produced by the other eight models as input features. The results show that the model outputs are mutually predictable, revealing systematic relationships across sentiment, topic, and emotion annotations.

Citation

If you use TTLabTweetCorpus or its TTLabTweetCrawler component in academic work, please cite the associated accepted KONVENS 2026 paper:

@inproceedings{bagci:et:al:2026,
  title     = {Extending a Parliamentary Corpus with {MP}s{\textquoteright} Tweets:
               Automatic Annotation and Evaluation Using {TTLABTWEETCORPUS}},
  author    = {Mevlüt Bagci and Ali Abusaleh and Daniel Baumartz and Alexander Mehler
               and Giuseppe Abrami and Maxim Konca},
  booktitle = {KONVENS 2026 - Context matters: NLP beyond Text},
  year      = {2026},
  address   = {Hamburg, Germany},
  keywords  = {Corpora, Tweets, Political Data, Text, Media classification},
  note      = {Accepted}
}

Machine-readable citation metadata is available in CITATION.cff, and the BibTeX entry is available separately in CITATION.bib. The publication URL and DOI will be added when available.

X User ID Collector

This command-line tool resolves X usernames to numeric user IDs through the official X API v2. It is based on the repository's original name_collector/collect_user_id.py, but it does not contain or save a bearer token.

Requirements

• Python 3.10 or newer
• An X developer account and a bearer token with access to GET /2/users/by

Installation

Open a terminal in this directory and create an isolated Python environment:

cd TTLabTweetCorpus
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install --editable ./python

Usage

Pass usernames directly

ttlab-collect-user-ids example_user research_account public_account \
  --output output/user_ids.json \
  --bearer-token "YOUR_BEARER_TOKEN"

Usernames may include a leading @.

Pass the token through an environment variable

This avoids exposing the token in the shell's process list and command history:

export X_BEARER_TOKEN="YOUR_BEARER_TOKEN"
ttlab-collect-user-ids example_user research_account --output output/user_ids.json

Read usernames from a file

For a text file, place one username on each line:

example_user
@research_account
public_account

Then run:

ttlab-collect-user-ids \
  --input usernames.txt \
  --output output/user_ids.json \
  --bearer-token "YOUR_BEARER_TOKEN"

JSON input is also supported. The tool recognizes:

• a list of username strings;
• objects with an X_User_Name list;
• the existing output_users.json structure containing X properties;
• lists of objects containing username, X_User_Name, or X.

Example using an existing username dataset:

ttlab-collect-user-ids \
  --input ../name_collector/data/X_User_Names.json \
  --output output/research_user_ids.json \
  --bearer-token "YOUR_BEARER_TOKEN"

Direct usernames and --input may be combined. Duplicate names are removed automatically. More than 100 usernames are split into separate API requests and combined into one output file.

Output

The output preserves the X API response structure:

{
  "data": [
    {
      "id": "123456",
      "name": "Example Name",
      "username": "example"
    }
  ],
  "errors": []
}

Unknown or unavailable usernames appear in errors. Authentication, permission, rate-limit, network, and invalid-response errors are printed to standard error and cause a non-zero exit status.

Security

Never add a real bearer token to source code, README files, or version control. Prefer the X_BEARER_TOKEN environment variable for regular use. If a token has previously been committed or shared, revoke and regenerate it in the X Developer Portal.

Tweet collection

The ttlab-collect-tweets command supports user timelines and search queries. Every API response page is stored as compressed JSON so that tweet data and API includes remain intact.

Display all available options:

ttlab-collect-tweets --help
ttlab-collect-tweets per-user --help
ttlab-collect-tweets per-query --help

Collect tweets per user

Use either --username or --user-id:

ttlab-collect-tweets per-user \
  --username example_user \
  --output output/example_user \
  --start-time 2025-01-01T00:00:00Z \
  --end-time 2025-02-01T00:00:00Z \
  --max-tweets 500 \
  --exclude replies,retweets \
  --bearer-token "YOUR_BEARER_TOKEN"

Using a numeric user ID:

ttlab-collect-tweets per-user \
  --user-id 123456789 \
  --output output/example_user \
  --max-pages 5 \
  --bearer-token "YOUR_BEARER_TOKEN"

Collect tweets per query

The value of --query is passed to X without modification. Operators such as from:, has:media, and -is:retweet can therefore be combined freely:

ttlab-collect-tweets per-query \
  --query "climate_policy has:media -is:retweet lang:de" \
  --output output/climate_policy_query \
  --search-endpoint recent \
  --max-tweets 1000 \
  --bearer-token "YOUR_BEARER_TOKEN"

To request only tweets containing a photo, GIF, or video without writing the X search operator manually, add --media-only:

ttlab-collect-tweets per-query \
  --query "WM2026" \
  --media-only \
  --output output/wm2026_media \
  --max-tweets 100 \
  --bearer-token "YOUR_BEARER_TOKEN"

Internally, this sends WM2026 has:media to X. If the query already contains has:media, has:images, or has:video_link, the tool does not append another media operator.

To exclude retweets without writing the search operator manually, add --exclude-retweets:

ttlab-collect-tweets per-query \
  --query "WM2026" \
  --media-only \
  --exclude-retweets \
  --output output/wm2026_original_media \
  --max-tweets 100 \
  --bearer-token "YOUR_BEARER_TOKEN"

This sends WM2026 has:media -is:retweet to X. An existing -is:retweet operator is not duplicated.

Use --search-endpoint all for full-archive search if the X API plan associated with the token permits it.

Token through the environment

The environment variable works for both collection modes and avoids putting the token in command history:

export X_BEARER_TOKEN="YOUR_BEARER_TOKEN"
ttlab-collect-tweets per-user \
  --username example_user \
  --output output/example_user \
  --max-tweets 100

Shared collection parameters

Both modes accept:

• --start-time and --end-time
• --since-id and --until-id
• --pagination-token
• --page-size (10 to 100)
• --max-pages and --max-tweets
• --timeout and --max-retries
• --no-wait-rate-limit
• --tweet-fields, --expansions, --media-fields, --poll-fields, --user-fields, and --place-fields

Run the subcommand-specific --help command for the authoritative parameter list and defaults.

Default data coverage

The defaults request all currently documented public tweet fields supported by both user timelines and search, including Articles, cards, Communities, context annotations, edit information, long-form Notes, media metadata, public metrics, referenced tweets, paid-partnership labels, suggested source links, and withholding information. Related Article media, attached media, polls, authors, mentions, places, and referenced-tweet media are expanded automatically.

The defaults intentionally exclude non_public_metrics, organic_metrics, and promoted_metrics for tweets and media. X documents these fields as requiring user-context authentication, so requesting them with an app-only bearer token can fail. They remain available through --tweet-fields and --media-fields when the supplied authentication supports them. The scopes field is requested, but X only returns it when relevant to a promoted post.

GerParCor download and tweet-sentence matching

The following workflow downloads parliamentary documents and compares complete tweets with the individual sentences already annotated in GerParCor XMI files.

Preview available documents

Use --list-only to query an inclusive date range without downloading files:

ttlab-download-gerparcor \
  --parliament Bundestag \
  --start 2010-01-01 \
  --end 2010-01-31 \
  --list-only \
  --insecure

The --insecure option disables TLS certificate verification and should only be used when the GerParCor server cannot otherwise be reached.

Download XMI documents

ttlab-download-gerparcor \
  --parliament Bundestag \
  --start 2010-01-01 \
  --end 2010-01-31 \
  --output gerparcor_documents \
  --insecure

Existing output files are retained, so the command can safely be rerun.

Download documents manually

Documents can also be selected and downloaded through the GerParCor web interface. Select a country, division, and parliament, navigate through the result pages, and use the download icon in a document row to retrieve its XMI file. The automated command is more convenient for reproducible date-range downloads; the web interface is useful for inspecting and downloading individual documents.

Retain the best sentence matches

This command embeds every complete tweet and every individual XMI sentence, then retains the five sentences with the highest cosine similarity per tweet:

ttlab-match-tweets-xmi \
  --tweets tweet_json_directory \
  --xmi xmi_directory \
  --output output/top_sentence_matches.json.gz \
  --top-k 5 \
  --minimum-score 0.4 \
  --device cpu

Retain every comparison

For small datasets, all tweet-sentence comparisons can be stored:

ttlab-match-tweets-xmi \
  --tweets tweet_json_directory \
  --xmi xmi_directory \
  --output output/all_sentence_matches.json.gz \
  --all-matches \
  --minimum-score=-1 \
  --include-text \
  --device cpu

All-matches output contains tweet_count * sentence_count comparison records. Use compressed .json.gz output and prefer top-k matching for large corpora. Without --include-text, results contain compact source locations: a tweet JSON file and JSON Pointer, plus an XMI file, XMI ID, and begin/end offsets.

Select an embedding model

The default model is sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2. Any compatible Sentence Transformers model can be selected explicitly:

ttlab-match-tweets-xmi \
  --tweets tweet_json_directory \
  --xmi xmi_directory \
  --output output/custom_model_matches.json.gz \
  --model sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 \
  --top-k 10 \
  --device cuda

The model is downloaded on first use and reused from the local cache. Use --device cpu when CUDA is unavailable. Run both commands with --help for their complete parameter reference:

ttlab-download-gerparcor --help
ttlab-match-tweets-xmi --help

Embed downloaded media as Base64

ttlab-download-media reads the .json.gz pages created by ttlab-collect-tweets. It downloads every available photo, video, and animated GIF referenced in includes.media, converts the binary content to Base64, and writes enriched .json.gz pages to a separate directory. The original API downloads are not modified.

ttlab-download-media \
  --input output/wm2026_media \
  --output output/wm2026_media_base64

For a single page:

ttlab-download-media \
  --input output/wm2026_media/tweets_query_WM2026_page_0001.json.gz \
  --output output/wm2026_media_base64

Use --recursive when the input directory contains nested directories. Videos are selected from variants using the highest available MP4 bitrate. Photos use their direct url. Duplicate URLs encountered during one run are downloaded only once.

The enriched media object has this structure:

{
  "media_key": "13_123456",
  "type": "photo",
  "embedded_media": {
    "source_url": "https://pbs.twimg.com/media/example.jpg",
    "content_type": "image/jpeg",
    "size_bytes": 12345,
    "base64": "/9j/4AAQSkZJRg..."
  }
}

Useful options:

• --timeout 120 changes the timeout per media request.
• --max-bytes 104857600 rejects individual files larger than 100 MiB.
• --fail-fast stops on the first failed download. By default, errors are recorded as embedded_media_error and processing continues.

Base64 increases binary data size by approximately one third before gzip compression. Videos can therefore produce very large JSON files and require substantial memory because each media file is encoded in memory.

Extract Base64 media to files

ttlab-extract-media converts the embedded Base64 content back into image and video files. It uses tweet IDs as filenames:

• One attachment: 123456789.jpg
• Multiple attachments: 123456789_1.jpg, 123456789_2.mp4

Run it on the enriched WM2026 pages:

ttlab-extract-media \
  --input output/wm2026_original_media_base64 \
  --output output/wm2026_extracted_media

The extractor handles primary tweets from data and referenced tweets from includes.tweets. It maps each tweet's attachments.media_keys to the matching object in includes.media, decodes embedded_media.base64, and chooses the file extension from content_type.

Identical files already present in the output directory are skipped, which makes reruns safe. The tool refuses to overwrite an existing file with different content. Use --recursive if the enriched input files are in nested directories.

Documentation website

A static project website is available in docs/. Preview it locally with:

python3 -m http.server 8000 --directory docs

Then open http://localhost:8000 in a browser.

When TTLabTweetCorpus is published on GitHub, open Settings → Pages and select GitHub Actions under Build and deployment → Source. The Deploy documentation workflow publishes the static /docs directory; no separate build step or external framework is required.

Licensing

• The software is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0-only).
• The original documentation and GitHub Page are licensed under CC BY 4.0.
• Project logos, third-party names and marks, X posts, API responses, and downloaded media are excluded from those licenses. See THIRD_PARTY_NOTICES.md.

Copyright 2026 Text Technology Lab, Goethe University Frankfurt.