Collection of Java-based Web crawlers

Prerequisites

• Maven 3
• JDK 21

How to build

mvn clean install

Common crawler functionality

• Your crawler should extend WebCrawler base crawler class
• DTO class which describes collected data should implement CrawlerData marker interface

Crawler for Orthodox torrent tracker pravtor.ru

Check PravtorRuWebCrawler for details

To make search - use run-search script in pravtor.ru-crawler folder.
Collected data will be placed into result.xls file in sandbox folder

Crawler for vacancies aggregator rabota.by

Check RabotaByWebCrawler for details

To make search - use run-search script in rabota.by-crawler folder.

Crawler for Onlíner (CPU catalog, looking for AM4 CPUs)

Check OnlinerByCpuCrawler for details. It reads the JSON-LD ItemList from catalog pages (filters: socket_cpu[0]=am4, price[from]=1).

To run — use run-search in the onliner.by-crawler folder after mvn package (output JSON path and optional args are set in the script).

Crawler for dating site tabor.ru

Check TaborRuWebCrawler for details. It searches women profiles in Minsk (age configurable; defaults 25–45, 01-run-search.bat uses 30–45) that have a cover photo, opens each profile, and downloads profile details & that photo. More details: tabor.ru-crawler/README.md.

To run — use 01-run-search in the tabor.ru-crawler folder after mvn package (output: profiles.json, photos/, log file tabor-crawler.log).

Args: outputJson [pagesCap] [delayMs] [minAge] [maxAge] — e.g. profiles.json -1 250 30 45.

Local dashboard: 02-run-dashboard → http://localhost:8080
Stop: 03-stop-dashboard (or Ctrl+C).

Video with description of the project