Research Crawling Engineer
True Talent Solutions Partners · Posted Nov 8, 2026
- Remote
- Remote
- $160K–$250K per year
- 2+ years
Build and maintain web crawlers and data pipelines for AI research.
Compensation
Annual salary range: $160,000 to $250,000 USD. Visa sponsorship is not available.
Responsibilities
- Build and maintain web crawlers across social, travel, and multilingual websites.
- Design high-throughput, fault-tolerant systems capable of collecting millions to billions of URLs per day.
- Address anti-bot systems, rate limits, and dynamic websites using browser-based and other crawling techniques.
- Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
- Collaborate with research teams to create datasets suited to model training and research needs.