Research Crawling Engineer

About the Role

Join a data infrastructure team building large-scale web data collection systems for AI research and model development. This role focuses on reliable, high-volume crawling and dataset preparation across diverse and changing web environments.

What You'll Do

• Build and maintain web crawlers across social, travel, and multilingual websites.

• Design high-throughput, fault-tolerant systems capable of collecting millions to billions of URLs per day.

• Address anti-bot systems, rate limits, and dynamic websites using browser-based and other crawling techniques.

• Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.

• Collaborate with research teams to create datasets suited to model training and research needs.

• Monitor crawl coverage, performance, and data quality, adapting systems as websites change.

• Optimize crawling infrastructure for cost, latency, and reliability across cloud or bare-metal environments.

What We're Looking For

• At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, with 2 or more years considered for candidates with a PhD.

• Experience operating data pipelines at terabyte scale or larger, including high-volume crawling or data acquisition.

• Strong programming skills in Go, Rust, Python, Java, or C++.

• Deep knowledge of HTTP, networking, browser behavior, distributed systems, and parallel processing.

• Hands-on experience navigating anti-bot systems, rate limits, and JavaScript-heavy websites.

• Experience with headless browsers such as Playwright or Puppeteer, proxy systems, and IP rotation.

• Background in data quality evaluation, web data processing, or cloud and bare-metal infrastructure.

• Experience with NLP pipelines or machine learning dataset curation is a plus.

Compensation & Benefits

Annual salary range: $160,000 to $250,000 USD. Visa sponsorship is not available.

Location

Remote.