Research Crawling Engineer

About the Role

Join a growing data infrastructure team building large-scale web data systems for AI research and model development. You will develop reliable crawling and data acquisition infrastructure across diverse websites, helping research teams create datasets for training and evaluating advanced models.

What You'll Do

• Build and maintain web crawlers across social, travel, multilingual, and other domains.

• Design high-throughput, fault-tolerant collection systems that process millions to billions of URLs per day.

• Adapt crawling methods to anti-bot systems, rate limits, and dynamic or JavaScript-heavy websites.

• Build pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.

• Work with research teams to align data collection and dataset curation with modeling needs.

• Monitor crawl coverage, performance, and data quality, and improve systems as websites change.

• Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.

What We're Looking For

• At least 2 years of experience building web crawlers or large-scale data pipelines; 4 or more years is preferred, with flexibility for candidates holding a PhD.

• Experience operating data pipelines at terabyte scale or greater, including distributed systems or parallel processing.

• Strong programming skills in Go, Rust, Python, Java, or C++.

• Deep knowledge of HTTP, networking, browser behavior, and techniques for handling anti-bot systems, rate limits, and dynamic websites.

• Familiarity with headless browsers such as Playwright or Puppeteer, proxy systems, and request orchestration.

• Experience with large-scale data quality evaluation, NLP pipelines, or machine learning dataset curation is a plus.

• Experience running data workloads on cloud platforms or bare-metal infrastructure.

Compensation & Benefits

Base salary is $160,000 to $250,000 annually. Visa sponsorship is not available.

Location

Fully remote.