- London, England, United Kingdom
- Hybrid
- 5+ years
Web Access Engineer for the Crawler team building reliable fetching and rendering systems for AI-powered search.
Responsibilities
- Own fetching and rendering improvements end to end, from investigation through implementation and production rollout, partnering with crawler engineers and SREs.
- Investigate fetch and rendering failures across DNS, TLS, HTTP, caching, redirects, and client-side rendering, and classify root causes.
- Improve page loading, completion detection, and content capture to preserve relevant content and handle dynamic pages reliably.
- Ensure the crawler respects site-imposed access restrictions and rate limits, identifies itself clearly, and retries only transient technical errors.
- Build tools to reproduce failures, compare crawler output against expected page content, and turn findings into regression tests.
Requirements
- Nice to have: CDN development
- Nice to have: caching systems
- Nice to have: web performance systems
- Nice to have: operating crawlers
- Nice to have: headless browsers