Lessons from crawling the web at scale 2026w17

Today I want to provide a project update on Greppr and all of the recent optimizations I have made.

If you’ve been following the journey lately, you know I’ve been deep in the trenches operating as a solo founder, specifically building out the core infrastructure for Greppr.org. Today, I want to step away from high-level management theory and getting straight into the weeds of systems engineering.

Over the last few weeks, my 40-worker crawling swarm pushed the Greppr index well past the 41 million document mark. But scaling a standalone Solr architecture to over half a terabyte isn't just about ingesting data; it's about surviving the absolute chaos of the open web. Today, I want to break down the technical gauntlet I just ran: language issues, crawler tar pits, NSFW filtering, rich media filtering, and deduplication of index entries via canonical URLs.

Let’s get into it.

Language re-direction workarounds with proxies

As I host my web crawlers in Germany, many websites try to be clever and re-direct my web crawlers to German-versions of their websites. Despite correctly using the Accept-language HTTP header in my requests to request English content, it became clear over time that Greppr was full of German versions of popular websites.

The reason for this is IP geo-location: many websites, especially those that use global Content Delivery Networks or CDNs, we match your IP to a source country, then redirect you to a country-specific localized version of their website. This is nice for end-users, but terrible for web crawlers that will also follow those re-directs.

The solution is to use proxy servers in the country that speaks your target language. For example, now Greppr crawlers running in Germany use proxy servers in the USA, effectively giving my crawlers a US IP address. Now the websites will map my IP to an English-language version of their sites.

Crawler tar pits

When you are running a swarm of concurrent workers, a few hanging threads will absolutely destroy your indexing velocity. I found my workers were getting trapped by 60-second "politeness" delays when hitting HTTP 429 rate limits, and sitting idle for 70 seconds waiting for dead servers to return 503 errors.

To fix this, I forced the crawler to fail fast. I bypassed the underlying library's sleep loops by configuring a hard 5-second cURL timeout and wrote a custom middleware to enforce this. This middleware intercepts 429 and 503 responses, morphing them into standard 400 Bad Requests at the network layer to force an immediate drop, while passing the true original status code through a custom HTTP header for accurate logging. I also deployed a shared cache for robots.txt files to intercept and cache Web Application Firewall blocks, preventing the swarm from endlessly thrashing secured domains. I know when I’m not welcome, and always honour robots.txt instructions from each website I try to index.

NSFW filtering

A search engine needs rigorous hygiene to maintain search quality. To keep the index clean from the start, I implemented a fast, memory-efficient filter that executes immediately after the HTML extraction phase.

The workers parse both the raw URL string and the page's title tag against a strict blocklist of explicit keywords. If a match triggers, the worker drops the payload from memory instantly and unsets the URL from the active queue. This ensures that zero compute cycles are wasted running statistical n-gram language detection or pushing garbage data into the Solr commit buffer.

Rich media filtering

Crawlers can easily get bogged down downloading massive files that have absolutely zero text search value. CDNs hosting static assets are a massive trap.

To combat this, I added an early-stage filter to parse the incoming URL path and check the file extension against a banned list before the worker ever makes a network connection. By explicitly dropping requests for things like PDFs, MP4s, WOFF fonts, and large images, the workers conserve massive amounts of network bandwidth and keep their CPU cycles focused entirely on parsing text-rich HTML.

Deduplication of index entries via canonical URLs

Search relevance dies when your index is bloated with five different URL parameters all pointing to the exact same article.

To solve this, the crawler now extracts the link rel="canonical" tag during the DOM parsing step. If the canonical URL differs from the URL the worker just fetched, the system immediately issues a destructive delete command to the Solr index and the database to purge the duplicate link. It then rewrites its internal state to target the canonical link moving forward, saving the record using optimistic locking to ensure we only ever store the single, authoritative version of a document.

Closing Thoughts

Building a search engine from the root up forces you to confront exactly how the Internet actually operates: it’s messy, it’s full of spam, and it requires ruthless, automated pruning. The hardest part about running a search engine is not the search, it’s the crawling and indexing. But when you get the infrastructure tuned just right, watching those workers chew through the web is an incredible thing to see.

If you want to test the raw speed of the index yourself, hit up greppr.org. And if you are navigating your own infrastructure challenges or operating as a solo founder, let's connect.

Until next time.