One thing that nobody’s talking about:
It’s been long assumed that one of Google’s fundamental moats is their infrastructure: that it is hard to crawl, index, store, and digest the entire internet
Basically all the big labs are disproving this. Their crawlers are everywhere
— John Loeber 🎢 (@johnloeber) May 20, 2025
Only people who didn’t know anything about anything thought that was true.
I personally — and I mean with my own funds — could build enough infrastructure to crawl 90%+ of the web every few days and store the metadata. It’s just not that much data. It’d take about $200,000 of gear and connectivity. That’d buy me a dozen petabytes of storage and a couple dozen 10Gb links. And that’s enough. The software is all open source.
Google’s moat has nothing to do with the ability to crawl or digest anything and more to do with their former search algo dominance.