Crawling December: HTTP caching

Allow us to cache, pretty please.
As the internet grew over the years, so did how much Google crawls. While Google’s crawling
infrastructure supports heuristic caching mechanisms, in fact always had, the number of requests
that can be returned from local caches has decreased: 10 years ago about 0.026% of the total
fetches were cacheable, which is already not that impressive; today that number is 0.017%.

Read more at Read More