Robots Refresher: Future-proof Robots Exclusion Protocol

In the previous posts about the Robots Exclusion Protocol (REP) we explored what’s already
possible to do with its various components — namely robots.txt and the URI level controls.
In this post we will explore how the REP can play a supporting role in the ever-evolving relation
between automatic clients and the human web.

Read more at Read More