neverland

Note (2019-07-02 20:44)

Please be advised that the following was written or last updated a while ago and may therefore contain outdated information or opinions I no longer hold. 请知悉下文自写作或上次更新已届相当时限，或包含过时信息及已摒弃观点。

Google 把解析爬虫规则的代码公开了看起来他们似乎发现 robot.txt 很多都是手写的专门留了一段代码来对付 “user agent” “disallow” 和 ”sitemap” 的各种错误拼法… https://github.com/google/robotstxt/blob/59f3643d3a3ac88f613326dd4dfc8c9b9a545e45/robots.cc#L680-L704