Google released its robots.txt parsing and matching library as open source in the hope of its now public code will help encourage web developers to agree on a standard way to spell out the proper etiquette for web crawlers.
https://www.theregister.co.uk/2019/07/02/google_robot_txt/
https://github.com/google/robotstxt
https://www.theregister.co.uk/2019/07/02/google_robot_txt/
https://github.com/google/robotstxt