Founded in 1993, The Motley Fool is a financial services company dedicated to making the world smarter, happier, and richer.

For years, websites included information about what kind of crawlers were not allowed on their site with a robots.txt file.