"We can't let only the people who want to do it for profit or want to do it for their own gain be making the decisions," Heap ...

For years, websites included information about what kind of crawlers were not allowed on their site with a robots.txt file.