For years, websites included information about what kind of crawlers were not allowed on their site with a robots.txt file.

A Vancouver payment processing company must pay over $3.3 million to the B.C. Securities Commission (BCSC) after admitting it ...