The occurrences are surprising since Google has enforced certain implementations to prevent these links from being indexed, Kaiser said. The robots.txt file at http://www.google.com/robots.txt contains the relevant entries to prevent the indexing.
As a general rule, all search engines adhere to the entries within robots.txt files. In addition, AdSense links should not be indexed since the entries at http://pagead2.googlesyndication.com/robots.txt clearly state not to index these links.
"It is incomprehensible why Google's robots.txt file is disregarded by MSN, especially due to the obvious reason that it is only detrimental towards Microsoft," Kaiser said.
After finding these existing links, cyberpromote's inquiries pertaining to Google's AdSense program produced more than one million links within the MSN index. The links for the AdWords program produced an even more startling number with over 4 million indexed results.
It is likely that Microsoft will investigate the reasons as to why the robots.txt files have been ignored. A problematic issue concerns MSN's lack of identifying these links clearly as advertising links.
.png&h=140&w=231&c=1&s=0)
.png&h=140&w=231&c=1&s=0)
.png&w=120&c=1&s=0)
EDUtech AU
.png&w=120&c=1&s=0)
Security Exhibition & Conference 2025
Integrate Expo 2025
Digital As Usual Cybersecurity Roadshow: Brisbane edition
iTnews Benchmark Security Awards 2025
.jpg&h=140&w=231&c=1&s=0)
.jpg&h=140&w=231&c=1&s=0)
