Here is one of my web log entries, perhaps from my Yakkity Yak page, What's New page, or one of my Astounding Adventures from my Geocaching section:

Yahoo! Goes Fishin'!
Wednesday, 28 April 2004 7:10 PM MDT
Yakkity Yak
On a whim, I perused my web server's access and error logs. I was surprised to note some requests for documents that don't exist on my site. The requests asked for the following files (which don't exist on my site):
  • audits.html
  • addguest.html
  • services.html.en
  • schedule.html
  • employment.html.en
  • services.html.es
  • index.html.en
  • about.html.en
  • employment.html.es
  • browse.html
I double-checked the IP addresses of the clients (all on the same class C subnet), and they belong to "Inktomi Corporation" -- you know, the one acquired by Yahoo! The access log section that records the HTTP user agent agreed:

66.196.72.70 - - [27/Apr/2004:21:06:03 -0600] "GET /audits.html HTTP/1.0" 404 209 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

So is that the "new thing" to do? Go fishing for documents even when there are no links to any such documents? Will other search engines start doing this too?

Interesting.
[Permanent Link] Track Back: 0 Post Comment