Was checking into my server logs this weekend after I made a new post to see which spiders would crawl it first. To probably nobody’s surprise, Googlebot was quick on the scene, followed by MSNBot, and Slurp didn’t roll through until a good 24 hours after it was posted. Slurp is just not a fan of daveshap.com.
One thing I did find very interesting was that MSNBot always checks robots.txt before crawling any pages, whereas Googlebot only checked it once the whole weekend. The grand totals were 37 crawls of robots.txt by MSNBot, 8 by Slurp, and 1 by Googlebot. Seems like Googlebot is a much more efficient crawler than the other two. For some reason I’m not surprised.