Simon Gelfand

I recently discovered new additions to Google Sitemaps. Eager to explore the new features on one of my sites I came across some interesting information.

Below is a graph displaying the activity of Googlebot’s average page crawl rate covering a 90 day period.

Notice the large drop in crawls during August?

Looking back I remember this was when my web site was starting to see a large rise in traffic and becoming a big strain on my shared hosting account. Towards the end of August we changed servers and hooray Google is back up and running full speed and still getting stronger.

But what can we learn from this?

Scrolling down the ‘crawl rate’ page we see a couple of more charts that can help us learn more on how Googlebot crawls and gives a ‘crawl quota’ to your web site.

Time spent downloading a page (in milliseconds)

Here we see that the server was so slow it was causing the Googlebot to stay on every page for more than 5 seconds!!! Thus killing our daily quota and causing less page crawls, index’s and of course less traffic.

It is quite possible that you are unaware of these problems and Google Sitemap’s helps you discover them. There have been many more features added during the last couple of months, here is a quick breakdown:

- Preferred Domain
- Crawl Speed Setup
- Full Page Error Reporting
- Search Queries and Positioning Reporting

In short if you don’t have a Google Sitemap setup, get it done now.
http://www.google.com/webmasters/sitemaps/