CloudFlare and Your Google Crawl Stats

by Michael on November 20, 2013 · 0 comments

About six weeks ago, I started using CloudFlare on another site of mine. For those that are unaware, CloudFlare is a content delivery network (CDN) and distributed DNS service that is intended to improve site performance and security.

All has been well since enabling the service, though I recently noticed something in Google’s Webmaster Tools that I thought was worth sharing. In short, though Google still visits my site with the same frequency, they’re downloading far less data.

I’ve included a screenshot below. Pay particular attention to early October on the lower panel. See the dropoff? It’s pretty clear.

At first glance, this makes sense. CloudFlare is caching and re-serving my images so there’s going to be less in the way of downloading from my site. That’s true from the perspective of my own server, but these graphs are from Google’s perspective.

When the Googlebot visits my site, it should be gobbling up everything that’s there. From their perspective, it shouldn’t really matter if the images (for example) are served by from a CDN or directly from my own server. Should it?

Said another way, data is data. When they download it, it seems like it should be counted. But that doesn’t appear to be happening. Or maybe they really are downloading a whole lot less data. But how/why?

Perhaps CloudFlare is somehow signaling to Google that the cached images haven’t changed and so the Googlebot is skipping over them instead of re-downloading like it used to. I’m really not sure what’s going on.

The good news, as noted above, is that the crawl frequency hasn’t really changed. If anything, the number of pages that they’re crawling per day has been slowly increasing over the same time period.

Given the above, I’m not particularly worried about the patterns that I’m seeing in Webmaster Tools. But I thought it was worth documenting my observations in case others have noticed something similar.

{ 0 comments… add one now }

Leave a Comment

Previous post:

Next post: