Learning by Doing

One of the most effective ways to increase the visibility of your content is to make sure it’s indexed regularly by Google. However, the Googlebot sometimes has a hard time with database-driven websites like WordPress blogs, so it helps if you tell Google which URLs to visit. The way to do that is with an XML sitemap. There are a couple different kinds of sitemaps, which work with different search engines, but I’m only going to talk about the XML sitemap supported by Google and Yahoo. There’s also a Google sitemap generator for WordPress, but if you’re like me, you try to keep the number of active plug-ins to a minimum to make your site as fast as possible.

Not only will a sitemap ensure Google has the freshest content from your site, but it will also make your site run faster by telling the Googlebot that it doesn’t need to crawl your back archives with the same frequency as your front page. This is especially important for shared hosting situations like Dreamhost. Because the Googlebot alone can use 50% of the CPU of the shared server, if your site isn’t configured properly, you could bog down the server for everyone else and even get your site taken offline¹.

To set this up you’ll need an account with Google Webmaster Tools, the downloadable sitemap generator, and a hosting account that uses Analog logging and offers python support. I use Dreamhost. If you need a host, check ‘em out (and use promo code “Synthesis” to get your first year for $60).

First, download the program and upload it to the base directory of your website. Unzip the package and open up config-example.xml. In config-example.xml are the parameters that control how the URL list that makes up the sitemap is generated. You’ll need to rename this to config.xml for it to work. There are two steps to setting up config.xml: Including URLs, and excluding URLs. Because sitemap_gen doesn’t do any crawling itself, you have to supply it with a list of URLs. One simple way to do this is with a text listing of URLs, but manually adding to this list every time you wrote a new post would get tedious. Conveniently, sitemap_gen can parse logfiles, so you can use your logs as the URL list. The frequency with which URLs appear in your logs also allows sitemap_gen to assign a priority score to each URL, letting the Googlebot know which pages to update more frequently and which pages it doesn’t need to crawl as often.

Next, find the section in config.xml that says, “The “site” node describes your basic web site.” In this section, you want to replace http://www.example.com with the path to your site. Replace /var/www/docroot/sitemap.xml.gz or whatever comes after store_into with the name of your sitemap. I used sitemap.xml.gz, to generate a compressed sitemap for google to read.

Moving down the file, find the INPUTS section. This is where you will specify which URLs to include in the sitemap. This part if broken up into sections which contain different link inclusion mechanisms. You can only use one mechanism at a time, so delete or comment out the sections until you get to the one that talks about accesslogs. Remove two of the three example statements in brackets in this section, and modify the remaining one to contain the full path to your access logs. You can use the * character to specify all the logs in the directory like so <accesslog path="/path/to/logs/access.log*" encoding="UTF-8" /> . Delete the remaining sections in the INPUTS section.

The next section is the filters section. This is where you will specify which URLs to exclude. You can do a lot of fancy stuff here, but the most important thing for WordPress is to remove URLs that lead to non-content pages, like wp-login, for example². In these statements you tell sitemap_gen which URLs to add or remove from the list, using normal wildcards or regular expressions. I recommend keeping this as simple as possible, avoiding the use of pass statements because those act like short circuits and will leave matching URLs in the list no matter what you specify later, and in conjunction with regular expressions, this can sometimes be non-intuitive and hard to debug.

Here’s my filters section:

<filter action="drop"  type="regexp"  pattern="/wp-admin/"    />  
  <filter action="drop"  type="regexp"  pattern="/wp-login/"       /> 
  <filter action="drop"  type="regexp"  pattern="wp-cron\.php"    />    
  <filter action="drop"  type="regexp"  pattern="wp-login\.php"      />  
  <filter action="drop"  type="regexp"  pattern="/doc/"        />
  <filter action="drop"  type="regexp"  pattern="/noexist_" />       
  <filter action="drop"  type="regexp"  pattern="/\?p=[\d]"      />  
  <filter action="drop"  type="regexp"  pattern="/\?s=[a-zA-Z0-9]" />       
  <filter action="drop"  type="regexp"  pattern="/Photos/tags/.*\.html" />       
  <filter action="drop"  type="regexp"  pattern="/Photos/tags/.*/tags/"    />    
  <filter action="drop"  type="regexp"  pattern="/wp-content/"  />
  <filter action="drop"  type="regexp"  pattern="/wp-includes/" />
  <filter action="drop"  type="regexp"  pattern="/stats/" />
  <filter action="drop"  type="regexp"  pattern="/_vti_bin/" />
  <filter action="drop"  type="regexp"  pattern="/MSOffice/" />
  <filter action="drop"  type="regexp"  pattern="/dh_phpmyadmin/"/> 
  <filter action="drop"  type="regexp"  pattern="/htmledit/" />
  <filter action="drop"  type="regexp"  pattern="/robots\.txt" />
  <filter action="drop"  type="regexp"  pattern="/sitemap\.xml"/> 
  <filter action="drop"  type="regexp"  pattern="/xmlrpc\.php" />
  <filter action="drop"  type="wildcard"  pattern="*.jpg"         />
  <filter action="drop"  type="wildcard"  pattern="*.tif"         />
  <filter action="drop"  type="wildcard"  pattern="*.tiff"        /> 
  <filter action="drop"  type="wildcard"  pattern="*.bmp"       />  
  <filter action="drop"  type="wildcard"  pattern="*.ico"         />
  <filter action="drop"  type="wildcard"  pattern="*.js"         />
  <filter action="drop"  type="wildcard"  pattern="*.css"       />  
  <filter action="drop"  type="wildcard"  pattern="*.gif"        /> 
     <!-- Exclude URLs within UNIX-style hidden files or directories       -->
  <filter action="drop"  type="regexp"    pattern="/\.[^/]*"   />  

That’s all fairly straightforward, I hope, but two things merit explaining. The section below

<filter action="drop"  type="regexp"  pattern="/\?p=[\d]"     />   
  <filter action="drop"  type="regexp"  pattern="/\?s=[a-zA-Z0-9]"    />    
  <filter action="drop"  type="regexp"  pattern="/Photos/tags/.*\.html"    />    
  <filter action="drop"  type="regexp"  pattern="/Photos/tags/.*/tags/"     />   

is an example of one way to remove redundant URLs from your list. You don’t need the “Pretty URL” to your site and the /p?number URL both, and if you’ve changed that setting recently, they will both show up in your logs. The /\?p=[\d] string tells site_gen to exclude any URL of the form /p?some number. Also, you don’t necessarily need search result pages to appear in the list, so the next line takes care of that. The following two lines are for use with the Flickr Photo Gallery plugin. This plugin allows you to browse your tags just as you would at Flickr, but this creates a URL problem when the site is crawled, resulting in 90% of your logs being composed of redundant crap. Those two lines remove all the URLs pertaining to the gallery except gallery pages and display pages for a single tag.

The next thing worth mentioning is the lines below, which are generated when someone using IE visits your page with the discussion toolbar loaded. IE looks to see if your site supports it, which mine doesn’t.

<filter action="drop" type="regexp" pattern="/_vti_bin/" />
<filter action="drop" type="regexp" pattern="/MSOffice/" />

After processing your logs and applying some intelligent filter rules to exclude URLs that aren’t content-containing parts of your site, you’re ready to submit. Run python sitemap_gen.py --config=config.xml --testing, extract the sitemap.xml file from sitemap.xml.gz, and load it in your browser. Look through it and make sure your rules have worked as expected, then run the command again, removing the –testing part. If you want to get fancy, you can set this up as a cron job. If you do, run it on access.log.0, yesterdays logs, around 2am. That way you don’t miss any traffic as the logging switches over at midnight.

Finally, log into Google webmaster tools and submit your sitemap to Google!

To see how must of your traffic is coming from the Googlebot, SSH to your server and run tail -10000 access.log| awk '{print $1}' | sort | uniq -c |sort -n from the same directory as your access.log files. The first number is the connections, the second is the IP making those connections. IPs that start with 66.249 are the Googlebot. If 66.249 is the last entry, and the number of connections is very high(over a thousand, say) and many times bigger than the number of connections for the second most frequent IP, you probably need to do something before the hosting company does something for you, like ban Google from accessing your site.
I’m not exactly sure if it would be better to leave some things in, but set to a zero priority, however I have non-content stuff removed for now. Really, the non-content pages should probably be excluded in robots.txt