It’s actually fairly easy to upgrade using subversion, so here’s what I did.

Updated betablog(my nonpublic mirror of my working site) to trunk

using svn sw http://svn.automattic.com/wordpress/trunk/ .
Files changed:
A wp-admin/js/edit-posts.js
A wp-includes/js/wp-lists.js
A wp-includes/images/wlw
A wp-includes/images/wlw/wp-icon.png
A wp-includes/images/wlw/wp-watermark.png
A wp-includes/images/wlw/wp-comments.png
A wp-includes/wlwmanifest.xml

No files I have edited are part of this update, so I shouldn’t have to re-edit anything. svn update should incorporate the changes into files, instead of overwriting the whole file, unless there are overlapping changes. The only local modification I’ve made to this is to add my analytics tag to footer.php, but this hasn’t been modified in the repository version.

[smithers]$ svn status -u
* 6188 wp-includes/functions.php
* 6188 wp-admin/setup-config.php
? wp-config.php
? sitemap_gen.py
? robots.txt
? wp-content/uploads
X wp-content/plugins/akismet
? wp-content/plugins/code-markup.php
M 6188 wp-content/themes/default/footer.php
? config.xml
? google5422237c5650fd97.html
* 6188 wp-settings.php
? .htaccess
Status against revision: 6294

Performing status on external item at 'wp-content/plugins/akismet'
Status against revision: 23396

I’ll add a comment to wp-admin/setup-config.php to see if the file is over written or just modified.

I don’t understand just yet how to diff my copy against the repository. In other words, I can get a list of what I’ve changed since last update, but I can’t get a line-by-line list of what they’ve changed without doing an update. However, doing an update should tell me if my changes overlap, and then I can manually incorporate them, but this probably won’t ever be an issue because the only thing I’ve changed in the core files is the tag output. The analytics stuff is at the end of the file, where it’s not likely to overlap.

Added comment to wp-admin/setup-config.php

[smithers]$ svn status -u
* 6188 wp-includes/functions.php
M * 6188 wp-admin/setup-config.php
? wp-config.php
? sitemap_gen.py
? robots.txt
? wp-content/uploads
X wp-content/plugins/akismet
? wp-content/plugins/code-markup.php
M 6188 wp-content/themes/default/footer.php
? config.xml
? google5422237c5650fd97.html
* 6188 wp-settings.php
? .htaccess

[smithers]$ svn update
U wp-includes/functions.php
U wp-settings.php
G wp-admin/setup-config.php

U means repository file changed, but I didn’t change it locally. (it had status * before update)
G means it changed both places(i.e. it had status M * before update)

Removed change from wp-admin/setup-config.php
[smithers]$ svn status -u
? wp-config.php
? sitemap_gen.py
? robots.txt
? wp-content/uploads
X wp-content/plugins/akismet
? wp-content/plugins/code-markup.php
M 6294 wp-content/themes/default/footer.php
? config.xml
? google5422237c5650fd97.html
? .htaccess
M 6294 wp-admin/setup-config.php
Status against revision: 6294

So footer.php remains modified, because it wasn’t updated, and setup-config.php remains M, because even though I removed the change I made, making it an exact copy of the repository version, it’s still different from the version in my .svn directory, which is fine, because I don’t plan on checking any changes into the repository any time in the forseeable future.

Ah, svn update doesn’t change the version number, it just updates the existing code, so totally new files don’t get added, and WP still thinks you’re running the old version, even though subversion reports the revision number correctly. svn update, then svn sw is perhaps a good idea, then, so that repository changes get incorporated locally, then new files get added. I don’t know if svn sw will overwrite, but I know one way to find out. I’ll edit wp-admin/setup-config.php in my other blog that hasn’t been updated yet, and instead of svn update, I’ll do svn sw.

That results in a C status for the changed file, so that means I’ll have to manually incorporate changes. This is most likely due to the fact my comment was at the top of the file.

[smithers]$ svn sw http://svn.automattic.com/wordpress/trunk/ .
C wp-admin/setup-config.php
U wp-admin/link-manager.php
U wp-admin/page.php
U wp-admin/export.php
U wp-admin/edit-pages.php
U wp-admin/categories.php
Updated to revision 6294.

If you get a conflict, you need to do one of three things:

Once you’ve resolved the conflict, you need to let Subversion know by running svn resolved. This removes the three temporary files and Subversion no longer considers the file to be in a state of conflict.

You have to be in the directory to use svn resolved, or you get an error.

[smithers]$ svn resolved /wp-admin/setup-config.php
svn: warning: '/wp-admin' is not a working copy

[smithers]$ cd wp-admin
[smithers]$ svn resolved setup-config.php
Resolved conflicted state of 'setup-config.php'

Some themes are in subversion, too, and since themes are more often customized, it makes sense to use subversion for them, especially the complicated K2 theme.

[smithers]$ svn status -u
M * 500 theloop.php
* 500 page.php
* 500 page-archives.php
* 500 single.php
* 500 k2.pot
* 500 header.php
* 500 attachment.php
Status against revision: 582

So my changes to theloop.php will need to be merged. Let's try svn update.

[smithers]$ svn update
D app/display/sbm/backup.php
D app/display/sbm/header.php
D css/rollingarchives.css
A css/options.css
A css/humanmsg.css
A css/rollingarchives.css.php
A css/header.php
C theloop.php
A images/sbmmanager/buttoncircle.png
A images/sbmmanager/deletebutton.png
A images/sbmmanager/undo.png
U single.php
A js/jquery.easing.js.php
A js/jquery.humanundo.js.php
A js/jquery.humanmsg.js.php
Updated to revision 582.
So I got the dreaded C for theloop.php, but I guess I should have expected that.
Rollingarchives.css is gone, interesting, and humanundo.js functions have been added. Single.php hasn't been changed, thankfully, nor has header, footer, or style.css.

Because of the revision marks in theloop.php, the file requires resolution of the changes before it will work again. In my case, almost the whole file has changed, so I'll have to re-add the tag code to the new version.

Lots of changes to theloop.php. One thing k2 does that I don't agree with is that it only shows tags on single post pages. I think it makes more sense to just stick the tags in the place of the categories at the top.

Problems:

Current status:

Useful subversion commands:

svn status -u shows which files will be updated when you run svn update
* means files will be updated
M means files are files I've changed
X means files are unversioned, but specified as an external, so this applies to WP supplied plug-ins.
? means files aren't being tracked in subversion. Files I've created, such as the sitemap generator, which isn't part of wordpress, will have ? Status.

The way to do it is to set up an OpenID, that way the only person you have to blame is yourself, and perhaps your hosting provider. Another benefit of doing this touches on a big interest of mine: owning your identity online. You see, in the OpenID scheme, your blog URL is all you provide upon login. The rest of the information is exchanged by encrypted API handshake between the OpenID provider(your site) and the site you’re logging into(the OpenID consumer).

The setup is not for the faint of heart, but it’s not that bad, and I took good notes on how I did it.

To set up your site as an OpenID provider, you need to following things:

Not only will a sitemap ensure Google has the freshest content from your site, but it will also make your site run faster by telling the Googlebot that it doesn’t need to crawl your back archives with the same frequency as your front page. This is especially important for shared hosting situations like Dreamhost. Because the Googlebot alone can use 50% of the CPU of the shared server, if your site isn’t configured properly, you could bog down the server for everyone else and even get your site taken offline¹.

To set this up you’ll need an account with Google Webmaster Tools, the downloadable sitemap generator, and a hosting account that uses Analog logging and offers python support. I use Dreamhost. If you need a host, check ‘em out (and use promo code “Synthesis” to get your first year for $60).

First, download the program and upload it to the base directory of your website. Unzip the package and open up config-example.xml. In config-example.xml are the parameters that control how the URL list that makes up the sitemap is generated. You’ll need to rename this to config.xml for it to work. There are two steps to setting up config.xml: Including URLs, and excluding URLs. Because sitemap_gen doesn’t do any crawling itself, you have to supply it with a list of URLs. One simple way to do this is with a text listing of URLs, but manually adding to this list every time you wrote a new post would get tedious. Conveniently, sitemap_gen can parse logfiles, so you can use your logs as the URL list. The frequency with which URLs appear in your logs also allows sitemap_gen to assign a priority score to each URL, letting the Googlebot know which pages to update more frequently and which pages it doesn’t need to crawl as often.

Next, find the section in config.xml that says, “The “site” node describes your basic web site.” In this section, you want to replace http://www.example.com with the path to your site. Replace /var/www/docroot/sitemap.xml.gz or whatever comes after store_into with the name of your sitemap. I used sitemap.xml.gz, to generate a compressed sitemap for google to read.

Moving down the file, find the INPUTS section. This is where you will specify which URLs to include in the sitemap. This part if broken up into sections which contain different link inclusion mechanisms. You can only use one mechanism at a time, so delete or comment out the sections until you get to the one that talks about accesslogs. Remove two of the three example statements in brackets in this section, and modify the remaining one to contain the full path to your access logs. You can use the * character to specify all the logs in the directory like so <accesslog path="/path/to/logs/access.log*" encoding="UTF-8" /> . Delete the remaining sections in the INPUTS section.

The next section is the filters section. This is where you will specify which URLs to exclude. You can do a lot of fancy stuff here, but the most important thing for WordPress is to remove URLs that lead to non-content pages, like wp-login, for example². In these statements you tell sitemap_gen which URLs to add or remove from the list, using normal wildcards or regular expressions. I recommend keeping this as simple as possible, avoiding the use of pass statements because those act like short circuits and will leave matching URLs in the list no matter what you specify later, and in conjunction with regular expressions, this can sometimes be non-intuitive and hard to debug.

Here’s my filters section:

<filter action="drop"  type="regexp"  pattern="/wp-admin/"    />  
  <filter action="drop"  type="regexp"  pattern="/wp-login/"       /> 
  <filter action="drop"  type="regexp"  pattern="wp-cron\.php"    />    
  <filter action="drop"  type="regexp"  pattern="wp-login\.php"      />  
  <filter action="drop"  type="regexp"  pattern="/doc/"        />
  <filter action="drop"  type="regexp"  pattern="/noexist_" />       
  <filter action="drop"  type="regexp"  pattern="/\?p=[\d]"      />  
  <filter action="drop"  type="regexp"  pattern="/\?s=[a-zA-Z0-9]" />       
  <filter action="drop"  type="regexp"  pattern="/Photos/tags/.*\.html" />       
  <filter action="drop"  type="regexp"  pattern="/Photos/tags/.*/tags/"    />    
  <filter action="drop"  type="regexp"  pattern="/wp-content/"  />
  <filter action="drop"  type="regexp"  pattern="/wp-includes/" />
  <filter action="drop"  type="regexp"  pattern="/stats/" />
  <filter action="drop"  type="regexp"  pattern="/_vti_bin/" />
  <filter action="drop"  type="regexp"  pattern="/MSOffice/" />
  <filter action="drop"  type="regexp"  pattern="/dh_phpmyadmin/"/> 
  <filter action="drop"  type="regexp"  pattern="/htmledit/" />
  <filter action="drop"  type="regexp"  pattern="/robots\.txt" />
  <filter action="drop"  type="regexp"  pattern="/sitemap\.xml"/> 
  <filter action="drop"  type="regexp"  pattern="/xmlrpc\.php" />
  <filter action="drop"  type="wildcard"  pattern="*.jpg"         />
  <filter action="drop"  type="wildcard"  pattern="*.tif"         />
  <filter action="drop"  type="wildcard"  pattern="*.tiff"        /> 
  <filter action="drop"  type="wildcard"  pattern="*.bmp"       />  
  <filter action="drop"  type="wildcard"  pattern="*.ico"         />
  <filter action="drop"  type="wildcard"  pattern="*.js"         />
  <filter action="drop"  type="wildcard"  pattern="*.css"       />  
  <filter action="drop"  type="wildcard"  pattern="*.gif"        /> 
     <!-- Exclude URLs within UNIX-style hidden files or directories       -->
  <filter action="drop"  type="regexp"    pattern="/\.[^/]*"   />  

That’s all fairly straightforward, I hope, but two things merit explaining. The section below

<filter action="drop"  type="regexp"  pattern="/\?p=[\d]"     />   
  <filter action="drop"  type="regexp"  pattern="/\?s=[a-zA-Z0-9]"    />    
  <filter action="drop"  type="regexp"  pattern="/Photos/tags/.*\.html"    />    
  <filter action="drop"  type="regexp"  pattern="/Photos/tags/.*/tags/"     />   

is an example of one way to remove redundant URLs from your list. You don’t need the “Pretty URL” to your site and the /p?number URL both, and if you’ve changed that setting recently, they will both show up in your logs. The /\?p=[\d] string tells site_gen to exclude any URL of the form /p?some number. Also, you don’t necessarily need search result pages to appear in the list, so the next line takes care of that. The following two lines are for use with the Flickr Photo Gallery plugin. This plugin allows you to browse your tags just as you would at Flickr, but this creates a URL problem when the site is crawled, resulting in 90% of your logs being composed of redundant crap. Those two lines remove all the URLs pertaining to the gallery except gallery pages and display pages for a single tag.

The next thing worth mentioning is the lines below, which are generated when someone using IE visits your page with the discussion toolbar loaded. IE looks to see if your site supports it, which mine doesn’t.

<filter action="drop" type="regexp" pattern="/_vti_bin/" />
<filter action="drop" type="regexp" pattern="/MSOffice/" />

After processing your logs and applying some intelligent filter rules to exclude URLs that aren’t content-containing parts of your site, you’re ready to submit. Run python sitemap_gen.py --config=config.xml --testing, extract the sitemap.xml file from sitemap.xml.gz, and load it in your browser. Look through it and make sure your rules have worked as expected, then run the command again, removing the –testing part. If you want to get fancy, you can set this up as a cron job. If you do, run it on access.log.0, yesterdays logs, around 2am. That way you don’t miss any traffic as the logging switches over at midnight.

Finally, log into Google webmaster tools and submit your sitemap to Google!

To see how must of your traffic is coming from the Googlebot, SSH to your server and run tail -10000 access.log| awk '{print $1}' | sort | uniq -c |sort -n from the same directory as your access.log files. The first number is the connections, the second is the IP making those connections. IPs that start with 66.249 are the Googlebot. If 66.249 is the last entry, and the number of connections is very high(over a thousand, say) and many times bigger than the number of connections for the second most frequent IP, you probably need to do something before the hosting company does something for you, like ban Google from accessing your site.
I’m not exactly sure if it would be better to leave some things in, but set to a zero priority, however I have non-content stuff removed for now. Really, the non-content pages should probably be excluded in robots.txt

EDIT: I couldn’t resist, I’m trying K2

I would have thought Google would have known better than to try to subvert such an important browser function. Bad Google, Bad!

The good: revision control, easy collaboration, seamless output to many formats, rich editing features.
The bad: post metadata isn’t preserved, non-standard browser UI, no way(I know of) to put the post on a separate page.

Maybe I could get the best of both by sticking the RSS feed of revisions on a separate WP page.

Because the widgets work in the default theme, but not in Tiga, there must be some weirdness with the theme, but I should be able to paste the widget code into sidebar.php in the theme directory.

I don’t think wp-admin/widgets.php works with tiga, because it expects wp-content/plugins/widgets.php. I’ll have to check that soon, and in the mean time, I could probably just paste the code in.

Replacing tiga’s sidebar.php with the default’s works, but the formatting is screwed up. I need to figure out what parts of the default sidebar need to be reproduced in tiga’s.

Specifically, the jerome’s keywords plugin doesn’t work with both Tiga and wordpress2.2, though they work with either alone.
Many of the fancier sidebar widgets don’t work, like the one that displays RSS feeds.

Not only that, but I can’t delete posts.