Beware Trac and Unicode

Posted by max on September 25, 2007

I just blew a frightening amount of time (mostly unbilled) tracking down the most aggravating problem I’ve seen in a while.

The way that the “production” version of Trac (which is at 0.10) works with Unicode is a bit dim. Somewhere laced in there it likes to use the default encoding of latin-1 instead of utf-8. (It’s not in the standard siteconfig.py location, though.)

The errors I was getting when trying to sync Trac to the SVN repository were like this:

2007-09-16 22:51:59,995 Trac[cache] WARNING: Revision 11 already cached: 'latin-1' codec can't encode character u'\ufffd' in position 43: ordinal not in range(256)

I tried a number of approaches, including rebuilding swig, subversion, python, and some others. None solved the damn problem, though I obviously encountered new ones along the way. Amy managed to refocus me properly on the goal by verifying that the problem was indeed caused by Unicode characters in filenames the Subversion repository, and by picking at the Trac code that inserts SVN data into its database. In attacking that problem I converted the Trac database to the utf-8 character set from latin-1, which would have been a problem. That still didn’t fix it, though.

It wasn’t until I tried resyncing using trac 0.11dev’s trac-admin tool that it finally worked.

Trac’s site does warn that 0.10 still has some problems with Unicode. There’s even a ticket pointing out a fix which I really didn’t want to apply myself, knowing it would get rebroken at some point. (One commenter noted that “this should be backported to the 0.10 line”, but unfortunately that doesn’t seem to have happened.)

Update: I’m fairly confident that, had I started with a utf8 MySQL database to begin with instead of latin1, I would have avoided all of these woes. But that wasn’t my starting point. (Fun fact: Python calls them “latin-1″ and “utf-8″, while MySQL calls them “latin1″ and “utf8″. Strangely enuf, that difference didn’t cause me any problems.)

Popularity: 11% [?]

Theoretically Related Posts
  • How to learn (and not get burned by) LVM
  • 12 Steps to Testing in Rails
  • Trackbacks

    Use this link to trackback from your own site.

    Comments

    Leave a response

    1. Tamer Salama Thu, 27 Sep 2007 11:50:33 UTC

      Have you tried RedMine? I’ve been using it for a current project and is a great tool. Moreover, it’s on Rails :)

    2. max Thu, 04 Oct 2007 21:01:47 UTC

      Tamer,

      Thanks for the pointer. I am curious about similar Rails tools; I had been looking at Collaboa before. Trac seems to have a lot of momentum behind it, but after this experience, I’m left slightly queasy by it.

    Comments