Moving to TG2, Part 1

This is the start of a series on the work we're doing upgrading BandRadar's components. BandRadar's development started in 2006, and used the default components of TurboGears at the time: SQLObject for ORM, kid for templating, tgwidgets for widgeting, mochikit for js library, and CherryPy as appserver.

Basically all of these components are now obsolete, or at least no longer recommended for new TG/TG2 projects. But they still work fine, so why change?

Good question. I'm hoping that re-examining the choices we made in implementing BandRadar version 1 will bear dividends in terms of simplifying the code or making it easier to add new functionality, but honestly it's not clear at this point whether time spent on this work will have more value than spending it on extending the current code. It also depends on how hard moving to the new components is. Thankfully, TurboGears will let us transition to new components in stages, instead of all at once.

The plan:

1. SQLObject to SQLAlchemy (Elixir)
   
2. kid to Genshi
3. tg.widgets to ToscaWidgets
4. CherryPy to Pylons/mod_wsgi

Currently working on #1. Converting classes derived from class SQLObject to class Entity and defining the relationships has been relatively painless. All our table names are not what Elixir is expecting, but it's easy to tell it what they should be. The first little bit of trouble is that SQLObject did not create db constraints for required fields, it keeps track of these itself and throws an error if a required field is omitted when creating an object. SA appears to rely on the database for this (which makes sense) but our existing db schema (originally created by SO) will need to be updated by hand for the new not-NULL constraints. We've also added a fair number of additional methods to the objects on model.py, and these will need to be reimplemented in SA-ese. Finally, our SO model includes a feature where all modifications to some tables are logged to a separate table (so we could recover from vandalism if necessary)... this will need to be reimplemented in SA, or we need to jump to versioned objects. I'd prefer the former, frankly, but I don't know if SA has a feature equivalent to sqlobject.events, or if I could just override __setattr__() in my model objects.

New Logo!

As you may have noticed, we have a new logo. This logo was professionally designed for us by Andrew Harrington, and I think he did a great job. Of course we will be continuing to refine the site's look, but I think the logo is a great starting point.

Watch out for 500 errors due to gclid

One kind of 500 seeing I've been seeing is this:

GET /
(traceback)
TypeError: index() got an unexpected keyword argument 'gclid'

I shrugged these off (stupid bots) but no these are VERY BAD.

After a little Googling, I discovered that Google AdWords adds this parameter to requests to track when people click through your ads. (We do a small amount of ads on Google.) Well every one of these that we've been paying for (!) have been getting 500 errors instead of the main page because of this added parameter! I have tg.strict_parameters set.

If you run a site and are using AdWords, make sure you aren't getting these errors. I would never have known if I hadn't turned on error reporting (see here).

More stuff coming soon

Although BandRadar is a labor of love more than a business (currently anyways) we have still been adding features and users at a slow but steady pace over the past few months. For example, we keep adding more artist info, as we can get it from online APIs such as last.fm and Musicbrainz, as well as album links to Cdbaby and Amazon. We also are currently on our third site design, with more improvements in that area on the way.

We also recently acquired an iPhone, and are seeing great possibilities for making the site more useful from portable wireless devices. I think one common scenario is:

(out at a bar)
Guy 1: Let's go check out a band!
Guy 2: Sure! Do you know who's playing around here tonight?
Guy 1: Uhh no... does anyone have a paper?

Of course if you have a web-enabled mobile device you could check BandRadar, but right now we're giving you ALL the events tonight -- too much to wade through on a tiny screen. What you really want to know is what's nearby, and a little about each one to know if it's something you'd like to check out.

This isn't hard to do. we could add a m.bandradar.com with a simplified webpage, enter your location and it finds the closest events. But what would be better would be to use the built-in GPS or cell-location features (like the iPhone has) so people could skip the "type in my location" step. That's the most painful part, really. It would go from taking 30 seconds (while your friends are waiting for you to quit futzing with your iPhone) to maybe 3 seconds. That difference boosts its usefulness tremendously, and it will be used much more.

Unfortunately, all these devices may know where they are, but they may not have a way to tell us this information. On the iPhone, its Maps app supports location, but it's written by Apple of course. Once the iPhone SDK is released it may be possible to write our own app with location info, but that's a big unknown. That still doesn't help people who own all the other location-aware devices out there now, or coming soon.

But we're thinking about these things. This is a killer feature that we personally would find very useful, and really want to make it happen if we can pull it off.

Error reporting is really nice to have

One thing that has really helped me feel better about the site lately is adding error reporting. That is, every time users get an error, I get an email. It's documented here: ErrorReporting.

I think most people want whole-site error reporting (documented as Method 3). The code listed worked great, but something irks me about cutting and pasting code out of the docs in order to add that feature. I felt compelled to review it line-by-line, which I don't think I would have felt the need to do otherwise.

Could this be made a standard part of TG, possibly?

EDITED: BTW spiders are really good as an automated testing system and finding bad links, as one finds after enabling error reporting.

Site makeover

Just went live with the new site makeover! W00t! No more harsh green, we went for a more mellow ochre (?) palette, and a blocky, vintage feel inspired from old concert posters.

We still have some kinks to work out when it comes to IE -- please bear with us, or even better, use Firefox.

In other news, I was out at a Karmedy gig at the Red Room on Thursday, the place lost power! Never had that happen at a gig before. Totally missed Karmedy (sup Josh!) and only caught half of The Unnamed's set (who were frickin amazing, btw)

A persistent dict class

One big part of the website is importing events, with artist names associated with it. However, sometimes a band is referred to in slightly a different way -- not enough to be significant to a person, but enough to fool a computer into thinking it's a new band. The way I initially fixed this was a artist_name_fixup dict in the importing module. The dict contained a mapping between the incorrect name and the correct one. So for example, "Static X" mapped to "Static-X" (the latter is the actual band name, according to their website.)

But this meant that every time a new fixup was needed, the source file containing the dict needed to be changed. OK, so put it in the database. Well, we check this dict a LOT, so it would be nice to not hit the DB every time.

The persistent dict solves this by saving added entries in the database, but also keeping the dict in-memory. First, we define the fixup DB table in our model:

class ArtistNameFixup(SQLObject):
    name = UnicodeCol(alternateID=True)
    value = UnicodeCol()

Then, the new class (in our util.py module):

class PersistentDict(dict):
    def __init__(self, model):
        super(PersistentDict, self).__init__()
        self.model = model
        for row in model.select():
            super(PersistentDict, self).__setitem__(row.name, row.value)

    def __setitem__(self, name, value):
        try:
            r = self.model.byName(name)
            r.value = value
        except SQLObjectNotFound:
            r = self.model(name=name, value=value)
        super(PersistentDict, self).__setitem__(name, value)

    def __delitem__(self, name):
        self.model.byName(name).destroySelf()
        super(PersistentDict, self).__delitem__(name)

    def update(self, other_dict):
        for name, value in other_dict.iteritems():
            try:
                r = self.model.byName(name)
                r.value = value
            except SQLObjectNotFound:
                r = self.model(name=name, value=value)
        super(PersistentDict, self).update(other_dict)

Finally in the import module, we add:

import util
artist_fixup_dict = util.PersistentDict(ArtistNameFixup)

When we start the TG application, the artist_fixup_dict will be populated from the database. Subsequent additions and deletions will be reflected both in the in-memory version, as well as in the db table, so if the app is reloaded, it won't lose anything.

Finally, note that the PersistentDict code is making assumptions about the SO model it is initialized with -- that the table has "name" and "value" fields. With a little more code, these could be made configurable, but I don't need that flexibility, so I haven't added it.