The BandRadar Blog

Moving to TG2, Part 1

2009-01-04T18:29:00.001-08:00

This is the start of a series on the work we're doing upgrading BandRadar's components. BandRadar's development started in 2006, and used the default components of TurboGears at the time: SQLObject for ORM, kid for templating, tgwidgets for widgeting, mochikit for js library, and CherryPy as appserver.

Basically all of these components are now obsolete, or at least no longer recommended for new TG/TG2 projects. But they still work fine, so why change?

Good question. I'm hoping that re-examining the choices we made in implementing BandRadar version 1 will bear dividends in terms of simplifying the code or making it easier to add new functionality, but honestly it's not clear at this point whether time spent on this work will have more value than spending it on extending the current code. It also depends on how hard moving to the new components is. Thankfully, TurboGears will let us transition to new components in stages, instead of all at once.

The plan:

SQLObject to SQLAlchemy (Elixir)
kid to Genshi
tg.widgets to ToscaWidgets
CherryPy to Pylons/mod_wsgi

Currently working on #1. Converting classes derived from class SQLObject to class Entity and defining the relationships has been relatively painless. All our table names are not what Elixir is expecting, but it's easy to tell it what they should be. The first little bit of trouble is that SQLObject did not create db constraints for required fields, it keeps track of these itself and throws an error if a required field is omitted when creating an object. SA appears to rely on the database for this (which makes sense) but our existing db schema (originally created by SO) will need to be updated by hand for the new not-NULL constraints. We've also added a fair number of additional methods to the objects on model.py, and these will need to be reimplemented in SA-ese. Finally, our SO model includes a feature where all modifications to some tables are logged to a separate table (so we could recover from vandalism if necessary)... this will need to be reimplemented in SA, or we need to jump to versioned objects. I'd prefer the former, frankly, but I don't know if SA has a feature equivalent to sqlobject.events, or if I could just override __setattr__() in my model objects.

New Logo!

2008-02-19T14:52:00.000-08:00

As you may have noticed, we have a new logo. This logo was professionally designed for us by Andrew Harrington, and I think he did a great job. Of course we will be continuing to refine the site's look, but I think the logo is a great starting point.

Watch out for 500 errors due to gclid

2008-02-18T10:23:00.000-08:00

One kind of 500 seeing I've been seeing is this:

GET /
(traceback)
TypeError: index() got an unexpected keyword argument 'gclid'

I shrugged these off (stupid bots) but no these are VERY BAD.

After a little Googling, I discovered that Google AdWords adds this parameter to requests to track when people click through your ads. (We do a small amount of ads on Google.) Well every one of these that we've been paying for (!) have been getting 500 errors instead of the main page because of this added parameter! I have tg.strict_parameters set.

If you run a site and are using AdWords, make sure you aren't getting these errors. I would never have known if I hadn't turned on error reporting (see here).

More stuff coming soon

2008-02-04T12:12:00.000-08:00

Although BandRadar is a labor of love more than a business (currently anyways) we have still been adding features and users at a slow but steady pace over the past few months. For example, we keep adding more artist info, as we can get it from online APIs such as last.fm and Musicbrainz, as well as album links to Cdbaby and Amazon. We also are currently on our third site design, with more improvements in that area on the way.

We also recently acquired an iPhone, and are seeing great possibilities for making the site more useful from portable wireless devices. I think one common scenario is:

(out at a bar)
Guy 1: Let's go check out a band!
Guy 2: Sure! Do you know who's playing around here tonight?
Guy 1: Uhh no... does anyone have a paper?

Of course if you have a web-enabled mobile device you could check BandRadar, but right now we're giving you ALL the events tonight -- too much to wade through on a tiny screen. What you really want to know is what's nearby, and a little about each one to know if it's something you'd like to check out.

This isn't hard to do. we could add a m.bandradar.com with a simplified webpage, enter your location and it finds the closest events. But what would be better would be to use the built-in GPS or cell-location features (like the iPhone has) so people could skip the "type in my location" step. That's the most painful part, really. It would go from taking 30 seconds (while your friends are waiting for you to quit futzing with your iPhone) to maybe 3 seconds. That difference boosts its usefulness tremendously, and it will be used much more.

Unfortunately, all these devices may know where they are, but they may not have a way to tell us this information. On the iPhone, its Maps app supports location, but it's written by Apple of course. Once the iPhone SDK is released it may be possible to write our own app with location info, but that's a big unknown. That still doesn't help people who own all the other location-aware devices out there now, or coming soon.

But we're thinking about these things. This is a killer feature that we personally would find very useful, and really want to make it happen if we can pull it off.

Error reporting is really nice to have

2008-02-04T11:51:00.000-08:00

One thing that has really helped me feel better about the site lately is adding error reporting. That is, every time users get an error, I get an email. It's documented here: ErrorReporting.

I think most people want whole-site error reporting (documented as Method 3). The code listed worked great, but something irks me about cutting and pasting code out of the docs in order to add that feature. I felt compelled to review it line-by-line, which I don't think I would have felt the need to do otherwise.

Could this be made a standard part of TG, possibly?

EDITED: BTW spiders are really good as an automated testing system and finding bad links, as one finds after enabling error reporting.

Site makeover

2007-04-23T20:56:00.000-07:00

Just went live with the new site makeover! W00t! No more harsh green, we went for a more mellow ochre (?) palette, and a blocky, vintage feel inspired from old concert posters.

We still have some kinks to work out when it comes to IE -- please bear with us, or even better, use Firefox.

In other news, I was out at a Karmedy gig at the Red Room on Thursday, the place lost power! Never had that happen at a gig before. Totally missed Karmedy (sup Josh!) and only caught half of The Unnamed's set (who were frickin amazing, btw)

A persistent dict class

2007-04-20T14:54:00.000-07:00

One big part of the website is importing events, with artist names associated with it. However, sometimes a band is referred to in slightly a different way -- not enough to be significant to a person, but enough to fool a computer into thinking it's a new band. The way I initially fixed this was a artist_name_fixup dict in the importing module. The dict contained a mapping between the incorrect name and the correct one. So for example, "Static X" mapped to "Static-X" (the latter is the actual band name, according to their website.)

But this meant that every time a new fixup was needed, the source file containing the dict needed to be changed. OK, so put it in the database. Well, we check this dict a LOT, so it would be nice to not hit the DB every time.

The persistent dict solves this by saving added entries in the database, but also keeping the dict in-memory. First, we define the fixup DB table in our model:

class ArtistNameFixup(SQLObject):
    name = UnicodeCol(alternateID=True)
    value = UnicodeCol()

Then, the new class (in our util.py module):

class PersistentDict(dict):
    def __init__(self, model):
        super(PersistentDict, self).__init__()
        self.model = model
        for row in model.select():
            super(PersistentDict, self).__setitem__(row.name, row.value)

    def __setitem__(self, name, value):
        try:
            r = self.model.byName(name)
            r.value = value
        except SQLObjectNotFound:
            r = self.model(name=name, value=value)
        super(PersistentDict, self).__setitem__(name, value)

    def __delitem__(self, name):
        self.model.byName(name).destroySelf()
        super(PersistentDict, self).__delitem__(name)

    def update(self, other_dict):
        for name, value in other_dict.iteritems():
            try:
                r = self.model.byName(name)
                r.value = value
            except SQLObjectNotFound:
                r = self.model(name=name, value=value)
        super(PersistentDict, self).update(other_dict)

Finally in the import module, we add:

import util
artist_fixup_dict = util.PersistentDict(ArtistNameFixup)

When we start the TG application, the artist_fixup_dict will be populated from the database. Subsequent additions and deletions will be reflected both in the in-memory version, as well as in the db table, so if the app is reloaded, it won't lose anything.

Finally, note that the PersistentDict code is making assumptions about the SO model it is initialized with -- that the table has "name" and "value" fields. With a little more code, these could be made configurable, but I don't need that flexibility, so I haven't added it.

Logging all table changes to another table

2007-04-15T20:24:00.000-07:00

Since the site is user-editable, it needs some way to log all changes to certain tables, so they can be undone, if hit by vandals adding bad data. What I came up with this: model classes that need this (not all do) inherit from Journalled (as well as SQLObject, of course). Journalled overrides the __setattr__ and set() methods, and creates entries in the UpdateLog table for each change.

class UpdateLog(SQLObject):
    created = DateTimeCol(default=datetime.now)
    changed_by = IntCol()
    table_name = UnicodeCol(length=12)
    table_id = IntCol()
    attrib_name = UnicodeCol(length=20)
    attrib_old_value = UnicodeCol()
    attrib_new_value = UnicodeCol()

    def _get_attrib_old_value(self):
        return pickle.loads(str(self._SO_get_attrib_old_value()))
    def _set_attrib_old_value(self, value):
        self._SO_set_attrib_old_value(pickle.dumps(value))
    def _get_attrib_new_value(self):
        return pickle.loads(str(self._SO_get_attrib_new_value()))
    def _set_attrib_new_value(self, value):
        self._SO_set_attrib_new_value(pickle.dumps(value))

The old and new values are stored as actual Python objects. IDs are stored as just Ints, no relationships, since entries here can come from more than one table.

# Classes inheriting from this will have changes stored in the UpdateLog table
class Journalled(object):

    def __setattr__(self, name, value):
        if name in self.sqlmeta.columns.keys():
            self._record_update({name:value})
        super(Journalled, self).__setattr__(name, value)

    def set(self, **kw):
        self._record_update(kw)
        super(Journalled, self).set(**kw)

    def _record_update(self, updates):
        for name, value in updates.iteritems():
            old_value = getattr(self, name, None)
            if old_value != value:
                try:
                    current_user = identity.current.user.id
                except:
                    current_user = None
                u = UpdateLog(
                    changed_by=current_user,
                    table_name=self.sqlmeta.table,
                    table_id=self.id,
                    attrib_name=name,
                    attrib_old_value=old_value,
                    attrib_new_value=value
                    )

Honestly, with this in place, I haven't actually written the code to roll back any vandalism. The data is all in the UpdateLog, so I can put it off until the website actually gets hit ;-)

SQLObject svn trunk also appears to have recently added table versioning support. Sounds good, but I haven't tried it yet.

Our data's getting better, but still not perfect

2007-04-09T10:28:00.000-07:00

We have worked hard over the past few months to substantially improve the amount and correctness of event data available through BandRadar. This is the kind of thing that doesn't really show up as a new feature, but is very important to the site being valuable to users.

Although we are pulling data from multiple sources automatically, a human is still involved in reviewing each item for correctness. Lately, the further-out events have been more complete on the site. However, if you see incorrect or missing entries (maybe for your band's next gig?) YOU can add them, or correct them! If there are other problems with the site, please leave a comment and we'll take a look right away.

We have some new user-visible features coming soon but I'll blog about them after they're actually up.

SelectResults are great

2007-04-08T23:03:00.000-07:00

One change I made as the website has evolved is to switch from using SQLObject RelatedJoin and MultipleJoin to their cousins, SQLRelatedJoin and SQLMultipleJoin. After having used them, I hazard to say that I think they should be the default. The non-SQL versions return lists of objects; the SQL versions return a SelectResult.

A SelectResult (which of course is also returned from Class.select()) doesn't actually contain the results of the select, but it knows how to get it. It waits to hit the database until actually dereferenced. If you have a SelectResult instead of an actual list of objects, it's not too late to further filter or modify the result!

I had several places in my code where I was iterating through a RelatedJoin, performing a filter in Python. Switching these to SQLRelatedJoins allowed me to ask the database to perform the filter instead, thus reducing the amount of code, as well as helping performance.

The one thing to watch out for: to determine if the SelectResults actually has any rows, you need to call class.results.count() instead of len(class.results), since class.results isn't a list anymore.

how to use the TurboGears scheduler

2007-04-01T20:35:00.000-07:00

This is a quick overview of how to use the TG scheduler to, for example, do nightly db cleanups, or similar.

First, put this in your controllers.py:


from turbogears import scheduler
import batch

def startup():
    scheduler.add_weekday_task(batch.task, range(1,8), (3,0))

turbogears.startup.call_on_startup.append(startup)

You can see I've put the code I'm going to run in another file, called batch.py, in a function called task(). You can also take a look at scheduler.py in the TG code for more details on the scheduler API, but above I'm basically asking to be called every day at 3 am.

Then, in batch.py:


def task():
    hub.threadingLocal = threading_local()
    hub.begin()
    # do stuff, like clean up your identity database. You can use your ORM objects here.
    hub.commit()
    hub.end()

Hope this helps!

undocumented new TG features

2007-04-01T19:48:00.000-07:00

BandRadar is implemented in TurboGears, a Python-based web framework. The interesting thing about TG is that it is like the Borg of web frameworks -- it incorporates several different projects for different areas of functionality, and acts to tie them all together nicely.

While there is the larger debate about which components to use, now that TG offers choices for templating and ORM, there are also smaller issues, because new versions of the old components are still coming out!

For example, when I first started implementing BandRadar in TurboGears 0.8, SQLObject would not clean up entries in an intermediate table used for many-to-many relationships. Therefore, I had to write destroySelf() methods for all classes with these that cleaned this up manually.

This isn't necessary anymore, SQLObject 0.8 deletes them for you. It also allows you to define cascade behavior for ForeignKeys. This is very nice, and I am glad to be rid of that code. But why did I have to find out about this new feature by happenstance? The SQLObject developers really need to have a changelog and release notes evident on their web page. These new features are documented, but I don't think many people make it a habit to skim docs for new tidbits -- this is why release notes are important.

However, ranting aside, there is now one less reason why I should take the time to switch to SQLAlchemy. ;-)

Testing...

2007-03-07T16:19:00.000-08:00

Hi this is a new blog, where I will talk about my website BandRadar, and its implementation with TurboGears. Unfortunately, the new blogger API isn't yet supported yet in Drivel, Doh!

Lately I've been adding AJAXish features to the website, in addition to the ability to track venues (you will receive a weekly email with upcoming events for each venue) as well as integrating Google Maps onto the site.