Peter's Blog

I am about to embark on some more web development so it's framework decision time again. It's a year since I dabbled with Turbogears and Django and they both seem to have had extensive development done since then. However, since a year has passed I am basically at the start of the learning curve again.

I still have bad memories about Turbogears, specifically the mess that seems to be the link between mod_python and the application itself, i.e. cherrypy and hacky glue scripts. I was left in the situation where I could only run one app on my server because the glue didn't permit any more.

I looked at Django again and I spent some time fiddling about trying to set it up on my site5 hosting. This involved scanning through umpteen forum postings, emailing site5 tech support (I am now trusted to run compilers and bash so my account is pretty powerful) but the fcgi link was refusing to work and I was frustrated about Django not appearing to generate log files or if it did, not telling me where it put them.

On a whim I bought a book about Ruby on Rails which is a similar kind of framework but is based on the Ruby language instead of python. I wanted a good book as I am tired of using tutorials and google searches to learn these things. Since someone is being paid to write the book they can take the time to explain things in more detail and express themselves better. I think there are books under development for the python frameworks but not a nice printed second edition that I can order from Amazon and be reading the next day.

I've never got into ruby before, I read a description of it by it's author/designer years ago and he came across to me as arrogant in the way that he was criticising python. It may just be a cultural thing, the guy is japanese. By contrast Guido von Rossam, the designer of python is one of those guys like John Carmack or Linus Torvolds who don't need to be arrogant, from the way they write you know they are smart.

Without ever having written a line of it, Ruby seems to me more perl derivative than anything else, like php but better designed. This may just be the use of sigils: $blah. However in Ruby the sigils seems to be useful in telling you the scope of the variable whereas in php they appear to serve no purpose except to tell the interpreter that an identifier is not a literal string and forgetting it normally leads to obscure bugs.

Ruby on Rails was the In Thing in 2006, so important that web hosting companies advertised support for it, it wasn't something that just happened to work if you messed about long enough. Rails is more mainstream. Ruby offers similar productivity improvements to Python, being essentially a typeless language you are freer to express yourself than in languages like Java or C# where the language keeps slapping your wrists with type checking. The frameworks for these languages are also pretty overbearing, Java frameworks apparently want you to define everything in XML, that odd designed-by-commitee compromise between awkward for humans to read and inefficient for computers to parse. Life is too short.

The book is 'Agile Web Development with Rails' 2nd edition and so is reasonably up to date. It is one of those bible books that become de facto for a particular platform, being co-authored by the guy that designed Rails. The book is very descriptive, walking you through development, it is not structured like a reference but the index looks very comprehensive which is useful when you don't have google. What I have read so far has been very interesting, the book is readable and Rails looks very well designed.

I cannot give a point by point comparison with the other frameworks yet but one example of nice design is the use of what they call 'migrations'. Suppose you want to add a new column to a table: you write a migration which is the code that adds the new column and another function to undo it. Each migration gets a number. You can then use the framework to update the database schema through however many migrations to get it to the version that you want. If you need to you can roll back the migrations to undo your changes. Get the migrations working in your development database and then apply them to the production database. Compare this to Django and Turbogears a year ago where if you wanted to add a column to your database you had to update the databases (development and production) and the code individually by hand (these two only had support for creating tables from table definitions, they didn't automate support changing table definitions).

I haven't got into the template language yet but it appears to be simply ruby embedded in html, no crippled little embedded languages. I haven't tried the Rails templates, I have no idea how it copes with the ability to have a 'master' template to layout all your pages with 'sub-templates' to modify the template according to each page.

Something nice that I haven't tried yet is 'InstantRails': a 50m download that you unzip, run and get a full development environment including Ruby, Rails, Apache and Mysql. It manages this without setting all these apps up so your pc.

If LAMP means Linux, Apache, Mysql and Php, is this 'LAMR' ?

Just found out that my new site5 server has python 2.4.3. This is Good News. Last time I had a site5 server it was running python 2.2 which was getting a bit ancient. 2.3 introduced generators which have been widely adopted. The latest version is 2.5 which is (flame bait) not exactly a must have, >= 2.3 will do.

Another job for my todo list is to see if I can get django or turbogears running through cgi, but I am not optimistic.

Should I get desperate, site5 offer ruby-on-rails hosting but that would involve selling my soul in a manner akin to adopting asp.net

Here are some sites I am visiting when I am desperate for something interesting and I've exhausted my rss feeds:

http://www.google.com/ig : personalised google home. Add a clock and some flickr photo's, stick it on your second monitor and always know if dell have sent more spam to your gmail account. Add some bookmarks to my other top sites so they are ready wherever you may be.

http://del.icio.us/popular/free : the latest free stuff

http://digg.com/ : interesting articles, although the comments are rather inane. slashdot without depth (no nested comments).

http://tailrank.com/ : this one finds the articles that are most referenced by blogs and hence are presumably the most interesting (or bait the most flames). How did I find it? It crawls this site and I found it in the logs.

http://en.wikipedia.org/wiki/Special:Random : wikipedia random page. May need pressing a few times to avoid obscure star trek characters.

http://www.djangoproject.com/ : see if the magic removal branch has been released yet (django in joke).

Still umming and arring about django vs turbogears. I knocked up an online questionnaire for our intranet in TurboGears in a couple of days and it is now running. I'm starting another intranet program so I thought I'd give django a try. Once again I had to go through the tutorial to remember all the steps to set the project up. I fiddled with batch files to set up environment variables and everything, hacked the source to get it to run under WingIDE, started on the database model, all the while seeming to struggle with django itself. I got the admin screens up and running and was figuring out how to stop a password being displayed in a password field. As far as I could see, this involved editing the templates for the admin screen. It's about here that I got tired and went back to turbogears as I couldn't be bothered to tackle the django template system.

In about the same amount of time with turbogears, the project was underway and I was focused on getting the main template running. There is something about turbogears that is to me more 'pythonic' in terms of things working the way you expect and not having to waste time figuring out how to do things. kid Just Works, it is fundamentally simple and keeps out of the way, leaving you to get on with your work.

SQLObject does not appear to handle relations as nicely as the Django ORM: in django it appears I can delete an object and have the associated many-to-many mapping in the database deleted automatically whereas SQLObject does not delete it automatically and neither the documentation or the tests show me how to delete it explicitly. There is a 'remove' method in there but the object model is too complex for me how to work out how to reach it. The SQLObject documentation is comprehensive but is essentially a list of examples (not a bad thing), it does not seem to include a detailed api listing. I will have to fiddle around at the interactive prompt and see if I can figure out the incantation. Then again, the django ORM api is currently being made more intuitive. If I was going to do a lot of database work I'd be tempted to use my own db wrapper: I'd have to write SQL but I don't care, I know how to write SQL, I don't know how to use these librarys.

I have looked through the tests for both projects as example code and in both the tests are a bit cursory. The django tests are far more readable than the SQLObject tests (doctestish vs unittestish). The django tests also benefit from being commented.

Right now I'm back with turbogears as the project I am working on will not have many relationships and they will rarely need deleting.

Members of both projects have left comments on this blog, making me feel guilty about picking one over the other. I like them both equally, choosing one over the other is hard. Right now I'm further along the TurboGears learning curve which for me is not as steep as the Django learning curve.

So I develop an online questionnaire (and learnt to spell that word) for the online intranet in turbogears, it takes maybe a day total to have a nice sqlite database driven app that walks you through the questions, allows you to go back and forth, handles errors, themed exactly like the main intranet (drupal) really nice.

After developing under windows I casually copy the files to the ubuntu intranet server and try installing it in a virtual path under the root domain (e.g. intranet.com/subdirectory). Then I mess around for hours because the mod_python support for cherrypy appears to be hacked on, the support for mapping applications to virtual paths in turbogears using servo.webpath is broken and cherrypy support for virtual paths boils down to this hack which it states is buggy but doesn't say what the bugs are! The main bug I see is that IT DOESN'T MAKE ANY DIFFERENCE.

Argh.

I can get the app to work but only the index, none of the other methods in the controller are recognised.

While googling for this I came across an interesting django vs turbogears review here. He makes an interesting point: django sites survive slashdotting. I look at the cherrypy site and see where cherrypy is being used: um, sorry but these sites don't look all that impressive compared to the django equivalents.

So I'm disillusioned with cherrypy: while the raw api is clean, it looks a bit hacky and the documentation is poor ('in progress') by comparison to the other elements of the turbogears stack. I have no idea whether django supports virtual paths but it Just Worked under mod_python. Should I switch back to that, given that they are working on cleaning up their database api? I'd be tempted to stick with kid even if I went back to django, like this guy.

Dunno what to do, almost tempted to learn ruby, I'm that fed up with it all.

UPDATE: It took me a few days to figure the problem out: I had an extra .htaccess file I didn't know was there, causing mischief and interfering with whatever I did in the main config file. I've got rid of it and turbogears/ cherrypy is working fine.

The turbogears server.webpath="/subdir" does not seem to be working (as mentioned in the turbogears issue tracker) but if I hack my controllers.py file I can work around it:

class subdir:
    @@turbogears.expose(html="tgpcw.templates.welcome")
    def index(self):
        import time
        return dict(now=time.ctime())

class Root(controllers.Root):
    subdir = subdir()

I enabled this in the apache2 config file thusly:

<Location "/hands">
    SetHandler mod_python
    PythonHandler mpcp
    PythonDebug On
    PythonPath "['/var/www/tgpcw'] + sys.path"
    PythonOption cherrysetup tgpcw_start::mp_setup

    AllowOverride All
    Order deny,allow
    Deny from all
    Allow from all
</Location>

where tgpcw is the name of my turbogears stuff.

My unhappiness with django is growing. I have a model containing the following classes:

class Post:

class Tag:

I have a many-to-many relationship between these two classes. Trying to code something, if I do something like the following:

class Post:
   def DoSmethingWithTags( self):
       oTags = tags.get_tags()

it fails because it needs:

from django.models.blog import posts,tags

but I cannot put this in the model file or module import starts recursing (or so I believe). This kind of thing is just a pain and I don't want to spend my time working around architectural oddities. I tried creating a library module in the model directory but then it couldn't be found when I tried to import it (?). Python path weirdness, and django is fussy about the python path setup, more so than any other package I can think of, usually everthing is relative to a current directory and is happy.

So I gave Turbogears a look. I watched the wiki in 20 minutes film, seems ok but a lot to learn (like django), including it's own architectural oddities.

I installed it directly onto Windows which was very easy.

Next stumbling block is to get it to run under WingIDE. This was more fiddly. Like django it has autoreload but unlike django there is a config entry to disable it. Next problem is that it is multi-threaded which gives WingIDE a hard time. Here is my solution:

Add the following to the turbogears dev.cfg file:

   1  #
   2  # Run in a single thread
   3  #
   4  server.class="_cphttpserver.embedded_server"
   5  server.threadPool=1
   6  
   7  #
   8  # Disable auto reload.
   9  #
  10  autoreload.on=0
Toggle Line Numbers

but this is not enough as it still runs two threads, one to watch the keyboard. Don't care about this if running under a debugger with a stop button: I can't stop it cleanly but this is not a big deal for development. Making it strictly single threaded involves overriding the _cpserver.Server class. I changed my turbogears xxx-start.py file as follows:

   1  cherrypy.root = Root()
   2  #cherrypy.server.start()
   3  
   4  import cherrypy._cpserver
   5  
   6  class NonThreadedServer( cherrypy._cpserver.Server):
   7      def start_http_server(self, blocking=True):
   8          """Start the requested HTTP server."""
   9          if self.httpserver is not None:
  10              msg = ("You seem to have an HTTP server still running."
  11                     "Please call server.stop_http_server() "
  12                     "before continuing.")
  13              warnings.warn(msg)
  14  
  15          if self.httpserverclass is None:
  16              return
  17  
  18          if cherrypy.config.get('server.socketPort'):
  19              host = cherrypy.config.get('server.socketHost')
  20              port = cherrypy.config.get('server.socketPort')
  21  
  22              cherrypy._cpserver.wait_for_free_port(host, port)
  23  
  24              if not host:
  25                  host = 'localhost'
  26              onWhat = "http://%s:%s/" % (host, port)
  27          else:
  28              onWhat = "socket file: %s" % cherrypy.config.get('server.socketFile')
  29  
  30          # Instantiate the server.
  31          self.httpserver = self.httpserverclass()
  32  
  33          # HTTP servers MUST be started in a new thread, so that the
  34          # main thread persists to receive KeyboardInterrupt's. This
  35          # wrapper traps an interrupt in the http server's main thread
  36          # and shutdowns CherryPy.
  37          def _start_http():
  38              try:
  39                  self.httpserver.start()
  40              except (KeyboardInterrupt, SystemExit), exc:
  41                  self.interrupt = exc
  42  #        threading.Thread(target=_start_http).start()
  43  
  44          #
  45          # Run in a single thread to make debugging a joy
  46          #
  47          self.state = cherrypy._cpserver.STARTED
  48          # Changing the socket timeout makes WingIDE more responsive
  49          self.httpserver.socket.settimeout( 0.1)
  50          self.httpserver.start()  # Should not return from here!!
  51  
  52          if blocking:
  53              self.wait_for_http_ready()
  54  
  55          cherrypy.log("Serving HTTP on %s" % onWhat, 'HTTP')
  56  
  57  # _cphttpserver uses this global variable so have to poke it
  58  cherrypy.server = NonThreadedServer()
  59  cherrypy.server.start()
Toggle Line Numbers

I wasn't able to make django debuggable in WingIDE without hacking it's source: it didn't seem to have as many useful hooks as cherrypy/turbogears.

I've been quiet on the Django progress front. There has been a little but I got bogged down in unit testing, mainly learning the shortcomings of the python unit testing module.

Some django observations:

• can debug it very nicely in WingIDE using my trick.
• problem with the many-to-many relationship: if you try to set the members of a relationship using the set_XXX method (where XXX = name of class members you want to add) you may get an error if your models do not include a field call 'id' (lower case) due to nasty hard-wired property name in the code. I had one called Id and I had to change the capitalisation.
• the database api is a bit like the template system: I'd rather use something else but it'll do. I can mix in other api's if I want to get thinks done quick without wasting time single stepping through library code.

My latest Safari book is Python in a Nutshell a very comprehensive reference guide to python, plenty to learn. In some places, like the standard library section, it just summarises what the module does in a way that is not so enlightning but I have picked up some stuff in passing that I was not aware of before. The standard python documentation is very dry and it can be hard to find a useful descriptions. Python in a Nutshell does explain things in a more understandable way and it does include more example code. I may well keep the safari book in my bookshelf permanently (or buy it) to use in parallel with the python documentation.

I read the section on testing which described the doctest and unittest modules. I have tried docstring in the past, it involves putting test scripts in the docstrings of each function and to my mind this made the code bloated and hard to read.

I hadn't tried unittest before so I gave it a try with some django code, specificly code to convert a string of tag names into correct database entries. I started off doing it the Right Way by writing the tests first, before I started coding. It was a good exercise, just thinking of what I wanted and what could go wrong before I started to code.

The unittest module requires creating TestCase classes with three phases of testing:

• a setUp function to initialise things
• various tests in arbitrary order (because each test is a method and the methods are stored in a dictionary there is no guarantee on what order they will be executed)
• a tearDown function to close things up

In my case of manipulating the database I needed setup and teardown code for each individual test to ensure that the correct tags and tag->post mappings were created for each tag string under test. I could have created multiple TestCase classes, with seperate setUp and tearDown functions for each individual test but that seemed like too much typing. The problem here was really due to the arbitrary order in which the tests could be performed: each test cannot rely on what state the preceeding test left the database in so each test has to set the database up first.

As I slept last night I tossed and turned with this on my mind. Writing lots of setUp and tearDown code just to get around the arbitrary order. I could write just one big test function that went through a sequence of tests but it felt like cheating. To make this worse, I looked at the django tests and they used something similar to docstrings but in seperate files, a straight sequence of tests as if typed into the command prompt. It looked good and reading it is like reading example code. However, I have two problems with working this way:

• it needs more editing to create the script than unittest
• because all the code is inside a string, it is all one colour, I lose syntax highlighting. Do I need editor macro's to toggle the string delimiters on and off?

I think what I will do is use unittest without excessive setup and teardown code and just write long test functions. The aim of the exercise after all is to test, not to write test code.

Conclusion: testing: keep it simple and don't lose sleep over it.

Update: single stepping through the unittest code with WingIDE, it seems that the test case methods are sorted by name using the cmp function so they are probably alphabetic. Hence I could just name my functions test_001, test_002 etc or I write something to determine line numbers, or I define an array of test functions...

Been playing more with komodo, just exploring I haven't used it in anger (too many distractions, life getting in the way). I have been looking into setting up my Django project. Here is what I have been able to do so far:

• Create a project file containing all the python files: django itself, my application and the template files. Komodo is quite happy editing all types of files, even the html in the template files. I have yet to put the css and supporting graphic files into subversion but when I do I will be able to put the css file in the project as well: that's all source files for a web application, edited using the same tool, organised in one project.
• There is a nice 'Run' facillty to run command line programs from within komodo. It is just so easy to run a command and add it to the toolbox for your project so that the tool is there while working on the project. This means the commands to open an ssh tunnel to mysql, do a svn update on Django, check my own files into my subversion repository, clean up .pyc files etc are a double click away. While it is entirely possible to set this up in Vim, it's to fiddly to bother with. Komodo makes it so easy. Komodo personal has no support for source control but it is easy enough to set up commands for your subversion operations.
• You can add shortcuts to URLs to your project so while you are working on your web app you quickly see the appropriate page in your browser. Ok, you could put shortcuts in your browser but they would be there all the time, polluting your shortcut namespace, and you would have to explicity swap from your editor/IDE to the browser to get to the shortcuts. What I am trying to say is that what komodo is doing is not rocket science, is not impossible any other way (even in vim) it's just that it is done in a way that is very easy to set up.

There is just one thing that is annoying me about komodo. To run my django app I need to run the django-admin.py file. I have yet to find a way to tell komodo that I always want this file exected. By default it always executes the file in the current tab of the editor. In my case I will not be editing django-admin.py as it is part of django itself, but I have to have this file open in the editor and switch to it's tab before I can debug. I tried recording a macro to switch to django-admin.py and then run but this doesn't work. I tried defining a debug configuration but each file gets it's own set of debug configurations. It looks horribly like I have to set up debug configuration for every source file that is likely to be the current file when I want to debug.

Back to the good stuff, there is a useful mode for editing css files: you tell komodo which html file you want to see it with and komodo splits the screen to show a css editor and a web browser preview window. It is a very nice setup for working with css files apart from it not helping you to write your css.

So far I am finding it quite impressive.

I decided to persist with trying to get a decent django development going using either komodo or wingide. I'm really hankering for a windows based python IDE, the open source ones I have tried (spe, pythonwin, idle, boa constructor) just don't cut it for me.

I have spent some time with the trial versions of both with django and both had the same problem: I set a breakpoint in a view, I run the django_admin.py script with the runserver parameter, I go to my browser and load the right url but it doesn't stop at the breakpoint.

Well today I found the problem: django_admin.py actually forks another process to run django and the debuggers cannot handle this, they are working on the original process. The new process is forked as part of the auto-reload mechanism which will kill the forked process if any of the source files changes, allowing the original script to fork a new process using the new modules. Crude but effective.

It is possible to stop it doing this but only by editing the code. I changed django_src/django/core/management.py as follows:

Change this:

    from django.utils import autoreload
    autoreload.main(inner_run)

to this:

    inner_run()

and the problem disappears. How nice it would be to have a config setting to disable the auto-reload...

Now which IDE would I chose? Well based on the fact that I am only considering the personal versions (about a tenth of the cost of the professional versions) it goes hands-down to Komodo. Why?

• it can handle more languages than just python (php, perl etc)
• you can inspect variables in the debugger by putting the mouse over a symbol and looking at the tooltip
• there is an interactive python shell available when you stop at breakpoints: to me this is invaluble. Effectively wingide personal only gives you the ability to look at current variable values, you cannot change them or do command line experiments.
• komodo has a very nice regular expression tool

Wingide may have some plus points over komodo but for now komodo is the most useful tool and it is the one I would rather spend time in.

I will carry on with the trials for now, make the final decision when the trial periods are up.

Still hunting for a decent python debugging environment I have convinced myself that Komodo cannot properly debug code that is exec'ed or eval'ed: it says so in the release notes. I tried resorting to IDLE for a while but for some bizarre reason it would not stop at breakpoints so I gave up on that as well.

So I'm back to my vim python debug setup. It seems nothing has changed in the art since I developed that back in April.

So I was working on an application with a GUI that did a lot of processing, an awkward thing to test and debug: I have to enter the test parameters into the GUI every time and start the processing. I realise then that I should just rip out the processing, seperate it from the user interface and call it from a test harness. Voila, it's no longer a GUI app and it isn't in an exec'ed module, it is a simple script file. Vast scope for debugging.

Then I realise that I can do something similar to test and debug django. I looked at the standard view code, the code that converts an url into html. One of my view methods looks like this:

def index(request):
    oBlogList = posts.get_list(order_by=['-Created'], limit=5)
    AddTags( oBlogList)
    AddCommentCount( oBlogList)

    return render_to_response('blog/posts', {'posts_title': 'Index',
                                                'blog_list': oBlogList,
                                                'root': g_strRoot
                                                })

On a hunch I tried doing this interactively in a console and examining the object returned by render_to_response. This turned out to be an object that had a member called 'content' holding the html for the page.

The upshot of all this is that again I can write simple tests to generate the html without worrying about an http server. I can either compare the html generated against what I want or use this trick to stick the html straight into a browser. I no longer have to worry about the server and any weird tricks it may be playing to confuse and confound debuggers.

It remains to be seen how much can be developed like this, how to handle forms, sessions, workflows etc but for now the moral is: divide and conquer.

On a sudden whim I decided to try developing my new django blog on my windows laptop then uploading it to the server when it is working. The setup is slightly complex and I have only just set it up so lets try to describe it in detail.

• I already had the source in subversion so I installed tortoise svn a nice Windows svn client so I could access the source from windows, upload it into subversion when it was tested and read it back into the working image on the server.
• Also using Tortoise I extracted the latest django source onto the laptop.
• I set up the windows environment variables according to the django installation instructions.
• Because my templates directories are different under windows and ubuntu I put a conditional section in my settings module:

     1  import os
     2  
     3  if os.name != 'nt':
     4      TEMPLATE_DIRS = (
     5          # Put strings here, like "/home/html/django_templates".
     6          '/usr/local/lib/Django/django_templates',
     7      )
     8  else:
     9      TEMPLATE_DIRS = (
    10          # Put strings here, like "/home/html/django_templates".
    11          'c:\\python24\\lib\\site-packages\\Django\\django_templates',
    12      )
  Toggle Line Numbers

• I installed the mysql client on the laptop, along with the appropriate version of MySQLdb
• I didn't want to run the database server on my laptop so I use an ssh tunnel to connect to the server: I have to run this in a command console while I am accessing the database:

  ssh -L 3306:127.0.0.1:3306 me@myserver.org
  

  For security the mysql server does not accept connections from outside the box it is running on, except via ssh. I am using the live database on the server and for development. I could create another instance of the database for testing.
• To get to files needed by the theme (e.g. style.css) I had to tweek my templates under windows such that they were loaded from the server using the full absolute url rather than a relative url. I didn't want to make django server them or put apache on my laptop.
• django_admin.py just worked, no problems.

Why did I do all this? Well I'm thinking again of using komodo as a development environment with a proper debugger. I don't have a lot of time to devote to all this, every minute is precious, so it will help to have productive tools available.

My embroyo Django blog got to the stage where I wanted to theme it. I started off by giving it the theme from this site which requires serving of various graphic files as well as the .css file. This means setting apache up for serving static files as well as dynamic pages. Django can serve static files but it is better to use apache as that is what apache is good at. The django advise is to use a differrent server for the static files, something lean and mean, but I'd prefer to use the one server. I get about 500 page loads/day, performance is not totally paramount to me.

I achieved this using the following apache virtualhost setup (in /etc/apache2/sites-available/mysite for debian and apache2).

<VirtualHost *>
    ServerName sitename.org
    ServerAlias sitename.org *.sitename.org
    ServerAdmin webmaster@localhost

    DocumentRoot /var/www/sitename.org
    <Directory />
        Options -Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>

    #
    # Most stuff is handled by django
    #
    <Location "/">
        SetHandler python-program
        PythonHandler django.core.handlers.modpython
        PythonPath "['/usr/local/lib/Django'] + sys.path"
        SetEnv DJANGO_SETTINGS_MODULE django_local.settings
    </Location>

    #
    # Theme files are static and not done by django
    #
    <Location "/theme">
        SetHandler none
    </Location>

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    ErrorLog /var/log/apache2/sitename.org/error.log

    CustomLog /var/log/apache2/sitename.org/access.log combined
    ServerSignature On

</VirtualHost>

Here I use the Location keyword to tell mod_python to handle all accesses to the site. Then I override this for the subdirectory /theme where I set the handler to none, causing static content to be served.

By far my favourite feature of django is the auto-reload feature whereby if you change some of the code the server will automatically reload it and you don't need to reboot the server. Having to do continual reboots was a bugbear with pyds. In django it has worked flawlessly so far.

I didn't mention the Django templating system in my citique yesterday. It seems ok, as templating systems go, my favourite python templating system being cheetah.

The main advantage of cheetah that I have seen is that you can embed more python code in cheetah templates, or at least use more complex expressions to retrieve values. The django template system seems to restrict you to retrieving values of object properties: you cannot call functions as it complains about '(' characters. I think this is designed to force you to separate logic from presentation which is a Good Thing.

The Django system supports inheritance so a specific web page can inherit from a template describing the overall layout which is cool. I have yet to figure out how or whether it is possible to the equivalent of #include or import to put common sub-blocks in some pages (conceptually similar to a subroutine call). The 'load' tag might be the way.

The django system looks good enough to me to use in preference to cheetah: that's the coolest thing, I have the choice and I could even use both if I wanted.

I've been playing with django recently. Django is a web application framework written in python. It means I can rewrite this website in python rather than use php: there are things that I would like to do that I cannot be bothered to do in php (or perl or C or assembly language), in python they would be fun.

I went through the django tutorial once and then I went through it again, using it as guidance in starting my new application. I have got it to a pretty basic point where it lists postings, posting can be clicked on for details and comments, and tags are supported already. It is not ready to go online by any means but I am happy with the progress and how django has helped me in a few hours.

Anyway, django pros:

• very good documentation I'd say. Lots of background descriptions and examples, not just bare api descriptions. Didn't have to read the source code (much).
• I like the system for mapping an url to a view using regular expressions. This is so flexible and powerful, it sold me compared to the cherrypy/turbogears way of doing things.
• The generic admin interface is handy. It essentially writes data-entry forms for you, although like any automatic form generation they are not ideal. They are more convenient than just using the webmin mysql module to edit the raw tables.
• The online documentation mentions automatic document generation. By reading the source I found this was at localhost:8000/admin/doc

Django Cons:

• The process of creating a website is kinda complex, it is probably best to be guided by the tutorial each time.
• I don't like a directory structure being imposed on me, especially a complex one.
• I don't like the way it uses environment variables to tell it which project it is dealing with. Seems fiddy if you are dealing with multiple sites.
• I think what I'm leading up to is why isn't it simply based on a script that imports the django modules, imports a personal config module and then goes on from there? Like every other python app I can think of?
• The way that the database schema is defined in python objects and these then generate the corresponding schema in the database is nice but it is very annoying that it doesn't support modification of the model. Once you have created it you are back to manually editing the object model and the database. BTW Don't tell me to design a perfect database schema upfront, I'm a hacker, that's not how I work.

The database update thing started to be a stumbling block for me. I could use my own DBTable module for easy database access in my web app but then I lose the django database administration screens. So I decided to write a script to do the upgrade automatically. This is not a totally robust solution, it isn't even totally automatic, you have to list the tables to upgrade (they could be introspected from the django.models.blog but there didn't seem to be a clean list of database table names in there).

This uses my DBTable module to access the database, I didn't use the django database api as that is by definition in a state of flux during an upgrade. This script dumps the database data out into a python script and then shells the django utilities to update the database schema. The auto-generated python script is then executed to populate the new empty database. I decided to generate a python script (first cut used a pickle file) as it allows me to manually edit the way data is added to the database if necessary (e.g. adding default values for new fields or whatever).

   1  #
   2  # Update django
   3  #
   4  import os
   5  import MySQLdb
   6  import DBTable
   7  import sys
   8  
   9  #
  10  # Tables that need backing up and restoring, in order of dependency
  11  #
  12  g_strMyTables = (
  13      "blog_tags",
  14      "blog_posts",
  15      "blog_posts_Tags",
  16      "blog_comments",
  17  )
  18  
  19  def BackupData():
  20      """
  21      Backup all data to python file.
  22      """
  23      oConnection = MySQLdb.connect('localhost', 'django', 'django')
  24      oConnection.select_db( 'django')
  25  
  26      oFile = open( '/tmp/django_data.py', 'w')
  27  
  28      oFile.write( '''import MySQLdb
  29  import DBTable
  30  import datetime
  31  
  32  oConnection = MySQLdb.connect('localhost', 'django', 'django')
  33  oConnection.select_db( 'django')
  34  ''')
  35  
  36      for strTable in g_strMyTables:
  37          oFile.write( "oDB = DBTable.DBTable( oConnection, '%s')\n" % strTable)
  38  
  39          oDB = DBTable.DBTable( oConnection, strTable)
  40          oDB.Select()
  41          for oRow in oDB:
  42              oFile.write( "oDB.Insert( %s)\n" % str(oRow))
  43  
  44  def RestoreData():
  45      """
  46      Restore all data from backup file.
  47      """
  48      oFile = open( '/tmp/django_data.py')
  49      exec oFile
  50  
  51  #
  52  # Do a dry run on generating sql before we do anything drastic.
  53  #
  54  if os.system( 'django-admin.py sqlall blog') > 0:
  55      raise 'Failed to create sql'
  56  
  57  #
  58  # Backup old data.
  59  #
  60  BackupData()
  61  
  62  #
  63  # Clear the old schema.
  64  #
  65  if os.system( 'django-admin.py sqlclear blog | mysql -u django -pdjango django') > 0:
  66      raise 'Failed to clear sql'
  67  
  68  #
  69  # Load the new schema.
  70  #
  71  if os.system( 'django-admin.py sqlall blog | mysql -u django -pdjango django') > 0:
  72      raise 'Failed to create sql'
  73  
  74  #
  75  # Restore data.
  76  #
  77  RestoreData()
  78  
  79  print 'Upgraded'
Toggle Line Numbers

Once the backup file is generated in /tmp/drupal_data.py it can be run at any time to repopulate the database with data.

Going through django tutorial for a second time and was disheartened to get a weird error when trying to log in as admin:

pcw@intranet:~/django_src$ ~/django_admin.py runserver
Validating models...
0 errors found.

Starting server on port 8000 with settings module 'djangotutorial.settings'.
Go to http://127.0.0.1:8000/ for Django.
Quit the server with CONTROL-C (Unix) or CTRL-BREAK (Windows).
[01/Nov/2005 07:15:52] "GET /admin/ HTTP/1.1" 200 1615
Traceback (most recent call last):
  File "/home/pcw/django_admin.py", line 131, in ?
    main()
  File "/home/pcw/django_admin.py", line 108, in main
    ACTION_MAPPING[action](addr, port)
  File "/usr/lib/python2.4/site-packages/django/core/management.py",
    line 656, in runserver
    autoreload.main(inner_run)
  File "/usr/lib/python2.4/site-packages/django/utils/autoreload.py",
    line 71, in main
    reloader_thread()
  File "/usr/lib/python2.4/site-packages/django/utils/autoreload.py",
    line 44, in reloader_thread
    mtime = os.stat(filename).st_mtime
OSError: [Errno 2] No such file or directory:
'/usr/lib/python2.4/site-packages/django/templatetags/adminmedia.py'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.4/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.4/threading.py", line 636, in __exitfunc
    self._Thread__delete()
  File "/usr/lib/python2.4/threading.py", line 522, in __delete
    del _active[_get_ident()]
KeyError: -1209573248
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.4/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.4/threading.py", line 636, in __exitfunc
    self._Thread__delete()
  File "/usr/lib/python2.4/threading.py", line 522, in __delete
    del _active[_get_ident()]
KeyError: -1209573248
Unhandled exception in thread started by
Error in sys.excepthook:

Original exception was:

ugly, huh? Looks like three stack traces, two of which are the same. Where to start. Well I think this line is the clue:

OSError: [Errno 2] No such file or directory:
'/usr/lib/python2.4/site-packages/django/templatetags/adminmedia.py'

Looking in /usr/lib/python2.4/site-packages/django/templatetags/ I see adminmedia.pyc but no adminmedia.py. I think what has happened is that updating django by doing

svn update

does not clear out the python .pyc files that are generated automaticaly by python. The django autoreload code is seeing the .pyc files and looking for the corresponding .py files. I deleted the .pyc files and life went on.