Peter's Blog

I am about to embark on some more web development so it's framework decision time again. It's a year since I dabbled with Turbogears and Django and they both seem to have had extensive development done since then. However, since a year has passed I am basically at the start of the learning curve again.

I still have bad memories about Turbogears, specifically the mess that seems to be the link between mod_python and the application itself, i.e. cherrypy and hacky glue scripts. I was left in the situation where I could only run one app on my server because the glue didn't permit any more.

I looked at Django again and I spent some time fiddling about trying to set it up on my site5 hosting. This involved scanning through umpteen forum postings, emailing site5 tech support (I am now trusted to run compilers and bash so my account is pretty powerful) but the fcgi link was refusing to work and I was frustrated about Django not appearing to generate log files or if it did, not telling me where it put them.

On a whim I bought a book about Ruby on Rails which is a similar kind of framework but is based on the Ruby language instead of python. I wanted a good book as I am tired of using tutorials and google searches to learn these things. Since someone is being paid to write the book they can take the time to explain things in more detail and express themselves better. I think there are books under development for the python frameworks but not a nice printed second edition that I can order from Amazon and be reading the next day.

I've never got into ruby before, I read a description of it by it's author/designer years ago and he came across to me as arrogant in the way that he was criticising python. It may just be a cultural thing, the guy is japanese. By contrast Guido von Rossam, the designer of python is one of those guys like John Carmack or Linus Torvolds who don't need to be arrogant, from the way they write you know they are smart.

Without ever having written a line of it, Ruby seems to me more perl derivative than anything else, like php but better designed. This may just be the use of sigils: $blah. However in Ruby the sigils seems to be useful in telling you the scope of the variable whereas in php they appear to serve no purpose except to tell the interpreter that an identifier is not a literal string and forgetting it normally leads to obscure bugs.

Ruby on Rails was the In Thing in 2006, so important that web hosting companies advertised support for it, it wasn't something that just happened to work if you messed about long enough. Rails is more mainstream. Ruby offers similar productivity improvements to Python, being essentially a typeless language you are freer to express yourself than in languages like Java or C# where the language keeps slapping your wrists with type checking. The frameworks for these languages are also pretty overbearing, Java frameworks apparently want you to define everything in XML, that odd designed-by-commitee compromise between awkward for humans to read and inefficient for computers to parse. Life is too short.

The book is 'Agile Web Development with Rails' 2nd edition and so is reasonably up to date. It is one of those bible books that become de facto for a particular platform, being co-authored by the guy that designed Rails. The book is very descriptive, walking you through development, it is not structured like a reference but the index looks very comprehensive which is useful when you don't have google. What I have read so far has been very interesting, the book is readable and Rails looks very well designed.

I cannot give a point by point comparison with the other frameworks yet but one example of nice design is the use of what they call 'migrations'. Suppose you want to add a new column to a table: you write a migration which is the code that adds the new column and another function to undo it. Each migration gets a number. You can then use the framework to update the database schema through however many migrations to get it to the version that you want. If you need to you can roll back the migrations to undo your changes. Get the migrations working in your development database and then apply them to the production database. Compare this to Django and Turbogears a year ago where if you wanted to add a column to your database you had to update the databases (development and production) and the code individually by hand (these two only had support for creating tables from table definitions, they didn't automate support changing table definitions).

I haven't got into the template language yet but it appears to be simply ruby embedded in html, no crippled little embedded languages. I haven't tried the Rails templates, I have no idea how it copes with the ability to have a 'master' template to layout all your pages with 'sub-templates' to modify the template according to each page.

Something nice that I haven't tried yet is 'InstantRails': a 50m download that you unzip, run and get a full development environment including Ruby, Rails, Apache and Mysql. It manages this without setting all these apps up so your pc.

If LAMP means Linux, Apache, Mysql and Php, is this 'LAMR' ?

Just found out that my new site5 server has python 2.4.3. This is Good News. Last time I had a site5 server it was running python 2.2 which was getting a bit ancient. 2.3 introduced generators which have been widely adopted. The latest version is 2.5 which is (flame bait) not exactly a must have, >= 2.3 will do.

Another job for my todo list is to see if I can get django or turbogears running through cgi, but I am not optimistic.

Should I get desperate, site5 offer ruby-on-rails hosting but that would involve selling my soul in a manner akin to adopting asp.net

Here are some sites I am visiting when I am desperate for something interesting and I've exhausted my rss feeds:

http://www.google.com/ig : personalised google home. Add a clock and some flickr photo's, stick it on your second monitor and always know if dell have sent more spam to your gmail account. Add some bookmarks to my other top sites so they are ready wherever you may be.

http://del.icio.us/popular/free : the latest free stuff

http://digg.com/ : interesting articles, although the comments are rather inane. slashdot without depth (no nested comments).

http://tailrank.com/ : this one finds the articles that are most referenced by blogs and hence are presumably the most interesting (or bait the most flames). How did I find it? It crawls this site and I found it in the logs.

http://en.wikipedia.org/wiki/Special:Random : wikipedia random page. May need pressing a few times to avoid obscure star trek characters.

http://www.djangoproject.com/ : see if the magic removal branch has been released yet (django in joke).

Still umming and arring about django vs turbogears. I knocked up an online questionnaire for our intranet in TurboGears in a couple of days and it is now running. I'm starting another intranet program so I thought I'd give django a try. Once again I had to go through the tutorial to remember all the steps to set the project up. I fiddled with batch files to set up environment variables and everything, hacked the source to get it to run under WingIDE, started on the database model, all the while seeming to struggle with django itself. I got the admin screens up and running and was figuring out how to stop a password being displayed in a password field. As far as I could see, this involved editing the templates for the admin screen. It's about here that I got tired and went back to turbogears as I couldn't be bothered to tackle the django template system.

In about the same amount of time with turbogears, the project was underway and I was focused on getting the main template running. There is something about turbogears that is to me more 'pythonic' in terms of things working the way you expect and not having to waste time figuring out how to do things. kid Just Works, it is fundamentally simple and keeps out of the way, leaving you to get on with your work.

SQLObject does not appear to handle relations as nicely as the Django ORM: in django it appears I can delete an object and have the associated many-to-many mapping in the database deleted automatically whereas SQLObject does not delete it automatically and neither the documentation or the tests show me how to delete it explicitly. There is a 'remove' method in there but the object model is too complex for me how to work out how to reach it. The SQLObject documentation is comprehensive but is essentially a list of examples (not a bad thing), it does not seem to include a detailed api listing. I will have to fiddle around at the interactive prompt and see if I can figure out the incantation. Then again, the django ORM api is currently being made more intuitive. If I was going to do a lot of database work I'd be tempted to use my own db wrapper: I'd have to write SQL but I don't care, I know how to write SQL, I don't know how to use these librarys.

I have looked through the tests for both projects as example code and in both the tests are a bit cursory. The django tests are far more readable than the SQLObject tests (doctestish vs unittestish). The django tests also benefit from being commented.

Right now I'm back with turbogears as the project I am working on will not have many relationships and they will rarely need deleting.

Members of both projects have left comments on this blog, making me feel guilty about picking one over the other. I like them both equally, choosing one over the other is hard. Right now I'm further along the TurboGears learning curve which for me is not as steep as the Django learning curve.

So I develop an online questionnaire (and learnt to spell that word) for the online intranet in turbogears, it takes maybe a day total to have a nice sqlite database driven app that walks you through the questions, allows you to go back and forth, handles errors, themed exactly like the main intranet (drupal) really nice.

After developing under windows I casually copy the files to the ubuntu intranet server and try installing it in a virtual path under the root domain (e.g. intranet.com/subdirectory). Then I mess around for hours because the mod_python support for cherrypy appears to be hacked on, the support for mapping applications to virtual paths in turbogears using servo.webpath is broken and cherrypy support for virtual paths boils down to this hack which it states is buggy but doesn't say what the bugs are! The main bug I see is that IT DOESN'T MAKE ANY DIFFERENCE.

Argh.

I can get the app to work but only the index, none of the other methods in the controller are recognised.

While googling for this I came across an interesting django vs turbogears review here. He makes an interesting point: django sites survive slashdotting. I look at the cherrypy site and see where cherrypy is being used: um, sorry but these sites don't look all that impressive compared to the django equivalents.

So I'm disillusioned with cherrypy: while the raw api is clean, it looks a bit hacky and the documentation is poor ('in progress') by comparison to the other elements of the turbogears stack. I have no idea whether django supports virtual paths but it Just Worked under mod_python. Should I switch back to that, given that they are working on cleaning up their database api? I'd be tempted to stick with kid even if I went back to django, like this guy.

Dunno what to do, almost tempted to learn ruby, I'm that fed up with it all.

UPDATE: It took me a few days to figure the problem out: I had an extra .htaccess file I didn't know was there, causing mischief and interfering with whatever I did in the main config file. I've got rid of it and turbogears/ cherrypy is working fine.

The turbogears server.webpath="/subdir" does not seem to be working (as mentioned in the turbogears issue tracker) but if I hack my controllers.py file I can work around it:

class subdir:
    @@turbogears.expose(html="tgpcw.templates.welcome")
    def index(self):
        import time
        return dict(now=time.ctime())

class Root(controllers.Root):
    subdir = subdir()

I enabled this in the apache2 config file thusly:

<Location "/hands">
    SetHandler mod_python
    PythonHandler mpcp
    PythonDebug On
    PythonPath "['/var/www/tgpcw'] + sys.path"
    PythonOption cherrysetup tgpcw_start::mp_setup

    AllowOverride All
    Order deny,allow
    Deny from all
    Allow from all
</Location>

where tgpcw is the name of my turbogears stuff.

My unhappiness with django is growing. I have a model containing the following classes:

class Post:

class Tag:

I have a many-to-many relationship between these two classes. Trying to code something, if I do something like the following:

class Post:
   def DoSmethingWithTags( self):
       oTags = tags.get_tags()

it fails because it needs:

from django.models.blog import posts,tags

but I cannot put this in the model file or module import starts recursing (or so I believe). This kind of thing is just a pain and I don't want to spend my time working around architectural oddities. I tried creating a library module in the model directory but then it couldn't be found when I tried to import it (?). Python path weirdness, and django is fussy about the python path setup, more so than any other package I can think of, usually everthing is relative to a current directory and is happy.

So I gave Turbogears a look. I watched the wiki in 20 minutes film, seems ok but a lot to learn (like django), including it's own architectural oddities.

I installed it directly onto Windows which was very easy.

Next stumbling block is to get it to run under WingIDE. This was more fiddly. Like django it has autoreload but unlike django there is a config entry to disable it. Next problem is that it is multi-threaded which gives WingIDE a hard time. Here is my solution:

Add the following to the turbogears dev.cfg file:

   1  #
   2  # Run in a single thread
   3  #
   4  server.class="_cphttpserver.embedded_server"
   5  server.threadPool=1
   6  
   7  #
   8  # Disable auto reload.
   9  #
  10  autoreload.on=0

but this is not enough as it still runs two threads, one to watch the keyboard. Don't care about this if running under a debugger with a stop button: I can't stop it cleanly but this is not a big deal for development. Making it strictly single threaded involves overriding the _cpserver.Server class. I changed my turbogears xxx-start.py file as follows:

   1  cherrypy.root = Root()
   2  #cherrypy.server.start()
   3  
   4  import cherrypy._cpserver
   5  
   6  class NonThreadedServer( cherrypy._cpserver.Server):
   7      def start_http_server(self, blocking=True):
   8          """Start the requested HTTP server."""
   9          if self.httpserver is not None:
  10              msg = ("You seem to have an HTTP server still running."
  11                     "Please call server.stop_http_server() "
  12                     "before continuing.")
  13              warnings.warn(msg)
  14  
  15          if self.httpserverclass is None:
  16              return
  17  
  18          if cherrypy.config.get('server.socketPort'):
  19              host = cherrypy.config.get('server.socketHost')
  20              port = cherrypy.config.get('server.socketPort')
  21  
  22              cherrypy._cpserver.wait_for_free_port(host, port)
  23  
  24              if not host:
  25                  host = 'localhost'
  26              onWhat = "http://%s:%s/" % (host, port)
  27          else:
  28              onWhat = "socket file: %s" % cherrypy.config.get('server.socketFile')
  29  
  30          # Instantiate the server.
  31          self.httpserver = self.httpserverclass()
  32  
  33          # HTTP servers MUST be started in a new thread, so that the
  34          # main thread persists to receive KeyboardInterrupt's. This
  35          # wrapper traps an interrupt in the http server's main thread
  36          # and shutdowns CherryPy.
  37          def _start_http():
  38              try:
  39                  self.httpserver.start()
  40              except (KeyboardInterrupt, SystemExit), exc:
  41                  self.interrupt = exc
  42  #        threading.Thread(target=_start_http).start()
  43  
  44          #
  45          # Run in a single thread to make debugging a joy
  46          #
  47          self.state = cherrypy._cpserver.STARTED
  48          # Changing the socket timeout makes WingIDE more responsive
  49          self.httpserver.socket.settimeout( 0.1)
  50          self.httpserver.start()  # Should not return from here!!
  51  
  52          if blocking:
  53              self.wait_for_http_ready()
  54  
  55          cherrypy.log("Serving HTTP on %s" % onWhat, 'HTTP')
  56  
  57  # _cphttpserver uses this global variable so have to poke it
  58  cherrypy.server = NonThreadedServer()
  59  cherrypy.server.start()

I wasn't able to make django debuggable in WingIDE without hacking it's source: it didn't seem to have as many useful hooks as cherrypy/turbogears.

I've been quiet on the Django progress front. There has been a little but I got bogged down in unit testing, mainly learning the shortcomings of the python unit testing module.

Some django observations:

• can debug it very nicely in WingIDE using my trick.
• problem with the many-to-many relationship: if you try to set the members of a relationship using the set_XXX method (where XXX = name of class members you want to add) you may get an error if your models do not include a field call 'id' (lower case) due to nasty hard-wired property name in the code. I had one called Id and I had to change the capitalisation.
• the database api is a bit like the template system: I'd rather use something else but it'll do. I can mix in other api's if I want to get thinks done quick without wasting time single stepping through library code.

My latest Safari book is Python in a Nutshell a very comprehensive reference guide to python, plenty to learn. In some places, like the standard library section, it just summarises what the module does in a way that is not so enlightning but I have picked up some stuff in passing that I was not aware of before. The standard python documentation is very dry and it can be hard to find a useful descriptions. Python in a Nutshell does explain things in a more understandable way and it does include more example code. I may well keep the safari book in my bookshelf permanently (or buy it) to use in parallel with the python documentation.

I read the section on testing which described the doctest and unittest modules. I have tried docstring in the past, it involves putting test scripts in the docstrings of each function and to my mind this made the code bloated and hard to read.

I hadn't tried unittest before so I gave it a try with some django code, specificly code to convert a string of tag names into correct database entries. I started off doing it the Right Way by writing the tests first, before I started coding. It was a good exercise, just thinking of what I wanted and what could go wrong before I started to code.

The unittest module requires creating TestCase classes with three phases of testing:

• a setUp function to initialise things
• various tests in arbitrary order (because each test is a method and the methods are stored in a dictionary there is no guarantee on what order they will be executed)
• a tearDown function to close things up

In my case of manipulating the database I needed setup and teardown code for each individual test to ensure that the correct tags and tag->post mappings were created for each tag string under test. I could have created multiple TestCase classes, with seperate setUp and tearDown functions for each individual test but that seemed like too much typing. The problem here was really due to the arbitrary order in which the tests could be performed: each test cannot rely on what state the preceeding test left the database in so each test has to set the database up first.

As I slept last night I tossed and turned with this on my mind. Writing lots of setUp and tearDown code just to get around the arbitrary order. I could write just one big test function that went through a sequence of tests but it felt like cheating. To make this worse, I looked at the django tests and they used something similar to docstrings but in seperate files, a straight sequence of tests as if typed into the command prompt. It looked good and reading it is like reading example code. However, I have two problems with working this way:

• it needs more editing to create the script than unittest
• because all the code is inside a string, it is all one colour, I lose syntax highlighting. Do I need editor macro's to toggle the string delimiters on and off?

I think what I will do is use unittest without excessive setup and teardown code and just write long test functions. The aim of the exercise after all is to test, not to write test code.

Conclusion: testing: keep it simple and don't lose sleep over it.

Update: single stepping through the unittest code with WingIDE, it seems that the test case methods are sorted by name using the cmp function so they are probably alphabetic. Hence I could just name my functions test_001, test_002 etc or I write something to determine line numbers, or I define an array of test functions...

Been playing more with komodo, just exploring I haven't used it in anger (too many distractions, life getting in the way). I have been looking into setting up my Django project. Here is what I have been able to do so far:

• Create a project file containing all the python files: django itself, my application and the template files. Komodo is quite happy editing all types of files, even the html in the template files. I have yet to put the css and supporting graphic files into subversion but when I do I will be able to put the css file in the project as well: that's all source files for a web application, edited using the same tool, organised in one project.
• There is a nice 'Run' facillty to run command line programs from within komodo. It is just so easy to run a command and add it to the toolbox for your project so that the tool is there while working on the project. This means the commands to open an ssh tunnel to mysql, do a svn update on Django, check my own files into my subversion repository, clean up .pyc files etc are a double click away. While it is entirely possible to set this up in Vim, it's to fiddly to bother with. Komodo makes it so easy. Komodo personal has no support for source control but it is easy enough to set up commands for your subversion operations.
• You can add shortcuts to URLs to your project so while you are working on your web app you quickly see the appropriate page in your browser. Ok, you could put shortcuts in your browser but they would be there all the time, polluting your shortcut namespace, and you would have to explicity swap from your editor/IDE to the browser to get to the shortcuts. What I am trying to say is that what komodo is doing is not rocket science, is not impossible any other way (even in vim) it's just that it is done in a way that is very easy to set up.

There is just one thing that is annoying me about komodo. To run my django app I need to run the django-admin.py file. I have yet to find a way to tell komodo that I always want this file exected. By default it always executes the file in the current tab of the editor. In my case I will not be editing django-admin.py as it is part of django itself, but I have to have this file open in the editor and switch to it's tab before I can debug. I tried recording a macro to switch to django-admin.py and then run but this doesn't work. I tried defining a debug configuration but each file gets it's own set of debug configurations. It looks horribly like I have to set up debug configuration for every source file that is likely to be the current file when I want to debug.

Back to the good stuff, there is a useful mode for editing css files: you tell komodo which html file you want to see it with and komodo splits the screen to show a css editor and a web browser preview window. It is a very nice setup for working with css files apart from it not helping you to write your css.

So far I am finding it quite impressive.

I decided to persist with trying to get a decent django development going using either komodo or wingide. I'm really hankering for a windows based python IDE, the open source ones I have tried (spe, pythonwin, idle, boa constructor) just don't cut it for me.

I have spent some time with the trial versions of both with django and both had the same problem: I set a breakpoint in a view, I run the django_admin.py script with the runserver parameter, I go to my browser and load the right url but it doesn't stop at the breakpoint.

Well today I found the problem: django_admin.py actually forks another process to run django and the debuggers cannot handle this, they are working on the original process. The new process is forked as part of the auto-reload mechanism which will kill the forked process if any of the source files changes, allowing the original script to fork a new process using the new modules. Crude but effective.

It is possible to stop it doing this but only by editing the code. I changed django_src/django/core/management.py as follows:

Change this:

    from django.utils import autoreload
    autoreload.main(inner_run)

to this:

    inner_run()

and the problem disappears. How nice it would be to have a config setting to disable the auto-reload...

Now which IDE would I chose? Well based on the fact that I am only considering the personal versions (about a tenth of the cost of the professional versions) it goes hands-down to Komodo. Why?

• it can handle more languages than just python (php, perl etc)
• you can inspect variables in the debugger by putting the mouse over a symbol and looking at the tooltip
• there is an interactive python shell available when you stop at breakpoints: to me this is invaluble. Effectively wingide personal only gives you the ability to look at current variable values, you cannot change them or do command line experiments.
• komodo has a very nice regular expression tool

Wingide may have some plus points over komodo but for now komodo is the most useful tool and it is the one I would rather spend time in.

I will carry on with the trials for now, make the final decision when the trial periods are up.