Peter's Blog

• While the database schema is modified using 'migrations', Rails appears to create a file called /db/schema.rb that contains a definitive reference of the current database schema. I like this, I don't want to have to search through all the migration files or use phpmyadmin or whatever to remind me what columns are at my disposal.
• Disappointed to see that Komodo 4 only comes in 'Komodo IDE' ($245) and 'Komodo Edit' ($0) versions: the latter version is an editor, no debugging facilities, Regular Expression play area, source code control, integrated consoles etc. The cut-down 'personal' version has been discontinued. It may be viable to combine komodo edit with a standalone ruby debugger. I like Komodo's auto-complete features, especially in HTML which when combined with the integrated preview makes prettifying web pages more productive.
• To run Rails under the Komodo debugger you have to open the script/server file in the Rails app and run that. After about three minutes of nothing appearing to happen the script dies complaining that it cannot open the log/mongrel.log file. It's pathetically slow, unusable, forget it.
• Rails creates a README file within an application that says:

  Rails creates a README file within an application that says:

  Breakpoint support is available through the script/breakpointer
  client. This means that you can break out of execution at any
  point in the code, investigate and change the model, AND then
  resume execution! Example:
  
    class WeblogController < ActionController::Base
      def index
        @posts = Post.find(:all)
        breakpoint "Breaking out from the list"
      end
    end
  

  sound good, who needs a debugger. When I try this I get an error page dumped saying:
  sound good, who needs a debugger. When I try this I get an error page dumped saying:

  Breakpoints are not currently working with Ruby 1.8.5
  

  which is pretty poor, they let it out the door with known bugs? Things aren't looking good, looks like debugging will be down to printf's to the log file.
• Installing ruby-debug and trying that:
  1. run

     C:\InstantRails\rails_apps\depot>gem install \Documents\Downloads\ruby-debug-0.7
     .4-mswin32.gem
     Successfully installed ruby-debug, version 0.7.4
     Installing ri documentation for ruby-debug-0.7.4-mswin32...
     Installing RDoc documentation for ruby-debug-0.7.4-mswin32...
     

  2. add this to envoronment.rb

     if ENV['RAILS_ENV'] == 'development'
         require 'ruby-debug'
     
         SCRIPT_LINES__ = {}
         Debugger.start
     end
     

  3. add this to top of script that needs debugging:

     require 'ruby-debug;
     

  4. put this where you want to stop and debug

     debugger
     

     this gives you a crude command line debugger. It works, better than nothing. It's the equivalent of the pdb stuff in python but the ruby crowd are raving about it because the standard ruby debugger is very slow.
• Ok so I'm using a standard issue five year old work pc (1.1GHz Win2K) but ruby seems very slow, even compared to python. I'm writing this while a load of RDoc documentation is generated after installing the Radiant CMS using ruby gems. Wouldn't be so bad if I knew how to read the documents it was diligently formatting.
• Also found Mephisto Rails blog system which is reputedly better than typo but it runs very sluggishly on this pc and I don't see anything remarkable about it. It uses something called liquid for a template engine which may be it's killer feature (Raidant appears to use something home-grown).
• Many ruby developers seem to use trac: which is written in python of course. Oh the irony.

• Been progressing through the book. Very interesting so far. Ruby on Rails seems to rely a lot on conventions, i.e. you click on a link on the site (e.g. /blah/do_something), this invokes a controller with a particular name (e.g. blah) containing an action function (e.g. do_something) and rails assumes that you have a view in a file with the same name as the action (e.g. do_something.rhtml) which rails uses to render the view. It's neat, it saves some typing but there is a lot going on behind the scenes, other frameworks may make you specify file names which makes things easier to understand.
• The book goes into the use of Ajax and special effects like flashing text which is all supported in Rails. It's all very 2007 and Web 2.0. From having a site where clicking a button causes a new page to be loaded, adding ajax to modify the existing page looks very straightforward. The book gives hints on how to design your site such that people with javascript disabled can still use it.
• I like the use of 'partial's to define parts of the page so that, for example, I could define sidebar blocks in partials. A partial is to a page like a method is to an application, a reusable snippet.
• I am on the Komodo 4.0 trial and this has support for Ruby on Rails. The advantage of this over RadRails is that it is a more general purpose tool and is good for editing html, css, python et al. I like it for html and css editing since the code completion works nicely, I don't need to keep looking up keywords. Komodo is cheaper than Dreamweaver which I have tried and like but just cannot justify buying.
• Something useful to me would be to be able to edit one of the Rails partial files and to be able to preview it within komodo to make sure that it looks pretty, and to do this without having to run the whole application. If I edit the .rhtml file such as:

     1  <table>
     2      <tr>
     3          <th>boogie</th>
     4          <th>Twatto</th>
     5      </tr>
     6      <tr>
     7          <td><%= some :ruby, :stuff %></td>
     8          <td><%= some :ruby, :blah %></td>
     9      </tr>
    10  </table>
  

  then the preview works fine but what about when I use css? There doesn't seem to be a feature that says 'I want to use this css file when preview this file', it works the other way, I say 'when I preview this css file, use this other file'. However this does not work as my partial does not include the line to load the css file (that goes in the layout file) and komodo doesn't load it automatically. The workaround I have found for this is to add the line to the .rhtml file to load the css file:

     1  <link href="css-1.css" media="screen" rel="Stylesheet" type="text/css" />
     2  
     3  <table>
     4      <tr>
     5          <th>boogie</th>
     6          <th>Twatto</th>
     7      </tr>
     8      <tr>
     9          <td><%= some :ruby, :stuff %></td>
    10          <td><%= some :ruby, :blah %></td>
    11      </tr>
    12  </table>
  

  this works nicely. The line would be deleted once the css was perfected. Since Komodo supports code snippets it could be set up to add the link line very easily.
• The Komodo blurb mentions support for Ruby on Rails but nowhere can I find what this amounts to: can it launch a server for debugging? Is it any more than basic Ruby support? (UPDATE: found here, at the bottom, the Rails support is simply Rails running as a Ruby app, nothing special, no ability to launch a Rails server). RadRails definitely allows you to launch a rails server. It also supports rake, plugins and suchlike, it looks good but with some learning curve and mediocre documentation. RadRails doesn't do code completion and doesn't natively handle html, it would be necessary to go back to it's eclipse roots and install a plugin. RadRails has an overengineered feel that I don't like.

Used WingIDE some more. Some notes:

• hasn't showed any bugginess, it is stable.
• turns out a lot of it is written in python and it may even be used to develop itself, which would explain why it is so robust. I can't help thinking that the python could do with some more optimisation, given how slowly it runs on my work pc (Pii 450/256M ram).
• the stack dump which shows the values of variables while debugging is not able to show the values of properties that are, for example, determined by a call to __getattr__: it could be that WingIDE avoids calling the code it is debugging to try to show the values of these variables. This is a safe approach and may be why komodo struggled. However, sometimes it is annoying as these tricks are quite common. I tried installing the professional version and using the debug probe to introspect these objects and this works but the professional version is five times more expensive than the personal version. I found that it is possible to invoke pdb from within wing: add

  import pdb
  

  at the top of your module and

  pdb.set_trace()
  

  where you want your breakpoint. Ok it's hastle but 9/10 times examining the stack data is all you need, you don't always need the debug probe.
• it would be nice if code completion worked in the debug probe command line. Otherwise the facilities of the debug probe seem to be the same as pdb.
• The professional version also includes a 'module browser' which shows you the contents of all modules. However, it only works while your program is running and it only shows symbol names: it does not show docstrings or any useful information. By comparison, the object browser in komodo can be used while editing, komodo scans your modules and builds up a database.
• The professional version has a 'Source Assistant' which shows you the docstring for any symbol you click on in the editor.
• While using WingIDE I haven't felt the need to add any debug print statements: that is something!!!

Conclusion: the professional version is nice, not sure it is five times nicer than the personal version.

Oh, and I'd say komodo is out of it now, the question is WingIDE personal or Profesional?

Following on from my disillusionment with komodo I have been trying other options for a python debugger.

I looked at Ipython again and was playing with it and came across a problem/annoyance: the command history was not stored between sessions, even though the documentation implied that it was. Since this was handled by the readline library which is written in python I thought I would look through the source. I found a file there called wingdbstub which is the remote debug stub file used by wingide to debug remote applications. I considered this an endorsement so I fired up Wingide and gave it a try. Sure enough, I was able to run IPython and stop in the readline library in Wingide and look see what was happening. I must say that Wingide performed flawlessly while I was using it. It is not as pretty as komodo and doesn't do as much but I haven't found any actual bugs.

I was able to determine that IPython has two command historys, it's own (the %hist command) that is not saved across sessions and readlines which is accessed via ctrl-P and ctrl-N and is saved across sessions.

I spent some more time with Wingide and found it quite usable.

Then I decided to give winpdb a try. This is an open source python debugger that uses wxpython. It debugged my simple script that komodo failed on so I tried debugging IPython with it. It gave this a valiant go but when single stepping through IPython it has some strange problem where it decides to stop within a docstring at the head of ipython/__init__.py and after that runs instead of stepping any further.

Komodo makes a better attempt at debugging IPython but then I found another problem: if you stop in the debugger and double-click on an identifier in the source code, you can get python to crash and the following error to appear in the console output:

C:\Program Files\ActiveState Komodo
3.5\lib\support\dbgp\pythonlib\dbgp\client.py:1449:
RuntimeWarning: tp_compare didn't return -1 or -2 for exception
  i = max(0, len(stack) - 1)

the answer to this one may be to NOT double click on identifiers (doctor, doctor it hurts when I do this: well don't do it). Sometimes komodo will show values of variables in a tool-tip, sometimes it doesn't, sometimes I double click the variable in an effort to prompt it.

Right now, WingIDE is looking like the most stable product.

You have to worry when you have been using a tool for only a couple of days and you find bugs in the latest version.

If you single step through the following in komodo:

import BeautifulSoup

s = """\xe2"""

bs = BeautifulSoup.BeautifulSoup( s)

print bs

then when you get to the print statement the debugger gives up the ghost. It's stops showing local variables and it won't single step any further.

The problem appears to be something to do with the way BeautifulSoup overrides the __str__ function which the debugger is calling to display the value of bs. The strange character code generated by drupal date formatting triggers the problems.

wingide and the python debugger have no problems with this.

If I replace the odd character code with the correct html entity (as drupal should have in the first place) the komodo is happy:

import BeautifulSoup

s = """\xe2"""

bs = BeautifulSoup.BeautifulSoup( s.replace('\xe2', 'â'))

print bs

I think my final conclusion is that Komodo's debugger badly needs debugging itself. I'm not sure it is stable to a degree where I would want to spend good money on it. The whole point to me is to get a decent debugger, if I want to compromise I can stick with pdb.

Update: wingide finds an exception within BeautifulSoup which somewhere gets ignored and life goes on. Wingide seems generally better in when you single step, the step finishes and you are on the next line. Komodo has a tendency to go and not come back.

Been playing more with komodo, just exploring I haven't used it in anger (too many distractions, life getting in the way). I have been looking into setting up my Django project. Here is what I have been able to do so far:

• Create a project file containing all the python files: django itself, my application and the template files. Komodo is quite happy editing all types of files, even the html in the template files. I have yet to put the css and supporting graphic files into subversion but when I do I will be able to put the css file in the project as well: that's all source files for a web application, edited using the same tool, organised in one project.
• There is a nice 'Run' facillty to run command line programs from within komodo. It is just so easy to run a command and add it to the toolbox for your project so that the tool is there while working on the project. This means the commands to open an ssh tunnel to mysql, do a svn update on Django, check my own files into my subversion repository, clean up .pyc files etc are a double click away. While it is entirely possible to set this up in Vim, it's to fiddly to bother with. Komodo makes it so easy. Komodo personal has no support for source control but it is easy enough to set up commands for your subversion operations.
• You can add shortcuts to URLs to your project so while you are working on your web app you quickly see the appropriate page in your browser. Ok, you could put shortcuts in your browser but they would be there all the time, polluting your shortcut namespace, and you would have to explicity swap from your editor/IDE to the browser to get to the shortcuts. What I am trying to say is that what komodo is doing is not rocket science, is not impossible any other way (even in vim) it's just that it is done in a way that is very easy to set up.

There is just one thing that is annoying me about komodo. To run my django app I need to run the django-admin.py file. I have yet to find a way to tell komodo that I always want this file exected. By default it always executes the file in the current tab of the editor. In my case I will not be editing django-admin.py as it is part of django itself, but I have to have this file open in the editor and switch to it's tab before I can debug. I tried recording a macro to switch to django-admin.py and then run but this doesn't work. I tried defining a debug configuration but each file gets it's own set of debug configurations. It looks horribly like I have to set up debug configuration for every source file that is likely to be the current file when I want to debug.

Back to the good stuff, there is a useful mode for editing css files: you tell komodo which html file you want to see it with and komodo splits the screen to show a css editor and a web browser preview window. It is a very nice setup for working with css files apart from it not helping you to write your css.

So far I am finding it quite impressive.

I decided to persist with trying to get a decent django development going using either komodo or wingide. I'm really hankering for a windows based python IDE, the open source ones I have tried (spe, pythonwin, idle, boa constructor) just don't cut it for me.

I have spent some time with the trial versions of both with django and both had the same problem: I set a breakpoint in a view, I run the django_admin.py script with the runserver parameter, I go to my browser and load the right url but it doesn't stop at the breakpoint.

Well today I found the problem: django_admin.py actually forks another process to run django and the debuggers cannot handle this, they are working on the original process. The new process is forked as part of the auto-reload mechanism which will kill the forked process if any of the source files changes, allowing the original script to fork a new process using the new modules. Crude but effective.

It is possible to stop it doing this but only by editing the code. I changed django_src/django/core/management.py as follows:

Change this:

    from django.utils import autoreload
    autoreload.main(inner_run)

to this:

    inner_run()

and the problem disappears. How nice it would be to have a config setting to disable the auto-reload...

Now which IDE would I chose? Well based on the fact that I am only considering the personal versions (about a tenth of the cost of the professional versions) it goes hands-down to Komodo. Why?

• it can handle more languages than just python (php, perl etc)
• you can inspect variables in the debugger by putting the mouse over a symbol and looking at the tooltip
• there is an interactive python shell available when you stop at breakpoints: to me this is invaluble. Effectively wingide personal only gives you the ability to look at current variable values, you cannot change them or do command line experiments.
• komodo has a very nice regular expression tool

Wingide may have some plus points over komodo but for now komodo is the most useful tool and it is the one I would rather spend time in.

I will carry on with the trials for now, make the final decision when the trial periods are up.

Still hunting for a decent python debugging environment I have convinced myself that Komodo cannot properly debug code that is exec'ed or eval'ed: it says so in the release notes. I tried resorting to IDLE for a while but for some bizarre reason it would not stop at breakpoints so I gave up on that as well.

So I'm back to my vim python debug setup. It seems nothing has changed in the art since I developed that back in April.

So I was working on an application with a GUI that did a lot of processing, an awkward thing to test and debug: I have to enter the test parameters into the GUI every time and start the processing. I realise then that I should just rip out the processing, seperate it from the user interface and call it from a test harness. Voila, it's no longer a GUI app and it isn't in an exec'ed module, it is a simple script file. Vast scope for debugging.

Then I realise that I can do something similar to test and debug django. I looked at the standard view code, the code that converts an url into html. One of my view methods looks like this:

def index(request):
    oBlogList = posts.get_list(order_by=['-Created'], limit=5)
    AddTags( oBlogList)
    AddCommentCount( oBlogList)

    return render_to_response('blog/posts', {'posts_title': 'Index',
                                                'blog_list': oBlogList,
                                                'root': g_strRoot
                                                })

On a hunch I tried doing this interactively in a console and examining the object returned by render_to_response. This turned out to be an object that had a member called 'content' holding the html for the page.

The upshot of all this is that again I can write simple tests to generate the html without worrying about an http server. I can either compare the html generated against what I want or use this trick to stick the html straight into a browser. I no longer have to worry about the server and any weird tricks it may be playing to confuse and confound debuggers.

It remains to be seen how much can be developed like this, how to handle forms, sessions, workflows etc but for now the moral is: divide and conquer.

On a sudden whim I decided to try developing my new django blog on my windows laptop then uploading it to the server when it is working. The setup is slightly complex and I have only just set it up so lets try to describe it in detail.

• I already had the source in subversion so I installed tortoise svn a nice Windows svn client so I could access the source from windows, upload it into subversion when it was tested and read it back into the working image on the server.
• Also using Tortoise I extracted the latest django source onto the laptop.
• I set up the windows environment variables according to the django installation instructions.
• Because my templates directories are different under windows and ubuntu I put a conditional section in my settings module:

     1  import os
     2  
     3  if os.name != 'nt':
     4      TEMPLATE_DIRS = (
     5          # Put strings here, like "/home/html/django_templates".
     6          '/usr/local/lib/Django/django_templates',
     7      )
     8  else:
     9      TEMPLATE_DIRS = (
    10          # Put strings here, like "/home/html/django_templates".
    11          'c:\\python24\\lib\\site-packages\\Django\\django_templates',
    12      )
  

• I installed the mysql client on the laptop, along with the appropriate version of MySQLdb
• I didn't want to run the database server on my laptop so I use an ssh tunnel to connect to the server: I have to run this in a command console while I am accessing the database:

  ssh -L 3306:127.0.0.1:3306 me@myserver.org
  

  For security the mysql server does not accept connections from outside the box it is running on, except via ssh. I am using the live database on the server and for development. I could create another instance of the database for testing.
• To get to files needed by the theme (e.g. style.css) I had to tweek my templates under windows such that they were loaded from the server using the full absolute url rather than a relative url. I didn't want to make django server them or put apache on my laptop.
• django_admin.py just worked, no problems.

Why did I do all this? Well I'm thinking again of using komodo as a development environment with a proper debugger. I don't have a lot of time to devote to all this, every minute is precious, so it will help to have productive tools available.