Peter's Blog

Attempted to upgrade to latest Wing IDE (2.1.3) which was a big mistake: it won't run scripts and the python shell will not work, says "waiting for shell to restart".

• Tried upgrading from python2.4 to python2.5, no joy
• Tried googling, fixes mentioned don't work. No patch for this problem on ftp site.
• Tried reinstalling WingIDE, no joy
• Tried setting project settings to point to python.exe
• Tried steps in Wing IDE manual "1.14.2.0. Failure to Start Debug", no joy. "Source / Show Analysis Stats" shows no problems. PYTHONPATH blank.
• Ran filemon, told me wing launches python ok, then writes to an error log. Inspected that, it shows command used to launch python. Trying running that command explicitly, ran ok. Python is happy and healthy.
• Uninstalled python 2.4 and python 2.5, cleaned directories, cleaned registry, reinstalled 2.5, no joy.
• Deleted user specific settings and managed to zap the licence key. Now wingide won't accept the licence key, says I have activated it too many times. I'm supposed to email sales to sort it out.

Wasted enough time on this now so it's back to VIM.

If I wanted a fix next week I could try the wingide mail group but I can't be bothered, I'm bitter and disillusioned but glad I didn't fork out for the professional version.

Update: by a strange coincidence, a new beta version of komodo is out. Time to re-evaluate.

WingIde 2.1 beta is out and... it has vi keystrokes! python ide, vi keystrokes, it just doesn't get any better.

Since I have been using Wing IDE I have not looked back. Annoyances:

• Even on a >1Ghz processor it feels sluggish
• It has an indentation analysis thing that examines the indentation in a file. You copy some dubiously indented code from somewhere and paste it into your beautifully crafted code and it decides to reindent your code with tabstop of 8 or something daft.
• ugly gtk user interface
• when editing anything other than python the tab key does nothing and I have to press ctrl-shift-dot to indent. Still, with vi keystrokes >> seems to work.

But for python development, even django and turbogears I wouldn't be without it.

Coding away with TurboGears and found that glorious nirvava where whatever I tried Just Worked. Kid, SQLObjects and cherrypy are nice and clean and turbogears has generated a nice boilerplate framework. Going through the code of the various librarys while debugging, there is magic in the way they are working but it is not causing me big troubles as the api's they present are nice and clean. Python is such a lovely language I always felt it would be great for web development and now it feels like I have the tools.

I commented previously on SQLObjects support for introspection and I was wrong here as the help command (possibly docstrings in general?) is broken and generates an exception but the online documents are fairly comprehensive.

I have a heavy cold and hence had one of those tortured nights sleep with recurring dreams. I kept seeing Kid's XML element tree sitting in memory, branches being swapped in and out and the resulting document changing in wonderful ways. It was not a bad dream. I woke up thinking it was a great way to manipulate a document compared to primitive string substitution. I have never really tried DOM models before, being reluctant to use all that memory, but now that memory is cheap I shouldn't be so cautious. Playing with SAX parsers is a pain, if I have to write another state machine I'll scream.

Another discovery, the TurboGears command `tg-admin shell` somehow picked up IPython and I was able to play with the database api whil admiring the pretty colours.

It was nice until I tried porting a theme from php and ran into an annoying works-in-ie-but-not-firefox problem. There are some things that even the nicest development environments cannot help with. I tried using html-kit to clean up the template code and it is indeed easy to preview kid templates in a browser, they Just Work.

Tip for the Day: although kid templates are xml it is better to tell WingIDE they are html as it avoids a limitation of the xml syntax highlighting where it treats Processing Instructions as errors and marks most of the file as a syntax error, making the editor run pretty sluggishly.

I've been giving TurboGears a try as a means of reimplementing this blog in python. I have been trying django but I felt the need to try something else, mainly prompted by frustrations with the database api.

TurboGears is similar in the tools that it provides for web application development. It is essentially an amalgum of a number of existing python projects:

kid

a templating system based on XML.

cherrypy

web application framework, essentially maps urls to python method calls

SQLObject

an SQL database wrapper

Mochikit

a javascript library which I have tried before

Turbogears itself provides 'glue' to put these together. It has tools for creating project files, setting up and using the database etc. The TurboGears administration script creates a boilerplate application that can be running in no time. It all feels very similar to django, the main thing missing I can see is the administration screens but they are working on something called 'catwalk' which I think will do this. At the end of the day there are many database front ends that can provide administration (phpmyadmin, webmin mysql module etc) which are not so end-user friendly but good enough for me.

The librarys that turbogears has chosen appear to be very good in their own rigght. Their designs are clean and they are well documented. Each has greater depth than the django equivalents.

Kid looks powerful: it is an intersting mix of xml and python: your kid scripts can be used like python modules: they compile to .pyc files and so are presumably only parsed once. You can define xml snippets of boilerplate code, such as an html list, and call it like a function from elsewhere in your template. As it is XML based, if your template is not XML compliant the XML parser (expat at the lowest level) slaps your wrist. An interesting side effect of the pythonic nature of it is that if a problem appears in the python embedded in your template, Wing IDE's debugger will stop on that line. Useful! Kid does not support inheritance like django's template system, it effectively gives you macro templates that could be very powerful. The syntax is clearer than django, you can use python code inline )like cheetah) and the template substitution allows you to use the same substitution twice (if, for example, you want to use the same title string in two places, header and page title): a pedantic limitation in django. Looking through the documentation, Kid shows great attention to xhtml compliance and generation of conformant XHTML. This is not the greatest concern to me, I don't lose sleep over whether my XHTML is standards compliant, I am more worried about whether browsers will display it properly, but it is nice to know that someone somewhere has thought about all this for me.

SQLObject looks very clean and the objects can be introspected nicely. You work with the classes you define directly and not with meta classes that appear from nowhere. SQLObject has also introduced me to SQLite a really nice, simple SQL database library. This looks very solid, certainly good enough for development without the hastle of getting mysql running. It would probably be good enough for production use for me, if I can get it to run SQL dumps for backup (which I prefer to backing up binarys). Certainly I will be using it in the places where I can't be bothered to set up mysql (user's, permissions, passwords, yawn).

CherryPy: TurboGears creates enough boilerplate code that I haven't needed to look at the documents for this yet. It maps urls to method calls simply enough but one thing I like about django is the regular expression based url mapping which is totally flexible but there is nothing to stop me adding such a mapping layer.

I've ported the basics of my blog (the theme I use in drupal, lists of posts etc) to TurboGears. I had to study the Kid documentation to get the TurboGears generated code to use the head from the master.kid file so I could put my css import in one fixed place. By default the head here is ignored. The solution to this conundrum is to modify master.kid like this:

<head py:match="item.tag=='{http://www.w3.org/1999/xhtml}head'">
    <meta content="text/html; charset=UTF-8" http-equiv="content-type" py:replace="''"/>
    <title>Peter's Better Blog | ${strPageTitle}</title>
    <link rel="stylesheet" href="/static/css/style.css" />
</head>

i.e. turn it into a match template. I had the django inheritance thing in my head (not a million miles away from the drupal phptemplate engine) and I had to read the kid documentation to realise it simply works by sequentially replacing blocks of XML, in this case any later head block is replaced by the block above.

Used WingIDE some more. Some notes:

• hasn't showed any bugginess, it is stable.
• turns out a lot of it is written in python and it may even be used to develop itself, which would explain why it is so robust. I can't help thinking that the python could do with some more optimisation, given how slowly it runs on my work pc (Pii 450/256M ram).
• the stack dump which shows the values of variables while debugging is not able to show the values of properties that are, for example, determined by a call to __getattr__: it could be that WingIDE avoids calling the code it is debugging to try to show the values of these variables. This is a safe approach and may be why komodo struggled. However, sometimes it is annoying as these tricks are quite common. I tried installing the professional version and using the debug probe to introspect these objects and this works but the professional version is five times more expensive than the personal version. I found that it is possible to invoke pdb from within wing: add

  import pdb
  

  at the top of your module and

  pdb.set_trace()
  

  where you want your breakpoint. Ok it's hastle but 9/10 times examining the stack data is all you need, you don't always need the debug probe.
• it would be nice if code completion worked in the debug probe command line. Otherwise the facilities of the debug probe seem to be the same as pdb.
• The professional version also includes a 'module browser' which shows you the contents of all modules. However, it only works while your program is running and it only shows symbol names: it does not show docstrings or any useful information. By comparison, the object browser in komodo can be used while editing, komodo scans your modules and builds up a database.
• The professional version has a 'Source Assistant' which shows you the docstring for any symbol you click on in the editor.
• While using WingIDE I haven't felt the need to add any debug print statements: that is something!!!

Conclusion: the professional version is nice, not sure it is five times nicer than the personal version.

Oh, and I'd say komodo is out of it now, the question is WingIDE personal or Profesional?

Following on from my disillusionment with komodo I have been trying other options for a python debugger.

I looked at Ipython again and was playing with it and came across a problem/annoyance: the command history was not stored between sessions, even though the documentation implied that it was. Since this was handled by the readline library which is written in python I thought I would look through the source. I found a file there called wingdbstub which is the remote debug stub file used by wingide to debug remote applications. I considered this an endorsement so I fired up Wingide and gave it a try. Sure enough, I was able to run IPython and stop in the readline library in Wingide and look see what was happening. I must say that Wingide performed flawlessly while I was using it. It is not as pretty as komodo and doesn't do as much but I haven't found any actual bugs.

I was able to determine that IPython has two command historys, it's own (the %hist command) that is not saved across sessions and readlines which is accessed via ctrl-P and ctrl-N and is saved across sessions.

I spent some more time with Wingide and found it quite usable.

Then I decided to give winpdb a try. This is an open source python debugger that uses wxpython. It debugged my simple script that komodo failed on so I tried debugging IPython with it. It gave this a valiant go but when single stepping through IPython it has some strange problem where it decides to stop within a docstring at the head of ipython/__init__.py and after that runs instead of stepping any further.

Komodo makes a better attempt at debugging IPython but then I found another problem: if you stop in the debugger and double-click on an identifier in the source code, you can get python to crash and the following error to appear in the console output:

C:\Program Files\ActiveState Komodo
3.5\lib\support\dbgp\pythonlib\dbgp\client.py:1449:
RuntimeWarning: tp_compare didn't return -1 or -2 for exception
  i = max(0, len(stack) - 1)

the answer to this one may be to NOT double click on identifiers (doctor, doctor it hurts when I do this: well don't do it). Sometimes komodo will show values of variables in a tool-tip, sometimes it doesn't, sometimes I double click the variable in an effort to prompt it.

Right now, WingIDE is looking like the most stable product.

You have to worry when you have been using a tool for only a couple of days and you find bugs in the latest version.

If you single step through the following in komodo:

import BeautifulSoup

s = """\xe2"""

bs = BeautifulSoup.BeautifulSoup( s)

print bs

then when you get to the print statement the debugger gives up the ghost. It's stops showing local variables and it won't single step any further.

The problem appears to be something to do with the way BeautifulSoup overrides the __str__ function which the debugger is calling to display the value of bs. The strange character code generated by drupal date formatting triggers the problems.

wingide and the python debugger have no problems with this.

If I replace the odd character code with the correct html entity (as drupal should have in the first place) the komodo is happy:

import BeautifulSoup

s = """\xe2"""

bs = BeautifulSoup.BeautifulSoup( s.replace('\xe2', 'â'))

print bs

I think my final conclusion is that Komodo's debugger badly needs debugging itself. I'm not sure it is stable to a degree where I would want to spend good money on it. The whole point to me is to get a decent debugger, if I want to compromise I can stick with pdb.

Update: wingide finds an exception within BeautifulSoup which somewhere gets ignored and life goes on. Wingide seems generally better in when you single step, the step finishes and you are on the next line. Komodo has a tendency to go and not come back.

I decided to persist with trying to get a decent django development going using either komodo or wingide. I'm really hankering for a windows based python IDE, the open source ones I have tried (spe, pythonwin, idle, boa constructor) just don't cut it for me.

I have spent some time with the trial versions of both with django and both had the same problem: I set a breakpoint in a view, I run the django_admin.py script with the runserver parameter, I go to my browser and load the right url but it doesn't stop at the breakpoint.

Well today I found the problem: django_admin.py actually forks another process to run django and the debuggers cannot handle this, they are working on the original process. The new process is forked as part of the auto-reload mechanism which will kill the forked process if any of the source files changes, allowing the original script to fork a new process using the new modules. Crude but effective.

It is possible to stop it doing this but only by editing the code. I changed django_src/django/core/management.py as follows:

Change this:

    from django.utils import autoreload
    autoreload.main(inner_run)

to this:

    inner_run()

and the problem disappears. How nice it would be to have a config setting to disable the auto-reload...

Now which IDE would I chose? Well based on the fact that I am only considering the personal versions (about a tenth of the cost of the professional versions) it goes hands-down to Komodo. Why?

• it can handle more languages than just python (php, perl etc)
• you can inspect variables in the debugger by putting the mouse over a symbol and looking at the tooltip
• there is an interactive python shell available when you stop at breakpoints: to me this is invaluble. Effectively wingide personal only gives you the ability to look at current variable values, you cannot change them or do command line experiments.
• komodo has a very nice regular expression tool

Wingide may have some plus points over komodo but for now komodo is the most useful tool and it is the one I would rather spend time in.

I will carry on with the trials for now, make the final decision when the trial periods are up.

Following my thoughts yesterday, here are some VIM python scripts to add python breakpoint and debugging features to VIM. With this set up the F7 key will set a breakpoint on a line of code, Shift-F7 will remove all breakpoints and Shift-F12 will execute a script in the python debugger. This only runs on windows as far as I know, because it uses the 'start' command to launch the debugger in a seperate process without VIM waiting for it to finish. This allows you to look through the source code (and fix it) while the debugging is still in progress.

This goes in a python block in _vimrc:

   1  def SetBreakpoint():
   2      import re
   3  
   4      nLine = int( vim.eval( 'line(".")'))
   5  
   6      strLine = vim.current.line
   7      strWhite = re.search( '^(\s*)', strLine).group(1)
   8  
   9      vim.current.buffer.append(
  10         "%(space)spdb.set_trace() %(mark)s Breakpoint %(mark)s" %
  11           {'space':strWhite, 'mark': '#' * 30}, nLine - 1)
  12  
  13      for strLine in vim.current.buffer:
  14          if strLine == "import pdb":
  15              break
  16      else:
  17          vim.current.buffer.append( 'import pdb', 0)
  18          vim.command( 'normal j1')
  19  
  20  vim.command( 'map <f7> :py SetBreakpoint()<cr>')
  21  
  22  def RemoveBreakpoints():
  23      import re
  24  
  25      nCurrentLine = int( vim.eval( 'line(".")'))
  26  
  27      nLines = []
  28      nLine = 1
  29      for strLine in vim.current.buffer:
  30          if strLine == 'import pdb' or strLine.lstrip()[:15] == 'pdb.set_trace()':
  31              nLines.append( nLine)
  32          nLine += 1
  33  
  34      nLines.reverse()
  35  
  36      for nLine in nLines:
  37          vim.command( 'normal %dG' % nLine)
  38          vim.command( 'normal dd')
  39          if nLine < nCurrentLine:
  40              nCurrentLine -= 1
  41  
  42      vim.command( 'normal %dG' % nCurrentLine)
  43  
  44  vim.command( 'map <s-f7> :py RemoveBreakpoints()<cr>')
  45  
  46  def RunDebugger():
  47      vim.command( 'wall')
  48      strFile = vim.eval( "g:mainfile")
  49      vim.command( "!start python -m pdb %s" % strFile)
  50  
  51  vim.command( 'map <s-f12> :py RunDebugger()<cr>')
Toggle Line Numbers

This relies on using the runscript plugin, and modifying the function 'SetMainScript' as follows:

function s:SetMainScript()
  let s:mainfile = bufname('%')
  let g:mainfile = bufname('%')
  echo s:mainfile . ' set as the starting program.'
endfunction

This allows F11 to be used to select which file is executed to start debugging. F12 can still be used to launch the script outside of the debugger.

Now I don't need WingIDE or a pc upgrade.

Further to my search for a python debugger I gave Stani's Python Editor a try. Observations:

• ran windows installer but it didn't create start menu entrys. To get it to run I had to:
  • rename c:\python24\scripts\spe to c:\python24\scripts\spe.py
  • run spe.py
• it started up running extremelt slowly: mouse moved in 2 inch steps. Don't know what it was up to.
• opened a python file and it was happy, ran at full speed.
• tried making the pane at the bottom of the screen smaller and it didn't seem to resize the source window, ugly space was left around it.
• ran python source checker and nothing happened. Later on, when I ran the program, the results of the source check appeared.
• directory browser is rooted at c:/python24 and doesn't appear to let me look elsewhere.
• it doesn't have a debugger (to me the whole point).
• runs at full speed (unlike WingIDE).
• it does seem to have lots of features, not sure they are all useful to me.

Conclusion: interesting work in progress but I don't need an editor, I want a debugger. I'm not criticising it for not being a debugger, I read it was an IDE so I asssumed it had a debugger. They appear to be planning to add one, maybe I should come back later.

I found a very good intro to using Python's built in debugger here. If I list my essential debugging features as:

• stop on exception for introspection
• set breakpoint in source code

Then I can easily enough define VIM macros to do it.

stop on exception for introspection

• launch script via the python debugger

  python -m pdb myscript.py
  

set breakpoint in source code

• Add import pdb to start of file, add pdb.set_trace() where I want my breakpoint.

There is this but it doesn't work on windows.

I'm still on the WingIDE trial and I am still using it. It is a big help using a debugger instead of print statements. Thoughts/observations:

• it still runs like treacle.
• when I edit a file in VIM and write it to disk, WingIDE immediately asks if I want to reload it. Other IDEs I use take the less annoying approach of waiting until I switch back to the IDE.
• still not crashed or shown any severe bugginess.
• in a language like python I find it strange that WingIDE does not allow me to 'edit and continue' as Visual C++ and Visual Basic 6 have been doing for about 8 years. Whenever I edit anything I have to stop and restart. If it can reload modules then I haven't found out how. To make this even more irritating, Wing always stops on the first line of the first file when the program restarts so I have to press the go button. Given the slowness Wing, I end up waiting for this halt so I am ready to press the button and get going again.
• I had a quick look a Komodo but, as I suspected, I would have to use the ActiveState python distribution. I stopped using that years ago as it's release cycle lagged behind 'real' python's and 'real' python was 100% satisfying. ActiveState spent years dicking around trying to make perl work with COM, making it difficult to embed in anything else in the process. I'm not a total fan of ActiveState.

I've been trying Wing IDE a development environment for Python. I've tried some in the past and always fell back to using VIM and print statements:

IDLE

cannot debug GUI applications (IDLEfork might, haven't tried it)

PythonWin

cannot debug GUI applications, doesn't appear to have evolved in the last few years

Boa Constructor

haven't tried it for a while, back then it was flaky and didn't work with the contemporary wxPython (which is a moving target anyway). Recent reports say the debugger is still flaky.

pdb

python's own debugger, command line based

I was interested in Wing IDE (WingIDE?) as it would be nice to have a decent debugger. Simple things like:

• double click exception stack dump to take me to point of exception in source
• breakpoints (!)
• single step

I tried the cheap 'personal' edition and here's a potted review:

• on pentium 2 450M with 256M ram on windows XP it is very sluggish. It uses 40M of memory so it is probably swapping. On 1.5G Centrino with 512M it is much more useable. Single steps still take a second or so each (like walking through mud). When it runs the target the target runs at full speed. The help file makes mention of it being slow when it is loading symbols on startup but the slowness I see effects it all the time.
• editor is ok in a Visual Studio kind of way. It's not VIM but I can still use VIM in the same way that I still use it with Visual Studio. Write code in VIM, correct it in IDE.
• editor has auto-completion (fills in the rest of an identifier for you) but it's kinda slow to come up and VIM has a way to do this that is simplistic but useful (CTRL-N for the uninitiated).
• first project I ran generated an exception in the startup code, something mysterious in Swig generated code. Wing is supposed to intelligently tell the difference between exceptions that you trap yourself in your code and ones that you don't trap: presumably it only drops into the debugger on an unhandled exception. This particular exception was in the Swig stuff so it may have been under some C code, in which case the exception always stops in the debugger (in Wing terminology this is a 'false positive'). However, Wing has a useful option to learn which exceptions you want to always ignore so I set this up and was not bothered by it again. The details of the exception were presumably stored in the project file.
• IDE and debugger works as you would expect from standards set by Turbo C 15 years ago.
• in the personal edition, when the debugger comes in you cannot evaluate expressions or examine variables, you can only look at the values of variables in the stack trace. There is a python shell but it is independent from the script you are debugging so it is only useful for experiments.
• from visual studio I am used to hovering the cursor over a variable name and seeing it's value in a tool tip, or being able to right click and see it's value. Wing does not do this, it seem you have to look through the stack trace to find the value of variable.
• coolest feature is remote debugging. You can edit a script to add a line to import a Wing module and from then on, if the script raises an exception it will find the Wing IDE and you can start debugging. This is useful if, for example, you are debugging CGI scripts that are launched from Apache: the scripts can find Wing IDE and you can debug them, they don't have to be launched from Wing IDE itself.
• it works fine debugging wxPython and has been solid so far.
• I left it running a program for an hour. The program generated lots of print output which was caught and displayed in the debugger. After an hour the debugger was using over 100M of ram: it doesn't seem to want to limit the size of the debug buffer. Maybe there is an option to stop this but it's odd that it should do this by default.

Conclusion: I can live with it's lethargy just to have a useable debugger. I'm still on the free trial, time will tell if I want to shell out $35 for the personal edition (less than £20, not bad).

There are other open source alternatives I haven't looked at:

spe: looks like work in progress

Eclipse: written in java, mainly for java