Peter's Blog

Going through django tutorial for a second time and was disheartened to get a weird error when trying to log in as admin:

pcw@intranet:~/django_src$ ~/django_admin.py runserver
Validating models...
0 errors found.

Starting server on port 8000 with settings module 'djangotutorial.settings'.
Go to http://127.0.0.1:8000/ for Django.
Quit the server with CONTROL-C (Unix) or CTRL-BREAK (Windows).
[01/Nov/2005 07:15:52] "GET /admin/ HTTP/1.1" 200 1615
Traceback (most recent call last):
  File "/home/pcw/django_admin.py", line 131, in ?
    main()
  File "/home/pcw/django_admin.py", line 108, in main
    ACTION_MAPPING[action](addr, port)
  File "/usr/lib/python2.4/site-packages/django/core/management.py",
    line 656, in runserver
    autoreload.main(inner_run)
  File "/usr/lib/python2.4/site-packages/django/utils/autoreload.py",
    line 71, in main
    reloader_thread()
  File "/usr/lib/python2.4/site-packages/django/utils/autoreload.py",
    line 44, in reloader_thread
    mtime = os.stat(filename).st_mtime
OSError: [Errno 2] No such file or directory:
'/usr/lib/python2.4/site-packages/django/templatetags/adminmedia.py'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.4/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.4/threading.py", line 636, in __exitfunc
    self._Thread__delete()
  File "/usr/lib/python2.4/threading.py", line 522, in __delete
    del _active[_get_ident()]
KeyError: -1209573248
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.4/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.4/threading.py", line 636, in __exitfunc
    self._Thread__delete()
  File "/usr/lib/python2.4/threading.py", line 522, in __delete
    del _active[_get_ident()]
KeyError: -1209573248
Unhandled exception in thread started by
Error in sys.excepthook:

Original exception was:

ugly, huh? Looks like three stack traces, two of which are the same. Where to start. Well I think this line is the clue:

OSError: [Errno 2] No such file or directory:
'/usr/lib/python2.4/site-packages/django/templatetags/adminmedia.py'

Looking in /usr/lib/python2.4/site-packages/django/templatetags/ I see adminmedia.pyc but no adminmedia.py. I think what has happened is that updating django by doing

svn update

does not clear out the python .pyc files that are generated automaticaly by python. The django autoreload code is seeing the .pyc files and looking for the corresponding .py files. I deleted the .pyc files and life went on.

Gave someone a pc with a spanking new Windows 98 install on it and the first thing they did was install software on it and broke it. After a reboot Explorer gave a page fault in explorer.exe and the system was totally useless. This even happened in safe mode.

I suspected that the software installed much more recent versions of the system files and broke the otherwise 1998 vintage installation. I did a google and found q296211 which looked like a solution.

I ran through the procedure, using a dos boot disk to extract system files from CAB files until I came to comctl32.dll when the extract utility kept running out of memory. I tried installing himem.sys to no avail.

So I tried knoppix. It wouldn't boot on the pc in question so I had to whip the hard drive out and put it in another pc that I knew ran knoppix. Once in there I had to edit /etc/fstab to add rw access to the drive and then mount it. I then used the cabextract utility that comes as standard with knoppix:

cabextract base4.cab --filter comctl32.dll

I repeated for all the files mentioned in the article as I didn't want to mess about too much.

Fortunately the hard disk contained a copy of the windows 98 setup files, including all the cabs, otherwise I would have had a problem with knoppix occupying the cd drive.

Anyway I unmounted the drive, put it in it's box and problem solved. Windows 98 popped up a dialog to say someone had been tampering with comctl32.dll but when I rebooted and it didn't complain again.

Conclusion: knoppix is better than a dos prompt for fixing windows.

Had a Windows 98 PC that would not validate with the domain controller. Could not determine whether the problem was networking or security. However, the security policy was set to only allow the pc to be used if it validated on the network. Once it had failed there was no way to test whether the networking was up.

To get around this I went into safe mode and searched the registry for 'validated'. I found a key called

HKLM/Network/Logon/MustBeValidated

I set this to 0 and rebooted and I was able to use the computer even after validation failed.

Conclusion: windows 98 security is more of an inconvenience.

I've been playing with django recently. Django is a web application framework written in python. It means I can rewrite this website in python rather than use php: there are things that I would like to do that I cannot be bothered to do in php (or perl or C or assembly language), in python they would be fun.

I went through the django tutorial once and then I went through it again, using it as guidance in starting my new application. I have got it to a pretty basic point where it lists postings, posting can be clicked on for details and comments, and tags are supported already. It is not ready to go online by any means but I am happy with the progress and how django has helped me in a few hours.

Anyway, django pros:

• very good documentation I'd say. Lots of background descriptions and examples, not just bare api descriptions. Didn't have to read the source code (much).
• I like the system for mapping an url to a view using regular expressions. This is so flexible and powerful, it sold me compared to the cherrypy/turbogears way of doing things.
• The generic admin interface is handy. It essentially writes data-entry forms for you, although like any automatic form generation they are not ideal. They are more convenient than just using the webmin mysql module to edit the raw tables.
• The online documentation mentions automatic document generation. By reading the source I found this was at localhost:8000/admin/doc

Django Cons:

• The process of creating a website is kinda complex, it is probably best to be guided by the tutorial each time.
• I don't like a directory structure being imposed on me, especially a complex one.
• I don't like the way it uses environment variables to tell it which project it is dealing with. Seems fiddy if you are dealing with multiple sites.
• I think what I'm leading up to is why isn't it simply based on a script that imports the django modules, imports a personal config module and then goes on from there? Like every other python app I can think of?
• The way that the database schema is defined in python objects and these then generate the corresponding schema in the database is nice but it is very annoying that it doesn't support modification of the model. Once you have created it you are back to manually editing the object model and the database. BTW Don't tell me to design a perfect database schema upfront, I'm a hacker, that's not how I work.

The database update thing started to be a stumbling block for me. I could use my own DBTable module for easy database access in my web app but then I lose the django database administration screens. So I decided to write a script to do the upgrade automatically. This is not a totally robust solution, it isn't even totally automatic, you have to list the tables to upgrade (they could be introspected from the django.models.blog but there didn't seem to be a clean list of database table names in there).

This uses my DBTable module to access the database, I didn't use the django database api as that is by definition in a state of flux during an upgrade. This script dumps the database data out into a python script and then shells the django utilities to update the database schema. The auto-generated python script is then executed to populate the new empty database. I decided to generate a python script (first cut used a pickle file) as it allows me to manually edit the way data is added to the database if necessary (e.g. adding default values for new fields or whatever).

   1  #
   2  # Update django
   3  #
   4  import os
   5  import MySQLdb
   6  import DBTable
   7  import sys
   8  
   9  #
  10  # Tables that need backing up and restoring, in order of dependency
  11  #
  12  g_strMyTables = (
  13      "blog_tags",
  14      "blog_posts",
  15      "blog_posts_Tags",
  16      "blog_comments",
  17  )
  18  
  19  def BackupData():
  20      """
  21      Backup all data to python file.
  22      """
  23      oConnection = MySQLdb.connect('localhost', 'django', 'django')
  24      oConnection.select_db( 'django')
  25  
  26      oFile = open( '/tmp/django_data.py', 'w')
  27  
  28      oFile.write( '''import MySQLdb
  29  import DBTable
  30  import datetime
  31  
  32  oConnection = MySQLdb.connect('localhost', 'django', 'django')
  33  oConnection.select_db( 'django')
  34  ''')
  35  
  36      for strTable in g_strMyTables:
  37          oFile.write( "oDB = DBTable.DBTable( oConnection, '%s')\n" % strTable)
  38  
  39          oDB = DBTable.DBTable( oConnection, strTable)
  40          oDB.Select()
  41          for oRow in oDB:
  42              oFile.write( "oDB.Insert( %s)\n" % str(oRow))
  43  
  44  def RestoreData():
  45      """
  46      Restore all data from backup file.
  47      """
  48      oFile = open( '/tmp/django_data.py')
  49      exec oFile
  50  
  51  #
  52  # Do a dry run on generating sql before we do anything drastic.
  53  #
  54  if os.system( 'django-admin.py sqlall blog') > 0:
  55      raise 'Failed to create sql'
  56  
  57  #
  58  # Backup old data.
  59  #
  60  BackupData()
  61  
  62  #
  63  # Clear the old schema.
  64  #
  65  if os.system( 'django-admin.py sqlclear blog | mysql -u django -pdjango django') > 0:
  66      raise 'Failed to clear sql'
  67  
  68  #
  69  # Load the new schema.
  70  #
  71  if os.system( 'django-admin.py sqlall blog | mysql -u django -pdjango django') > 0:
  72      raise 'Failed to create sql'
  73  
  74  #
  75  # Restore data.
  76  #
  77  RestoreData()
  78  
  79  print 'Upgraded'

Once the backup file is generated in /tmp/drupal_data.py it can be run at any time to repopulate the database with data.

I didn't mention the Django templating system in my citique yesterday. It seems ok, as templating systems go, my favourite python templating system being cheetah.

The main advantage of cheetah that I have seen is that you can embed more python code in cheetah templates, or at least use more complex expressions to retrieve values. The django template system seems to restrict you to retrieving values of object properties: you cannot call functions as it complains about '(' characters. I think this is designed to force you to separate logic from presentation which is a Good Thing.

The Django system supports inheritance so a specific web page can inherit from a template describing the overall layout which is cool. I have yet to figure out how or whether it is possible to the equivalent of #include or import to put common sub-blocks in some pages (conceptually similar to a subroutine call). The 'load' tag might be the way.

The django system looks good enough to me to use in preference to cheetah: that's the coolest thing, I have the choice and I could even use both if I wanted.

By far my favourite feature of django is the auto-reload feature whereby if you change some of the code the server will automatically reload it and you don't need to reboot the server. Having to do continual reboots was a bugbear with pyds. In django it has worked flawlessly so far.

Due to some finger trouble in Salamander I accidently deleted some files. Oops.

I could have installed an undelete utility such as freeundelete but downloading it or installing it could have damaged the remains of the deleted files on the disk (as far as windows is concerned the space the files are occupying can be used to store something else) so I thought I'd try knoppix. It transpires that Knoppix 4 includes a utility called ntfsundelete to undelete ntfs files so I gave it a try:

• reboot pc in knoppix. I used knoppix 2 as the start parameters to start in text mode so I didn't have to wait for a gui.
• ran:

  ntfsundelete /dev/hda1
  

  and it listed the potentially undeletable files on my ntfs partition. There were a lot of them.
• ran:

  ntfsundelete /dev/hda1 -m *.py
  

  and it listed just the python files, as the file I most wanted to undelete was python.
• ran:

  ntfsundelete /dev/hda1 -m *.py -u
  

  and it undeleted the python files. They were in the current directory, knoppix's ram drive, i.e. it didn't try to write them back to the ntfs disk. This is very good, it means that if it failed it is unlikely to have ruined the ntfs drive.
• look at my file:

  cat blah.py
  

  and argh I get funny characters and rubbish at the end of the file. Panic. Dissillusionment.
• Open file in vim and it seems that the file is there but a load of rubbish has been added to the end. Maybe ntfsundelete does not now how long the file is and can only restore the whole sectors it was in? Conjecture. Anyway, in vim, delete the rubbish at the end of the file.
• Use sftp to copy the file somewhere safe.

Conclusion: success, albeit not as clean as I would prefer. I do like the idea of being able to undelete the data with the partition read-only so if I fail miserably I can try again with some other tool.

Knoppix 2 Windows 0.

knoppix is a version of linux that boots from a CD. This means you can run linux without having to install it. Apart from being a Linux showcase it is very useful for fixing problems in Windows (resetting passwords, undeleting files etc).

Had a manager with a shiny new Dell Inspiron 220 laptop which was working ok but suddenly the Wifi stopped connecting. It reported a weak signal (1Mbps) when trying to connect to either of the company wireless access points.

On a hunch (I cannot recall my reasoning but I'm not sure it was desperation) I tried giving it a static IP address rather than using automatic settings (i.e. DHCP). Sure enough, this cured the problem, connected fine. I looked in the dhcp server logs and while it was not complaining that the IP address pool was all used, it did seem that all the IP addresses available were being used. I increased the number of IP addresses available to be allocated by the dhcp server and problem solved.

The big mystery to me is why Windows XP would report this as a problem with a weak signal? Isn't the access to the dhcp server and the allocation of the IP address at a layer above the wireless connection? Anyway, it's useful to be warned that the wireless diagnostics are misleading.

I am working on a project where I have to test the support of an old RS232 (serial) based communications protocol. I was interested in writing some python scripts to implement the protocol and a simple test harness. I can remember more than ten years ago doing this in C on QNX and it taking a few weeks. I did it in python in less than a day. I used the pyserial module to handle the serial. Here is a simple loopback test I did in pyserial:

   1  
   2  import serial.serialwin32
   3  import time
   4  
   5  oComms1 = serial.serialwin32.Serial( port= 'COM1', baudrate=9600, bytesize=8,
   6                                       parity='N', stopbits=1, timeout=100, xonxoff=0,
   7                                       rtscts=0, writeTimeout=None, dsrdtr=None)
   8  oComms2 = serial.serialwin32.Serial( port= 'COM2', baudrate=9600, bytesize=8,
   9                                       parity='N', stopbits=1, timeout=100, xonxoff=0,
  10                                       rtscts=0, writeTimeout=None, dsrdtr=None)
  11  
  12  #
  13  # Say hello to myself
  14  #
  15  oComms1.write( 'Hello Peter')
  16  
  17  #
  18  # Poll serial for five seconds.
  19  #
  20  nStart = time.time()
  21  while time.time() - nStart < 5:
  22      nRx = oComms2.inWaiting()
  23      if nRx > 0:
  24          #
  25          # Print what is received.
  26          #
  27          print oComms2.read( nRx)

I tested this by using a loopback cable, connecting COM1 to COM2.

After doing this I wondered whether there was a way to do the loopback in software, a COM driver that implemented loopback. A quick google and I found HW VSP which can do what I want and more as well. This program creates a new virtual COM port, COM5 by default, and relays any data sent to this port to an IP socket somewhere, i.e. it can send it to another computer over the network, or it could connect back to the computer it is running on. By running another instance on COM6 and running it as a server, the two virtual COM ports can communicate with each other, equivalent to a loopback. Or, should I so desire, I could have the test harness run on a different computer.

I ran the two instances like this:

First Instance:

Port

COM5

IP Address

IP address of computer it was running on

Port

5000

Server Port

5001

HW VSP Works as the server only (on settings page)

On

Second Instance:

Port

COM6

IP Address

IP address of computer it was running on

Port

5001

Server Port

5000

HW VSP Works as the server only (on settings page)

Off

Could this be the end of the need for a pile of serial cables, gender benders, swapper boxes and soldering irons?