Peter's Blog

Redefining the Impossible

Items filed under python


I blogged long ago about validating users against a windows domain server using winbind. The users were logging into a web application running on a linux box and winbind allowed their main windows passwords to be used.

Well two problems have come up since those distant days:

1) Through two upgrades of the linux installation (dapper drake->feisty fawn) the python winbind module has become broken when trying to use python version 2.4. It may work with the latest python 2.5 but I am using an old version of turbogears that relies on many old libraries, not all of which are still available in version 2.4 builds. I've tried rebuilding stuff but there seems to be a deep dependency problem between the python module and the winbind library.

2) I want to use the same trick in ruby but I cannot find a winbind gem.

Solution: shell out to a command line tool:

Python version:

bOk = False

if os.system( 'ntlm_auth --username="%s" --password="%s"' % (strUserName.strip(), strPassword.strip())):
    bOk = True

Ruby version:

bOk = system( 'ntlm_auth --username="#{strUserName.strip}" --password="#{strPassword.strip}"')

This is slightly horrible not just because it runs a shell but also because it will send plain-text passwords over the network, unencrypted so any packet sniffer can see them.

This seems to work from with fastcgi based web application. I'm not sure mod_python would allow system calls but I think I prefer fastcgi anyway as it's less flaky.

UPDATE: it is slightly more horrible because it doesn't escape special characters in the username or password: for example it is broken if the password contains and ampersand.


Filed under: linux python runy

2 Comments

Not sure any of this is recommended practise but it appears to work so I'm noting it here.

Overriding Array Methods

   1   class MyArray
   2     #
   3     # Initialise array
   4     #
   5     def initialize
   6        @zog = []
   7     end
   8  
   9     #
  10     # Read from array
  11     #
  12     def [](x)
  13         return @zog[x]
  14     end
  15  
  16     #
  17     # Assign to array
  18     #
  19     def []=(x,y)
  20       @zog[x] = y
  21     end
  22   end

Used thus:

   1  irb(main):021:0* a = MyArray.new
   2  => #<MyArray:0x30d8074 @zog=[]>
   3  irb(main):024:0> a[0] = 1
   4  => 1
   5  irb(main):025:0> a[0]
   6  => 1
   7  irb(main):026:0> a[1] = 2
   8  => 2
   9  irb(main):027:0> a[0]
  10  => 1
  11  irb(main):028:0> a[1]
  12  => 2
  13  irb(main):029:0>

Interesting that ruby doesn't seem to be fussy about array indices:

irb(main):031:0* g = []
=> []
irb(main):032:0> g[5] = 2
=> 2
irb(main):033:0> g[2]
=> nil

Contrast to python:

>>> a = []
>>> a[5] = 2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list assignment index out of range

I think if I had the choice I would rather have my wrist slapped: the ruby way looks like an opportunity for obscure bugs.

Method Missing

It took me ages to work this out but it's simple. You have a class and you want to hook into calls to any undefined methods. Maybe you want to determine the method names at runtime. The hook is called 'method_missing':

   1  class Klack
   2    def method_missing( strName, *args)
   3      print "Calling undefined method #{strName} with arguments #{args.inspect}"
   4    end
   5  end
   6  
   7  irb(main):054:0> k = Klack.new
   8  => #<Klack:0x309dd84>
   9  irb(main):055:0> k.wibble( :wobbles, :banana)
  10  Calling undefined method wibble with arguments [:wobbles, :banana]=> nil

args appears to be a simple array holding the arguments the method was called with.

Note how useful the .inspect method is. It's equivalent to repr in python in making anything into a human readable string. Also gotta love how easy it is to insert the values of variables into strings.

UPDATE: this does seem to make for very fragile code. Just about any problem in your method_missing method can lead to a stack overflow. Or maybe it's a rails problem?

Wots the colon?

The colon thing is handy:

b = :blah

:blah is like a string constant, something like a string, easier to type but less complex in it's implementation. Have to be careful with them though:

   1  irb(main):058:0> b = :blah
   2  => :blah
   3  irb(main):059:0> b[1]
   4  NoMethodError: undefined method `[]' for :blah:Symbol
   5          from (irb):59
   6  irb(main):060:0> b == 'blah'
   7  => false
   8  irb(main):061:0> b == :blah
   9  => true
  10  irb(main):062:0> b.to_s == 'blah'
  11  => true

They don't have common string functions and :blah is NOT comparable to 'blah'.

Regular Expression Gotcha

The =~ operator for testing a string against a regular expression returns the offset of the match.

irb(main):086:0> o = "poop1" =~ /poop(\d+)(=?)/
=> 0

In this case the result is zero as the match starts at character offset zero. The operator would return nil if there was no match.

If you want access to the expression match object then this does the job:

irb(main):088:0> o = /poop(\d+)(=?)/.match( "poop1")
=> #<MatchData:0x3085748>
irb(main):089:0> o[1]
=> "1"
irb(main):090:0>

o is the expression match object where you can examine the juicy details of the match. These are also available in global varibles such as $1, $~ etc but everyone knows that using global variables is sloppy.

Zero is True

The =~ result works nicely with an if operator to detect a match because of the strangest ruby design decision: zero is true:

irb(main):095:0> if 0
irb(main):096:1>  print 'true'
irb(main):097:1> else
irb(main):098:1*  print 'false
irb(main):099:1> end
true=> nil

nil is false but 0 is true.

This is totally at odds with python where 0 and None are both False (with a capital F). I think even basic has 0 as false.

Processing Arrays

The collect method looks useful for writing one-liners:

irb(main):103:0> [1,2,3].collect { |x| x + 1}
=> [2, 3, 4]

Each item in the array is processed by the block and the results returned in a new array. To me this is more readable than the python version:

[ i + 1 for i in [1,2,3]]

Filed under: noob python ruby

2 Comments

So how does one go about mining the information in the World of Warcraft Armory? Inspired by applications like the Warcraft Signature Generator I had to find out.

Here is a python script to interrogate the armory. This will download an xml document containing all the information about a character.

   1  # Interrogate wow armory
   2  
   3  import urllib2
   4  import xml.dom.minidom
   5  
   6  #
   7  # Character we are nosing into
   8  #
   9  strRealm = 'Aerie Peak'
  10  strCharacter = 'Pookypoo'
  11  
  12  #
  13  # Look the data up in the european armory. Change this for US.
  14  #
  15  strUrl = "http://armory.wow-europe.com/character-sheet.xml?r=%s&n=%s" %
  16            (strRealm.replace( ' ', '+'), strCharacter)
  17  
  18  #
  19  # Open url.
  20  # Need to specify firefox as user agent as this makes the server return an XML file.
  21  # If this is not done we get html.
  22  #
  23  oOpener = urllib2.build_opener()
  24  oOpener.addheaders = [
  25     ('user-agent',
  26      'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-GB; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4'),
  27     ]
  28  
  29  req = urllib2.Request( strUrl)
  30  strFile = oOpener.open(req).read()
  31  
  32  #
  33  # Now have raw xml file, can print it if interested
  34  #
  35  # print strFile
  36  
  37  #
  38  # Use xml dom to parse the file
  39  #
  40  oDoc = xml.dom.minidom.parseString( strFile)
  41  
  42  #
  43  # Quick hack to display certain attributes of certain elements.
  44  #
  45  strAttributes = (( 'character', 'level'),
  46                      ( 'character', 'guildName'),
  47                      ( 'baseStats', 'agility'))
  48  
  49  for strElement, strAttribute in strAttributes:
  50      oElement = oDoc.getElementsByTagName( strElement)[0]
  51      strValue = oElement.getAttribute( strAttribute)
  52      print strElement, strAttribute, strValue

This particular script will display the characters level, guild name and agility. There is much more information in there, it's just a matter of finding a use for it. I might get around to automatically updating the character levels shown on this site as the sig generator that inspired this gets hammered a lot and is not totally reliable. UPDATE: not criticising here, it's just a very popular site with heavy loading. The tools there are very nice and more than my imagination could come up with.


Filed under: games python warcraft wow

3 Comments

Right now I'm working in Python yet I am already missing Ruby Blocks.

What I am doing is I am sending commands to firmware and processing the results. I have common code like this:

def DoCommand():
   try:
      SendCommand()
      oResults = GetResults()
      #
      # Process the results
      #
   except:
      ProcessError()

Different types of command yield different results and I have to duplicate all this with different result handling code. In ruby I could do it like this:

   1  def DoCommand()
   2     begin
   3        SendCommand()
   4        oResults = GetResults()
   5        #
   6        # Process the results
   7        #
   8        yield( oResults)
   9     rescue
  10        ProcessError()
  11     end
  12  end

and call it like this

DoCommand() { |oResults| print oResults }

i.e. I pass the result handling code as a block and the block is executed within the exception handler.

There are many other ways I could structure this but using blocks seems like a nice efficient way to do it using a minimum of typing. For example, in python I could do it like this:

   1  def DoCommand( oResultHandler):
   2     try:
   3        SendCommand()
   4        oResults = GetResults()
   5        #
   6        # Process the results
   7        #
   8        oResultHandler( oResults)
   9     except:
  10        ProcessError()
  11  
  12  def HandleOneKindOfResult( oResults)
  13    print oResults
  14  
  15  DoCommand( HandleOneKindOfResult)

I have to define multiple functions like HandleOneKindOfResult and I have to think of names to give them where in ruby these are just nameless blocks.

So why am I doing what I am doing in python and not ruby? Two reasons:

  • short deadline
  • I am using wxPython, pyserial and pyexe and don't know of or have experience of any ruby equivalents.

Filed under: python ruby

2 Comments

After the short-lived Ruby on Rails Daily News theme died after three episodes I have been quiet on the subject. I have used Ruby on Rails for an Intranet application and I find I really like it. The system has a great depth of design, as if every little thing has been thought about, yet it is all done in a very simple way. Need a new static page on your website? Just create a .rhtml file for it, give the file the right name and it will be there on your site, no wiring required. Modifying your database schema? Want to add a column to a table? Create a migration that will both add the column and delete it if you change your mind. Once the migration is loaded the column is there in your mysql, sqlite or whatever database and also available in your object model.

The application I have built has made the users happy but needed what seems like a tiny amount of code. I haven't used the Ruby language much but you do seem to be able to express yourself more succinctly than you can in python, mainly due to the use of blocks.

My main reason for picking up Ruby-on-Rails apart from it being mainstream and commercial was that I can buy books about it and the Pragmatic Programmers books are very good, both "Agile Web Development with Ruby on Rails" and "Programming Ruby".

While the python web development platforms Django and Turbogears have no mature blogging packages ready (I can only find various "works in progress") Ruby on Rails has at least two, Typo and Mephisto. Typo comes with InstantRails and Mephisto appears to have a very small codebase given what it does. Both could be the basis for a CMS and I am very tempted to go this route since the Ruby-on-Rails platform is robust and well documented and the Ruby language is at least as good as Python.

Summary: Ruby on Rails is worth learning ruby for.


Filed under: python rails ruby

2 Comments

I don't normally put gui's on my python scripts but I am doing something for a client and cannot rely on them using the command line. All my script needs is the name of a directory to work with.

The following is a distillation of how to show the dialog for browsing for a directory under python and the win32com module.

   1  from win32com.shell import shell
   2  
   3  oNeedlesslyComplex = shell.SHBrowseForFolder(0, # parent HWND
   4                                  None, # root PIDL.
   5                                  "Choose directory to convert", # dialog title
   6                                  0, # flags
   7                                  None, # callback function
   8                                  None) # 'data' param for the callback
   9  
  10  if oNeedlesslyComplex[0] == None:
  11      pass
  12      # cancel pressed
  13  
  14  #
  15  # Get selected folder from weird return value.
  16  #
  17  strPath = shell.SHGetPathFromIDList(oNeedlesslyComplex[0])

win32com is powerful but little documented.


Filed under: python windows


Been doing some python programming. Since my hassles I am now getting to grips with python 2.5 and the first thing I did required a fast database so I reached for sqlite. Looking on the pysqlite website there was no build for python 2.5. Odd. On a hunch/half remembered thing I simply tried it from the python 2.5 console:

import sqlite3

and there it was, it comes with the python standard library. Cool. The interface follows standard python dbi so it's easy for me to use.

   1  import sqlite3
   2  
   3  oDB = sqlite3.connect( "c:\\desktop\\my database")
   4  oCursor = oDB.cursor()
   5  oCursor.execute( "Select * from table")
   6  while 1:
   7      oRow = oCursor.fetchone()
   8      if not oRow:
   9          break
  10      print oRow
  11  

It's fast, the job I am doing involves creating tens of thousands of records and sqlite is managing about 60,000 in a minute (1.1GHz AMD cpu). This is writing to disk files, creating the database in memory wasn't much faster (probably because my pc has only 512M and is swapping).

Sqlite is a straight database library, python is calling it directly via function calls. There is no inter-process communications, no COM overheads, no network overheads, this is pretty much as fast as databases get.

It is still SQL so there is still the overhead of parsing sql for the database to perform. Something I am getting into is this syntax:

strFirstName = "Fred"
strSurname = "Bloggs"

oCursor.execute( """SELECT * from table
                    WHERE firstname = ?, surname = ?""",
                   (strFirstName, strSurname))

The ?s get replaced with the parameters in the tuple. This is great as it saves a lot of messing around with quoting strings and escapes and whatever and it makes you more resistant to sql injection attacks, should that worry you at all. This technique has got to be faster than building sql strings holding the parameter values, passing them to sqlite and it having to parse them out again. Everybody wins.


Filed under: python


Just found out that my new site5 server has python 2.4.3. This is Good News. Last time I had a site5 server it was running python 2.2 which was getting a bit ancient. 2.3 introduced generators which have been widely adopted. The latest version is 2.5 which is (flame bait) not exactly a must have, >= 2.3 will do.

Another job for my todo list is to see if I can get django or turbogears running through cgi, but I am not optimistic.

Should I get desperate, site5 offer ruby-on-rails hosting but that would involve selling my soul in a manner akin to adopting asp.net


Filed under: django python site5 turbogears


Needed to recover a password from my FileZilla settings. It transpires that this is not very difficult. The passwords are not strongly encoded which some regard as a security flaw but the developers seem to acknowledge that if you want security, don't trust a computer.

   1  #
   2  # Dump filezilla site manager, including account name, host, user and password.
   3  #
   4  import _winreg
   5  
   6  def DecodePassword( strPass):
   7      """Decode a filezilla password"""
   8      strKey = "FILEZILLA1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ"
   9  
  10      nPassLen = len(strPass) / 3
  11      nOffset = nPassLen % len(strKey)
  12  
  13      strDecodedPass = ""
  14  
  15      for i in range(nPassLen):
  16          c = int(strPass[i * 3:(i * 3) + 3])
  17          c2 = ord(strKey[(i + nOffset) % len(strKey)])
  18          c3 = chr((c ^ c2))
  19  
  20          strDecodedPass += c3
  21  
  22      return strDecodedPass
  23  
  24  #
  25  # Walk through registry, decoding site details.
  26  #
  27  oReg = _winreg.ConnectRegistry( None, _winreg.HKEY_CURRENT_USER)
  28  oLicenceKey = _winreg.OpenKey( oReg, r'SOFTWARE\FileZilla\Site Manager')
  29  
  30  nIndex = 0
  31  while 1:
  32      try:
  33          strSite = _winreg.EnumKey( oLicenceKey, nIndex)
  34      except EnvironmentError:
  35          break
  36  
  37      oSiteKey = _winreg.OpenKey( oLicenceKey, strSite)
  38      strHost = _winreg.QueryValueEx( oSiteKey, u'Host')[0].encode( 'ascii')
  39      strUser = _winreg.QueryValueEx( oSiteKey, u'User')[0].encode( 'ascii')
  40      strPassword = DecodePassword( _winreg.QueryValueEx( oSiteKey, u'Pass')[0].encode( 'ascii'))
  41  
  42      print strSite, strHost, strUser, strPassword
  43  
  44      nIndex += 1

Filed under: filezilla python

3 Comments

I still want to set up a system what will automatically record tv programs for playback on my pocketpc. For various reasons this is becoming more important: in a nutshell, I can only take so much ITV.

Been trying to setup mythtv on a new pc I have acquired. I have tried various windows PVR packages in the past but none of them were attractive, being flaky and annoying (including meedio which of now available for free as yahoo go). Mythtv was attractive because of it's flexibility.

I installed a clean kubuntu install in the box which went quite smoothly and after some exploration I established that the kernel already supported my hauppauge nova-t usb. I found an application called kaffeine that was already installed which was able to display tv and this essentially Just Worked out-of-the-box, albeit with lip-sync issues (probably because it needs the proprietary nvidia drivers).

Still desiring mythtv, I found an ubuntu repository that has mythtv packages for version 0.18 and installed that. I looked in the ubuntu package readme which says something to the effect that whoever set it up had no experience with mythtv and didn't know what he was doing. Thanks for the warning. I set it up using the mythtv-setup application, started the mythtv backend and tried to start mythweb. I am mainly interested in this as a way of remotely scheduling recordings, I care not about using the myth frontend. Mythweb wouldn't work, it complained about being a different version to the back end. The php code appears to display this error if there are any problems communicating with the back end but I decided I wouldn't mess around getting an old version to work, I would build the latest 0.19. After a few hours of installing dependant packages it built but when I tried running it I got a segmentation error. At this point I gave up with mythtv, I just don't have the time to nurse it into life. mythtv seems bloated and fragile. there is the option of knoppmyth, a dedicated mythtv distribution, but I'm not sure how cutting edge this is, whether it is any good as a general purpose linux distribution or whether the kernel will support my tv card without having to fiddle with compiling it.

I had a brief look at freevo but sourceforge was down (what an advert for oss) but found that freevo used command line tools to do the recording so I am currently investigating that approach: knocking up simple python scripts to do just what I want. I would rather debug these than mythtv (hell is other peoples source code).

Incidentally, I was browsing through some ruby source yesterday and for a few files there I was wondering whether ruby had a comment character.


3 Comments