Peter's Blog

Redefining the Impossible

Items filed under php


I still want to set up a system what will automatically record tv programs for playback on my pocketpc. For various reasons this is becoming more important: in a nutshell, I can only take so much ITV.

Been trying to setup mythtv on a new pc I have acquired. I have tried various windows PVR packages in the past but none of them were attractive, being flaky and annoying (including meedio which of now available for free as yahoo go). Mythtv was attractive because of it's flexibility.

I installed a clean kubuntu install in the box which went quite smoothly and after some exploration I established that the kernel already supported my hauppauge nova-t usb. I found an application called kaffeine that was already installed which was able to display tv and this essentially Just Worked out-of-the-box, albeit with lip-sync issues (probably because it needs the proprietary nvidia drivers).

Still desiring mythtv, I found an ubuntu repository that has mythtv packages for version 0.18 and installed that. I looked in the ubuntu package readme which says something to the effect that whoever set it up had no experience with mythtv and didn't know what he was doing. Thanks for the warning. I set it up using the mythtv-setup application, started the mythtv backend and tried to start mythweb. I am mainly interested in this as a way of remotely scheduling recordings, I care not about using the myth frontend. Mythweb wouldn't work, it complained about being a different version to the back end. The php code appears to display this error if there are any problems communicating with the back end but I decided I wouldn't mess around getting an old version to work, I would build the latest 0.19. After a few hours of installing dependant packages it built but when I tried running it I got a segmentation error. At this point I gave up with mythtv, I just don't have the time to nurse it into life. mythtv seems bloated and fragile. there is the option of knoppmyth, a dedicated mythtv distribution, but I'm not sure how cutting edge this is, whether it is any good as a general purpose linux distribution or whether the kernel will support my tv card without having to fiddle with compiling it.

I had a brief look at freevo but sourceforge was down (what an advert for oss) but found that freevo used command line tools to do the recording so I am currently investigating that approach: knocking up simple python scripts to do just what I want. I would rather debug these than mythtv (hell is other peoples source code).

Incidentally, I was browsing through some ruby source yesterday and for a few files there I was wondering whether ruby had a comment character.


3 Comments

Moved Peter's Blog to new oneandone dedicated server. Observations:

The speed may be due to:

  • dedicated server to itself
  • mod_php instead of cgi on site5.

In summary, woo hoo.


Filed under: blogging oneandone php site5


Something is starting to really irritate me. Whenever I make some change on this site through Drupal such as editing a page, I press the submit button and the browser shows me the page as it was before the edit. I have to press refresh to see the changes. This happens all over, if I look at the logs, add a page, look again at the logs, nothing has changed until I refresh the page.

Clues to the cause:

  • I have caching disabled in Drupal and it should not be caching updated pages anyway.
  • My browser is connected directly, no caching proxy.
  • If I run ethereal I can see that the browser is sending a GET for the updated page and receiving the old version
  • If I look at the site apache access logs I do not see this GET reaching the server

Something in between is caching the pages. J'accuse Site5. I've looked all through their site and I see nothing about this but the forums tend to be full of noobs (like most forums). I tried editing the .htaccess file to disable mod_cache if it was loaded but that just caused an internal server error. It is highly likely that they are running some form of cache the other side of apache (squid?), I'd probably do the same to save the load on a shared server for other peoples sites. However, I find it intrusive when I'm the victim.

Hosting contract is coming to an end, I still have time to evaluate a new hosting solution. I'm highly tempted by a cheap dedicated server. It's expensive but:

  • I've plenty of power: I can run X and use a vnc terminal.
  • I can do what I like in python. I have no interest in doing php at all and Site5's python support is weak (python 2.2.3, no generators sad). They now support ruby on rails but I'm more interested in php than ruby (weak python clone).
  • I learn to secure a linux box properly. I think my linode server was hacked through an xmlrpc weakness in drupal which has been fixed now.
  • It's all mine. With VPS's and shared hosting, you are having to share disk and cpu with other people. I would only have to share network bandwidth with them.
  • I can sell CPU time/hosting/web sites should I feel the desire

One thing about dedicated servers: I cannot believe that to get the server rebooted you have to email some guy who has to run to the box and press the reset button. How primitive is that? If I screw things up to the extent that shutdown -r now does not work I will either learn to prototype hacks on a local box or give up and become an estate agent.

UPDATE: er, it wasn't site5's fault, it was Microsofts.


Filed under: drupal linode php python ruby site5

7 Comments

After fixing postfix installation all was looking good until I realised the web server running Drupal was only showing the home page: clicking on other pages kept giving the home page. I looked in /var/log/apache/error.log and found I was getting this error on each click:

/usr/sbin/apache: relocation error: /usr/lib/php4/20020429/mysql.so:
undefined symbol: php_sprintf

A nasty one. Some googling and forum browsing gave me the clue to the solution: I had an unholy mix of Apache 1.3 and Apache 2.0 installed on the box, apache 1.3 was running and finding php compiled for Apache 2.0 (or something like that).

The solution was:

  • run rcconf and disable apache (1.3) and enable apache2
  • stop apache 1.3
  • install mod_php4 for apache2
  • enable php in /etc/apache2/apache2.conf
  • enable mysql.so and gd.so in /etc/php4/apache2/php.ini
  • start apache2

and sanity was restored (if an intranet can be described as that).


Filed under: apache linux mysql php ubuntu

4 Comments

I updated my linode to Ubuntu hoary hedgehog. I followed the upgrade steps here except I didn't install the ubuntu-desktop package as its a GUI-less server (although I may regret that one day).

Notes:

  • it only took a few minutes to upgrade 250 packages.
  • installing one of the packages failed with:
    udev requires a kernel >= 2.6.8, upgrade aborted.
    dpkg: error processing /var/cache/apt/archives/udev_0.050-3ubuntu7_i386.deb (--u
    npack):
     subprocess pre-installation script returned error exit status 1
    Errors were encountered while processing:
     /var/cache/apt/archives/udev_0.050-3ubuntu7_i386.deb
    E: Sub-process /usr/bin/dpkg returned an error code (1)
    
    which appeared to be because I am running a 2.4 kernel. I am not sure how free I am to change kernels on a Linode. I ignored the error and life went on.
  • libphp4 module installation failed because it could not find a php.ini file. I create one for it to fiddle with and this time it was happy.
  • I did shutdown -r now to reboot but this only powered down the linode. I had to go into the Linode control panel to boot it again.
  • It came up and I was happy. If it hadn't I would have to do a clean install. I MUST get around to adding a rescue partition on the virtual hard disk.
  • It had lost the hostname and I had to use the hostname command to reset it.
  • It had decided to upgrade me to apache2 and install the generic index.html which overrode my drupal index.php so I deleted it.
  • I had to enable php in /etc/apache2/apache2.conf by uncommenting:
    AddType application/x-httpd-php .php
    
  • I had to enable mysql and gd in /etc/php4/apache2/php.ini by uncommenting:
    extension=mysql.so
    extension=gd.so
    
  • It has python 2.4 smile
  • It still has subversion 1.06 sad I wanted to try out 1.2 which is why I upgraded Ubuntu.

Filed under: drupal linode mysql php ubuntu


Linode looks interesting. It is a hosting service whereby you get a virtual linux box all of your own. It is on a server and it is shared with 40 other people but you get 64M of ram to yourself, 3G of disk space that you partition yourself as you see fit and a selection of linux distributions to choose from. You install linux, have root access and basically can install whatever software you like on it (even painful gentoo compilation). It is like having your own linux box out there on the web.

It can be used not just for a web server but ftp, mail, proxy, DNS server, backup server, you name it. It sounds more interesting than Site5 which gives plenty of power except there is no root access, cannot use wget to get packages, no compiler, two year old version of Python, no fastcgi or mod_python just slow cgi, etc etc. Linode costs $20 or £10.50 a month which is more expensive and the support would not extend to patching your kernel like Site5 would do. Then again, unlike a shared host, if some other tosser you share with uses up all the mysql connections with a flaky script your account does not suffer (happened for the second time to my knowledge this sunday).

I'd be tempted to go with this rather than renew my site5 deal.


2 Comments

This python code generates syntax highlighted python code in html format. I know about SilverCity but I want this for my Site5 account where I cannot install executable code. The code below was highlighted using the code itself: spooky.

It is a simplistic solution but it should not be confused by multiline strings, comment characters in strings etc. I started off by trying to use the ply python lex as a tokeniser and processing the tokens but that persisted in confusing multiline string characters with normal strings and while thinking about it I realised that I could live without it. I don't know how slow this is: if using it on a website with heavy traffic you will want to cache the output.

#
# Syntax Highlighting
#

import re
import cgi

# Regular expression rules for simple tokens
strStyles = (
    ('PUNC', re.compile( r'<<|>>|<=|>=|!=|==|[-+*|^~/%=<>\[\]{}(),.:]'), None),
    ('NUMBER', re.compile( r'0x[0-9a-fA-F]+|[+-]?\d+(.\d+)?([eE][+-]\d+)?|\d+'),
                            'color: red'),
    ('KEYWORD', re.compile( r'def|class|break|continue|del|exec|finally|pass|' +
                            r'print|raise|return|try|except|global|assert|lambda|' +
                            r'yield|for|while|if|elif|else|and|in|is|not|or|import|' +
                            r'from|True|False'), 'font-weight: bold'),
    ('MULTILINE', re.compile( r'r?u?(\'\'\'|""")'), 'color: darkred'),
    ('STRING', re.compile( r'r?u?\'(.*?)(?<!\\)\'|"(.*?)(?<!\\)"'), 'color: red'),
    ('IDENTIFIER', re.compile( r'[a-zA-Z_][a-zA-Z0-9_]*'), None),
    ('COMMENT', re.compile( r'\#.*\r?\n'), 'color: green; font-style: italic'),
    ('WHITESPACE', re.compile( r'[ \t\r\n]+'), None),

# if all else fails...
    ('UNKNOWN', re.compile( r'.'), None)
)

class Highlight:
    """
    Syntax highlight some python code.
    """
    def __init__( self):
        self.strOutput = []
        self.strSpanStyle = None

    def Highlight( self, strData):
        """
        Syntax highlight some python code.
        Returns html version of code.
        """

        i = 0
        strMultiline = ''

        #
        # While input is not exhausted...
        #
        while i < len(data):
            #
            # Compare current position with all possible display types.
            #
            for strTok, oRE, strStyle in strStyles:
                oMatch = oRE.match( data, i)
                if oMatch:
                    #
                    # Input matches this type.
                    #
                    strValue = cgi.escape( oMatch.group())
                    if strTok == 'MULTILINE':
                        #
                        # Multiline string token
                        #
                        if strMultiline == '':
                            #
                            # If not inside a multiline string then start one now.
                            #
                            self.ChangeStyle( strStyle)
                            self.strOutput.append( strValue)
                            #
                            # Remember you are in a string and remember how it was
                            # started (""" vs ''')
                            #
                            strMultiline = oMatch.group(1)
                        else:
                            #
                            # Multiline Token found within a multiline string
                            #
                            if oMatch.group() == strMultiline:
                                #
                                # Token is end of multiline so stop here.
                                #
                                self.strOutput.append( strMultiline)
                                strMultiline = ''

                            else:
                                #
                                # Not the same multiline token as started so just output it
                                #
                                self.strOutput.append( strValue)
                    else:
                        #
                        # Other token, not multiline
                        #
                        if strMultiline != '':
                            #
                            # In multiline mode so output the raw text of the token
                            #
                            self.strOutput.append( strValue)
                        else:
                            #
                            # Not in multiline mode so change display style as appropriate
                            # and output the text.
                            #
                            self.ChangeStyle( strStyle)
                            self.strOutput.append( strValue)
                    i += len( oMatch.group())
                    break
            else:
                #
                # Token not found so dump out raw text. This doesn't have to be bullet proof.
                #
                self.ChangeStyle( None)
                self.strOutput.append( data[i])
                i += 1

        #
        # Terminate any styles in use.
        #
        self.ChangeStyle( None)

        return "".join( self.strOutput)

    def ChangeStyle( self, strStyle):
        """
        Generate output to change from existing style to another style only.
        """

        #
        # Output minimal formatting code: only output anything is the style has
        # actually  changed.
        #
        if self.strSpanStyle != strStyle:
            if self.strSpanStyle != None:
                self.strOutput.append( '</span>')
            if strStyle != None:
                self.strOutput.append( '<span style="%s">' % strStyle)
            self.strSpanStyle = strStyle

Used like this:

import sys
data = open( sys.argv[0]).read()
strHighlighted = Highlight().Highlight( data)

print """<html>

<head>
<title>It works</title>
</head>
<body>
<pre>
%s
</pre>
</body>

</head>
""" % strHighlighted

Filed under: hosting php python site5

3 Comments

I've been contemplating some python web development. This is mainly for the following reasons:

  • I'd like my blog to have features for threading subjects based on keywords, a kind of related articles list. This would be dynamically updated, old articles could link to newer articles. This is possible in Drupal but the taxonomy system is laborious to configure through the web interface. There is a new keyword feature (folksonomies) but it appears to be a library for other modules to use. There are some modules that promise this kind of thing but they are not what I am looking for. The most important thing is that old articles are displayed with links to new articles, which involves messing around with themeing to add links to a cached page or figuring out how to flush the cache and put articles through a filter that would add the links.
  • I'd rather spend my limited programming time being creative with technologies I am already familiar with. I don't like php enough to want to learn it any more. I know and like python.
  • I don't want to spend my time trying to figure out how the Drupal code base works. Ok, I'm lazy and I have a short attention span and I'm not very interested in how it works.
  • If I do this as a spare time project I want to enjoy it, I don't want it to feel like a maintenance exercise. I can do that at work.
  • I have found that using librarys etc written by third partys can give you 95% of what you want, then you spend weeks trying to hack them to get the extra 5%. The only exception to this rule I can think of is the python standard library which is way cool.

So I looked at a few web application frameworks. This is a little lightweight and short on detail, I did most of it a few weeks ago and I didn't take notes.

cherrypy: Requires python 2.3 and my hosting service will only give me python 2.2 with cgi. I think generator functions are way cool and I'd love to use them. Do I stand any chance of getting my hosting service to upgrade? I'll email them and ask.

quixote: got the demo going, although I had to fix some of the urls within it as they appeared to be broken. It worked easily enough with cgi, the 1.3 version worked with python 2.2. However, again they have just released version 2 which has a couple of yield statements in it, limiting it python 2.3.

webware: I don't think I could see the advantage of 'servlets' and their web site (presumably powered by webware but it only says 'python powered') is not an impressive display of it's capabilities.

zope/plone: I've toyed with zope in the past (version 2) and found it's learning curve to be almighty. It may be very capable but you have to learn the zope way to do everything. The developers seem to be concentrating on inventing new object-orientated paradigms rather than keeping things simple and I hate huge complex object models. I don't like the web-based source editor (what, no syntax highlighting?: although I could use mozex and edit in vim). Also, non-cgi.

I was most interested in quixote until I came to the python 2.3 stumbling block with the new version 2. I don't want to settle for an old version. This led me on to think about what these frameworks actually give me:

  • map urls to class/method calls
    • big deal
  • session management
    • do I need it for my simple purposes?
  • templating systems
  • interface to cgi/mod_python/fastcgi/whatever
    • I can only have cgi anyway
  • easy form generation
    • classes to build the contents of a form. How many forms will I need? Can't templates do this?
  • already used on the internet and exposed to hackers
    • lets not be too ambitious with what we do. Keep drupal in the background for data entry.
  • learning curves

Which has led me to... straight cgi. Run my own scripts from cgi, using the cgi module in the standard python library to get the url parameters and generate a nice error page when things go wrong. I can get it going easily enough and below, with just a few hours hacking, is something that can display nodes from the drupal database. The output is themed through the Cheetah template engine and gives me the same look as my drupal theme. Given that it is only cgi it is not massively scalable, but I understand every line of it and that means a lot in terms of development time. Given that it generates pages that look like the drupal pages, I can chop and change, leaving some bits to drupal (data entry) and others in python (presentation). It's a lash up but so what?

The main challenge is porting my wilki module to python, particularly the geshi syntax highlighting.

I am not planning to rewrite Drupal and I probably won't release Yet Another Python Web Framework. I may not even finish it. It is something for me to play with.

   1  #!/usr/bin/python
   2  #
   3  import Cheetah.Template
   4  import cgi
   5  import cgitb; cgitb.enable()
   6  import time
   7  import datetime
   8  import os
   9  import pdo
  10  
  11  def ThemeAndOutput( strPageTitle, strContent, oParams={}):
  12      "Theme the site"
  13  
  14      strSideBar = """
  15      <div class="block">
  16      <h2>Reports</h2>
  17      <ul>
  18          <li><a href="Intranet.cgi">Drupal Emulator</a></li>
  19      </ul>
  20      </div>"""
  21  
  22      oDict = {
  23          'Root': '/SMB',
  24          'Url': '/Intranet/Intranet.cgi',
  25          'Title': 'Drupal Emulator',
  26          'Slogan': 'There are no Problems, only Challenges',
  27          'PrimaryLinks': [ '<a href="http://intranet.org">Intranet</a>',
  28                              '<a href="http://www.google.co.uk">Google</a>'],
  29          'PageTitle': strPageTitle,
  30          'SideBar': strSideBar,
  31          'Footer': 'Copyright 2005 Peter Wilkinson'
  32      }
  33  
  34      #
  35      # Read page template and shove in the content.
  36      #
  37      strTemplate = open( 'PageTemplate.html').read() % { 'Content': strContent}
  38  
  39      #
  40      # Run template and content through cheetah in one go.
  41      # Is this faster? Doubt it.
  42      #
  43      oHtml = Cheetah.Template.Template( strTemplate, [oDict, oParams])
  44  
  45      print str(oHtml)
  46  
  47  def MainPage( oForm):
  48      "Main welcome page: tell user what is going on"
  49  
  50      #
  51      # Determine how many links to show.
  52      #
  53      strFrom = oForm.getfirst( 'from')
  54      if strFrom == None:
  55          strLimit = ' LIMIT 10'
  56      else:
  57          try:
  58              nLimit = int(strFrom)
  59              strLimit = ' LIMIT %d, 10' % nLimit
  60          except:
  61              strLimit = ' LIMIT 10'
  62  
  63      c = pdo.connect( 'Module=MySQLdb;user=drupal;passwd=secret;db=drupal')
  64      rs = c.open( 'SELECT * FROM node ORDER BY created DESC %s' % strLimit)
  65  
  66      strTitles = []
  67      while rs.next():
  68          strTitles.append( '<a href="Intranet.cgi?node=%s">%s</a>' %
  69                (rs.fields['nid'].value, rs.fields['title'].value))
  70  
  71      strPage = """
  72  <ul>
  73  #for $strTitle in $titles
  74  <li>$strTitle</li>
  75  #end for
  76  </ul>
  77      """
  78  
  79      ThemeAndOutput( "Welcome", strPage, { 'titles': strTitles})
  80  
  81  def NodePage( oForm):
  82      "Display a node"
  83  
  84      strNode = oForm.getfirst( 'node')
  85      if strNode == None:
  86          ErrorPage()
  87          return
  88      else:
  89          try:
  90              nNode = int(strNode)
  91          except:
  92              ErrorPage()
  93              return
  94  
  95      c = pdo.connect( 'Module=MySQLdb;user=drupal;passwd=secret;db=drupal')
  96      rs = c.open( 'SELECT * FROM node WHERE nid = %d' % nNode)
  97  
  98      if rs.next():
  99          strTitle = rs.fields['title'].value
 100          strBody = rs.fields['body'].value
 101      else:
 102          ErrorPage()
 103          return
 104  
 105      strPage = """
 106  <h2>$strTitle</h2>
 107  
 108  <pre>
 109  $strBody
 110  </pre>
 111      """
 112  
 113      ThemeAndOutput( strTitle, strPage,
 114            { 'strTitle': strTitle, 'strBody': strBody})
 115  
 116  def ErrorPage():
 117      strPage = """<p>There has been some kind of navigation error, the link
 118  you just clicked is broken</p>
 119  <p>Better <a href="mailto:someone@somewhere.com">complain about it</a>.</p>
 120      """
 121  
 122      ThemeAndOutput( "Drupal emulator", strPage)
 123  
 124  oForm=cgi.FieldStorage()
 125  
 126  node = oForm.getfirst( 'node')
 127  if node:
 128      NodePage( oForm)
 129  else:
 130      MainPage( oForm)
Toggle Line Numbers

The page template is kept in a seperate file. It is based on the phptemplate of my drupal theme and was ported over in less than an hour. Theming engines? A specialisation of templates.

   1  Content-Type: text/html
   2  
   3  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
   4   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
   5  <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
   6  <head>
   7  <title>$Title</title>
   8  <meta http-equiv="Content-Style-Type" content="text/css"/>
   9  <style type="text/css">@import url(style.css);</style>
  10  </head>
  11  <body>
  12      <div id="headerleft">
  13      </div>
  14      <div id="headermain">
  15          <div id="ie6hack">
  16          </div>
  17      </div>
  18      <div id="headertitle">
  19          <h1>
  20              <a href="$Url" title="$Title">
  21              $Title</a>
  22          </h1>
  23      </div>
  24      <div class="slogan">
  25          $Slogan
  26      </div>
  27      <div id="logo">
  28          <a href="$Url" title="$Title">
  29              <img src="ITL.gif"/>
  30          </a>
  31      </div>
  32      <div id="headerright">
  33      </div>
  34  #if len($PrimaryLinks) > 0
  35      <div id="primarylinks">
  36          <ul>
  37  #for $PrimaryLink in $PrimaryLinks
  38              <li>$PrimaryLink</li>
  39  #end for
  40          </ul>
  41      </div>
  42  #end if
  43  
  44      <div id="ie5forcemargininsidehack">
  45          <div id="main">
  46              <div class="tab">
  47                  <div class="tabbackground">
  48                      <div class="tableft">
  49                          <h3>
  50                              $PageTitle
  51                          </h3>
  52                      </div>
  53                  </div>
  54              </div>
  55              <div class="tabbody">
  56                  <div class="ieisbuggy">
  57                  </div>
  58                  <div id="content">
  59                      %(Content)s
  60                  </div>
  61                  <div class="ieisbuggy">
  62                  </div>
  63              </div>
  64          </div>
  65          <div id="footer">
  66              <div class="footerleft">
  67                  <p>
  68                  $Footer
  69                  </p>
  70              </div>
  71          </div>
  72      </div>
  73      <div id="sidebarright">
  74          <div id="sidebar">
  75              <div class="sidebarbox">
  76                  $SideBar
  77              </div>
  78          </div>
  79          <div id="sidebarbottom">
  80          </div>
  81      </div>
  82  </body>
Toggle Line Numbers


Those nice Site5 people appear to have increased the storage on my hosting account from 1.5G to 3G without telling me or charging me.

This is the same as their latest hosting packages.

How very nice of them.


Filed under: hosting php site5

4 Comments

Following my thoughts yesterday, here are some VIM python scripts to add python breakpoint and debugging features to VIM. With this set up the F7 key will set a breakpoint on a line of code, Shift-F7 will remove all breakpoints and Shift-F12 will execute a script in the python debugger. This only runs on windows as far as I know, because it uses the 'start' command to launch the debugger in a seperate process without VIM waiting for it to finish. This allows you to look through the source code (and fix it) while the debugging is still in progress.

This goes in a python block in _vimrc:

   1  def SetBreakpoint():
   2      import re
   3  
   4      nLine = int( vim.eval( 'line(".")'))
   5  
   6      strLine = vim.current.line
   7      strWhite = re.search( '^(\s*)', strLine).group(1)
   8  
   9      vim.current.buffer.append(
  10         "%(space)spdb.set_trace() %(mark)s Breakpoint %(mark)s" %
  11           {'space':strWhite, 'mark': '#' * 30}, nLine - 1)
  12  
  13      for strLine in vim.current.buffer:
  14          if strLine == "import pdb":
  15              break
  16      else:
  17          vim.current.buffer.append( 'import pdb', 0)
  18          vim.command( 'normal j1')
  19  
  20  vim.command( 'map <f7> :py SetBreakpoint()<cr>')
  21  
  22  def RemoveBreakpoints():
  23      import re
  24  
  25      nCurrentLine = int( vim.eval( 'line(".")'))
  26  
  27      nLines = []
  28      nLine = 1
  29      for strLine in vim.current.buffer:
  30          if strLine == 'import pdb' or strLine.lstrip()[:15] == 'pdb.set_trace()':
  31              nLines.append( nLine)
  32          nLine += 1
  33  
  34      nLines.reverse()
  35  
  36      for nLine in nLines:
  37          vim.command( 'normal %dG' % nLine)
  38          vim.command( 'normal dd')
  39          if nLine < nCurrentLine:
  40              nCurrentLine -= 1
  41  
  42      vim.command( 'normal %dG' % nCurrentLine)
  43  
  44  vim.command( 'map <s-f7> :py RemoveBreakpoints()<cr>')
  45  
  46  def RunDebugger():
  47      vim.command( 'wall')
  48      strFile = vim.eval( "g:mainfile")
  49      vim.command( "!start python -m pdb %s" % strFile)
  50  
  51  vim.command( 'map <s-f12> :py RunDebugger()<cr>')
Toggle Line Numbers

This relies on using the runscript plugin, and modifying the function 'SetMainScript' as follows:

function s:SetMainScript()
  let s:mainfile = bufname('%')
  let g:mainfile = bufname('%')
  echo s:mainfile . ' set as the starting program.'
endfunction

This allows F11 to be used to select which file is executed to start debugging. F12 can still be used to launch the script outside of the debugger.

Now I don't need WingIDE or a pc upgrade.


Filed under: php python vim windows wingide

5 Comments

Further to my search for a python debugger I gave Stani's Python Editor a try. Observations:

  • ran windows installer but it didn't create start menu entrys. To get it to run I had to:
    • rename c:\python24\scripts\spe to c:\python24\scripts\spe.py
    • run spe.py
  • it started up running extremelt slowly: mouse moved in 2 inch steps. Don't know what it was up to.
  • opened a python file and it was happy, ran at full speed.
  • tried making the pane at the bottom of the screen smaller and it didn't seem to resize the source window, ugly space was left around it.
  • ran python source checker and nothing happened. Later on, when I ran the program, the results of the source check appeared.
  • directory browser is rooted at c:/python24 and doesn't appear to let me look elsewhere.
  • it doesn't have a debugger (to me the whole point).
  • runs at full speed (unlike WingIDE).
  • it does seem to have lots of features, not sure they are all useful to me.

Conclusion: interesting work in progress but I don't need an editor, I want a debugger. I'm not criticising it for not being a debugger, I read it was an IDE so I asssumed it had a debugger. They appear to be planning to add one, maybe I should come back later.

I found a very good intro to using Python's built in debugger here. If I list my essential debugging features as:

  • stop on exception for introspection
  • set breakpoint in source code

Then I can easily enough define VIM macros to do it.

stop on exception for introspection

  • launch script via the python debugger
    python -m pdb myscript.py
    

set breakpoint in source code

  • Add import pdb to start of file, add pdb.set_trace() where I want my breakpoint.

There is this but it doesn't work on windows.


Filed under: php python vim windows wingide

1 Comment

How to get Python CGI running on a Site5 hosting account:

  • Example code, stored in a file in the ~/www/cgi-bin directory:
       1  #!/usr/bin/python
       2  # CGI test
       3  #
       4  import cgi
       5  import cgitb; cgitb.enable()
       6  
       7  strUser = 'Peter'
       8  
       9  #
      10  # Template of html output
      11  #
      12  strHtml = """Content-Type: text/html\n
      13  
      14  <html>
      15  <head>
      16      <title>
      17          This is so cool
      18      </title>
      19  </head>
      20  <body>
      21  <h1>
      22      Careful with that axe Eugene.
      23  </h1>
      24  Hello %s
      25  </body>
      26  </html>
      27  """ % strUser
    
    Toggle Line Numbers
  • chmod the file 700
    chmod 700 FileName
    

The file name does not need the .py extension as it is running as a straight executable.


Filed under: hosting php python site5


Read about the BBCode Firefox Extension which adds right menu for adding BBCode markup. I had already looked at BBCode and decided that it isn't much simpler than plain html markup and I'd rather only remember one of them. So I went looking to see if there is a similar firefox extension to do HTML markup for this blog and... the BBCode extension can be set up to do html markup as well.


Filed under: blog firefox php


Putty is a simply great ssh client and works nicely with open-ssh, which is found in Ubuntu Linux, Site5 and just about everywhere.

A nice feature of ssh is the ability to generate a public key that can be used to log into a server without having to give a password, or as extra secutiry in addition to the password.

Here is a procedure for creating ssh keys that can be used in both open-ssh and putty:

  • On windows, install the open-ssh package with Cygwin
  • execute the command
    ssh-keygen -t ssh-dss
    
    to generate the dss key. You may need to create the directory ~/.ssh in Cygwin bash for this to work. This will create a file in this directory called id_dsa.pub
  • use sftp/ssh to copy the id_dsa.pub file to your ssh server box. Put the contents of this file (which is one big long line) at the end of a file called ~/.ssh/authorised_keys2, adding it to any other keys that are already there.
  • back on windows, execute the command 'puttygen', from the putty site.
  • In putty gen, use file/load private key to load in the file ~/.ssh/id_dsa
  • Choose 'save private key' and store it somewhere handy where putty can find it. You may be prompted to enter a passphrase. This is a password used in addition to the key when connecting to the server. If the passphrase is blank then you don't have to enter it, the connection will be automatic.
  • Open putty and enter the details of the server you want to connect to (address etc)
  • In the 'connection' settings, enter your login name in 'Auto-login username'.
  • In Connection/SSH/Auth, in the box 'Private key file for authentication' load the putty private key file.
  • Save this configuration so you don't have to do it again.
  • Click 'open'

Your life won't be the same again.


2 Comments

Since I mentioned awstats on this blog I've been getting attempts to access the awstats.pl script on this site. awstats.pl is not accessable through this domain, it is provided by Site5 but I have to log in to netadmin to get to them.

Anyway, I had a quick search to see if there was a way to hack in via awstats and sure enough there is. The trick mentioned in this article is the one they are trying to get in with:

200.223.55.134 - - [11/Feb/2005:14:44:54 -0500] "GET
/stats/cgi-bin/awstats.pl?configdir=|echo%20;echo%20;id;echo%20;echo%20|
HTTP/1.0" 404 6186 "-" "Mozilla/4.0 (compatible; MSIE 6.0b; Windows NT 5.0)"

this is trying to execute the command id which shows the uid, gid and groups of the account it runs in. I guess this is probing for this vulnerability and seeing whether it gives root access.

The break-in attempts are coming from a variety of IPs, as is usual they are using proxys so there is no point trying to block them. They are getting 403s anyway, they aren't consuming much bandwidth.

Moral: keep an eye on your access logs, see what folk are up to.


1 Comment

Since Gmail have given me 100 invites I have decided to give my site it's own gmail account:
images/mail.png
. I don't quite have enough faith in gmails spam filters to put the raw email address here yet. prattboy@gmail.com would agree.

I realised today that I can just set up the auto-forwarding in gmail to forward this email to my main gmail account so I don't have to go through the tedious process of logging out of one gmail account and logging into another.

I haven't tried Gmails new POP service yet. I only see that as a way to create my own email archive. Gmail's user interface is good enough, it's main shortcoming for me is not being able to simply paste pictures into emails, you have to mess around attaching them.

My Site5 account gives me unlimited email accounts or something but this is a simpler option. If I do decide to put up the raw mailto then it's gmail that will have to handle the spam.

If anyone reading this wants a gmail invite then just ask. I think they are so common these days that I doubt I'll get any takers.

A nod to this site for the email icon generator.


Filed under: email gmail hosting php site5

2 Comments

I saw in my logs that someone was searching on my site on how to create a Drupal block for flickr. It's easy:

  • On your flickr account, create a badge
  • Copy the badge code
  • in Drupal, create a block of type 'php' and paste the flickr code into it.

This will show selected or random photos in a selection of sizes. It works very nicely.

I chose to centre the pictures in the block by surrounding the code in a centred div:

<div style="text-align: center">
[flick code]
</div>

Filed under: drupal flickr php

5 Comments

I never did get around to trying to install awstats. I've been using Statcounter but I fancied trying awstats with reverse DNS turned on. I can't do this on my Site5 host as they don't like reverse DNS. I didn't install it on Gentoo as that looked like big time hastle.

I realised today that installing awstats under Ubuntu should be as simple as installing the awstats package and it almost is. I can install it on my home server, download my Site5 access logs there and let awstats format them up.

Here are the steps I had to take to install it:

  • Install awstats package
  • Edit a file called /etc/awstats/awstats.hostname.conf where hostname is the hostname. Put something like this in it:
    LogFile="/var/log/apache/access.log"
    LogFormat=1
    DNSLookup=1
    DirData="/var/cache/awstats/"
    DirCgi="/cgi-bin"
    DirIcons="/icon"
    SiteDomain="hostname"
    AllowToUpdateStatsFromBrowser=1
    AllowFullYearView=3
    
  • Make a directory called /var/cache and chmod it 777 so it can be used from the web server
  • Copy icons to web directory:
    cp -r /usr/share/awstats/icon /var/www/icon
    
  • Run this to update databases:
    /usr/lib/cgi-bin/awstats.pl -config=hostname -update
    
  • In your web browser, go to the url:
    http://hostname/cgi-bin/awstats.pl?config=hostname
    
  • Study the stats in quiet awe
  • Edit crontab to update stats automatically every night:
    crontab -e
    0 1 * * * /usr/lib/cgi-bin/awstats.pl -config=hostname -update
    

5 Comments

I wanted to use my favourite VIM script runscript under Ubuntu Linux. This script is my python IDE as it adds the following key presses:

F11
mark script to execute with python
F9
open output window
F12
run script, show results (stdout) in output window

However the F9 button was not working. This turned out to be because it was trying to open a file called 'Output window' which does not equate nicely to a unix file name. Running

:%s/Output window/Output_window/g

on runscript.vim in vim itself fixed the problem.


Filed under: linux php python ubuntu vim

2 Comments

This code goes in a Drupal block set to type 'php' and it shows a Drupal search form. When it loads it examines the 'referer' string that was used to find your site and, if the referrer was google, it will extract the query term and pre-populate the search box with it. This way your visitor can do a local search if they are not immediately inspired by what google led them to.

   1  <?php
   2  $referer = urldecode($_SERVER['HTTP_REFERER']);
   3  if (preg_match('|^http://(www)?\.?google.*|i', $referer)) {
   4    $query_terms = preg_replace('/^.*q=([^&]+)&?.*$/i','$1', $referer);
   5  } else {
   6    $query_terms = "";
   7  }
   8  
   9  $strHtml = <<< EOL2
  10  <form action="search" method="post">
  11    <div id="search">
  12    <input class="form-text" type="text" size="15" value="$query_terms" name="keys"
  13                            alt="Enter the terms you wish to search for." />
  14    <input class="form-submit" type="submit" value="Search" />
  15    </div>
  16  </form>
  17  
  18  EOL2;
  19  return $strHtml;
  20  ?>
Toggle Line Numbers

Here is an alternative version that shows a google site search box instead of a drupal one. It is better in that it uses googles search engine, which is more likely to work with the query term it arrives from, but it does put adverts on your results and relies on your site being thoroughly indexed.

The code below is set up to do a site search on my site so references to www.bisiand.me.uk should be changed.

   1  <?php
   2  $referer = urldecode($_SERVER['HTTP_REFERER']);
   3  if (preg_match('|^http://(www)?\.?google.*|i', $referer)) {
   4    $query_terms = preg_replace('/^.*q=([^&]+)&?.*$/i','$1', $referer);
   5  } else {
   6    $query_terms = "";
   7  }
   8  
   9  $strHtml = <<<EOL
  10  <!-- Search Google -->
  11  <center>
  12  <div style="text-align: center;">
  13  <FORM method=GET action=http://www.google.com/custom>
  14  <TABLE bgcolor=#FFFFCC cellspacing=0 border=0>
  15  <tr valign=top><td>
  16  <A HREF=http://www.google.com/search>
  17  <IMG SRC=http://www.google.com/logos/Logo_40wht.gif border=0 ALT=Google align=middle></A>
  18  </td>
  19  </tr><tr>
  20  <td>
  21  <INPUT TYPE=text name=q size=31 maxlength=100 style="width:100px" value="$query_terms">
  22  <INPUT type=submit name=sa VALUE="Google Search">
  23  <INPUT type=hidden name=cof VALUE="http://www.bisiand.me.uk">
  24  <input type=hidden name=domains value="www.bisiand.me.uk"><br>
  25  <input type=radio name=sitesearch value=""> Google
  26  <input type=radio name=sitesearch value="www.bisiand.me.uk" checked> Peters Blog
  27  </td></tr></TABLE>
  28  </FORM>
  29  </div>
  30  </center>
  31  <!-- Search Google -->
  32  EOL;
  33  return $strHtml;
  34  
  35  ?>
Toggle Line Numbers

Filed under: blog drupal google php


Bisiand.me.uk is mine again! 123-reg got around to reading the fax I sent on Saturday and today I've been able to set up the nameservers to point to Site5.

Leave things to brew for a day or two and my old bisiand.me.uk visitors will be back to join the new crazy frog crowd.

This has made me happy.


Filed under: hosting php site5

1 Comment

I've been keeping an eye on my visitor logs to see how much my domain name problems have effected my traffic. According to Statcounter they had been climbing but yesterday there is a sudden dip. The Awstats logs provided by Site5 show no such dip.

I've seen a number of such dips in the Statcounter logs: their servers do not appear to be the most reliable. This is not a big complaint, I use them for free, more of a lamentation. Their professional service is too expensive for my simple ego brushing needs, $9 a month, but if I was paying that I would not want drop-outs approximately once a week.

The main advantage of Statcounter for me is that it counts visitors who have javascript enabled so it is essentially counting human beings rather than crawlers and referrer spam bots. It is also easy to set it up to ignore my own IP address. The Drupal statistics module does not have this feature but I could simply use phpmyadmin or another generic mysql database report generation tool to filter the drupal logs in any way I desire. The statistics module does list external referrers in reverse chronological order so it is useful for updating .htaccess referrer exclusion lists.


1 Comment

So how does the human clock know what time zone I am in?


Filed under: php


This site now has two URLs, bisiand.me.uk and (drum roll) petersblog.org.

I'm not keen on changing the site's name, given my hard-earned pagerank of 4 but I am fed up with 1&1, my former registrar, renewing unwanted domain names without notice and insisting on cancellation in writing. I wrote to them and told them to stick it all and now I'm worried whether they will let me tranfer bisiand.me.uk elsewhere. As it handles all my email as well as this site I need another domain to cover me if it goes away.

New registrar is 123-easy who are cheap and got me a working domain name in less than three hours.

I am using this trick to get Drupal to serve up the same site on multiple names. It's a cool trick, all the internal links point to the same site. I may have some hard-wired bisiand.me.uk links in there to fix.

Update:

I've modified the trick code to do two things:

  • Make sure all urls start with 'www'. I know google is fussy about this, I get different page ranks with and without.
  • If url appears to be a raw numeric (e.g. 1.2.3.4) then use proper site name.
   1  $base_url = 'http' . ($_SERVER['HTTPS'] == 'on' ? 's' : '') . '://';
   2  $strHost = $_SERVER['HTTP_HOST'];
   3  #
   4  # See if url starts with www
   5  #
   6  if( substr( $strHost, 0, 3) != 'www') {
   7      #
   8      # If addressing url is a numeric ip address then convert it to a site name
   9      #
  10      if( ctype_digit( substr( $strHost, 0, 1)) == TRUE) {
  11          $base_url .= "www.bisiand.me.uk";
  12      } else {
  13          #
  14          # For consistance, make sure url always starts with www.
  15          # It avoids confusion, google seems to care.
  16          #
  17          $base_url .= "www." . $strHost;
  18      }
  19  } else {
  20      #
  21      # Use url as given.
  22      #
  23      $base_url .= $strHost;
  24  }
  25  
  26  if ($dir = trim(dirname($_SERVER['PHP_SELF']), '\,/')) {
  27    $base_url .= "/$dir";
  28  }
Toggle Line Numbers

This works with drupal in the root directory, I haven't tested it with subdirectorys.

Test it out here, here, here and here.


Filed under: drupal google pagerank php


I was faced with writing a setup program for the first time in ages. This meant going back to Wise InstallBuilder 8.1. I couldn't face it, there was no wizard to do the boring stuff for me, it would take time, even for a pretty simple installation.

Around version 3, about 10 years ago, Wise was at a sweet spot with regard to maturity: it had the features I needed at the time and it was robust. It came on a single floppy which was impressive then, mind boggling now. Since that time it has become increasingly bloated, the interface more complex and confusing and each new point release seems to break existing scripts in new ways. Some problems I have had with it are:

  • a new version (point release) decided to set default installation language to German which didn't please our French clients (my fleeting glance testing just saw a foreign language and was satisfied). Had to edit the script to replace all French strings. Wise technical support said 'oh'.
  • I tried using one of the pre-configured scripts to install odbc and it didn't work. The setup program built but did not install odbc. When I looked inside the script the whole thing was commented out with a comment saying I should install odbc another way.

Version 8.1 is a couple of years old now and who knows, maybe they have sorted their act out now. I haven't bothered to upgrade it as the new features aren't worth having and they will probably break my existing scripts.

For this new setup program I decided to give Inno Setup a try as I had heard good things about it. They were true. It has a wizard to do the boring stuff and the script looks pretty easy to manage. It seems to take a long time to compile the executable as it doesn't do an incremental compile like Wise but it worked first time. The setup program it creates is simple but professional. Inno Setup looks as good to me now as Wise did 10 years ago.


Filed under: php


I have a system set up whereby I can forward posts from my private work blog to this public blog. I've just got this system running again since changing the work server to Ubuntu and upgrading Drupal to 4.5.1.

The system works by having the work server email a posting to a python script on the server that uses the xmlrpc api to post the message. I came across two problems.

  1. Ubuntu uses the postfix MTA. With the default setup email posted from php using the mail function just disappeared. I turned on php error logging by setting the following in /etc/php4/apache/php.ini:
    log_errors = On
    
    and was rewarded with the following entries in /var/log/apache/error.log
    sh: line 1: -t: command not found
    
    I worked out that this was due to the following in /etc/php4/apache/php.ini:
    ; For Unix only.  You may supply arguments as well (default: "sendmail -t -i").
    ;sendmail_path =
    
    I changed this to
    sendmail_path = /usr/sbin/sendmail -i -t
    
    and the problem was solved.
  2. When the new post arrived in Drupal it was stored and published but the creation date was rubbish, sometime last year, so the posting was not at the top of the listing. This turned out to be something I have come across before so I changed the function blogapi_new_post in modules/blogapi.module in a similar way to how I did before:
       1    // check for bloggerAPI vs. metaWeblogAPI
       2    if (is_array($params[3])) {
       3      $edit['title'] = $params[3]['title'];
       4      $edit['body'] = $params[3]['description'];
       5      _blogapi_mt_extra($edit, $params[3]);
       6  
       7      //
       8      // PCW
       9      //
      10      if (array_key_exists('created', $params[3])) {
      11          $edit['created'] = $params[3]['created'];
      12      } else {
      13          $edit['created'] = time();
      14      }
      15    }
      16    else {
      17      $edit['title'] = blogapi_blogger_title($params[3]);
      18      $edit['body'] = $params[3];
      19      //
      20      // PCW
      21      //
      22      $edit['created'] = time();
      23    }
    
    Toggle Line Numbers
    This explicitly sets a creation date if none is found in the posting parameters.


Been trying the Drupal image module, using it to create a family photo album. It's ok but I do have a few issues with it:

  • The html it generates for the table in a gallery page is incorrect. It follows the form:
   1  
   2  <table>
   3  <td></td><td></td><td></td>
   4  <tr class="gallery_album_upper_row">
   5  </tr>
   6  <tr class="gallery_album_lower_row">
   7  
   8  </tr>
   9  <td></td><td></td><td></td>
  10  </table>
  11  
Toggle Line Numbers

i.e. the td's are not between the tr's. It still displays ok.

  • The code is supposed to automatically regenerate thumbnails if it cannot find them. I wanted to alter the size of my thumbnails so I deleted the thumbnail files. However, the code generated a load of errors but no thumbnails. This turned out to be because of the following line in the function image_page:
$sql = "SELECT n.nid, n.title, n.teaser, n.body, i.thumb_path FROM {node} n, {term_node} t, {image} i
   WHERE n.nid = t.nid AND n.nid = i.nid AND t.tid = '". $tid ."' AND i.personal = 0 AND n.moderate = 0
   AND n.status = 1 ORDER BY ". _image_get_thumb_order();

This should be:

$sql = "SELECT n.nid, n.title, n.teaser, n.body, i.image_path, i.thumb_path FROM {node} n, {term_node} t, {image} i
   WHERE n.nid = t.nid AND n.nid = i.nid AND t.tid = '". $tid ."' AND i.personal = 0 AND n.moderate = 0
   AND n.status = 1 ORDER BY ". _image_get_thumb_order();

because the function theme_image_gallery_album expects image_path to be amongst the node data.

  • I take photos in both landscape and portrait format and I wanted the thumbnails to be the same area. As it is the code makes the height or width of all thumbnails the same size so landscapes might be 100x75 and portraits 100x133 (same width, vastly different height). I altered the code as follows to make landscape images (for example) 640x480 and portrait images 480x640. There may be a proper way to apply a custom theme but I've found Drupal theme code impenetrable so far.
       1  function _image_make_thumbnail($imagename, $thumbname) {
       2    $error = '';
       3    $lib = variable_get('image_lib', 'imagemagick');
       4    if (($image = image_open($imagename)) != false) {
       5      list($thx, $thy) = split('x', variable_get('image_default_thumb_size', '100'));
       6      $size = getimagesize($imagename);
       7      // pcw: make images same size, allowing for portrait mode
       8      if( $size[1] > $size[0]) {
       9          // portrait: swap required x and y dimensions
      10          $tmp = $thx;
      11          $thx = $thy;
      12          $thy = $tmp;
      13      }
      14      if (!is_numeric($thy)) {
      15        // Set default thumbnail width to 100 if nothing was specified
      16        $thx = is_numeric($thx) ? $thx : 100;
      17        // Compute thumbnail height to keep aspect ratio
      18        $thy = round($size[1] * $thx / $size[0]);
      19      }
      20      else if (!is_numeric($thx)){
      21        // Compute thumbnail width to keep aspect ratio
      22        $thx = round($size[0] * $thy / $size[1]);
      23      }
      24      image_scale($image, $thx, $thy);
      25      image_close($image, $thumbname);
      26    }
      27    else {
      28      $error = t("unable to open image %img", array("%img" => $imagename));
      29    }
      30    return $error;
      31  }
    
    Toggle Line Numbers
  • For the record, the font for the images titles can be altered in style.css thus:
    #image .title {
      font-size: 1.0em;
      font-weight: normal;
    }
    

Filed under: drupal php theme


The following will mass delete unapproved drupal comments, especially accursed comment spam. It is a quick and dirty sledgehammer approach, no subtlety.

  • Create a file in your root directory called, say, killspam.php containing the following:
    <?php
    include_once 'includes/bootstrap.inc';
    include_once 'includes/common.inc' ;
    
    db_query('DELETE FROM {comments} WHERE Status = 1');
    
    echo( "Done");
    
    ?>
    
  • Go to {your website name}/killspam.php in a web browser.

This should result in your unapproved comments being deleted. Note that there is no access control here, anyone can do it. Better to restrict access to the kill file or give it a name that no one can guess.


Filed under: drupal php


This gives nice Python help in Vim.

  • Download this plugin to the plugins directory
  • Add this command to _vimrc to tell it where pydoc can be found:
    let g:pydoc_cmd="python c:/python23/lib/pydoc.py"
    

To see documentation about, for example, the xmlrpclib module, enter:

:Pydoc xmlrpclib

Help comes up in a new window. Cool.

This command:

:PydocSearch xmlrpclib

is supposed to search through the documentation for a string. It's very slow and it throws up an error dialog about not being able to find tk84.dll but it does come up with results eventually. Not so cool.


Filed under: php python vim

2 Comments

I had a problem with using the Drupal 4.5.0 metaWeblogapi function to create posts. The new posts were created with dates that were about a month old. I fixed the problem by commenting out a line in the function _blogapi_mt_extra in blogapi.module.

  // dateCreated
  if ($struct['dateCreated']) {
// PCW: causes posts to be a month old
//    $node->created = iso8601_decode($struct['dateCreated'], 1);
  }

This function is being called erroneously for metaWeblog api. As no 'dateCreated' is passed (not a requirement) the iso8601_decode function returns nothing, setting $node->created to a value that provokes the wrong date later on.

This used to work in 4.4.0 but the code has all changed.


Filed under: drupal php