Peter's Blog

I was finding Rails development on my Site5 account slightly problematic. There were these issues:

• because of the way the shared hosting was set up you generally needed to trawl the site5 forums to find the special site5 specific hacks to do what you wanted. For example, many seem to have tried and failed to install trac so ultimately one has to compromise over what one can do.
• sometimes it was very slow: I'd find it to be sluggish and checking the load average on the server it would be in double figures: over 300 processes fighting for time with certain people hogging the cpu.

I now have a number of web projects on the go (five including this blog) and I fancied going back to dedicated hosting for the total control it would give me. However there was a problem here: dedicated hosting is expensive and I didn't fancy shelling out that much. After much agonosing I decided to return to using a Virtual Private Server (VPS). I chose to go with VPSLink as their prices were reasonable and it seems suited to what I want. They seem to specialise in VPS's and their site has a blog/wiki/forum which gives them a face compared to oneandone, godaddy or any other big name that only provides vps's to cover the market. I now share a server box with fifteen over guys, each with 512Megs of memory to call our own. Each of us has a 'virtual' server with root access and hence total control over what we do.

On my virtual server I chose to install a preconfigured Ubuntu 7 linux/Ruby on Rails/Lighttpd setup. It is very nice, logging into it with ssh it is hard to tell that the server is shared and is running on a different continent. My rails applications run nice and snappy (once they have done their caching).

I haven't used lighttpd before although I was aware of it as a new clean apache wannabe. While apache is getting bloaty and the configuration files have to me always been obtuse, requireing endless try-this-and-see-what-happens attrition, lighttpd's configuration file was immediately crystal clear: here's the url, there's the directory that serves it. Nuff said. The config file in my installation had a commented out Rails setup which worked on my first tweek.

The only problem I have had thus far with lighttpd is that

/etc/init.d/lighttpd stop

doesn't stop all the lighttpd processes and I have to kill the last one explicitly. I haven't looked into a cause and could even script my way around it if I needed to reboot the server that often.

One possible problem with VPSLink is that their systems are set up such that there is effectively no swap space: if the processes on your VPS use all the allocated memory then tough, they crash. Hence I chose a plan with hopefully sufficient memory (512M) and a light web server. I'm also hoping I don't get too many fcgi processes spawn but I'm not going to lose sleep over that. Should any of my sites get that successful that the server is continually crashing I just upgrade to a dedicated server. (I think I just invited everyone to a DDOS party).

At the time of writing this blog is still on site5. IF I ever get around to porting it to mehpisto then I may move it to the new server. The performance of Drupal on site5 is still acceptable.

Articles related to hosting a web site.

Got a Linode and I've installed ubuntu on it. So far it seems really quick: ssh login is fast and responsive, better than site5. This may be because the server does not have many users yet or maybe because it's 8:30 on a sunday morning.

It is just like my Ubuntu box at work but 10x faster.

I'm paying monthly for the linode, just trying it for now. If I'm still using it in August I may ditch site5 as linode is far cooler.

Damn, have to go out

Linode looks interesting. It is a hosting service whereby you get a virtual linux box all of your own. It is on a server and it is shared with 40 other people but you get 64M of ram to yourself, 3G of disk space that you partition yourself as you see fit and a selection of linux distributions to choose from. You install linux, have root access and basically can install whatever software you like on it (even painful gentoo compilation). It is like having your own linux box out there on the web.

It can be used not just for a web server but ftp, mail, proxy, DNS server, backup server, you name it. It sounds more interesting than Site5 which gives plenty of power except there is no root access, cannot use wget to get packages, no compiler, two year old version of Python, no fastcgi or mod_python just slow cgi, etc etc. Linode costs $20 or £10.50 a month which is more expensive and the support would not extend to patching your kernel like Site5 would do. Then again, unlike a shared host, if some other tosser you share with uses up all the mysql connections with a flaky script your account does not suffer (happened for the second time to my knowledge this sunday).

I'd be tempted to go with this rather than renew my site5 deal.

This python code generates syntax highlighted python code in html format. I know about SilverCity but I want this for my Site5 account where I cannot install executable code. The code below was highlighted using the code itself: spooky.

It is a simplistic solution but it should not be confused by multiline strings, comment characters in strings etc. I started off by trying to use the ply python lex as a tokeniser and processing the tokens but that persisted in confusing multiline string characters with normal strings and while thinking about it I realised that I could live without it. I don't know how slow this is: if using it on a website with heavy traffic you will want to cache the output.

#
# Syntax Highlighting
#

import re
import cgi

# Regular expression rules for simple tokens
strStyles = (
    ('PUNC', re.compile( r'<<|>>|<=|>=|!=|==|[-+*|^~/%=<>\[\]{}(),.:]'), None),
    ('NUMBER', re.compile( r'0x[0-9a-fA-F]+|[+-]?\d+(.\d+)?([eE][+-]\d+)?|\d+'),
                            'color: red'),
    ('KEYWORD', re.compile( r'def|class|break|continue|del|exec|finally|pass|' +
                            r'print|raise|return|try|except|global|assert|lambda|' +
                            r'yield|for|while|if|elif|else|and|in|is|not|or|import|' +
                            r'from|True|False'), 'font-weight: bold'),
    ('MULTILINE', re.compile( r'r?u?(\'\'\'|""")'), 'color: darkred'),
    ('STRING', re.compile( r'r?u?\'(.*?)(?<!\\)\'|"(.*?)(?<!\\)"'), 'color: red'),
    ('IDENTIFIER', re.compile( r'[a-zA-Z_][a-zA-Z0-9_]*'), None),
    ('COMMENT', re.compile( r'\#.*\r?\n'), 'color: green; font-style: italic'),
    ('WHITESPACE', re.compile( r'[ \t\r\n]+'), None),

# if all else fails...
    ('UNKNOWN', re.compile( r'.'), None)
)

class Highlight:
    """
    Syntax highlight some python code.
    """
    def __init__( self):
        self.strOutput = []
        self.strSpanStyle = None

    def Highlight( self, strData):
        """
        Syntax highlight some python code.
        Returns html version of code.
        """

        i = 0
        strMultiline = ''

        #
        # While input is not exhausted...
        #
        while i < len(data):
            #
            # Compare current position with all possible display types.
            #
            for strTok, oRE, strStyle in strStyles:
                oMatch = oRE.match( data, i)
                if oMatch:
                    #
                    # Input matches this type.
                    #
                    strValue = cgi.escape( oMatch.group())
                    if strTok == 'MULTILINE':
                        #
                        # Multiline string token
                        #
                        if strMultiline == '':
                            #
                            # If not inside a multiline string then start one now.
                            #
                            self.ChangeStyle( strStyle)
                            self.strOutput.append( strValue)
                            #
                            # Remember you are in a string and remember how it was
                            # started (""" vs ''')
                            #
                            strMultiline = oMatch.group(1)
                        else:
                            #
                            # Multiline Token found within a multiline string
                            #
                            if oMatch.group() == strMultiline:
                                #
                                # Token is end of multiline so stop here.
                                #
                                self.strOutput.append( strMultiline)
                                strMultiline = ''

                            else:
                                #
                                # Not the same multiline token as started so just output it
                                #
                                self.strOutput.append( strValue)
                    else:
                        #
                        # Other token, not multiline
                        #
                        if strMultiline != '':
                            #
                            # In multiline mode so output the raw text of the token
                            #
                            self.strOutput.append( strValue)
                        else:
                            #
                            # Not in multiline mode so change display style as appropriate
                            # and output the text.
                            #
                            self.ChangeStyle( strStyle)
                            self.strOutput.append( strValue)
                    i += len( oMatch.group())
                    break
            else:
                #
                # Token not found so dump out raw text. This doesn't have to be bullet proof.
                #
                self.ChangeStyle( None)
                self.strOutput.append( data[i])
                i += 1

        #
        # Terminate any styles in use.
        #
        self.ChangeStyle( None)

        return "".join( self.strOutput)

    def ChangeStyle( self, strStyle):
        """
        Generate output to change from existing style to another style only.
        """

        #
        # Output minimal formatting code: only output anything is the style has
        # actually  changed.
        #
        if self.strSpanStyle != strStyle:
            if self.strSpanStyle != None:
                self.strOutput.append( '</span>')
            if strStyle != None:
                self.strOutput.append( '<span style="%s">' % strStyle)
            self.strSpanStyle = strStyle

Used like this:

import sys
data = open( sys.argv[0]).read()
strHighlighted = Highlight().Highlight( data)

print """<html>

<head>
<title>It works</title>
</head>
<body>
<pre>
%s
</pre>
</body>

</head>
""" % strHighlighted

Those nice Site5 people appear to have increased the storage on my hosting account from 1.5G to 3G without telling me or charging me.

This is the same as their latest hosting packages.

How very nice of them.

How to get Python CGI running on a Site5 hosting account:

• Example code, stored in a file in the ~/www/cgi-bin directory:

     1  #!/usr/bin/python
     2  # CGI test
     3  #
     4  import cgi
     5  import cgitb; cgitb.enable()
     6  
     7  strUser = 'Peter'
     8  
     9  #
    10  # Template of html output
    11  #
    12  strHtml = """Content-Type: text/html\n
    13  
    14  <html>
    15  <head>
    16      <title>
    17          This is so cool
    18      </title>
    19  </head>
    20  <body>
    21  <h1>
    22      Careful with that axe Eugene.
    23  </h1>
    24  Hello %s
    25  </body>
    26  </html>
    27  """ % strUser
  

• chmod the file 700

  chmod 700 FileName
  

The file name does not need the .py extension as it is running as a straight executable.

Putty is a simply great ssh client and works nicely with open-ssh, which is found in Ubuntu Linux, Site5 and just about everywhere.

A nice feature of ssh is the ability to generate a public key that can be used to log into a server without having to give a password, or as extra secutiry in addition to the password.

Here is a procedure for creating ssh keys that can be used in both open-ssh and putty:

• On windows, install the open-ssh package with Cygwin
• execute the command

  ssh-keygen -t ssh-dss
  

  to generate the dss key. You may need to create the directory ~/.ssh in Cygwin bash for this to work. This will create a file in this directory called id_dsa.pub
• use sftp/ssh to copy the id_dsa.pub file to your ssh server box. Put the contents of this file (which is one big long line) at the end of a file called ~/.ssh/authorised_keys2, adding it to any other keys that are already there.
• back on windows, execute the command 'puttygen', from the putty site.
• In putty gen, use file/load private key to load in the file ~/.ssh/id_dsa
• Choose 'save private key' and store it somewhere handy where putty can find it. You may be prompted to enter a passphrase. This is a password used in addition to the key when connecting to the server. If the passphrase is blank then you don't have to enter it, the connection will be automatic.
• Open putty and enter the details of the server you want to connect to (address etc)
• In the 'connection' settings, enter your login name in 'Auto-login username'.
• In Connection/SSH/Auth, in the box 'Private key file for authentication' load the putty private key file.
• Save this configuration so you don't have to do it again.
• Click 'open'

Your life won't be the same again.

Since I mentioned awstats on this blog I've been getting attempts to access the awstats.pl script on this site. awstats.pl is not accessable through this domain, it is provided by Site5 but I have to log in to netadmin to get to them.

Anyway, I had a quick search to see if there was a way to hack in via awstats and sure enough there is. The trick mentioned in this article is the one they are trying to get in with:

200.223.55.134 - - [11/Feb/2005:14:44:54 -0500] "GET
/stats/cgi-bin/awstats.pl?configdir=|echo%20;echo%20;id;echo%20;echo%20|
HTTP/1.0" 404 6186 "-" "Mozilla/4.0 (compatible; MSIE 6.0b; Windows NT 5.0)"

this is trying to execute the command id which shows the uid, gid and groups of the account it runs in. I guess this is probing for this vulnerability and seeing whether it gives root access.

The break-in attempts are coming from a variety of IPs, as is usual they are using proxys so there is no point trying to block them. They are getting 403s anyway, they aren't consuming much bandwidth.

Moral: keep an eye on your access logs, see what folk are up to.

I realised today that I can just set up the auto-forwarding in gmail to forward this email to my main gmail account so I don't have to go through the tedious process of logging out of one gmail account and logging into another.

I haven't tried Gmails new POP service yet. I only see that as a way to create my own email archive. Gmail's user interface is good enough, it's main shortcoming for me is not being able to simply paste pictures into emails, you have to mess around attaching them.

My Site5 account gives me unlimited email accounts or something but this is a simpler option. If I do decide to put up the raw mailto then it's gmail that will have to handle the spam.

If anyone reading this wants a gmail invite then just ask. I think they are so common these days that I doubt I'll get any takers.

A nod to this site for the email icon generator.