Peter's Blog

I mentioned last week how I'd signed up for one of the site5 750g hosting packages and also enquired whether my company could use a similar account for remote backup. Since the latter made me aware of the limitations of the 750g account I decided to cancel my personal account under their 60 day guarantee.

They didn't quibble, they asked me to confirm that I had nothing that needed backing up and cancelled the account, refunding my money.

In all the times I've had to deal with them site5 has always given me great service.

I emailed site5 to ask them whether the 750g accounts could be used for my company offsite backups. The answer was no and it directed me to the site5 Resource Usage Policies on a wiki. It is also interesting to see that there is a limit of 25,000 inodes which equates roughly to 25,000 files. To fill 750g the files would need to be 30 megabytes each.

Conclusion: read the smallprint (and/or the wiki).

I couldn't resist the site5 deal for a hosting account with 750g of space for $7.50 a month over two years PLUS (or rather minus) a 13% Halloween discount. That's an awesome amount of backup space, enough to back up the family photo album of 8g 93 times AND the hosting account service includes my backups being backed up.

I bought a new domain name especially to associate with the new account but I don't feel like sharing it. I'm not sure what actual sites I want to host on it, as I say I got it primarily to back everything else up and I'll be spending the next few days writing rsync scripts to do rotating backups and waiting for 8g of family photos to upload at about 400kbps.

Speaking of broadband, the worst possible thing happened yesterday evening while I was waiting for access to the new site5 account: my broadband died. It was down all evening. First time this has happened in a year since I got it. Wife was busy, had 11 month old baby to take care of, what to do? Watched three episodes of Heroes on PVR. Really good if sometimes a bit talky. Unfortunately I was left at episode nine, Hiro (hero geddit?) was six months in the past trying to stop a rather sweet waitress losing the top of her head. Does he manage it? Dunno, I haven't recorded any more episodes and BBC are now a few months down the line. How to fill the gap? Please, NO SPOILERS. I've already spotted that the waitress happened to be learning japanese six months before she had the bad hair day...

To my tremendous relief the broadband was back up this morning. After a year I'm still with pipex, despite my initial hostility it has been so reliable (so far) that I'd rather let it be. Other WoW players moan about disconnections but these rarely effect me (sometimes at about 9am on Saturday mornings but not for months now). Of course I will be reviewing this if I get any more evenings offline ESPECIALLY now I've savoured the cream of my PVR backlog.

This site was offline yesterday while site5 moved the hosting to a server in Texas (yee ha). Everything was offline for a few hours during which time I changed the DNS so the domains (four in all on this host) pointed to the new server on a different IP address.

When the server came up my generic domain which I use for forgettable experiments was ok but the domain pointers (i.e. secondary mappings of domain names to the same ip address as the primary domain name), including petersblog.org were still broken. I tried to kick petersblog.org into life by deleting the domain pointer and creating it again but then I ran into site5's checking against a whois database, presumably to make sure I wasn't trying to hijack someone else's domain name. It seems that this whois database updates comparatively slowly: when I did a whois lookup the records were already up to date. It wouldn't let me create the domain pointer.

Another domain name on a domain pointer started working later in the day so I ought to have been more patient.

This morning I was able to create the domain pointer and hence I am writing this. Fortunately deleting and recreating the domain pointer has not zapped the awstats log history which is recording half a million hits on petersblog.org over the last year.

A third site was running on Ruby on Rails and Radiant and that was broken: it loaded a page that was totally blank. I presumed that the rails on the new site5 server is different or broken so I did something that was on my todo list anyway, move it to the vpslink server which I have as a better Rails platform.

I've had the site5 account for a year, the contract is up and I may be in a position to abandon it or upgrade. Right now they are doing a special offer of 750g for $7.50 a month (yes, that's 750 GIGABYTES) which is almost irresistable. 750g backup space anyone?

Site5 have been ok up to now for basic web hosting and php stuff. Their rails is not totally hassle free (you can get things running if you spend enough time poring through the forums) and I prefer root access for this. It is still hosting a couple of drupal sites including this blog, mainly because they are tied to php4 which I don't want on my vpslink server (less is more).

The vpslink box has apache randomly crashing, presumably through lack of burst memory. I'm not sure I want to commit myself to it long term. I'm tempted again to get a dedicated host and one of these huge bucket site5 accounts for rsync backups while they are still available.

Disclosure: I was in the site5 affiliate program at one time and even earned a few $$$ from it. I'm not actively in it now, although there may be some old affiliate links knocking around here. I'd rather not take their money and be in a better position to criticise them.

I've had a few emails from people wanting to pay to put links here but I haven't taken them up. You will note that this is one of the few sites on the net not plastered with google ads. I like it that way.

I've got a nice Rails development setup going now. Aptana is a very nice IDE, very powerful, very rich. The Rails development aspect is most useful in being able run applications in development mode on my PC. I use the mysql server on the deployment server through an ssh tunnel rather than install a database on the PC.

I have created a subversion repository on my VPSlink server. I have installed the subclipse plugin on Aptana and checked out my application onto the work PC. I work on the application in Aptana, polish it and when it is ready I commit the changes again through Aptana. Then on my server I deploy the code by getting the new version out of subversion. I don't use a simple rsync to deploy and I haven't got into capistrano (the rails deployment tool) going via svn works for me. I might consider putting up something like trac (or a rails based alternative) to give me a development wiki and web browsing of the repository.

I've been thinking about backups and the current plan is to backup the SVN repositorys and mysql dumps from my VPSLink server to my site5 server and vice versa. The VPSLink stuff is more in need of backing up since I am the administrator of that one, the site5 account includes daily backups. I was tempted by rsync.net a nice, simple, flexible and cheap remote backup option but since I already have two accounts with ssh access and 35G of space between them I don't think I need to spend more money. Rsync.net looks appealing but the main thing it is lacking for professional purposes would be Windows file permissions, otherwise I might consider it for backing up the servers at work: being able to recover all ones files is good, not having to spend a week fixing the spaghetti mess of windows access permissions is better.

One major change is that I have reverted from Lighttpd back to Apache on my VPSLink server. The main reason for this was that Lighttpd seems to have some limitations in terms of supporting things like drupal's urls. Lighttpd would be good for simple setups but getting multiple legacy php and rails applications set up is just as troublesome as with Apache so I've gone back to the devil I know.

I've had Apache fail to start with memory errors a few times. This could well be the 500M limit of the VPSLink coming into play. VPSLink is cheap because there is no swap, 500M is my absolute limit. If I start getting memory problems I will have to consider the options:

• I can upgrade the VPSLink to 1G memory but for similar money I could get a cheap dedicated server
• I could get a different VPS account, one that had burst memory (it's the bursts that kill VPSLink) but I do like the performance of VPSLink.
• I could fiddle about tuning the number of apache processes and suchlike but life is too short for that one

This blog is still on the site5 account. The server is being upgraded soon, maybe this will resolve the loading issues it has whenever I try doing some development on it (e.g run 'top' and shriek in horror).

I was finding Rails development on my Site5 account slightly problematic. There were these issues:

• because of the way the shared hosting was set up you generally needed to trawl the site5 forums to find the special site5 specific hacks to do what you wanted. For example, many seem to have tried and failed to install trac so ultimately one has to compromise over what one can do.
• sometimes it was very slow: I'd find it to be sluggish and checking the load average on the server it would be in double figures: over 300 processes fighting for time with certain people hogging the cpu.

I now have a number of web projects on the go (five including this blog) and I fancied going back to dedicated hosting for the total control it would give me. However there was a problem here: dedicated hosting is expensive and I didn't fancy shelling out that much. After much agonosing I decided to return to using a Virtual Private Server (VPS). I chose to go with VPSLink as their prices were reasonable and it seems suited to what I want. They seem to specialise in VPS's and their site has a blog/wiki/forum which gives them a face compared to oneandone, godaddy or any other big name that only provides vps's to cover the market. I now share a server box with fifteen over guys, each with 512Megs of memory to call our own. Each of us has a 'virtual' server with root access and hence total control over what we do.

On my virtual server I chose to install a preconfigured Ubuntu 7 linux/Ruby on Rails/Lighttpd setup. It is very nice, logging into it with ssh it is hard to tell that the server is shared and is running on a different continent. My rails applications run nice and snappy (once they have done their caching).

I haven't used lighttpd before although I was aware of it as a new clean apache wannabe. While apache is getting bloaty and the configuration files have to me always been obtuse, requireing endless try-this-and-see-what-happens attrition, lighttpd's configuration file was immediately crystal clear: here's the url, there's the directory that serves it. Nuff said. The config file in my installation had a commented out Rails setup which worked on my first tweek.

The only problem I have had thus far with lighttpd is that

/etc/init.d/lighttpd stop

doesn't stop all the lighttpd processes and I have to kill the last one explicitly. I haven't looked into a cause and could even script my way around it if I needed to reboot the server that often.

One possible problem with VPSLink is that their systems are set up such that there is effectively no swap space: if the processes on your VPS use all the allocated memory then tough, they crash. Hence I chose a plan with hopefully sufficient memory (512M) and a light web server. I'm also hoping I don't get too many fcgi processes spawn but I'm not going to lose sleep over that. Should any of my sites get that successful that the server is continually crashing I just upgrade to a dedicated server. (I think I just invited everyone to a DDOS party).

At the time of writing this blog is still on site5. IF I ever get around to porting it to mehpisto then I may move it to the new server. The performance of Drupal on site5 is still acceptable.

Researching online backup solutions: really must backup family photos properly. Options so far:

mozy: 2G free or $4.95 a month unlimited. Automated backup and offers file versioning for a month which would be handy. Files accessable from web.

carbonite: a different shade of mozy, horses for courses. Trial account but no long term free account.

xdrive: 5G free which is enough. Creates a virtual drive which you copy files to however you like or run their backup software. Web based access to files from anywhere, file sharing, lots of nice features. What puts me off? It's owned by AOL

flickr: could just upload multiple G of photos to flickr but since they decided to force me to get a yahoo account I haven't been using flickr (yahoo or AOL, I don't want the hassle). Also the upload wouldn't be as totally automatic as mozy or carbonite.

site5: I could cobble something together on my site5 hosting account: I have about 14G free and ssh access so I should be able to do something but it will be a lashup that I would have to nurse along.

Being a dedicated backup solutions, Mozy or Carbonite would almost force me to do backups and do them well. With anything that looked like another hard disk I would end up with yet another disorganised mess. Less flexibility than xdrive could be a good thing. And by avoiding AOL and yahoo I stick it to "the man".

Just found out that my new site5 server has python 2.4.3. This is Good News. Last time I had a site5 server it was running python 2.2 which was getting a bit ancient. 2.3 introduced generators which have been widely adopted. The latest version is 2.5 which is (flame bait) not exactly a must have, >= 2.3 will do.

Another job for my todo list is to see if I can get django or turbogears running through cgi, but I am not optimistic.

Should I get desperate, site5 offer ruby-on-rails hosting but that would involve selling my soul in a manner akin to adopting asp.net

I've been contemplating recently whether to save some money by changing from a dedicated server back to a hosting service, i.e. site5. I managed to break the server by unwisely doing an apt-get update/upgrade which introduced some horrible dependancy problem.

There seemed to me to be a few possible solutions to this:

• hack around in the apt/dpkg files to try to fix it
• wait for someone else to fix it
• compile it all up myself
• start again from scratch

In considering the latter approach I checked out the http://www.site5.com/affiliates/idevaffiliate.php%3Fid=75 site and found a limited offer of an 80% discount (eight oh) for 15G disk space and 1Tb of bandwidth if I signed up there and then: so I signed up there and then. This should save me over £700 over the next two years (seven hundred). Sold.

The 'condition' of the offer is that I have to tell people about it (the 'tell all your friends' special event). Since I don't have any friends I'll tell you.

The site is now back on site5.com and looking good. It 'feels' as fast as the dedicated server but maybe I'm the only one on it so far.

I got an email alert from SiteUptime to say:

Dear Peter Wilkinson,

This is an automated message from SiteUptime.

Alert Type: Site Not Available Result: Failed Time: September 12, 2005 09:09:43 PST HostName: petersblog.org Monitor Name: Peter's Blog Service: http

There was another email half an hour later to say the site was back again (the polling period is half an hour).

Looking on the server itself:

$ uptime
 09:01:45 up 21 days, 11:21,  1 user,  load average: 0.00, 0.00, 0.00

so the server didn't reboot.

Grepping through the apache access log for SiteUptime..

67.30.130.180 - - [12/Sep/2005:15:11:18 +0100] "HEAD / HTTP/1.0" 200 - "-" "SiteUptime.com"
67.30.130.180 - - [12/Sep/2005:15:41:20 +0100] "HEAD / HTTP/1.0" 200 - "-" "SiteUptime.com"
67.30.130.180 - - [12/Sep/2005:16:11:21 +0100] "HEAD / HTTP/1.0" 200 - "-" "SiteUptime.com"
67.30.130.180 - - [12/Sep/2005:16:41:22 +0100] "HEAD / HTTP/1.0" 200 - "-" "SiteUptime.com"
67.30.130.180 - - [12/Sep/2005:17:41:24 +0100] "HEAD / HTTP/1.0" 200 - "-" "SiteUptime.com"
67.30.130.180 - - [12/Sep/2005:18:11:25 +0100] "HEAD / HTTP/1.0" 200 - "-" "SiteUptime.com"
67.30.130.180 - - [12/Sep/2005:18:41:26 +0100] "HEAD / HTTP/1.0" 200 - "-" "SiteUptime.com"
67.30.130.180 - - [12/Sep/2005:19:11:27 +0100] "HEAD / HTTP/1.0" 200 - "-" "SiteUptime.com"
67.30.130.180 - - [12/Sep/2005:19:41:29 +0100] "HEAD / HTTP/1.0" 200 - "-" "SiteUptime.com"

Missing entry at 17:11.

Looking around this in the apache access log:

80.88.204.40 - - [12/Sep/2005:17:04:09 +0100]
80.88.204.40 - - [12/Sep/2005:17:04:09 +0100]
194.244.83.8 - - [12/Sep/2005:17:41:16 +0100]
194.244.83.8 - - [12/Sep/2005:17:41:17 +0100]

No traffic between 17:04 and 17:41.

Nothing in the error log. None of the other logs show anything suspicious.

I think my conclusion here is that there was a loss of connectivity within oneandone and my server was temporarily disconnected from the internet. Now lets think, if they promise 99% uptime does this mean the server is running or the server is running and connected to the internet?

Anyway, oneandone sent me this email this morning:

Dear Peter Wilkinson,

Please be advised that due to an upgrade of the 1&1 Data Centre, we will need to shut down your server for a short time while technicians perform an internal realignment of hardware.

This move will be made during the night from 18.09.2005 to 19.09.2005, between 11:00 PM and 06:00 AM.

Your data will not be affected by the move. However, as a precautionary measure, we recommend strongly that you first back up your data and server settings.

Best regards,

The 1&1 Team

so apparently they are shuffling things about and could well be the cause.

The alert emails were forwarded to my vodafone email account but the filters on that prevented me from getting notification text messages (until I got a message I didn't know what 'from' address to set the filter to) so I wasn't sent alarming and costly (2x10p!!) text messages.

Conclusion: I shouldn't panic, I haven't run a site monitor before, this kind of thing probably happened all the time I was on site5 and I was blissfully unaware.

I have closed my Site5 account, now that the years subscription has expired. I don't need it now that I am hosted on oneandone and things have been running smoothly for a while. I find it kind of sad as the Site5 account served me well. If I wanted shared hosting again I would definitely go back to them.

Moved Peter's Blog to new oneandone dedicated server. Observations:

• Much faster
• The cache problem has gone away (qed Site5?).

The speed may be due to:

• dedicated server to itself
• mod_php instead of cgi on site5.

In summary, woo hoo.

Something is starting to really irritate me. Whenever I make some change on this site through Drupal such as editing a page, I press the submit button and the browser shows me the page as it was before the edit. I have to press refresh to see the changes. This happens all over, if I look at the logs, add a page, look again at the logs, nothing has changed until I refresh the page.

Clues to the cause:

• I have caching disabled in Drupal and it should not be caching updated pages anyway.
• My browser is connected directly, no caching proxy.
• If I run ethereal I can see that the browser is sending a GET for the updated page and receiving the old version
• If I look at the site apache access logs I do not see this GET reaching the server

Something in between is caching the pages. J'accuse Site5. I've looked all through their site and I see nothing about this but the forums tend to be full of noobs (like most forums). I tried editing the .htaccess file to disable mod_cache if it was loaded but that just caused an internal server error. It is highly likely that they are running some form of cache the other side of apache (squid?), I'd probably do the same to save the load on a shared server for other peoples sites. However, I find it intrusive when I'm the victim.

Hosting contract is coming to an end, I still have time to evaluate a new hosting solution. I'm highly tempted by a cheap dedicated server. It's expensive but:

• I've plenty of power: I can run X and use a vnc terminal.
• I can do what I like in python. I have no interest in doing php at all and Site5's python support is weak (python 2.2.3, no generators ). They now support ruby on rails but I'm more interested in php than ruby (weak python clone).
• I learn to secure a linux box properly. I think my linode server was hacked through an xmlrpc weakness in drupal which has been fixed now.
• It's all mine. With VPS's and shared hosting, you are having to share disk and cpu with other people. I would only have to share network bandwidth with them.
• I can sell CPU time/hosting/web sites should I feel the desire

One thing about dedicated servers: I cannot believe that to get the server rebooted you have to email some guy who has to run to the box and press the reset button. How primitive is that? If I screw things up to the extent that shutdown -r now does not work I will either learn to prototype hacks on a local box or give up and become an estate agent.

UPDATE: er, it wasn't site5's fault, it was Microsofts.

This site went down for 3 hours this afternoon. I emailed Site5 after half an hour and still no response from them. Sites back up (or you wouldn't be reading this) but not happy.

I realised today that 99% uptime means 'down 3.6 days a year'.

I'm taking down their tacky adverts. Ironically I was recommending them to someone just a few days ago.

Update: I was actually perusing their site only today as I have to renew my hosting soon. They have stopped doing the mid-range package I was interested in and only have two shared hosting deals, I have a choice between 'not quite enough features' and 'too expensive'.

Update2: um, retraction, there was a reply in my email, for some reason it wasn't highighted and I didn't notice it. They lost all the servers at their Michican facility. I'll put back a small ad and agonise some more over how to host for the next year. If only site5 supported something more recent than python 2.2.3...

A google for 'peter blog' brings www.petersblog.org/blog/1 as result number 3! Out of 8,730,000! This url has a page rank of 4, even though www.petersblog.org is still a 3. This may explain why I've been getting a record number of visitors. Neither page has a significant number of backlinks so I'm surprised I have any page rank at all. Methinks pagerank isn't so important any more.

This site appears in the first page of results for Site5 and Linode and both of these go directly to the corresponding tag page. I have a theory that google is looking through the tag page, seeing the word 'linode' or 'site5' many times and assuming that these pages are highly relevant (which of course they are).

Moral: Search engine placings: it's all in the keywords.

n.b. changing from linode back to site5 hasn't hurt visitor numbers a bit, even though I changed www.bisiamd.me.uk to a simple redirect page instead of an alias of petersblog.org.

Update: this page has PR 5!

Site5 is a web hosting company. They host this web site. I have found them to be cheap, fairly reliable and generally responsive to problems.

This morning I got two emails from Linode, this one:

We're currently receiving a large number of UDP packets coming from your machine:

01:33:33.136745 70.85.129.118 > 24.110.229.206: udp (frag 57902:1480@1480+)
01:33:33.136871 70.85.129.118 > 24.110.229.206: udp (frag 57902:1480@2960+)
01:33:33.136873 70.85.129.118 > 24.110.229.206: udp (frag 57902:1480@4440+)

I've blocked the offending traffic for now. Please reply within 48 hours to avoid service interruption.

and this one:

More DoS traffic:

03:30:42.548612 70.85.129.118 > 194.146.152.98 : udp (frag 56782:1480@2960+)
03:30:42.548617 70.85.129.118 > 194.146.152.98 : udp (frag 56782:1480@4440+)
03:30:42.548874 70.85.129.118 > 194.146.152.98 : udp (frag 56782:1480@7400+)

Your Linode's networking has been disabled.

So my Linodes network access was gone and the site was down. I could log in through the lish console and this showed two tasks called 'bashex' owned by the www-data user i.e. apache taking most of the cpu time. I guess someone had hacked in and were up to no good.

Well, I have been pondering what to do about the Linode, in one way I love it but in another way I am not getting the uptime I would like. Ok, this security breech is MY problem, I didn't secure the server properly but then again, I don't want to spend my valuble spare time securing servers, I have more interesting things to do. Together with silly power problems in their racks, I have had more downtime in a month with Linode that I have in 10 or so months with Site5.

I don't know how long it would take me to secure the linode now it has been breeched, I would want only feel it was clean if I reinstalled everything, and then I would have to go into adding extra security.

I decided instead that no, Linode is not for me, I'll go back to Site5. There are a number of reasons for this:

• someone else does system admin and worries about backups and security and this is included in the price.
• I can afford a top-of-the-line shared hosting package for less than a bottom-of-the-line linode. This gives me, for example, four times as much disk space.
• Site5 have many tech support people ready 24 hours a day and pride themselves on 15 minute response times. Linode only appear to have 'caker' who is good at his job but presumably needs to sleep.

Disadvantages:

• I am not root, I am not free to install what I like.
• The python on the site5 server is version 2.2 which is too old to be useful to me.
• I cannot run any long-lived processes such as a python-based web server.
• It is not as cool

As my networking had been disconnected, my only access to the linode was via the lish console. I got all my data from my web sites off it as follows:

• cd /var/www
• mysqldump -u -p >petersblog.sql
• tar xvvzf www.tar.gz * .htaccess
• uuencode www.tar.gz www.tag.gz

This dumped the tar file as Ascii. By logging all console output to a log file I captured it all on my local pc. I uploaded it to site5 and

• uudecode putty.log
• tar xvfz www.tar.gz
• reload sql and away
• make sure drupal is up to latest version (once bitten...)

I had to change dns servers again and that took a few hours to propogate but this posting was done on Site5.

Now to tell Linode the bad news...

Linode hosting this blog went down for about five hours and I couldn't log into the linode site to reboot it. Once I could log in I had to manually reboot my server to get it up again. This is the third time this has happened this week (twice monday) but the other two times I could at least log into the linode console and reboot. This is really putting me off linode, I give it one more failure like this and I ditch it, I'm off back to site5 who I haven't had reliability problems with.

Update: I thought it would be only reasonable to raise a support ticket about this: only problem being that their online support ticket system is down. Things are not looking good.

My head is starting to rule my heart: from Site5 I can get four times the disk space for less money per month ($16.95 vs $19.99), responsive technical support that has always been available, administration, backup... I lose root login and the ability to do whatever I like but I don't have the spare time to take advantage of that anyway.

I've moved this site to my Linode. I have a couple more months before my Site5 account runs out so I have time to move back if necessary. I moved the site as follows:

• Dump the sql:

  cd ~/www
  mysqldump -u <user> -p drupal > pb.sql
  

• Zip the useful stuff:

  zip -r pb cron.php database/ fail.html favicon.ico files/ images/ includes/
  index.php misc/ modules/ pb.sql robots.txt scripts/ sites/ themes/ tmp/ xmlrpc.php
  

  Will I ever learn to love tar?
• copy the zip to new site by running sftp on new site.
• unzip the zip file in /var/www
• load the database into mysql:

  mysqladmin create drupal -u <user> -p
  mysql -u <user> -p drupal < pb.sql
  

• go to domain registrar and change DNS address of site from site5 to linode. It only took three hours or so for this to propogate enough for me to access the linode via the petersblog.org name.

And if you are reading this then you are reading the site from the Linode as I haven't posted it on the old site. I can leave the old site floating for a while until the old address is flushed from all the DNS caches out there. I'll leave bisiand.me.uk pointing there until the account expires: unless of couse something happens to convince me to stick with Site5.

I have no problems with Site5, I would recommend them to anybody, it's just that Linode is more technically challenging and hence more interesting and fun.

Got a Linode and I've installed ubuntu on it. So far it seems really quick: ssh login is fast and responsive, better than site5. This may be because the server does not have many users yet or maybe because it's 8:30 on a sunday morning.

It is just like my Ubuntu box at work but 10x faster.

I'm paying monthly for the linode, just trying it for now. If I'm still using it in August I may ditch site5 as linode is far cooler.

Damn, have to go out

Linode looks interesting. It is a hosting service whereby you get a virtual linux box all of your own. It is on a server and it is shared with 40 other people but you get 64M of ram to yourself, 3G of disk space that you partition yourself as you see fit and a selection of linux distributions to choose from. You install linux, have root access and basically can install whatever software you like on it (even painful gentoo compilation). It is like having your own linux box out there on the web.

It can be used not just for a web server but ftp, mail, proxy, DNS server, backup server, you name it. It sounds more interesting than Site5 which gives plenty of power except there is no root access, cannot use wget to get packages, no compiler, two year old version of Python, no fastcgi or mod_python just slow cgi, etc etc. Linode costs $20 or £10.50 a month which is more expensive and the support would not extend to patching your kernel like Site5 would do. Then again, unlike a shared host, if some other tosser you share with uses up all the mysql connections with a flaky script your account does not suffer (happened for the second time to my knowledge this sunday).

I'd be tempted to go with this rather than renew my site5 deal.

This python code generates syntax highlighted python code in html format. I know about SilverCity but I want this for my Site5 account where I cannot install executable code. The code below was highlighted using the code itself: spooky.

It is a simplistic solution but it should not be confused by multiline strings, comment characters in strings etc. I started off by trying to use the ply python lex as a tokeniser and processing the tokens but that persisted in confusing multiline string characters with normal strings and while thinking about it I realised that I could live without it. I don't know how slow this is: if using it on a website with heavy traffic you will want to cache the output.

#
# Syntax Highlighting
#

import re
import cgi

# Regular expression rules for simple tokens
strStyles = (
    ('PUNC', re.compile( r'<<|>>|<=|>=|!=|==|[-+*|^~/%=<>\[\]{}(),.:]'), None),
    ('NUMBER', re.compile( r'0x[0-9a-fA-F]+|[+-]?\d+(.\d+)?([eE][+-]\d+)?|\d+'),
                            'color: red'),
    ('KEYWORD', re.compile( r'def|class|break|continue|del|exec|finally|pass|' +
                            r'print|raise|return|try|except|global|assert|lambda|' +
                            r'yield|for|while|if|elif|else|and|in|is|not|or|import|' +
                            r'from|True|False'), 'font-weight: bold'),
    ('MULTILINE', re.compile( r'r?u?(\'\'\'|""")'), 'color: darkred'),
    ('STRING', re.compile( r'r?u?\'(.*?)(?<!\\)\'|"(.*?)(?<!\\)"'), 'color: red'),
    ('IDENTIFIER', re.compile( r'[a-zA-Z_][a-zA-Z0-9_]*'), None),
    ('COMMENT', re.compile( r'\#.*\r?\n'), 'color: green; font-style: italic'),
    ('WHITESPACE', re.compile( r'[ \t\r\n]+'), None),

# if all else fails...
    ('UNKNOWN', re.compile( r'.'), None)
)

class Highlight:
    """
    Syntax highlight some python code.
    """
    def __init__( self):
        self.strOutput = []
        self.strSpanStyle = None

    def Highlight( self, strData):
        """
        Syntax highlight some python code.
        Returns html version of code.
        """

        i = 0
        strMultiline = ''

        #
        # While input is not exhausted...
        #
        while i < len(data):
            #
            # Compare current position with all possible display types.
            #
            for strTok, oRE, strStyle in strStyles:
                oMatch = oRE.match( data, i)
                if oMatch:
                    #
                    # Input matches this type.
                    #
                    strValue = cgi.escape( oMatch.group())
                    if strTok == 'MULTILINE':
                        #
                        # Multiline string token
                        #
                        if strMultiline == '':
                            #
                            # If not inside a multiline string then start one now.
                            #
                            self.ChangeStyle( strStyle)
                            self.strOutput.append( strValue)
                            #
                            # Remember you are in a string and remember how it was
                            # started (""" vs ''')
                            #
                            strMultiline = oMatch.group(1)
                        else:
                            #
                            # Multiline Token found within a multiline string
                            #
                            if oMatch.group() == strMultiline:
                                #
                                # Token is end of multiline so stop here.
                                #
                                self.strOutput.append( strMultiline)
                                strMultiline = ''

                            else:
                                #
                                # Not the same multiline token as started so just output it
                                #
                                self.strOutput.append( strValue)
                    else:
                        #
                        # Other token, not multiline
                        #
                        if strMultiline != '':
                            #
                            # In multiline mode so output the raw text of the token
                            #
                            self.strOutput.append( strValue)
                        else:
                            #
                            # Not in multiline mode so change display style as appropriate
                            # and output the text.
                            #
                            self.ChangeStyle( strStyle)
                            self.strOutput.append( strValue)
                    i += len( oMatch.group())
                    break
            else:
                #
                # Token not found so dump out raw text. This doesn't have to be bullet proof.
                #
                self.ChangeStyle( None)
                self.strOutput.append( data[i])
                i += 1

        #
        # Terminate any styles in use.
        #
        self.ChangeStyle( None)

        return "".join( self.strOutput)

    def ChangeStyle( self, strStyle):
        """
        Generate output to change from existing style to another style only.
        """

        #
        # Output minimal formatting code: only output anything is the style has
        # actually  changed.
        #
        if self.strSpanStyle != strStyle:
            if self.strSpanStyle != None:
                self.strOutput.append( '</span>')
            if strStyle != None:
                self.strOutput.append( '<span style="%s">' % strStyle)
            self.strSpanStyle = strStyle

Used like this:

import sys
data = open( sys.argv[0]).read()
strHighlighted = Highlight().Highlight( data)

print """<html>

<head>
<title>It works</title>
</head>
<body>
<pre>
%s
</pre>
</body>

</head>
""" % strHighlighted

Those nice Site5 people appear to have increased the storage on my hosting account from 1.5G to 3G without telling me or charging me.

This is the same as their latest hosting packages.

How very nice of them.

How to get Python CGI running on a Site5 hosting account:

• Example code, stored in a file in the ~/www/cgi-bin directory:

     1  #!/usr/bin/python
     2  # CGI test
     3  #
     4  import cgi
     5  import cgitb; cgitb.enable()
     6  
     7  strUser = 'Peter'
     8  
     9  #
    10  # Template of html output
    11  #
    12  strHtml = """Content-Type: text/html\n
    13  
    14  <html>
    15  <head>
    16      <title>
    17          This is so cool
    18      </title>
    19  </head>
    20  <body>
    21  <h1>
    22      Careful with that axe Eugene.
    23  </h1>
    24  Hello %s
    25  </body>
    26  </html>
    27  """ % strUser
  Toggle Line Numbers

• chmod the file 700

  chmod 700 FileName
  

The file name does not need the .py extension as it is running as a straight executable.

Putty is a simply great ssh client and works nicely with open-ssh, which is found in Ubuntu Linux, Site5 and just about everywhere.

A nice feature of ssh is the ability to generate a public key that can be used to log into a server without having to give a password, or as extra secutiry in addition to the password.

Here is a procedure for creating ssh keys that can be used in both open-ssh and putty:

• On windows, install the open-ssh package with Cygwin
• execute the command

  ssh-keygen -t ssh-dss
  

  to generate the dss key. You may need to create the directory ~/.ssh in Cygwin bash for this to work. This will create a file in this directory called id_dsa.pub
• use sftp/ssh to copy the id_dsa.pub file to your ssh server box. Put the contents of this file (which is one big long line) at the end of a file called ~/.ssh/authorised_keys2, adding it to any other keys that are already there.
• back on windows, execute the command 'puttygen', from the putty site.
• In putty gen, use file/load private key to load in the file ~/.ssh/id_dsa
• Choose 'save private key' and store it somewhere handy where putty can find it. You may be prompted to enter a passphrase. This is a password used in addition to the key when connecting to the server. If the passphrase is blank then you don't have to enter it, the connection will be automatic.
• Open putty and enter the details of the server you want to connect to (address etc)
• In the 'connection' settings, enter your login name in 'Auto-login username'.
• In Connection/SSH/Auth, in the box 'Private key file for authentication' load the putty private key file.
• Save this configuration so you don't have to do it again.
• Click 'open'

Your life won't be the same again.

Since I mentioned awstats on this blog I've been getting attempts to access the awstats.pl script on this site. awstats.pl is not accessable through this domain, it is provided by Site5 but I have to log in to netadmin to get to them.

Anyway, I had a quick search to see if there was a way to hack in via awstats and sure enough there is. The trick mentioned in this article is the one they are trying to get in with:

200.223.55.134 - - [11/Feb/2005:14:44:54 -0500] "GET
/stats/cgi-bin/awstats.pl?configdir=|echo%20;echo%20;id;echo%20;echo%20|
HTTP/1.0" 404 6186 "-" "Mozilla/4.0 (compatible; MSIE 6.0b; Windows NT 5.0)"

this is trying to execute the command id which shows the uid, gid and groups of the account it runs in. I guess this is probing for this vulnerability and seeing whether it gives root access.

The break-in attempts are coming from a variety of IPs, as is usual they are using proxys so there is no point trying to block them. They are getting 403s anyway, they aren't consuming much bandwidth.

Moral: keep an eye on your access logs, see what folk are up to.

I realised today that I can just set up the auto-forwarding in gmail to forward this email to my main gmail account so I don't have to go through the tedious process of logging out of one gmail account and logging into another.

I haven't tried Gmails new POP service yet. I only see that as a way to create my own email archive. Gmail's user interface is good enough, it's main shortcoming for me is not being able to simply paste pictures into emails, you have to mess around attaching them.

My Site5 account gives me unlimited email accounts or something but this is a simpler option. If I do decide to put up the raw mailto then it's gmail that will have to handle the spam.

If anyone reading this wants a gmail invite then just ask. I think they are so common these days that I doubt I'll get any takers.

A nod to this site for the email icon generator.

I never did get around to trying to install awstats. I've been using Statcounter but I fancied trying awstats with reverse DNS turned on. I can't do this on my Site5 host as they don't like reverse DNS. I didn't install it on Gentoo as that looked like big time hastle.

I realised today that installing awstats under Ubuntu should be as simple as installing the awstats package and it almost is. I can install it on my home server, download my Site5 access logs there and let awstats format them up.

Here are the steps I had to take to install it:

• Install awstats package
• Edit a file called /etc/awstats/awstats.hostname.conf where hostname is the hostname. Put something like this in it:

  LogFile="/var/log/apache/access.log"
  LogFormat=1
  DNSLookup=1
  DirData="/var/cache/awstats/"
  DirCgi="/cgi-bin"
  DirIcons="/icon"
  SiteDomain="hostname"
  AllowToUpdateStatsFromBrowser=1
  AllowFullYearView=3
  

• Make a directory called /var/cache and chmod it 777 so it can be used from the web server
• Copy icons to web directory:

  cp -r /usr/share/awstats/icon /var/www/icon
  

• Run this to update databases:

  /usr/lib/cgi-bin/awstats.pl -config=hostname -update
  

• In your web browser, go to the url:

  http://hostname/cgi-bin/awstats.pl?config=hostname
  

• Study the stats in quiet awe
• Edit crontab to update stats automatically every night:

  crontab -e
  0 1 * * * /usr/lib/cgi-bin/awstats.pl -config=hostname -update
  

Bisiand.me.uk is mine again! 123-reg got around to reading the fax I sent on Saturday and today I've been able to set up the nameservers to point to Site5.

Leave things to brew for a day or two and my old bisiand.me.uk visitors will be back to join the new crazy frog crowd.

This has made me happy.

I've been keeping an eye on my visitor logs to see how much my domain name problems have effected my traffic. According to Statcounter they had been climbing but yesterday there is a sudden dip. The Awstats logs provided by Site5 show no such dip.

I've seen a number of such dips in the Statcounter logs: their servers do not appear to be the most reliable. This is not a big complaint, I use them for free, more of a lamentation. Their professional service is too expensive for my simple ego brushing needs, $9 a month, but if I was paying that I would not want drop-outs approximately once a week.

The main advantage of Statcounter for me is that it counts visitors who have javascript enabled so it is essentially counting human beings rather than crawlers and referrer spam bots. It is also easy to set it up to ignore my own IP address. The Drupal statistics module does not have this feature but I could simply use phpmyadmin or another generic mysql database report generation tool to filter the drupal logs in any way I desire. The statistics module does list external referrers in reverse chronological order so it is useful for updating .htaccess referrer exclusion lists.