Peter's Blog

Something is starting to really irritate me. Whenever I make some change on this site through Drupal such as editing a page, I press the submit button and the browser shows me the page as it was before the edit. I have to press refresh to see the changes. This happens all over, if I look at the logs, add a page, look again at the logs, nothing has changed until I refresh the page.

Clues to the cause:

• I have caching disabled in Drupal and it should not be caching updated pages anyway.
• My browser is connected directly, no caching proxy.
• If I run ethereal I can see that the browser is sending a GET for the updated page and receiving the old version
• If I look at the site apache access logs I do not see this GET reaching the server

Something in between is caching the pages. J'accuse Site5. I've looked all through their site and I see nothing about this but the forums tend to be full of noobs (like most forums). I tried editing the .htaccess file to disable mod_cache if it was loaded but that just caused an internal server error. It is highly likely that they are running some form of cache the other side of apache (squid?), I'd probably do the same to save the load on a shared server for other peoples sites. However, I find it intrusive when I'm the victim.

Hosting contract is coming to an end, I still have time to evaluate a new hosting solution. I'm highly tempted by a cheap dedicated server. It's expensive but:

• I've plenty of power: I can run X and use a vnc terminal.
• I can do what I like in python. I have no interest in doing php at all and Site5's python support is weak (python 2.2.3, no generators ). They now support ruby on rails but I'm more interested in php than ruby (weak python clone).
• I learn to secure a linux box properly. I think my linode server was hacked through an xmlrpc weakness in drupal which has been fixed now.
• It's all mine. With VPS's and shared hosting, you are having to share disk and cpu with other people. I would only have to share network bandwidth with them.
• I can sell CPU time/hosting/web sites should I feel the desire

One thing about dedicated servers: I cannot believe that to get the server rebooted you have to email some guy who has to run to the box and press the reset button. How primitive is that? If I screw things up to the extent that shutdown -r now does not work I will either learn to prototype hacks on a local box or give up and become an estate agent.

UPDATE: er, it wasn't site5's fault, it was Microsofts.

A google for 'peter blog' brings www.petersblog.org/blog/1 as result number 3! Out of 8,730,000! This url has a page rank of 4, even though www.petersblog.org is still a 3. This may explain why I've been getting a record number of visitors. Neither page has a significant number of backlinks so I'm surprised I have any page rank at all. Methinks pagerank isn't so important any more.

This site appears in the first page of results for Site5 and Linode and both of these go directly to the corresponding tag page. I have a theory that google is looking through the tag page, seeing the word 'linode' or 'site5' many times and assuming that these pages are highly relevant (which of course they are).

Moral: Search engine placings: it's all in the keywords.

n.b. changing from linode back to site5 hasn't hurt visitor numbers a bit, even though I changed www.bisiamd.me.uk to a simple redirect page instead of an alias of petersblog.org.

Update: this page has PR 5!

This morning I got two emails from Linode, this one:

We're currently receiving a large number of UDP packets coming from your machine:

01:33:33.136745 70.85.129.118 > 24.110.229.206: udp (frag 57902:1480@1480+)
01:33:33.136871 70.85.129.118 > 24.110.229.206: udp (frag 57902:1480@2960+)
01:33:33.136873 70.85.129.118 > 24.110.229.206: udp (frag 57902:1480@4440+)

I've blocked the offending traffic for now. Please reply within 48 hours to avoid service interruption.

and this one:

More DoS traffic:

03:30:42.548612 70.85.129.118 > 194.146.152.98 : udp (frag 56782:1480@2960+)
03:30:42.548617 70.85.129.118 > 194.146.152.98 : udp (frag 56782:1480@4440+)
03:30:42.548874 70.85.129.118 > 194.146.152.98 : udp (frag 56782:1480@7400+)

Your Linode's networking has been disabled.

So my Linodes network access was gone and the site was down. I could log in through the lish console and this showed two tasks called 'bashex' owned by the www-data user i.e. apache taking most of the cpu time. I guess someone had hacked in and were up to no good.

Well, I have been pondering what to do about the Linode, in one way I love it but in another way I am not getting the uptime I would like. Ok, this security breech is MY problem, I didn't secure the server properly but then again, I don't want to spend my valuble spare time securing servers, I have more interesting things to do. Together with silly power problems in their racks, I have had more downtime in a month with Linode that I have in 10 or so months with Site5.

I don't know how long it would take me to secure the linode now it has been breeched, I would want only feel it was clean if I reinstalled everything, and then I would have to go into adding extra security.

I decided instead that no, Linode is not for me, I'll go back to Site5. There are a number of reasons for this:

• someone else does system admin and worries about backups and security and this is included in the price.
• I can afford a top-of-the-line shared hosting package for less than a bottom-of-the-line linode. This gives me, for example, four times as much disk space.
• Site5 have many tech support people ready 24 hours a day and pride themselves on 15 minute response times. Linode only appear to have 'caker' who is good at his job but presumably needs to sleep.

Disadvantages:

• I am not root, I am not free to install what I like.
• The python on the site5 server is version 2.2 which is too old to be useful to me.
• I cannot run any long-lived processes such as a python-based web server.
• It is not as cool

As my networking had been disconnected, my only access to the linode was via the lish console. I got all my data from my web sites off it as follows:

• cd /var/www
• mysqldump -u -p >petersblog.sql
• tar xvvzf www.tar.gz * .htaccess
• uuencode www.tar.gz www.tag.gz

This dumped the tar file as Ascii. By logging all console output to a log file I captured it all on my local pc. I uploaded it to site5 and

• uudecode putty.log
• tar xvfz www.tar.gz
• reload sql and away
• make sure drupal is up to latest version (once bitten...)

I had to change dns servers again and that took a few hours to propogate but this posting was done on Site5.

Now to tell Linode the bad news...

Linode running sooo sweetly today, have to love it. Still don't totally trust it though. I've put this in my crontab:

0 1 * * * uptime | mail -s uptime <email address>

so it can email me it's uptime every day.

$ uptime
15:37:27 up 6 days, 54 min,  1 user,  load average: 0.00, 0.01, 0.00

Linode has been up for a week, looking ok after last weeks problems. Their support site is back online.

Looking around the forums I got some insight into the business model. Linode apparently rent dedicated servers from ThePlanet, splits them into 40 odd virtual servers and sells these on for $20 a month or more. Given that the server costs $250 a month from ThePlanet and can bring in $800 a month, that's a nice little earner.

Apparently some of the Linode site is hosted in the same racks and went down with mine.

Linode hosting this blog went down for about five hours and I couldn't log into the linode site to reboot it. Once I could log in I had to manually reboot my server to get it up again. This is the third time this has happened this week (twice monday) but the other two times I could at least log into the linode console and reboot. This is really putting me off linode, I give it one more failure like this and I ditch it, I'm off back to site5 who I haven't had reliability problems with.

Update: I thought it would be only reasonable to raise a support ticket about this: only problem being that their online support ticket system is down. Things are not looking good.

My head is starting to rule my heart: from Site5 I can get four times the disk space for less money per month ($16.95 vs $19.99), responsive technical support that has always been available, administration, backup... I lose root login and the ability to do whatever I like but I don't have the spare time to take advantage of that anyway.

Could not access this website, nor ssh into it, so decided to reboot the Linode that hosts it. It took a few minutes for the reboot to complete but this resolved the problem. Either:

• Linode sufferred a glitch
• My site sufferred a glitch

It's slightly worrying as Site5 never did this.

UPDATE: happened again. According to the Linode forums, the host my site is on (39) suffered from a power outage sometime sunday.

My statcounter stats indicate that the site has been down for some time, the number of hits is down.

On examination my linode was not dead but was running very slowly. I rebooted it again and it became responsive, however this site would not come up, I got a totally blank page. The apache2 error log said:

[client 1.2.3.4] PHP Fatal error:  Can't open file: 'sessions.MYI'. (errno: 145)\nquery:
SELECT u.*, s.* FROM users u INNER JOIN sessions s ON u.uid = s.uid WHERE s.sid =
'61a28f0a34b8ebb9f26304c033219df0' AND u.status < 3 LIMIT 0, 1 in
/var/www/includes/database.mysql.inc on line 66

trying a command line query on the sessions table confirmed that mysql couldn't read this table. Corrupt? I had a quick google for repair options and very quickly came across the following sql command:

repair table sessions;

This did the trick and my site was back up again.

So did my reboot corrupt the sessions table, did the power outage do it, was the slowness related to this and in what way... so many questions. The bottom line is, do I trust Linode any more?

I put postfix on my linode for mail handling. The installation went smoothly enough, loading the ubuntu package and editing /etc/postfix/main.cf to set up the domain names to be accepted. However, I had trouble receiving mail from outside. Trying to telnet in on port 25 told me that the smtp daemon was not listening.

The solution to this was to edit the /etc/postfix/master.cf file and change the lines:

127.0.0.1:smtp inet n   -       -       -       -       smtpd
::1:smtp       inet n   -       -       -       -       smtpd

to

smtp      inet  n       -       -       -       -       smtpd

which tells it to listen to any connection to the smtp port, not just local connections.

According to this site I haven't put up another open relay. (note to self: publish open relay test, lie about results and build nice list of open relays to sell to spammers).

I've moved this site to my Linode. I have a couple more months before my Site5 account runs out so I have time to move back if necessary. I moved the site as follows:

• Dump the sql:

  cd ~/www
  mysqldump -u <user> -p drupal > pb.sql
  

• Zip the useful stuff:

  zip -r pb cron.php database/ fail.html favicon.ico files/ images/ includes/
  index.php misc/ modules/ pb.sql robots.txt scripts/ sites/ themes/ tmp/ xmlrpc.php
  

  Will I ever learn to love tar?
• copy the zip to new site by running sftp on new site.
• unzip the zip file in /var/www
• load the database into mysql:

  mysqladmin create drupal -u <user> -p
  mysql -u <user> -p drupal < pb.sql
  

• go to domain registrar and change DNS address of site from site5 to linode. It only took three hours or so for this to propogate enough for me to access the linode via the petersblog.org name.

And if you are reading this then you are reading the site from the Linode as I haven't posted it on the old site. I can leave the old site floating for a while until the old address is flushed from all the DNS caches out there. I'll leave bisiand.me.uk pointing there until the account expires: unless of couse something happens to convince me to stick with Site5.

I have no problems with Site5, I would recommend them to anybody, it's just that Linode is more technically challenging and hence more interesting and fun.

Looking through the Linode control panel options I discovered that it is possible to request a 2.6 kernel instead of the standard 2.4. I tried this and it indeed booted up but ssh fails to work. I can still log in through the linode console which appears to emulate a tty console so you can watch the system boot and log in that way but if I try to ssh the connection gets suddenly closed with no clues as to why, not even in the sshd log wth full debugging enabled.

There is mention of some odd devpts thing in kernel 2.6 being to blame for these ssh problems but I haven't pinned it down yet.

I resized the disk space on my Linode to free up 400M for a debian linux install, to use to rescue the main ubuntu install if I ever break it. You can change the size of your partitions without losing all your data if you are using ext2 or ext3. Cool.

Linode is looking good.