Peter's Blog

I have some ideas for the note-taking and organising tool I want. My thoughts have been inspired by Dave Winer's OPML editor.

I started blogging as a way of organising notes but that has gone in an awkward direction: I write mini-articles and this takes time. What I am thinking of is the ability to just chuck down a sentence or two, maybe a web link, and just store it. Associated with these jottings would be a few tags that would enable me to organise them. The jottings would not necessarily have titles, I don't want to waste time thinking of a title. The intention of all this is to be light-footed, nothing to slow down the process of using it: the main reason I don't take down as many notes as I should is that I can't be bothered. The OPML editor is close in terms of being able to bang down notes with no titles but I have yet to explore it's categorising/tagging facilities.

One way I could use the OPML editor would be to compose an OPML file full of each days notes. I don't want to mess with a different file for each thing I work on, I'd rather use categories/tags to do that. At the end of the day the OPML could be consolidated into Drupal or whatever for archiving/viewing/editing as ultimately OPML is not a database. Posting articles directly into Drupal is too laborious, especially over the net: I don't want this to take more than a second or so. Another advantage of storing the articles in Drupal is that I can use my wilki syntax to add syntax highlighting.

I have to think about this some more.

Update: from the OPML Editor notes:

The core purpose of this program is to create outlines and share them with other people, in various forms.

Got Motorola V547 back from 'repair' but it still keeps disconnecting during calls or fails to dial with a 'call failure'. As the problem is intermittent there is little hope in convincing O2 or their repairers that the phone has a problem so we have given up and she has gone back to a Nokia 6610. No camera!

The v547 isn't totally useless: it still reminds her to take her pill every day.

Here is an endorsement for the Drupal Admin Block Module.

This simply adds a block to the page to tell me if there are comments ready to be moderated. This saves me from going to the administration menu to scan the log or going to administration/comments/approval queue. This isn't the biggest chore in the world but the Admin Block module has removed it for me anyway. The block only shows itself to me (as administrator) if there are comments to approve.

I have hacked my comment module to refuse any comment that attempts to create a link: no http, www, whatever. By refuse I mean it fails to validate, it doesn't get as far as the approval queue. This has so far proved quite effective at eliminating comment spam. Real users can read the error message and be reassured that I will edit a link to make it work when I approve the post. Any captcha haters reading, this should even work for blind people.

I found one advantage of using ssh (e.g. putty) over vnc: no error bells. I was using a terminal under vnc and the bells were driving me mad. I disabled them in vim with:

:set visualbell t_vb=

but still got them on the bash command line. Tried various things from google to disable them at the linux end but nothing worked. Tried disabling the sounds in vnc (Control Panel/Sounds, VNC = None) but not joy.

Don't especially want to unplug the speaker so I went with the flow and used ssh.

Sudden thought: maybe I WAS going mad?

Servant Salamander: coming to rely on this. Pretty much destined to register it. Today I discovered that I can get it to calculate space used in subdirectorys and then send a report of directory names and sizes, sorted by size, to the clipboard. Try doing that with Windows Explorer. I've stopped using filezilla, Salamander is a useful sftp client. Things I do wish it could do:

• copy directly from one sftp site to another. Similarly, since you can browse straight into a zip or tgz file, it would be nice to be able to copy a file from within an archive on one window to other window showing an sftp site. The user interface spoils you, you start thinking these things are possible, you try them and are thwarted.
• create tar.gz files. I have actually learned the incantation for this, I am becoming such a command line junkie:

  tar cvfz blah *
  

  It can do zips and rars and stuff.
• It sometimes goes a bit crazy with the confirmation dialogs, sometimes giving two or three in a row. I don't have faith enough to disable them, combining them into one dialog would be nice (e.g. yes I want to copy, I want to copy it there, I want to overwrite everything).

Thunderbird: since I fixed the exchange server I have been using this instead of outlook and have not looked back. I use an IMAP connection to the Exchange server and it is working just fine. I used to have problems with it when my network connection was flakier (it kept asking me for my password) but since I installed XP on my pc the network has been ok (one of those software decay things on my old Win2k install). I think outlook was tainted by spambayes which was pretty lethargic in moving spam around (to get around outlook's flaky message filtering it does lots of sleeps to keep out of the way). So you go to look at a message and find that spambayes is still in the process of moving it to the junk folder. Thunderbirds integrated bayesian spam filter is working for me. Another thing I like about thunderbird, the flag thing that pops up and goes away again when new mail arrives. The outlook taskbar thing always used to get stuck on, even after deleting new messages, to the extent that I disabled it.

Servant Salamander is a File Manager for Windows.

At work our main file server is a Windows 2000 server holding 70G of useful, interesting and irreplacable information. It gets dayly incremental tape backups kept offsite, twice weekly full backups, monthly archved backups, all stored on tape. The backup server has no tape drive. Come the fateful day when the main file server dies the first job would be getting the tape drive running on the backup server and then restoring the 70G so that everyone can get on with their work.

Since disk space is cheap I decided to implement a nightly backup copy from the main server to the backup server over the network. If the main server goes down our biggest problem is remapping drive F on everyone's pc to a different share (a hangover from Netware days, nobody uses the UNC path). Once that is done all the files are ready to use, no having to restore anything.

A quick google revealed that Microsoft had a tool to do such a backup in the Windows Server 2003 resource kit called robocopy. This is a command line tool (like all the most useful tools) and is similar to xcopy but, most importantly, can copy file/folder ownership, access permissions etc. We wouldn't want the backup copies to be readable by everyone or no-one but whoever could read them before.

The Windows 2003 version of robocopy works on windows 2000 server but apparently has some bug fixes. I downloaded the Windows Server 2003 resource kit from the microsoft site and I had to install it on my XP system as it refused to install on the Win2k box.

Below is the simple batch file I wrote to do the backup:

rem options:
rem     XJ              Do not copy mount points or junctions
rem     RH:hhmm-hhmm    Specify time range that robocopy will work in
rem     PF              Check above RH time per file
rem     COPYALL         Copy owner ship, auditing, dates, attributes, all the good stuff
rem     MIR             Mirror, delete files that not longer exist in source, recurse, copy empty
rem     LOG:file        Log output to file
rem     TEE             Log to console as well as log file
rem     W:n             Wait time between retries (in seconds)
rem     R:n             Number of retries
rem  NP		Do not show (or log) progress information
rem  NFL		Do not show (or log) each file processed
rem   XD dir		Exclude directory
"c:\RoboBackup\Robocopy" f:\ e:\DriveFBackup\Yesterday /XJ /RH:1800-0730 /PF /COPYALL /Z /MIR
/LOG:c:\RoboBackup.log /TEE /W:1 /R:1 /NP /XD "tmp" "Archived Files"

rem NOT USED
rem     ZB              Copy in backup mode, then restartable mode, can continue on failure
rem                     (robocopy.doc says this slows things down so only use if necessary)
rem     CREATE          Create directory structure but do not copy files. This should be done
rem                     for any empty target to build the directory structure as it reduces
rem                     disk fragmentation.

rem Email the log file to peter.
"c:\RoboBackup\blat" c:\RoboBackup.log -to me@work.com -subject "RoboBackup Output"

This script does a 'mirror' backup, it copies new and modified files (based on time stamp and size) and also will delete files in the backup that have been deleted in the original. Hence, after the script runs the backup should be identical to the master.

The last line uses blat a neat little command line mailing tool to email me the log file. This mailing normally fails as this thing is quite capable of generating 60M log files.

I use a batch file holding this to create the scheduler job:

at 18:00 c:\RoboBackup\RoboBackup.bat

I then have to edit the scheduler task in the gui application to tell it to run as administrator and to give it the administrator password. In an ideal world I would run it as the backup user but to achieve that I would have to make sure the 39000 odd directories all have backup user access.

So there you go, let yourself sleep a little bit easier tonight, using free tools (and an expensive operating system).

New version of Konfabulator out today. They've fixed a bug where the weather display always showed the moon, which did puzzle me at a background level. Today there is a nice big sun:

The sun/cloud patterns change every hour or so, according to the weather, which is cool when there are no windows where you are sitting.

Bit chilly for August.

My Microblogging tool is coming together. This is my vision for a blogging tool that is:

• Fast to post with so posting is as lightweight as possible
• Supports tagging to organise the posts. I have decided that tagging is preferable to putting articles in a rigid tree hierarchy.

I decided on the following architecture:

• VIM as an editor. For me this is by far the quickest way to do data entry
• I create however many posts in an single page and post them in one go
• I use Vim's python scripting to send the file full of posts to the server via an xmlrpc call
• The server implements a python cgi script to process the file full of posts. It splits the files into seperate posts and submits them to Drupal one by one. Drupal is the backend database and is also a reporting tool.

Most of the processing is done by the cgi script. I could post directly from VIM to Drupal, post-by-post but the Drupal blogapi does not support awtags and I'd rather write some python to do it that struggle modifying the php code. The cgi script also supports reading back a days-worth of postings so I can also edit existing posts.

Using VIM I was able to create a nice syntax-highlighing definition to make the posts easy to edit. Here is an example of how it looks in VIM:

This is the title
    23 August 2005 12:34
    [some stuff]
    This is a posting in Vim. I can create stuff in VIM much faster than I
    can type it in textarea in a web browser.

    As posts support basic html markup, html tags are <i>syntax highlighted</i>.

    It will syntax-highlight python:
    <python>
    for i in range( 10):
        print "Python rox"
    </python>

    It shows blocks with a green mark in the first column.

    It will syntax-highlight php:
    <php>
    $strStr = "What is the dollar for?";
    </php>

    It marks out my verbatim blocks:
    <verbatim>
    This is preformatted.
    </verbatim>

The format of the posts is a title line followed by date/timestamp and a list of tags, followed by the post itself, indented to delineate it. For me the syntax highlighting is the killer feature, I am almost word-blind without syntax highlighting. I stole the idea fromwas inspired by The Vim Outliner.

I won't post all the code here as it's too big to post inline and it is highly likely that nobody else is interested anyway. It's not quite finished, I still have a few issues to resolve:

• VIM macros to help in posting, such as filling in the date.
• Support for multiple blogs: work blog, personal note blog and this blog. How to cross post?
• Any way to handle images/attachments?

I call it the Microblog, micro to emphasise the fine-grained nature: not big laborious articles (like this one) but short pithy notes.

Something is starting to really irritate me. Whenever I make some change on this site through Drupal such as editing a page, I press the submit button and the browser shows me the page as it was before the edit. I have to press refresh to see the changes. This happens all over, if I look at the logs, add a page, look again at the logs, nothing has changed until I refresh the page.

Clues to the cause:

• I have caching disabled in Drupal and it should not be caching updated pages anyway.
• My browser is connected directly, no caching proxy.
• If I run ethereal I can see that the browser is sending a GET for the updated page and receiving the old version
• If I look at the site apache access logs I do not see this GET reaching the server

Something in between is caching the pages. J'accuse Site5. I've looked all through their site and I see nothing about this but the forums tend to be full of noobs (like most forums). I tried editing the .htaccess file to disable mod_cache if it was loaded but that just caused an internal server error. It is highly likely that they are running some form of cache the other side of apache (squid?), I'd probably do the same to save the load on a shared server for other peoples sites. However, I find it intrusive when I'm the victim.

Hosting contract is coming to an end, I still have time to evaluate a new hosting solution. I'm highly tempted by a cheap dedicated server. It's expensive but:

• I've plenty of power: I can run X and use a vnc terminal.
• I can do what I like in python. I have no interest in doing php at all and Site5's python support is weak (python 2.2.3, no generators ). They now support ruby on rails but I'm more interested in php than ruby (weak python clone).
• I learn to secure a linux box properly. I think my linode server was hacked through an xmlrpc weakness in drupal which has been fixed now.
• It's all mine. With VPS's and shared hosting, you are having to share disk and cpu with other people. I would only have to share network bandwidth with them.
• I can sell CPU time/hosting/web sites should I feel the desire

One thing about dedicated servers: I cannot believe that to get the server rebooted you have to email some guy who has to run to the box and press the reset button. How primitive is that? If I screw things up to the extent that shutdown -r now does not work I will either learn to prototype hacks on a local box or give up and become an estate agent.

UPDATE: er, it wasn't site5's fault, it was Microsofts.