Peter's Blog

After years of messing with backup scripts I have finally set up a decent backup system, courtesy of backuppc. Backuppc is a set of backup scripts too but much better than I could have knocked up.

Backuppc runs on a server and backs up client pcs over a network. It doesn't need any client software but supports various kinds of net file shares so it can, for example, backup windows pcs through smb file shares. The administration program is web based so you can run it anywhere.

It runs as a server pinging the clients once every hour. If it finds a client is running and if it hasn't been backed up then it runs a full or incremental backup according to it's schedule.

The files from a number of pcs are copied into a pool so that multiple gigabytes of windows bloat files only get stored once so it becomes quite efficient on space.

For each pc you can browse the daily backup archives going back for months. You can select a directory to restore and download a tar or zip file of the contents of the directory.

I've got it backing up all my systems:

• web sites including photos and git repositories
• personal documents, including scans
• irreplacable family photos
• iTunes music collection including all my rips
• /etc directory of all my Linux boxes
• loads of crap from /home/peter that I can't be bothered to filter out: better to accidently back up too much that to accidently filter out something irreplacable.

This is for:

• the home server box where backuppc runs backing up itself
• dedicated server over then VPN
• home desktop

It all adds up to 22gigs. I then rsync this archive to a second disk so my backups are disk-proof. I may dump it to a USB disk that can be grabbed if the house burns down (offsite backup isn't feasible).

Backuppc doesn't support tape backup which suits me as I don't have a tape drive. It can't do 'bare metal' restores but I'd rather reinstall the os and applications after that kind of failure, backuppc is backing up the data which is where my time is invested.

The backup pools include images of the original directories that wet backed up, albeit with an f at the start of every file name. I like this, it's not as if the files are locked inside some proprietary black box.

There are other free network backup systems such as amanda and bacula but they seemed more tape orientated and complex than backuppc.

I have my clients running rsync daemons so after the first backup the updates, both full and incremental take a minute or two for gigs of data. Since I run Linux I don't notice if my pc is being backed up while I am using it.

It's something I should have set up sooner as I really feel as if for the first time I have proper backups in place.

I set up a cron job to backup my web sites to my Mythical Convergence Box, just a simple rsync of the subversion repository and, to be safe, /var/www. The latter was 500Mbytes.

Next morning the cron job had run and my server was inaccessable. Oops, with 10g of WoW and 10g of family photos I had filled the main disk up, my 250g Myth disk being mounted on /var/lib/mythtv. The box hadn't crashed but it was in a kind of coma, you ask it to do something and it went urgh.

I had to reboot into recovery mode and move the photos to the mythtv disk, leaving me 68%. Observations:

• Interesting how hard the system failed when the disk filled up. Have to be careful.
• Linux recovery mode is actually useful! Whenever I've tried the (admittedly pre-vista) Windows recovery modes it has spent a quarter of an hour loading drivers for obscure disk controllers, then I've tried the automated recovery that never works, then I've rebooted, I've waited another quarter hour, then it's given me something with the raw power of a 1985 dos prompt, then I've given up and reformatted the disk.

Last night my cron job worked fine and my websites are so thoroughly backed up that my head is spinning.

I had my rsync scripts set up and running. I had backup and get scripts so I could backup my work to a server, go home, download the changes, do some work, upload the changes, download them again at work. Who needs a flash drive?

However there's this thing called 'finger trouble' that happens sometimes: you forget to backup the changes on one machine or another and suddenly, bam, your work vanishes. At least that is what I think happened, all I know is that twice now I have lost some work. Running backup and get scripts to synchronise two pc's against a server is too error prone for me.

So I set up unison which, once running, should be a less error prone solution. It will synchronise the files on my home and work pc's with the files on the server. Changes to the files may be propogated either way, depending on the time stamps. I can change the files on either pc and unison will help me resolve any conflicts I create. While it is synchronising it will prompt me with each file that has changed and ask me what to do with it so I can see if I am about to overwrite an evening's work.

I have the following unison profile file set up in my .unison directories on both pc's:

#
# Unison profile for project X
#
root = /cygdrive/c/projects/X
root = ssh://me@myserver.com//home/pcw/X

ignore = Name Downloads/*
ignore = Name Backup/*
ignore = Name Backups/*
ignore = Name Tmp/*
ignore = Name *.map
ignore = Name *~
ignore = Name *.obj
ignore = Name *.o
ignore = Name *.i

backup = Name *
backupdir = f:\users\pcw\BACKUP\X

If the profile file is called X.prf then I can synchronise this project by running

unison X

Because the profile is in my private .unison directory I don't even need to cd to the project directory.

The 'backup' and 'backupdir' lines at the end of the profile tell unison to maintain an extra local backup, in this case on our main file server such that the backups fall under our company tape backup procedure.

It has worked for a day now with fewer problems than rsync gave (and ironically unison uses rsync in it's underpinnings) and it is fast, taking less than a minute to work on 491 files.

To maintain straight backups unison would be overkill but where changes could be made to the files in more than one place it is looking good.

Of course, I could use unison with a flash drive rather than a file server but I don't have to remember to carry the server around with me.

And I know about synctoy but unison works easily through an ssh tunnel.

Another backing up setup is under construction. This time I'm backing up my project developed on a windows box to a linux server using rsync. It is using ruby as a scripting language. If this script is invoked thus:

ruby Backup.rb

then this script will merely update a simple backup of the source directory on the server. rsync is very fast and will only spend time uploading files that have actually changed and also it will only upload the changes to those files. I am using this on a 30M source archive and it takes less than a minute to update the backup. The backup files are just copies of the original files so easy to browse, diff, restore.

If the script is invoked thus:

ruby Backup.rb --backup

then it will use rsync's magic link-dest option to create historical archive of backups. What the script will do is this:

• Move the current main backup to an archive directory and give that archive directory a name that is date/time stamped.
• Create a new backup. Where files have not changed from the previous backup, the new backup will contain a link to the existing copy of the file in the archive, rather than consume more file space. The new backup directory will only contain files that are new or have been changed.

There is nothing in the script to limit the number of backups but they can be manually pruned every few months as required.

This is the script:

   1  #
   2  # This script is used to backup the source code to an offsite server
   3  #
   4  # If the parameter '--backup' is provided then create a backup directory
   5  #
   6  strMeAtMine = "me@myserver.org"
   7  strLocalFolder = "/cygdrive/c/projects/757/"
   8  strTargetFolder = "/home/pcw/757"
   9  strBackupFolder = "/home/pcw/Backup/757"
  10  
  11  #
  12  # If asked to backup then move previous backup to a backup directory
  13  # given a name related to date/time.
  14  #
  15  if ARGV.index( '--backup')
  16    bBackup = true
  17  
  18    strTimeStamp = Time.now.strftime( '%Y-%m-%d-%H-%M')
  19  
  20    strBackupDir = "#{strBackupFolder}/#{strTimeStamp}"
  21  
  22    strCommand = "ssh #{strMeAtMine} mv #{strTargetFolder} #{strBackupDir}"
  23  
  24    system( strCommand)
  25  else
  26    bBackup = false
  27  end
  28  
  29  #
  30  # Determine location of rsync exluded file list
  31  #
  32  strDir = File.dirname( __FILE__)
  33  strExcludeFile = File.join( strDir, "rsync-exclude.txt")
  34  
  35  #
  36  # Build command to invoke in an array of strings since this allows me to comment what
  37  # each parameter does.
  38  #
  39  strCommand = [ "rsync",
  40  #  	"-n",                                 # -n = dry run
  41    	"-v",                                 # verbose
  42                  "-a",                                 # archiving options
  43                  "--delete",                           # delete files no longer used from target
  44                  "--chmod=u=rwX",                      # set target file permissions
  45                  "--exclude-from=#{strExcludeFile}",   # exclude rubbish
  46                  "--delete-excluded",                  # delete excluded files from target
  47                  "-e ssh",                             # use ssh tunnel
  48                  strLocalFolder,                       # source
  49                  "#{strMeAtMine}:\"#{strTargetFolder}\"" # target
  50              ]
  51  
  52  if bBackup
  53    #
  54    # Backing up so instead of uploading everything again, link to files in the
  55    # backup directory where there are no changes.
  56    #
  57    strCommand.insert( -3, "--link-dest=#{strBackupDir}")
  58  end
  59  
  60  system( strCommand.join(" "))
Toggle Line Numbers

Since this is a backup of files from a windows box to linux I used the "--chmod=u=rwX" to specify simple file permissions on the linux end. Without this rsync was tending to create files with no access permissions for anyone.

The 'rsync-exclude.txt' file lives in the same directory as the script and the script uses some magic to find it. This is a list of stiff that doesn't need backing up:

/Downloads
*.map
Backup/*
Backups/*
*~

This uses the cygwin version of rsync which uses ssh to talk to the remote server. I have ssh set up with key files so I don't need to enter passwords.

Vista has a new and improved backup facility. This is what Microsoft has to say:

"The file backup feature in Windows Vista is targeted at the core consumer audience. To savvy users, it will appear quite simple, probably too simple for you to use. For most consumers, however, it will hopefully be the right balance of complexity and functionality to get them backing up their files, which is something virtually none of them did in Windows XP, no matter how capable the tool was. For core consumers, we wanted to eliminate the confusion caused by selecting individual folders and files from a long tree of checkboxes, trying to find or guess where other users kept their files, predicting where files would be stored in the future, etc. These choices added too much complexity to the backup process and discouraged users from using NTBackup."

I tried to use it, clicked ok in the UAP box and got this:

I'm not sure where this puts me in the demographics. I'm savvy enough to see that it doesn't look that simple to get it running and as a normal consumer I'd say the available functionality won't get me backing up my files.

Following my ruminations on online backup I'm giving mozy a try with the 2G free space plan. I selected 800M of stuff to backup and it's still working on uploading it. The signup wasn't problem free with one or two timeouts and the setup program was disdainful about my upload speed, saying it wasn't good enough but allowing me to proceed anyway. It will take a few days to backup everything. The backup seems to be interrupted whenever I use the computer (like Google Desktop Search) probably because Windows is so poor at multitasking disk intensive applications. How often do I stop using the computer for more than a minute? Not very often it seems (is it coffee time yet?).

There is a web-based interface that will let me restore files and would presumably let me download them to wherever I happened to be. It doesn't seem to let you view the files online in your browser which would be very handy, especially for pictures, you can only download the files. Overall it is not as flexible as xdrive promises but it could be handy one day.

Mozy adds a 'Mozy Remote Backup' virtual drive to windows explorer but for me it refuses to open: I click on it but the selection bounces away in buggy fashion. I presume it is supposed to show the folders and files I have backed up and allow me to restore them with drag and drop.

Maybe I should bite the AOL bullet and try xdrive. It would be more flexible even if backup was not totally automatic and didn't offer file versioning. It also gives 5G of space for free but I just cannot help feeling that AOL don't give anything away without a catch. Will they start emailing me .iso disk images?

What I should really be concentrating on is having something like Mozy to backup everyday work in progress and a source control system such as Subversion to handle the versioning of project files. That is the best option. The source control system only holds working code, I don't believe in checking half-finished rubbish in every day just so it is backed up. The backup system is there to save your behind if your hard disk goes down or you accidently delete a file.

Researching online backup solutions: really must backup family photos properly. Options so far:

mozy: 2G free or $4.95 a month unlimited. Automated backup and offers file versioning for a month which would be handy. Files accessable from web.

carbonite: a different shade of mozy, horses for courses. Trial account but no long term free account.

xdrive: 5G free which is enough. Creates a virtual drive which you copy files to however you like or run their backup software. Web based access to files from anywhere, file sharing, lots of nice features. What puts me off? It's owned by AOL

flickr: could just upload multiple G of photos to flickr but since they decided to force me to get a yahoo account I haven't been using flickr (yahoo or AOL, I don't want the hassle). Also the upload wouldn't be as totally automatic as mozy or carbonite.

site5: I could cobble something together on my site5 hosting account: I have about 14G free and ssh access so I should be able to do something but it will be a lashup that I would have to nurse along.

Being a dedicated backup solutions, Mozy or Carbonite would almost force me to do backups and do them well. With anything that looked like another hard disk I would end up with yet another disorganised mess. Less flexibility than xdrive could be a good thing. And by avoiding AOL and yahoo I stick it to "the man".

Since plugging Cobian Backup last week I have noticed that it leaves two processes running all the time, committing about 15M of memory, 7M of it for a UI to support a status icon, the rest for the backup service. Since the backup only runs once a day for a minute or so I don't see the point of committing resources all the time: the Windows scheduler service is running all the time anyway, so why run another service to do backups?

I've written a new batch file to do a rolling backup using 7zip, a very nice compression package:

rem
rem yet  another backup script
rem
erase "%2\Backup9.7z"
move "%2\Backup8.7z" "%2\Backup9.7z"
move "%2\Backup7.7z" "%2\Backup8.7z"
move "%2\Backup6.7z" "%2\Backup7.7z"
move "%2\Backup5.7z" "%2\Backup6.7z"
move "%2\Backup4.7z" "%2\Backup5.7z"
move "%2\Backup3.7z" "%2\Backup4.7z"
move "%2\Backup2.7z" "%2\Backup3.7z"
move "%2\Backup.7z" "%2\Backup2.7z"

"c:\program files\7-zip\7z" a -r -x@"c:\bin\exclude.txt" "%2\Backup.7z" "%1\*

Exclude.txt is a list of file extensions that should not be backed up.

A wrapper batch file is created to call the above for each backup source:

c:\bin\backup c:\desktop g:\backup\desktop

and this wrapper is run from the Windows scheduler.

I am using 7zip because of the way it allows a source directory to be specified: I'm not sure regular info-zip supports this so cleanly.

7Zip native zip format creates smaller archive files than regular zip (albeit more slowly) and yet salamander still allows me to view them like regular directories so for me there is no practical difference between zip and 7zip. 7zip can be told to create regular zip files is need be.

7Zip supports incremental backups and if my backup files became very large I would consider using it but the external hard disk that I backup to is 80G and has space aplenty so there isn't much to be gained from this.

Found Cobian Backup a nice windows tool for doing automated backups. What is nice about it is that it backs up to zip files so backups are nicely compressed in a non-proprietary format and restoring then does not need any special tools: I can browse the archives from salamander, view them and copy them individually or on mass. It can be set up to do incremental backups every day, a full backup every week and to keep backups of every backup so you have some kind of history.

I am using it simply to archive my work in progress: the computer I am using dies randomly once or twice a week and I don't trust it (it flashes a blue screen and reboots).

I set up the scheduler to do the backup every lunch time, backing up my desktop (where I chuck everything these days) to an external USB hard disk.

The only bugbear I have with it is that it is tedious to get it to exclude files by extension: you have to add them one at a time (click add, enter extension, ok, click add..) and with VC development there are many extensions that can be excluded from backup: my typical list of exclusions is:

• *.obj
• *.sbr
• *.bsc
• *.zip
• *.map
• *.wsp
• *.pdb
• *.pch
• *.tmp
• *.res
• *.ilk
• *.exp
• *.idb
• *.ncb
• *.opt
• *.plg

It seems to store it's configurations in C:\Program Files\Cobian Backup 7\DB\MainList.lst which is a text file so I edited it in Vim, rebooted Cobian and job done.

Many quality windows freeware tools listed here and here.

Another good source of free stuff is this.

I've been plying with unison recently, as a means of replicating my collection of 3000 photos between two computers: I nearly lost all the family photos once before, and I have been neglecting backups. With my new strategy of having the dell D410 fire up in it's docking station every night and doing backups, it can also synchronise the photo collection with the 500m which runs all the time.

The advantage of unison over rsync is that it synchronises a pair of directories, i.e. if you add, edit or delete files on one copy then the other copy will be updated accordingly. This is a bi-directional thing, changes made to the second copy will be replicated in the first. In the case of my photo's, I always use the 500m to copy them from the camera but I am likely to use the D410 to manipulate them and what I want is changes in one archive to be fed into the other.

The invocation of unison that I have come up with so far is:

unison -batch -fastcheck true <path1> <path2>

-batch

tells unison not to ask me stupid questions, it will reconcile problems automatically where it can. -force can be used toi resolve conflicts in favour of one of the paths.

-fastcheck true

tells it to compare files by timestamp and length rather than a full binary compare, as is the default under windows.

I have been using a native windows build of unison rather than the cygwin version, mainly because the windows version handles windows file names transparently, including network file names. cygwin unison via ssh does not seem to be able to find any of the file paths I give it (e.g. /cygdrive/c/blah, the cygwin form of c:\blah) and doesn't like network path names (e.g. \\server\share).

Still, it is not working properly for my photo collection but it may be because the network is not reliable enough to handle 3000 photos and 3G worth of data with absolutely no network problems. This may be another sign that the Wifi on the 500m is iffy (iffy wifi ha!) as the d410 in it's docking station has wired networking available.

Have I mentioned recently that the d410 is really nice? The 500m screen seems huge when I go back to it but the d410 is so fast, small and light.

What with it being the Christmas holidays I found a spare couple of hours to set up the Docking Station for my Dell D410 Laptop. I set it up thusly:

• 17" CRT monitor
• Packard bell keyboard
• Packard bell wireless mouse
• USB 2 Hub
• External USB hard Drive
• Wired network so I can play with wake-on-lan and access it from the internet

I need to buy a long audio cable so I can plug it into the hifi.

It makes a nice system, the Dell is faster than the desktop. I still use the desktop system for the following:

• Canon USB Scanner
• Nightly backup of remote server

I could do my scanning on the laptop and I could set it up to do the nightly server backup as well. Since I set the desktop machine up for nightly backup I haven't got around to getting it to shut down automatically so one of my bedtime tchores is to turn it off. Once that is done I don't need the desktop...

I would like to set the laptop up for a nightly backup to the external hard disk: turn itself on at 3am, backup, then switch itself off. I'd like to figure out the most efficient way to do that: weekly full, and daily incremental maybe. This could be combined with the remote server backup.

The only disadvantage of a laptop over a desktop for me is that it has no PCI slots: the only PCI slot I could use is my TV card and I only need that to capture my wedding video (2 years, 3 months, still haven't got around to doing that). I'm not into games now (did I mention I'm married?) so I'm not inclined to upgrade the graphics: I'd buy an XBox 360 for that.

For Christmas I also treated myself to a Toshiba 160G external USB hard disk. I decided against Freecom as the one I use at work isn't the greatest quality and I figured a toshiba would be nicer, and I was right, it is a nice little thing, good build quality. So far it has Just Worked and when video editing with my Dell D410 laptop it was certainly fast enough (7200rpm spin speed, faster than the Dell's 5400rpm internal disk).

The video I was working with was 13G long in it's raw .avi format and I only have 20G free in the partition on the laptop, hence I used the external disk. This also had the advantage that I could plug the disk into my desktop system and use that to blow a video CD (Pinnacle Studio doesn't recognise the CD in the laptop). It is running as I type but the rendering is very slow: looks like a frame every five seconds or so. The Dell ran much faster when generating an mpeg to the Toshiba hard disk, maybe five times faster.

I didn't realise how fast the Dell D410 was, or how slow the Desktop is.

I might set up the D410 docking station to use it as a desktop PC and automate backups to the Toshiba which came with some backup software that I haven't played with yet.

Backing up linux ubuntu intranet stuff to central windows server using rsync and smbfs. I did this as follows:

• Create new windows user with limited priviledges, apart from writing backup files. The intranet server will have permanent access to the windows server and access priviledges should not be too generous.
• Install smbfs:

  sudo apt-get install smbfs
  

• Add following to /etc/fstab

  //{server name}/{share name} /mnt/{target} smbfs rw,username={windows username}, \
            password={windows password},uid={linux username},gid={linux username} 0 0
  

  uid and gid are necessary to get write permission.
• Mount share:

  sudo mount /mnt/{target}
  

• Create a backup directory on the windows server. Since it is only a backup I made this compressed.
• Add a cron entry to copy stuff over every night:

  34 3 * * 1-5 rsync -a --delete --exclude="\\.*" \
            /home/pcw/StuffToBackup/ /mnt/{target}/users/PCW/BACKUP
  

  Windows does not like file names that start with a . so I exclude these.

I needed to set up an automated backup of a web site via ftp. I would have used rsync but that does not seem to support ftp. Looking through the ftp clients on ubuntu I found lftp which can be driven from the command line (and hence cron) and has a 'mirror' command that will replicate the files and directories on an ftp site.

It took some fiddling to find a command line that worked: the -c option to run a command string only seems to work if it is the only option. Anyway, here is a one-line lftp recipe (why is it so hard to put examples in man pages?):

lftp -c 'open ftpsite.com; user fred fredpassword; mirror --delete -p -X _*/*'

This does the following:

• -c runs the command string
• open ftpsite.com opens connection to the site
• user fred fredpassword logs in as fred with fredpassword
• mirror replicates the ftp site locally: default is root of ftp site to current directory
• --delete means delete files on backup if they have been deleted on the source
• -p means do not copy file permissions. The ftp server I was backing up uses Windows IIS and with -p the directories were created without the execute bit set !?!
• -X _*/* again, because it is windows the ftp server has a load of garbage directories with names that start in a _ containing stuff only known in the darkest bowels of msdn. This excludes them from the backup.

Once the web site is copied I use my normal stratagem for backing up.

I have developed my strategy for putting drupal database dumps into subversion every day. This was slightly more complicated than I though it would be. In principle I could just use mysqldump to dump the sql and put that in subversion but the problem is that mysqldump by default will output the insert statements to reconstruct a table in a single line in the file, even of that line grows to 600k long. If subversion does a line-by-line diff on the files it stores it will end up writing all 600k to it's transaction log.

To minimise the sizes of each diff I dump the mysql in two parts, the data and the structure. For the data dump I do the following:

• use --skip-opt to ensure that each row is inserted on a seperate line: this limits the line length. The --opt option is the default for my version of mysql (4.1.13) so I have to turn it off.
• remove comments
• strip out data for the 'accesslog', 'cache', 'search_index' and 'sessions' tables. I don't think these need backing up.
• sort the remaining lines into alphabetical order as I am not sure the sql dump is guaranteed to always dump the rows in the same order

The structure is dumped straight, I certainly don't sort the lines in the file!

This creates sql data files that should diff pretty optimally.

Here is the bash script that does this for me:

   1  #!/bin/bash
   2  
   3  function SqlDumpData {
   4      mysqldump -u secret -psecret --no-create-info --skip-opt --comments=0 $1 | \
   5        egrep -v "INSERT INTO \`(accesslog|cache|search_index|sessions)\`" | sort >$2
   6  }
   7  
   8  function SqlDumpStructure {
   9      mysqldump -u secret -psecret --no-data $1 >$2
  10  }
  11  
  12  cd /home/peterc/DatabaseDumps
  13  
  14  SqlDumpData petersblog petersblog_data.sql
  15  
  16  SqlDumpStructure petersblog petersblog_structure.sql
  17  
  18  svn commit -m "daily backup"
Toggle Line Numbers

This is all done on the server. Following this I can use my standard strategy to backup the subversion repository.

I have tested that the sql dumps can be reimported into mysql and give a functioning website.

This is the script I am using to back up my debian dedicated server to my ubuntu desktop. It uses ssh and rsync. It uses the cool rsync link-dest trick so that instead of creating multiple copies of the same file, it creates only one copy of the file with multiple hard links to it. I have my ssh keys set up so I don't need to give a password to log in via ssh.

This uses a 'pull' technique: the desktop reads the files from the server using this script.

This is not entirely efficent in that it will create a new set of backup files even if nothing changes: if you run the script ten times in a row then you will end up with ten identical sets of files. However, it backs up a web site that changes every day so running it once a day is valid.

Next job is to put selected files within the backup set into subversion. I decided against using subversion for everything, I can't see a way to automatically delete files bit I'd like to put the main sql dump into subversion.

   1  #!/bin/bash
   2  
   3  #
   4  # Rotate old backups:
   5  #   $1 = remote directory to backup
   6  #   $2 = local backup directory
   7  #
   8  function rotate {
   9      # Ripple old backups
  10      rm -rf $2/Backup9
  11      mv $2/Backup8 $2/Backup9
  12      mv $2/Backup7 $2/Backup8
  13      mv $2/Backup6 $2/Backup7
  14      mv $2/Backup5 $2/Backup6
  15      mv $2/Backup4 $2/Backup5
  16      mv $2/Backup3 $2/Backup4
  17      mv $2/Backup2 $2/Backup3
  18      mv $2/Backup1 $2/Backup2
  19      mv $2/latest $2/Backup1
  20  
  21      # Copy current version to latest, creating hard links where files have not changed.
  22      #
  23      rsync -avz --delete --exclude=.svn --link-dest=$2/Backup1 -e ssh $1/  $2/latest/
  24  
  25      #
  26      # Put a date stamp in the backup directory.
  27      #
  28      echo >`date +$2/latest/Backup-%Y-%m-%d` "Hello Peter"
  29  }
  30  
  31  rotate sshusername@ssh.server.address:/var/www/petersblog.org /home/peter/Backup/petersblog.org
  32  
Toggle Line Numbers

I have implemented a daily backup from my oneandone dedicated server to my home desktop pc. I used rsync over ssh using the notes I wrote myself which work with ubuntu as well as windows. It is only backing up the websites, I'm not backing up the configuration yet, I could add a backup of /etc only that has to run as root.The PC is set up to turn itself on everyday via the BIOS, it will run the backup, do any other daily stuff I can think of, then switch itself off.

At the server end, just before the backup schedule, the mysql databases for the websites are all dumped. The sql for this site is now 9M although a lot of that appears to be server logs.

Once the backup has reached the desktop pc I'd like to create some form of historical backup and I am deciding whether to create incremental differences or chuck everything into subversion. I am leaning towards the latter.

One option for the servers daily activities could be to download podcasts and dump them onto my phone...

At work our main file server is a Windows 2000 server holding 70G of useful, interesting and irreplacable information. It gets dayly incremental tape backups kept offsite, twice weekly full backups, monthly archved backups, all stored on tape. The backup server has no tape drive. Come the fateful day when the main file server dies the first job would be getting the tape drive running on the backup server and then restoring the 70G so that everyone can get on with their work.

Since disk space is cheap I decided to implement a nightly backup copy from the main server to the backup server over the network. If the main server goes down our biggest problem is remapping drive F on everyone's pc to a different share (a hangover from Netware days, nobody uses the UNC path). Once that is done all the files are ready to use, no having to restore anything.

A quick google revealed that Microsoft had a tool to do such a backup in the Windows Server 2003 resource kit called robocopy. This is a command line tool (like all the most useful tools) and is similar to xcopy but, most importantly, can copy file/folder ownership, access permissions etc. We wouldn't want the backup copies to be readable by everyone or no-one but whoever could read them before.

The Windows 2003 version of robocopy works on windows 2000 server but apparently has some bug fixes. I downloaded the Windows Server 2003 resource kit from the microsoft site and I had to install it on my XP system as it refused to install on the Win2k box.

Below is the simple batch file I wrote to do the backup:

rem options:
rem     XJ              Do not copy mount points or junctions
rem     RH:hhmm-hhmm    Specify time range that robocopy will work in
rem     PF              Check above RH time per file
rem     COPYALL         Copy owner ship, auditing, dates, attributes, all the good stuff
rem     MIR             Mirror, delete files that not longer exist in source, recurse, copy empty
rem     LOG:file        Log output to file
rem     TEE             Log to console as well as log file
rem     W:n             Wait time between retries (in seconds)
rem     R:n             Number of retries
rem  NP		Do not show (or log) progress information
rem  NFL		Do not show (or log) each file processed
rem   XD dir		Exclude directory
"c:\RoboBackup\Robocopy" f:\ e:\DriveFBackup\Yesterday /XJ /RH:1800-0730 /PF /COPYALL /Z /MIR
/LOG:c:\RoboBackup.log /TEE /W:1 /R:1 /NP /XD "tmp" "Archived Files"

rem NOT USED
rem     ZB              Copy in backup mode, then restartable mode, can continue on failure
rem                     (robocopy.doc says this slows things down so only use if necessary)
rem     CREATE          Create directory structure but do not copy files. This should be done
rem                     for any empty target to build the directory structure as it reduces
rem                     disk fragmentation.

rem Email the log file to peter.
"c:\RoboBackup\blat" c:\RoboBackup.log -to me@work.com -subject "RoboBackup Output"

This script does a 'mirror' backup, it copies new and modified files (based on time stamp and size) and also will delete files in the backup that have been deleted in the original. Hence, after the script runs the backup should be identical to the master.

The last line uses blat a neat little command line mailing tool to email me the log file. This mailing normally fails as this thing is quite capable of generating 60M log files.

I use a batch file holding this to create the scheduler job:

at 18:00 c:\RoboBackup\RoboBackup.bat

I then have to edit the scheduler task in the gui application to tell it to run as administrator and to give it the administrator password. In an ideal world I would run it as the backup user but to achieve that I would have to make sure the 39000 odd directories all have backup user access.

So there you go, let yourself sleep a little bit easier tonight, using free tools (and an expensive operating system).