Peter's Blog

Netbeans and cygwin ruby don't have a happy relationship. To use the cygwin ruby interpreter it is necessary to launch Netbeans (6.1) with the arguments:

-J-Druby.no.sync-stdio=true

as otherwise Netbeans will pass a windows-style file name to the cygwin ruby that the latter cannot recognise (C:\Projects\Blah instead of /cygdrive/c/Projects/Blah). Unfortunately the ruby interpreter configuration in Netbeans tries to be clever and insists you point it to a ruby executable, you can't give it a batch file that will hack things into working shape.

rspec on cygwin has some similar problems. Here are some hacks to the rspec plugin to make it work from netbeans:

In vendor/plugins/rspec/bin/bin, change:

$LOAD_PATH.unshift(File.expand_path(File.dirname(__FILE__) + "/../lib"))

to

$LOAD_PATH.unshift(File.dirname(__FILE__).sub( /C:\//, '/cygdrive/c/') + "/../lib")

In vendor/plugins/rspec/lib/spec/runner/example_group_runner.rb, change:

load file

to

load File.expand_path(file).gsub(/C:/, '/cygdrive/c')

Tip: to diagnose problems with 'require' not finding files, put this:

p $:

before the failing line to see what directories are in the load path. If you see 'C:\' and it's cygwin then fix it.

Unfortunately it takes a good 20 seconds for cygwin/rspec to run a trivial test script. The rspec server starts and runs but netbeans/rspec seem reluctant to use it (it runs tests ok from the command line).

Why am I still bothering with cygwin? Because I still want to test gcc C code by compiling it into ruby extensions and using ruby test codes (rspec or unit testing).

In tracing these problems I came across more reasons that Netbeans is worth the hassle:

• ctrl-tab to switch to last open tab. ctrl and hold tab brings up a handy little context menu to choose a tab from (kinda like windows task switching with tab).
• press ctrl and mouse over an identifier to bring up rdoc info about it. Ctrl-click to go to it's definition.
• alt-b to go back to where you came from.
• click on an 'end' and the matching begin/do/if is highlighted and vice versa, which is damn useful.
• select an identifier and all matching identifiers in your file are automatically highlighted.

I'm using cygwin bash as my default shell these days as my work is all on cygwin. To complete the cygwininess of the setup it is necessary to lose cmd.exe and finally free oneself of the limitations of Microsoft's 32 bit dos prompt.

To ~/.bashrc I add:

declare PATH=/bin:$PATH

because my cygwin /bin utilities get top priority.

For a shell I use rxvt. I set it up based on this page, the meaty bits of which I duplicate for my own personal reference, slightly tweeked to my preferences:

Step 1: Edit ~/.Xdefaults - the settings below create an 132x50 terminal window with a deep-blue-black background and yellow-white text (looks nice, easy on the eyes) and assign VIM-style color codes:

   1  ! ~/.Xdefaults - X default resource settings
   2  Rxvt*geometry: 130x50
   3  Rxvt*background: #000000
   4  Rxvt*foreground: #e0e0e0
   5  !Rxvt*borderColor: Blue
   6  !Rxvt*scrollColor: Blue
   7  !Rxvt*troughColor: Gray
   8  Rxvt*scrollBar: True
   9  Rxvt*scrollBar_right: True
  10  Rxvt*font: Lucida Console-16
  11  Rxvt*SaveLines: 2000
  12  Rxvt*loginShell: True
  13  ! VIM-like colors
  14  Rxvt*color0:    #000000
  15  Rxvt*color1:    #FFFFFF
  16  Rxvt*color2:    #00A800
  17  Rxvt*color3:    #FFFF00
  18  Rxvt*color4:    #0000A8
  19  Rxvt*color5:    #A800A8
  20  Rxvt*color6:    #00A8A8
  21  Rxvt*color7:    #D8D8D8
  22  Rxvt*color8:    #000000
  23  Rxvt*color9:    #FFFFFF
  24  Rxvt*color10:   #00A800
  25  Rxvt*color11:   #FFFF00
  26  Rxvt*color12:   #0000A8
  27  Rxvt*color13:   #A800A8
  28  Rxvt*color14:   #00A8A8
  29  Rxvt*color15:   #D8D8D8
  30  ! eof
Toggle Line Numbers

Step 2: Modify your c:\cygwin\cygwin.bat to invoke your shiny new Rxvt instead of cmd.exe:

@echo off
C:
chdir C:\cygwin\bin
set EDITOR=vi
set VISUAL=vi
set CYGWIN=codepage:oem tty binmode title
rxvt -e bash --login -i

Now make a shortcut to /cygwin/cygwin.bat and life is sweet.

Oh! That's nice, it remembers my command history from the last time I ran it. Almost as if whoever wrote it had to use it.

I've been using cygwin rsync for my backups and I have now got it working smoothly. One problem that I came across was cygwin's attempts to map the Unix security model onto the Windows security model. I won't go into details on what it gets up to but I can summarise by saying that it is either broken or defaults to something that is useless. It is continually creating files with access permissions equivalent to chmod 000 i.e. nobody has any access to them.

The simple solution has been to declare the environment variable

CYGWIN=nontsec

which turns off the whizz bang nt security model and falls back to a simpler model which simply propogates the windows 'read-only' flag and makes no attempts to worry about owner, group or any of that jazz. This suits me for what I am doing.

cygwin is a set of unix tools compiled and built with a library that allows them to run on Windows systems.

I decided to put a new backup strategy in place at work. I have my desktop PC running windows and an Ubuntu server. I wanted to back up my day-to-day work under windows to the server. I wanted incremental backup so I have the option to backtrack through file history if necessary.

rsync is a nice utility to copy an set of files from one pc to another and works under windows {via Cygwin) and Linux. It can copy over ssh and hence I can use my ssh keys to avoid having to log into the server or put my password in scripts. However it does not do incremental backups, it just duplicates.

rdiff-backup is a nice backup tool that can do cross-network incremental backups. It uses the rsync protocol so it is very efficient. It is also easy to use, no weird command line switches, just give it the name of the source and target directories. However, support for this on windows is not straightforward and it relies on using a cygwin version of python rather than the standard distribution.

So, a compromise solution, use both. I have set things up so that this is done every night when I go home:

cd c:\Projects
rsync -avz --exclude-from="rsync.cnf" -e ssh ./ pcw@rd-pcw2:Projects/ > backup.log
blat backup.log -to pcw@itl.co.uk

this copies files from my 'Projects' directory to the server. The "rsync.cnf" file is a set of things to exclude from the copy, e.g.:

#
# Doxygen output files
#
- Doxygen/

#
# Anything downloaded
#
- Download/
- lstfiles/
- ofiles/
- *.bak
- *.Bak

#
# Anything generated by py2exe
#
- build/
- dist/

#
# Anything in a folder called Old
#
- Old/

#
# VC build directorys
#
Debug/
Release/
debug/
release/

#
# Miscellaneous.
#
- *.obj
- *.tmp
- *.pyc
- setup/*.exe
- Output/setup.exe

After running this I use blat to email me what happened so I know it succeeded.

On the server I have crontab set up to run rdiff-backup every night after the files have been uploaded:

0 18 * * * rdiff-backup /home/pcw/Projects /home/pcw/Backup

This system gives me two full copies of my project files and incremental backups to boot.

Todo: rdiff-backup to a different disk, giving three copies.

Uploading images to this blog can be kinda fiddly. Say I have an image in the windows clipboard (e.g. a screen clip from OneNote and I want it on the blog, I have to fire up a graphics program, paste it in, save it to a file, fire up filezilla, upload it, delete the image file.

This little script does all this for me. It asks for a file name, writes the image in the clipboard to the file, uploads it and deletes the image. Job Done.

The uploading is done using pscp, a version of scp from the putty camp. It uses my putty private key so I don't have to give a password for access to the server (I didn't use the cygwin version of ssh as it didn't like having 'c:\' in file names). The script uses the Python Imaging Library to do the grabbing and wxPython to ask for the file name.

   1  #
   2  # Upload image in clipboard to server
   3  #
   4  import wx
   5  import ImageGrab
   6  import Image
   7  import os
   8  import tempfile
   9  
  10  class MyApp(wx.App):
  11      def OnInit(self):
  12          oImage = ImageGrab.grabclipboard()
  13  
  14          if not isinstance( oImage, Image.Image):
  15              #
  16              # In no way is wxPython an easy to use library.
  17              #
  18              dlg = wx.MessageDialog( None, 'No image in clipboard', 'Error', wx.OK)
  19              dlg.ShowModal()
  20              dlg.Destroy()
  21              return True
  22  
  23          dlg = wx.TextEntryDialog( None, 'Enter a file name', 'Upload Clipboard', '')
  24          if dlg.ShowModal() != wx.ID_OK:
  25              dlg.Destroy()
  26              return True
  27  
  28          strName = dlg.GetValue()
  29          dlg.Destroy()
  30  
  31          strTempFile = tempfile.mktemp( strName)
  32          oImage.save( open( strTempFile, 'wb'))
  33  
  34          strCmd = 'pscp -q -i "c:\my documents\puttykey.ppk" %s me@myserver.com:www/images/%s' % (strTempFile, strName)
  35  
  36          os.system( strCmd)
  37  
  38          os.unlink( strTempFile)
  39  
  40          return True
  41  
  42  app = MyApp(0)
  43  app.MainLoop()
Toggle Line Numbers

This is the result of alt-print screen and running this script:

The script itself inside VIM

ToDo:

• handle file names in clipboard
• detect upload errors
• allow resizing images

Putty is a simply great ssh client and works nicely with open-ssh, which is found in Ubuntu Linux, Site5 and just about everywhere.

A nice feature of ssh is the ability to generate a public key that can be used to log into a server without having to give a password, or as extra secutiry in addition to the password.

Here is a procedure for creating ssh keys that can be used in both open-ssh and putty:

• On windows, install the open-ssh package with Cygwin
• execute the command

  ssh-keygen -t ssh-dss
  

  to generate the dss key. You may need to create the directory ~/.ssh in Cygwin bash for this to work. This will create a file in this directory called id_dsa.pub
• use sftp/ssh to copy the id_dsa.pub file to your ssh server box. Put the contents of this file (which is one big long line) at the end of a file called ~/.ssh/authorised_keys2, adding it to any other keys that are already there.
• back on windows, execute the command 'puttygen', from the putty site.
• In putty gen, use file/load private key to load in the file ~/.ssh/id_dsa
• Choose 'save private key' and store it somewhere handy where putty can find it. You may be prompted to enter a passphrase. This is a password used in addition to the key when connecting to the server. If the passphrase is blank then you don't have to enter it, the connection will be automatic.
• Open putty and enter the details of the server you want to connect to (address etc)
• In the 'connection' settings, enter your login name in 'Auto-login username'.
• In Connection/SSH/Auth, in the box 'Private key file for authentication' load the putty private key file.
• Save this configuration so you don't have to do it again.
• Click 'open'

Your life won't be the same again.

This is the ssh command line to connect to a squid proxy server via ssh:

ssh -L 3128:127.0.0.1:3128 user@remoteaddress

using cygwin open-ssh.

What is this doing? Well ssh starts running on the local pc and creates a socket on port 3128 (this is given by the first 3128 on the command line). Firefox is then configured to use localhost:3128 as the http proxy. ssh then takes connections to that port and forwards them to the ssh server running remotely. It will go through any firewalls in between provided that port 22 (the ssh port) is open. It encrypts whatever is going through it. The ssh server connects to port 3128 on the remote pc which is the squid server port. The squid server acts as a nice caching proxy server and fetches whatever web pages you are looking for.

This works with Ubuntu after installing the standard open-ssh package and the squid proxy.

It must work or you wouldn't be reading this.

Even more cool: using VNC over SSH. This allows gives you a remote desk top

• install tightvnc package on remote server
• on local pc run the command:

  ssh -L 5900:127.0.0.1:5901 user@remoteaddress
  

• on server, run the command

  vncserver :1
  

  The first time you do this you will be asked for the login password.
• On local pc, run a vnc viewer such as ultravnc. Connect to 127.0.0.1:0 and enter the password

This gives you remote access to the server desktop. Easy really and better, in my humble opinion, that using remote X, especially with a local PC running cygwin as X on that is a bit buggy.

Note: for me, vncserver :1 worked as X was already running on the box. The 5901 in the ssh command caters for display 1 being on a port number 1 higher than the default of 5900.

UPDATE: this post which describes a more modern approach using xming which is a far better X server than cygwin.

Retracing earlier steps I find Ubunto works as a headless X box, running programs while displaying remotely on a windows 2000 pc (X windows has the definitions of 'client' and 'server' randomised).

Here are the steps it takes to do this, rationalised from what I wrote before:

• Install Cygwin on the windows box. In particular, install the openssh and X11 base packages.
• Edit the sshd_config file on the remote server and make sure this is set:

  X11Forwarding yes
  

  and restart sshd. On my Ubuntu this was how it was already set up.
• on the local pc run the cygwin command startxwin.bat. It is buried in:

  <cygwin base dir>\usr\X11R6\bin\startxwin.bat
  

  where <cygwin base dir> is wherever you installed cygwin. This should bring up a window with an X desktop in it.
• from a terminal window on the X desktop, run an ssh command of the form:

  ssh -X -l <username> <host address>
  

  the -X tells ssh to do magic X port forwarding.
• in the new ssh session run an X program, e.g.:

  xterm &
  

• and away you go

I tried running Synaptic Package Manager from this, i.e.

sudo synaptic &

and it worked, a bit sluggishly, until it got to the point where I tried to accept changes and install packages, where it died with an X error. It did this twice so it is probably a deep X problem. I will have to stick with dselect

Been trying IPython, an improved interactive python shell. I've played with some of these before (pythonwin, pycrust etc) but found them little better than the normal python interactive prompt.

IPython so far looks promising:

• tab completion of file names, class members etc
• %pdb command to catch exceptions in the pdb debugger. This is ok but only catches unhandled exceptions, if your code has an exception handler then that gets priority.
• %run command to run scripts from within the shell. Combined with %pdb this is a nice way to work and is equivalent to just running the script from the command line.
• can cd and ls from within ipython just as if using the command line (and cygwin).
• more stuff I haven't tried yet.

It has some quirks: killing a running script seems to take a lot of ctrl-c's and I suspect that sometimes it didn't reload a script properly but it does look like a definite improvement over the standard python interpreter.

If you have a file on an ssh enabled server that you want to edit with VIM and you don't want to mess around with ftp you can use VIM's ftp support.

However, I find the support for scp to be more useful. Having installed Cygwin and open ssh I have also acquired the scp command. VIM knows about this and can use it to edit files.

Suppose I want to edit the file 'interesting.txt' on the server 'stuff.com'? I use the VIM command:

:e scp://peter@stuff.com/interesting.txt

and start editing. When I am finished I just write the file normally:

:w

Because I have already created an ssh key on stuff.com (as shown here) I am not hastled for passwords.

Very nice for quick .css tweeks, editing drupal modules etc. I can edit the file locally in a windows GUI and upload it to nice secure linux server. Best of both worlds.

Downsides?

1. Win32 vim leaves a silly dos box with a 'Hit any key to close this window...' message on it when reading the file.
2. Have to know the exact path to the file.

Mission: to get rsync going over ssh. This will allow me to synchronise files on local client and remote server. Rsync is a nice fast incremental copy program designed for synchronising file sets between a client and server. The result is that both computers share identical versions of a set of files.

Steps:

• install openssh on client and server
• install rsync on client and server
• to avoid typing in passwords, create ssh key using:

  ssh-keygen -t dsa
  

• Log into server using sftp
• copy /home//.ssh/id_dsa.pub to the server
• on the server, merge contents of id_dsa.pub with ~/.ssh/authorized_keys2
• From client, try logging in using ssh or sftp. Key should be accepted.
• Start rsyncing.

Note: using a blank passphrase for ssh means you will not be prompted for a passphrase when you log in but this is a security risk: if anyone gets their hands on the key file they can get it.

Examples: Copy files in the directory 'logs' to the directory 'blah' on server ad-pc (files will be in blah/*).

rsync -avz logs/ -e ssh ad-pc:blah/

Copy the files back.

rsync -avz -e ssh ad-pc:blah/ logs/

Copy the directory 'logs' and files in it to the directory 'blah' on server ad-pc (files will be in blah/logs/*).

rsync -avz logs -e ssh ad-pc:blah/

Copy the directory 'logs' and files in it to the directory 'blah' on server ad-pc. Copy recursively and delete files on the receiver that do not exist on the sender (i.e. replicate deletion of files).

rsync -avzr --delete logs -e ssh ad-pc:blah/

Copy while logging in as a different user.

rsync -avz -e ssh <user>@ad-pc:www/modules/ modules/

Note the subtle use of the trailing /.

blah/

copy files to/from directory

blah

copy directory itself

Using cygwin on Windows 2000 I had a problem copying from the server to the local pc: the shell would hang real hard at the end of the transfer. I tried reinstalling the cygwin stuff from a different server (mirror.ac.uk) and found a much more recent version (2.6.2 as opposed to 2.5.5). This fixed the problem, which may have been this.

I have managed to get remote X windows working via SSH. From a Windows 2000 desktop I can run X applications on my server at home. The steps involved are approximately as follows (i.e. I frigged around a bit but I think this will work):

• Install Cygwin. In particular, install the openssh and X11 base packages.

• Edit the sshd_config file on the remote server and make sure this is set:

   X11Forwarding yes 

  and restart sshd. This is important for avoiding the need for xauth

• start X on the remote computer.

• on the remote computer run:

   xhost + 

  This turns off strict access control. I am not entirely sure this command is needed when using ssh X port forwarding but I don't think it will hurt. Both local and remote systems are behind firewalls with an encrypted ssh link between then and I am not paranoid enough to worry about extra security.

• on the local pc run the cygwin command startxwin.bat. This should bring up a window with a X desktop in it.

• from a terminal window on the X desktop, run an ssh command of the form:

   ssh -X -l <username> <host address> 

  the -X tells ssh to do magic X port forwarding ¹.

• again in the terminal window, do this to check things are ok:

   export | grep DISPLAY 

  you should see that the DISPLAY variable has been set to something like 127.0.0.1:10.0. Setting this up is the job of the sshd server. If it is not there then the X port forwarding is broken.

• run an X program, e.g.:

   xterm & 

You should get a new term on your desktop running from the remote computer. I got as far as running mozilla across this but it was too slow to be of any use (12kbytes/sec upload speed achieved by Bit Torrent yesterday).

Cygwin defaults to twm which is a little primitive. I tried WindowMaker but that crashed. fvwm2 works, looks a little prettier than twm but is not much more sophisticated.

It's cool, but an awful lot can be done with just a straight ssh console.

I have managed to get remote X windows working via SSH. From a Windows 2000 desktop I can run X applications on my server at home. The steps involved are approximately as follows (i.e. I frigged around a bit but I think this will work):

• Install Cygwin. In particular, install the openssh and X11 base packages.

• Edit the sshd_config file on the remote server and make sure this is set:

   X11Forwarding yes 

  and restart sshd. This is important for avoiding the need for xauth

• start X on the remote computer.

• on the remote computer run:

   xhost + 

  This turns off strict access control. I am not entirely sure this command is needed when using ssh X port forwarding but I don't think it will hurt. Both local and remote systems are behind firewalls with an encrypted ssh link between then and I am not paranoid enough to worry about extra security.

• on the local pc run the cygwin command startxwin.bat. This should bring up a window with a X desktop in it.

• from a terminal window on the X desktop, run an ssh command of the form:

   ssh -X -l <username> <host address> 

  the -X tells ssh to do magic X port forwarding ¹.

• again in the terminal window, do this to check things are ok:

   export | grep DISPLAY 

  you should see that the DISPLAY variable has been set to something like 127.0.0.1:10.0. Setting this up is the job of the sshd server. If it is not there then the X port forwarding is broken.

• run an X program, e.g.:

   xterm & 

You should get a new term on your desktop running from the remote computer. I got as far as running mozilla across this but it was too slow to be of any use (12kbytes/sec upload speed achieved by Bit Torrent yesterday).

Cygwin defaults to twm which is a little primitive. I tried WindowMaker but that crashed. fvwm2 works, looks a little prettier than twm but is not much more sophisticated.

It's cool, but an awful lot can be done with just a straight ssh console.

[1]	the howto on the Cygwin site says '-Y' which is rejected by ssh.

Discovered rsync and unison which are tools for syncing files and directories. unison appears to have better windows support via gtk (which makes it look like a naff tk app). It syncs locally ok and could be a nice way to do nightly backups. To do remote backups requires it to be set up to fork a ssh command. Unfortunately it does not provide good hooks for doing this in the setup file: trying to use plink (a command line version of putty) I get this:

 Received unexpected header from the server:  expected "U" but received "\013". This is probably because you have different versions of Unison installed on the client and server machines. 

Both appear to be version 2.9.1, the difference appears to be a windows/unix line delimiter problem. Plink has no options to fiddle with this. I might be able to fix it with Cygwin but that is becoming like installing linux on top of windows (I just wanna run ssh without installing 20M of unix environment).

What I really want is an ftp based solution as that would let be publish websites.