Peter's Blog

I have been using the Twitter gem to post tweets regarding new blog updates. During some testing I was seeing bare passwords in urls which prompted me into migrating to using Oauth to authenticate with twitter. However this proved to be a bit of a minefield. Oauth tends to be described in explicit technical details and the gems available have cursory examples based on the simplest use cases and they never explain how to get the tokens.

So here is how to use oauth with twitter in enough detail for other people to do it. The main problem is that even if you are writing, say, a command line twitter app you will still need to write a web based app to get twitter to provide you with the right authentication keys as the process involves being redirected through the twitter web site.

First go here and ask twitter nicely for your Consumer Token and your Consumer Secret. I won't bore you with what these are, if you want to be bored read the oauth docs.

Stick these in a yaml file like this:

---
accountname:
  consumer_secret: ulasdfjasdhflkshdflkjsdfhasdfasdfasdffs
  consumer_token: 9asdjfhkasjdfhkhkjhkjA

I put this in RAILS_ROOT/config/twitter.yml

Now install the twitter gem:

sudm gem install twitter

Here is a library to encapsulate reading and writing these tokens. I called it PetersTwit.rb and put it in RAILS_ROOT/lib:

   1  #
   2  # Simple wrapper for twitter oauth tokens
   3  #
   4  
   5  require 'rubygems'
   6  require 'twitter'
   7  require 'yaml'
   8  
   9  class PetersTwit
  10    #
  11    # Initialise oauth stuff. strConfig points to the yaml config file,
  12    # strAccount is the name of an account in the file (need not be same name
  13    # as twitter account)
  14    #
  15    def initialize( strConfig, strAccount)
  16      @strConfig = strConfig
  17      @strAccount = strAccount
  18  
  19      @oTokens = nil
  20  
  21      begin
  22        @oAllTokens = YAML::load_file( strConfig)
  23        @oTokens = @oAllTokens[@strAccount]
  24      rescue
  25        @oAllTokens = {}
  26      end
  27  
  28      if not @oTokens
  29        @oTokens = {
  30          'consumer_token' => nil,
  31          'consumer_secret' => nil,
  32          'access_token' => nil,
  33          'access_secret' => nil,
  34        }
  35  
  36        @oAllTokens[@strAccount] = @oTokens
  37      end
  38    end
  39  
  40    def consumer
  41      OAuth::Consumer.new(
  42         @oTokens['consumer_token'],
  43         @oTokens['consumer_secret'],
  44         {:site => 'http://twitter.com'}
  45      )
  46    end
  47  
  48    def SetAccessToken( strToken, strSecret)
  49      @oTokens['access_token'] = strToken
  50      @oTokens['access_secret'] = strSecret
  51      @oAllTokens[@strAccount] = @oTokens
  52      Save()
  53    end
  54  
  55  
  56    def Save
  57      YAML::dump( @oAllTokens, open( @strConfig, "w"))
  58    end
  59  
  60    def Auth
  61      oAuth = Twitter::OAuth.new( @oTokens['consumer_token'], @oTokens['consumer_secret'])
  62      oAuth.authorize_from_access( @oTokens['access_token'], @oTokens['access_secret'])
  63  
  64      return oAuth
  65    end
  66  end
Toggle Line Numbers

Now add the following actions to a handy rails controller you may have:

   1    def twitter_oauth_login
   2      oTwit = PetersTwit.new( File.join( RAILS_ROOT, 'config', 'twitter.yml'), 'therealpeter')
   3  
   4      request_token = oTwit.consumer.get_request_token
   5      session[:request_token] = request_token.token
   6      session[:request_token_secret] = request_token.secret
   7  # Send to twitter.com to authorize
   8      redirect_to request_token.authorize_url
   9    end
  10  
  11    def twitter_oauth_register
  12      oTwit = PetersTwit.new( File.join( RAILS_ROOT, 'config', 'twitter.yml'), 'therealpeter')
  13  
  14      request_token = OAuth::RequestToken.new( oTwit.consumer, session[:request_token], session[:request_token_secret])
  15  
  16      #
  17      # Exchange the request token for an access token.
  18      #
  19      access_token = request_token.get_access_token( :oauth_verifier => params[:oauth_verifier])
  20  
  21      oTwit.SetAccessToken( access_token.token, access_token.secret)
  22  
  23      redirect_to( :action => :index)
  24    end
Toggle Line Numbers

When you register your application with twitter you can say it is a web application and where it asks for a callback url you can point to the url of the twitter_oauth_register. I did this but it never called me back

Run the rails app and go to the twitter_oauth_login url which should redirect you to twitter where you log into your twitter account (I'm not phishing here, honest) and authorise twitter to let your application connect to twitter without any more tedious password nonsense.

In an ideal world twitter will redirect you back to your app but this is broken for me. Instead Twitter acts as if I asked for a desktop app and displays a seven digit PIN. This can still be used by manually entering the url for:

http://yourserver/your_controller/twitter_oauth_register?oauth_verifier=1234567

where 1234567 is the PIN that twitter gave you.

After this you should have values for access_token and access_secret appear in your yaml file. These are good, keep these safe as they will let you post to twitter without passwords and crap. You can use them anywhere: web app or command line app. They allow your app to access twitter under a particular twitter user name.

Posting to twitter from the command line is now easy:

require 'PetersTwit'
require 'twitter'

oPTwit = PetersTwit.new( File.join( RAILS_ROOT, 'config', 'twitter.yml'), 'therealpeter')
oTwit = Twitter::Base.new( oPTwit.Auth())

oTwit.update( 'OMG jonas brothers')

When developing this the biggest conceptual hurdles came in the handling of the exchange of request tokens and access tokens. What can be confusing is that this can only be done once: if you get a set of request tokens you have one chance to swap them for access tokens and if you screw it up you need a fresh set of request tokens. But you don't need to know that if you follow my recipe, it's an implementation detail.

I have a two and a half year old python/turbogears web application that is fairly mission critical yet thoroughly software rotted. It was built with a delicate mix of python eggs running on python 2.4 and ubuntu hardy. Any attempt to upgrade any part of it yeilded obscure errors.

A recent disk failure on another server had me worrying about this state of affairs: how to migrate it to a new server? Well, an idea hit me: copy all the python2.4 site-packages to the applications own directory on the new server and set it's path up to use them rather than try to install the correct hotch-potch of system libraries.

This didn't work immediately as the site-packages directory was full of broken symbolic links to the python setup tools and mysql gems. Installing these from ubuntu packages didn't work because the installation directories had changed between ubuntu hardy and jaunty.

In the end I avoided the need for python setup tools by unpacking all the python eggs, including the mysql stuff, into bare sets of python source. This worked surprisingly well, problem solved.

Now I had a working blob of code and libraries that only needed python2.4 and it's standard libraries to work. I quickly shoved it into git in it's functional glory.

The experience made me appreciate the concept of freezing gems with a rails app:

• everything goes into git with minimal external dependencies
• no worry that (for example) the gem for a specific version of mysql is no longer available anywhere when redeploying (the sqlite gem on windows is precarious).
• quicker redeploy on a clean server: no run server/get missing gem error/install gem/ run server... loop.

I find that every rails update tends to break at least one thing and I only discover them when I am doing a panicy server rebuild.

Someone left a quite justifiable complaint that they couldn't copy source code from this site without it being polluted with line numbers. I've found this myself looking up old treasures. What to do? I like the line numbers so they're staying. I use the ultraviolet gem for syntax highlighting which gives two choices: line numbers or no line numbers. I've seen various javascripty sites that allowed me just to select the code without the line numbers or even shove it into the clipboard but ultraviolet's output can't do that. I looked at some javascript libraries like SyntaxHighlighter but I don't really want to add a lot more javascript libraries to the site, I prefer the server side solution.

Back to basics. How could the user click something to toggle the line numbers in Ultraviolet's output? Well the Prototype javascript library that is already used makes it incredibly easy:

<div onclick="$$('.line-numbers').invoke( 'toggle');">
   Toggle Line Numbers
</div>

When the text 'Toggle Line Numbers' is clicked this searches for all elements with the class 'line-numbers' and toggles their visibility.

So I now munge the ultraviolet output to add that little gem. The code below is highlighted by itself and it doesn't disappear up it's own bottom.

   1   #
   2   # Syntax highlight some code.
   3   #
   4    def wilkiHighlight( strText, strLanguage)
   5      if strLanguage == 'bash'
   6        strLanguage = 'shell-unix-generic'
   7      end
   8  
   9      #
  10      # Only add line numbers on snippets of 10 lines or more
  11      #
  12      bLines = strText.split( "\n").length >= 10
  13      if Uv.syntaxes.index( strLanguage)
  14        strText = Uv.parse( strText, "xhtml", strLanguage, bLines, "lazy")
  15      else
  16        strText = Uv.parse( strText, "xhtml", 'plain_text', bLines, "lazy")
  17      end
  18      if bLines
  19        strText.sub!( "</pre>", "<div class=\"linetoggle\" onclick=\"$$('.line-numbers').invoke( 'toggle');\">Toggle Line Numbers</div></pre>")
  20      end
  21  
  22      return strText
  23    end
Toggle Line Numbers

Some css makes the toggle text look pretty:

.linetoggle {
    font-family: verdana;
    font-size: smaller;
    color: #808080;
    margin-top: 5px;
    padding-top: 5px;
    border-top: 1px solid #808080;
}

I am aware that all line numbers on a page will be toggled, not just individual snippets but I don't see how that is a big deal.

AMock was born of a need for testing Mocks with rspec in real-time systems. I found the mocking support in rspec was not very good in making sure that functions are called in a particular order. While it can do this the support is not strong, i.e. it won't tell you explicitly what method it was expecting to see called and what function was called unexpectedly, making debugging difficult. In testing real time systems the order in which functions are called is far more important than, for example, the order that values are gathered for displaying on a web page.

I understand that rspec's mocking is designed to be 'declarative' rather than 'imperative'. The testing that I do is very close to testing the implementation rather than nice Behavioural Driven Design. However it is still nice to know that my implementation has been tested.

AMock also has some syntactic sugar to make the expectations easy to define:

   1      oAmock = AMock::Mock.new( :Wibble)
   2  
   3      oAmock.should_receive do
   4        CallMyFunction( 27)
   5        DoSomethingElse( 27).and_return( 58)
   6        AnyWhichWay( 67).unordered.and_return( 93)
   7      end
   8  
   9      oAmock.AnyWhichWay( 67).should == 93
  10      oAmock.CallMyFunction( 27)
  11      oAmock.DoSomethingElse( 27).should == 58
  12  
  13      oAmock.verify?
Toggle Line Numbers

The functions listed in should_receive are in the order that they are expected to be called, apart from AnyWhichWay which can be called at any time.

The implementation of AMock is a bit of a hack, it could be cleaner but:

1. No time
2. It works.

I use this library for both Mocking and Stubbing.

AMock can be found on github.

My main twitter client is Tweetie and I don't scan it all the time so I often miss replies that are sent to me, sometimes for a few days (I don't check the 'mentions' page). Despite this sentiment I don't want to miss any replies as they can be quite important (e.g. direct questions).

I do check my email regularly so I thought it would be nice to be emailed if I had new replies. Surprisingly, twitter.com only does this for direct messages so I wrote something to do the job for me:

This runs from a cron job every half hour so hopefully I won't be accused of API abuse.

BTW this is the first time I have embedded a github gist in the blog so I don't know how nicely it will work.

My conscience is leading me more and more into formal testing and I have been trying out cucumber the latest ruby Behaviour Driven Development offering. Cucumber is still very new and there is little documentation available so here are some notes.

The basic idea of cucumber is along the lines that your tests are almost in spoken language rather that a programming language and hence non technical people like the marketing department, clients etc can read them and understand them, although I just cannot believe that they could ever write them in any useful way. The tests describe the behaviour of the systems and hence document it in a similar way to a requirements specification. If one decided, for example, to go with the herd one could take the behaviour listings as a specification for reimplementing everything in .NET.

Or something like that.

Anyway, here is a cucumber 'feature' that I have come up with for pctv. This describes the features of the system. Pctv shows a list of tv programs, one can click on a program and see a list of episodes of that program.

Feature: Looking at programme listing
    In order to watch programs
    I should be able to
    See listings of what is on

    Scenario: Look at programs
        Given I have some programmes
            And I visit "/"
        Then I should see "Dora the Explorer"
            And I should see "Loose Women"

    Scenario: Examine Dora the Explorer
        Given I have some programmes
            And I visit "/"
        When I follow "Dora the Explorer"
        Then I should see "Dora the Explorer"
            And I should see "Tomorrows Episode"
            And I should not see "Yesterdays Episode"
            And I should not see "Loose Women"
            And the "Series Link" checkbox should not be checked

    Scenario: Mark Dora series link
        Given I have some programmes
            And I visit "/"
        When I follow "Dora the Explorer"
            And I check "Series Link"
            And I press "Update"
        Then I should see "Dora the Explorer"
            And I should see "Tomorrows Episode"
            And I should not see "Yesterdays Episode"
            And I should not see "Loose Women"
            And the "Series Link" checkbox should be checked

    Scenario: Star Trek series link
        Given I have some programmes
            And I visit "/"
        When I follow "Star Trek"
        Then I should see "Star Trek"
            And the "Series Link" checkbox should be checked
            And I should see "Boldly Gone"
            And I should see "Boldly Going"
            And I should not see "Yesterdays Episode"
            And I should not see "Loose Women"

    Scenario: Unmark Star Trek series link
        Given I have some programmes
            And I visit "/"
        When I follow "Star Trek"
            And I uncheck "Series Link"
            And I press "Update"
        Then I should see "Star Trek"
            And the "Series Link" checkbox should not be checked
            And I should see "Boldly Gone"
            And I should see "Boldly Going"
            And I should not see "Yesterdays Episode"
            And I should not see "Loose Women"

    Scenario: Visit recordings page
        Given I have some programmes
            And I visit "/"
        When I follow "Recordings"
        Then I should see "Star Trek"
            And I should see "Boldly Gone"
            And I should see "Recorded"
            And I should not see "Boldly Going"

    Scenario: Examine Categories of Women
        Given I have some programmes
            And I visit "/"
        When I follow "Women"
        Then I should see "Loose Women"
            And I should not see "Dora the Explorer"
            And I should not see "Star Trek"

    Scenario: Search for Dora
        Given I have some programmes
            And I visit "/programmes"
        When I follow "Search"
            And I fill in "Search" with "Dora"
            And I press "Seek"
        Then I should see "Dora the Explorer"
            And I should not see "Loose Women"
            And I should not see "Star Trek"

I think this works at the level of being quite readable and describes nicely how one interacts with pctv.

This uses the webrat gem which simulates the interactions with a web browser (seeing text, clicking links etc).

The Given, When and Then keywords translate directly to code steps which establish the state of the system, an event that changes the system and the results of the change respectively. When cucumber is set up in a rails application with:

script/generate cucumber

it will generate a file called /features/step_definitions/webrat_steps.rb which contains some useful webrat examples such as:

When /^I press "(.*)"$/ do |button|
  clicks_button(button)
end

Then /^I should see "(.*)"$/ do |text|
  response.body.should =~ /#{text}/m
end

So the When term is converted into webrat operations and the Then term examines the html generated by the code. It should be noted that webrat does nothing in respect to examining the output, that can be done directly on the response object or it can be done using rspecs have_tag method:

Then /it should have the tag "(.*)" containing the text "(.*)"/ do |strTag, strText|
  response.should have_tag( strTag, /#{strText}/)
end

The above is one of mine and I'm a bit uneasy about it. I'm not sure if the cucumber level of testing should be above the implemntation and above the level of examining the html in the response or whether it should be at the level of what a user would actually see on a web site. The studying of the html should be at the level of view tests using rspec-rails but this is just a feeling.

Here is another one of mine:

Given /^I visit "(.*)"$/ do |url|
  visits(url)
end

This seems like an obvious one to need and I'm not sure why the generated webrat steps don't include it, it makes me wonder if I haven't quite grasped the philosophy.

Cucumber doesn't seem to support fixtures and googling for this I came across a thread where someone was asking about this issue and the author of cucumber said 'google for it': there is no FM to R. Other posters in the thread were more helpful but I still found that the fixtures were only loaded in the first scenario and for following scenarios the database stayed empty. I decided to just code something:

   1  Given /I have some programmes/ do
   2    strYesterday = (DateTime.now - 1).strftime( "%Y%m%d")
   3    strTomorrow = (DateTime.now + 1).strftime( "%Y%m%d")
   4  
   5    oChannel1 = Channel.create( :dvbname => 'Nick Jr')
   6    oChannel2 = Channel.create( :dvbname => 'ITV ONE')
   7  
   8    oCategory1 = Category.create( :name => 'Children')
   9    oCategory2 = Category.create( :name => 'Crap')
  10  
  11    oProgram = Programme.create( :title => "Dora the Explorer", :category => oCategory1)
  12    oEpisode = Episode.create( :programme => oProgram, :subtitle => "Episode 1", :description => "Yesterdays Episode", :channel => oChannel1, :starting => DateTime.parse( strYesterday + "080000"), :ending => DateTime.parse( strYesterday + "080000"))
  13    oEpisode2 = Episode.create( :programme => oProgram, :subtitle => "Episode 2", :description => "Tomorrows Episode", :channel => oChannel1, :starting => DateTime.parse( strTomorrow + "080000"), :ending => DateTime.parse( strTomorrow + "080000"))
  14  
  15    oProgram2 = Programme.create( :title => "Loose Women", :category => oCategory2)
  16    oEpisode2 = Episode.create( :programme => oProgram2, :description => "You did it where?", :channel => oChannel2, :starting => DateTime.parse( strTomorrow + "123000"), :ending => DateTime.parse( strTomorrow + "133000"))
  17  
  18    oProgram3 = Programme.create( :title => "Star Trek", :category => oCategory1, :series_link => true)
  19    oEpisode1 = Episode.create( :programme => oProgram3, :description => "Boldly Gone", :channel => oChannel2, :starting => DateTime.parse( strTomorrow + "013000"), :ending => DateTime.parse( strYesterday + "023000"),
  20                                  :recording_status => 'recorded', :filename => 'Star Trek.avi')
  21    oEpisode2 = Episode.create( :programme => oProgram3, :description => "Boldly Going", :channel => oChannel2, :starting => DateTime.parse( strTomorrow + "013000"), :ending => DateTime.parse( strTomorrow + "023000"))
  22  end
Toggle Line Numbers

I put the following in /features/support/env.rb

require "webrat/rails"

and run it using

spec features

and I get:

   1  Feature: Looking at programme listing  # features/programmes.feature
   2      In order to watch programs
   3      I should be able to
   4      See listings of what is on
   5    Scenario: Look at programs               # features/programmes.feature:6
   6      Given I have some programmes           # features/step_definitions/pctv_steps.rb:1
   7      And I visit "/"                        # features/step_definitions/pctv_steps.rb:24
   8      Then I should see "Dora the Explorer"  # features/step_definitions/webrat_steps.rb:85
   9      And I should see "Loose Women"         # features/step_definitions/webrat_steps.rb:85
  10  
  11    Scenario: Examine Dora the Explorer                     # features/programmes.feature:12
  12      Given I have some programmes                          # features/step_definitions/pctv_steps.rb:1
  13      And I visit "/"                                       # features/step_definitions/pctv_steps.rb:24
  14      When I follow "Dora the Explorer"                     # features/step_definitions/webrat_steps.rb:10
  15      Then I should see "Dora the Explorer"                 # features/step_definitions/webrat_steps.rb:85
  16      And I should see "Tomorrows Episode"                  # features/step_definitions/webrat_steps.rb:85
  17      And I should not see "Yesterdays Episode"             # features/step_definitions/webrat_steps.rb:89
  18      And I should not see "Loose Women"                    # features/step_definitions/webrat_steps.rb:89
  19      And the "Series Link" checkbox should not be checked  # features/step_definitions/webrat_steps.rb:97
  20  
  21    Scenario: Mark Dora series link                     # features/programmes.feature:22
  22      Given I have some programmes                      # features/step_definitions/pctv_steps.rb:1
  23      And I visit "/"                                   # features/step_definitions/pctv_steps.rb:24
  24      When I follow "Dora the Explorer"                 # features/step_definitions/webrat_steps.rb:10
  25      And I check "Series Link"                         # features/step_definitions/webrat_steps.rb:69
  26      And I press "Update"                              # features/step_definitions/webrat_steps.rb:6
  27      Then I should see "Dora the Explorer"             # features/step_definitions/webrat_steps.rb:85
  28      And I should see "Tomorrows Episode"              # features/step_definitions/webrat_steps.rb:85
  29      And I should not see "Yesterdays Episode"         # features/step_definitions/webrat_steps.rb:89
  30      And I should not see "Loose Women"                # features/step_definitions/webrat_steps.rb:89
  31      And the "Series Link" checkbox should be checked  # features/step_definitions/webrat_steps.rb:93
  32  
  33    Scenario: Star Trek series link                     # features/programmes.feature:34
  34      Given I have some programmes                      # features/step_definitions/pctv_steps.rb:1
  35      And I visit "/"                                   # features/step_definitions/pctv_steps.rb:24
  36      When I follow "Star Trek"                         # features/step_definitions/webrat_steps.rb:10
  37      Then I should see "Star Trek"                     # features/step_definitions/webrat_steps.rb:85
  38      And the "Series Link" checkbox should be checked  # features/step_definitions/webrat_steps.rb:93
  39      And I should see "Boldly Gone"                    # features/step_definitions/webrat_steps.rb:85
  40      And I should see "Boldly Going"                   # features/step_definitions/webrat_steps.rb:85
  41      And I should not see "Yesterdays Episode"         # features/step_definitions/webrat_steps.rb:89
  42      And I should not see "Loose Women"                # features/step_definitions/webrat_steps.rb:89
  43  
  44    Scenario: Unmark Star Trek series link                  # features/programmes.feature:45
  45      Given I have some programmes                          # features/step_definitions/pctv_steps.rb:1
  46      And I visit "/"                                       # features/step_definitions/pctv_steps.rb:24
  47      When I follow "Star Trek"                             # features/step_definitions/webrat_steps.rb:10
  48      And I uncheck "Series Link"                           # features/step_definitions/webrat_steps.rb:73
  49      And I press "Update"                                  # features/step_definitions/webrat_steps.rb:6
  50      Then I should see "Star Trek"                         # features/step_definitions/webrat_steps.rb:85
  51      And the "Series Link" checkbox should not be checked  # features/step_definitions/webrat_steps.rb:97
  52      And I should see "Boldly Gone"                        # features/step_definitions/webrat_steps.rb:85
  53      And I should see "Boldly Going"                       # features/step_definitions/webrat_steps.rb:85
  54      And I should not see "Yesterdays Episode"             # features/step_definitions/webrat_steps.rb:89
  55      And I should not see "Loose Women"                    # features/step_definitions/webrat_steps.rb:89
  56  
  57    Scenario: Visit recordings page        # features/programmes.feature:58
  58      Given I have some programmes         # features/step_definitions/pctv_steps.rb:1
  59      And I visit "/"                      # features/step_definitions/pctv_steps.rb:24
  60      When I follow "Recordings"           # features/step_definitions/webrat_steps.rb:10
  61      Then I should see "Star Trek"        # features/step_definitions/webrat_steps.rb:85
  62      And I should see "Boldly Gone"       # features/step_definitions/webrat_steps.rb:85
  63      And I should see "Recorded"          # features/step_definitions/webrat_steps.rb:85
  64      And I should not see "Boldly Going"  # features/step_definitions/webrat_steps.rb:89
  65  
  66    Scenario: Examine Categories of Women       # features/programmes.feature:67
  67      Given I have some programmes              # features/step_definitions/pctv_steps.rb:1
  68      And I visit "/"                           # features/step_definitions/pctv_steps.rb:24
  69      When I follow "Women"                     # features/step_definitions/webrat_steps.rb:10
  70      Then I should see "Loose Women"           # features/step_definitions/webrat_steps.rb:85
  71      And I should not see "Dora the Explorer"  # features/step_definitions/webrat_steps.rb:89
  72      And I should not see "Star Trek"          # features/step_definitions/webrat_steps.rb:89
  73  
  74    Scenario: Search for Dora                # features/programmes.feature:75
  75      Given I have some programmes           # features/step_definitions/pctv_steps.rb:1
  76      And I visit "/programmes"              # features/step_definitions/pctv_steps.rb:24
  77      When I follow "Search"                 # features/step_definitions/webrat_steps.rb:10
  78      And I fill in "Search" with "Dora"     # features/step_definitions/webrat_steps.rb:14
  79        Could not find field labeled "Search" (RuntimeError)
  80        /usr/lib/ruby/gems/1.8/gems/webrat-0.3.2/lib/webrat/core/flunk.rb:4:in `flunk'
  81        /usr/lib/ruby/gems/1.8/gems/webrat-0.3.2/lib/webrat/core/locators.rb:16:in `field_labeled'
  82        /usr/lib/ruby/gems/1.8/gems/webrat-0.3.2/lib/webrat/core/locators.rb:10:in `field'
  83        /usr/lib/ruby/gems/1.8/gems/webrat-0.3.2/lib/webrat/core/scope.rb:183:in `locate_field'
  84        /usr/lib/ruby/gems/1.8/gems/webrat-0.3.2/lib/webrat/core/scope.rb:41:in `fills_in'
  85        /usr/lib/ruby/gems/1.8/gems/webrat-0.3.2/lib/webrat/rails.rb:88:in `send'
  86        /usr/lib/ruby/gems/1.8/gems/webrat-0.3.2/lib/webrat/rails.rb:88:in `method_missing'
  87        /usr/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/integration.rb:498:in `__send__'
  88        /usr/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/integration.rb:498:in `method_missing'
  89        ./features/step_definitions/webrat_steps.rb:15:in `And /^I fill in "(.*)" with "(.*)"$/'
  90        features/programmes.feature:79:in `And I fill in "Search" with "Dora"'
  91      And I press "Seek"                     # features/step_definitions/webrat_steps.rb:6
  92      Then I should see "Dora the Explorer"  # features/step_definitions/webrat_steps.rb:85
  93      And I should not see "Loose Women"     # features/step_definitions/webrat_steps.rb:89
  94      And I should not see "Star Trek"       # features/step_definitions/webrat_steps.rb:89
  95      And I press "Seek"                     # features/step_definitions/webrat_steps.rb:6
  96      Then I should see "Dora the Explorer"  # features/step_definitions/webrat_steps.rb:85
  97      And I should not see "Loose Women"     # features/step_definitions/webrat_steps.rb:89
  98      And I should not see "Star Trek"       # features/step_definitions/webrat_steps.rb:89
  99  
 100  
 101  58 steps passed
 102  1 steps failed
 103  4 steps skipped
Toggle Line Numbers

The error above is because webrat can't find the Search box in my form, although it looks good to me. Not the easist thing to debug, dump the html into the console and peer at it, viewing source in a web browser is easier. Webrat has a function save_and_open_page that only seems to work in OSX.

UPDATE: found the problem with the search term. My search form was set up like this:

<% form_tag(:controller => :programmes, :action => :search) do %>
  <p>
    <%= label_tag "Search" %> <%= text_field_tag :search_term %>

    <%= submit_tag "Seek" %>
  </p>
<% end %>

I kinda assumed that webrat used the fact that the label was before the text field in the html to associate the two but it turns out there is a correct way to do this association. The following works, the :search_term parameter to label_tag makes it generate a 'for' attribute that points to the text input field:

<% form_tag(:controller => :programmes, :action => :search) do %>
  <p>
    <%= label_tag :search_term, "Search" %> <%= text_field_tag :search_term %>

    <%= submit_tag "Seek" %>
  </p>
<% end %>

Webrat is helping me validate my html.

I still have a few problems with this:

• in the example of a checkbox, I can set up a test to check a checkbox and I can see it checked but as an engineer I'm itching to look in the database to make sure it is checked. I have to resist the urge to test at the cucumber level and tell myself the view, controller and model tests would have that sorted.
• to test that a checkbox can be unchecked I stuffed the database with a record that is already checked. I could do these steps:

      Scenario: Mark Dora series link
          Given I have some programmes
              And I visit "/"
          When I follow "Dora the Explorer"
              And I check "Series Link"
              And I press "Update"
              And I uncheck "Series Link"
              And I press "Update"
          Then I should see "Dora the Explorer"
              And I should see "Tomorrows Episode"
              And I should not see "Yesterdays Episode"
              And I should not see "Loose Women"
              And the "Series Link" checkbox should not be checked
  

  and I have to rely on having already tested that checking works. Writing now and thinking more, this seems ok.
• my specs are lacking in justifications. This happens in most specifications, things are detailed with no details on why things are this way. For example, in my test I click on 'Dora the Explorer' and I don't see 'Loose Women' because clicking on Dora the Explorer takes me to the episodes of Dora and Dora doesn't usually feature loose women. The test above doesn't state this unless I add comments which breaks the idea of the formal definition language. Maybe it needs something like 'because'?

      Then I should not see "Loose Women" Because that is not an episode of Dora
      And I should not see "Yesterdays Episode" Because it should only show future episodes
  

  However since I can define the Then terms myself there is nothing stopping me adding Because to my tests and this could then force me to put in my justifications!
• tests where you don't want to see something are fragile. I could be happy that I don't see "Yesterdays Episode" and then I later break things to that the text "Yesterday's Episode" is generated but the test still passes.

I can definitely see some use for this in keeping auditors happy.

This is how I gather my twitterings into the blog. I just scrape the rss feed. The twitter api may allow for something more sophisticated but this Just Works.

It's just too easy in ruby/rails:

   1  class TwitterHandler
   2    def update( strFile)
   3      require 'open-uri'
   4      require 'simple-rss'
   5  
   6      oRss = SimpleRSS.parse( open( strFile))
   7  
   8      oRss.items.each do |oItem|
   9        strBody = oItem.description
  10        strBody.gsub!( /^petersblog: /, '')
  11        strTitle = "Twittering: #{oItem.pubDate.strftime( "%d %B %Y %H:%M")}"
  12  
  13        if not Post.find_by_title( strTitle)
  14          oPost = Post.new( :title => strTitle, :body => strBody, :created => oItem.pubDate)
  15          oPost.Taglist = "twittering"
  16        end
  17      end
  18    end
  19  end
  20  
  21  The above is only wrapped in a class because that is how the first rspec example I read was laid out.
  22  
  23  Then there is a rake task to run the above:
  24  
  25  <ruby>
  26  task (:twitter => :environment) do
  27    require 'lib/TwitterHandler'
  28  
  29    oTwitter = TwitterHandler.new
  30    oTwitter.update( "http://twitter.com/statuses/user_timeline/16738335.rss")
  31  end
Toggle Line Numbers

Then a cron job:

13,28,43,58 * * * *  bash -c "cd /var/www/PetersBlogger/current; rake -s RAILS_ENV=production twitter"

This polls every 15 minutes. The '-s' stops rake outputting a needless 'In directory blah' message which cron dutifully emails to me.

I'm enjoying using twitter, I like the medium, short and pithy. Not really suitable for code snippets though.

I was working on this site and noticed how slow it was, particularly on rss feeds. This was just after managing to completely screw up my ruby gem installation (apt-get remove rubygems/apt-get install rubygems didn't reinstall rubygems) which I only cured by upgrading the server to ubuntu hardy. I wasn't sure whether I had blown something in the upgrade (the server did a fsck when I rebooted it but maybe only because it had been 'up' for 100 days) or whether the code was just slow.

How to pinpoint the cause of the slowness? Profiling. Some research revealed I can profile ruby code as follows:

• install a ruby profiler:

  gen install ruby-prof
  

• insert some profiling code around your suspect slow code (in my case around the rss formatting):

     1  gem install ruby-prof
     2  
     3     require 'ruby-prof'
     4  
     5     RubyProf.start
     6  
     7     {code to profile}
     8  
     9   result = RubyProf.stop
    10     printer = RubyProf::FlatPrinter.new(result)
    11     printer.print(File.open( '/tmp/prof.txt', "w"), 0)
  Toggle Line Numbers

• run script/server and try downloading rss
• wait because it is very slow
• examine the profile

  vim /tmp/prof.txt
  

• lament

This told me that 70% of my time was being spent in a gsub in my wilki code that I added last week. gsub was being called 1800 times (!) and when I commented out the new gsub call the rss was rendered much faster.

The thing is, my wilki formatting is supposed to be cached, it should only be done when a posting is modified, not every time the post is rendered. Turns out to be because updating a field in an ActiveRecord doesn't automatically cause it to be saved because it is dirty, an explicit save helped. Problem solved, now downloading the rss feed doesn't max out my cpu for 20 seconds

NB (clarification): the rss doesn't go through the rails fragment cache.

My websites are all migrating to a CMS kinda vibe where pages are edited through the web browser. Easy for me, easy for anyone else. I've already got the technology to upload pictures but it has an annoying problem and it seems to be built into the attachment_fu plugin. The problem is that one must choose how big one wants the pictures to be when they are first uploaded and the size is defined in the database model. attachment_fu will then resize the image accordingly. That is all very well if one is organised and prepared and knows in advance how big they want their pictures or are running a simple gallery where all pictures are the same size.

But what about where one is trying to do some nice layouts on a page and wants to adjust the sizes of the pictures? Say I decide I want a particular picture on a page to be 100 pixels wide? I have two options:

• I can go into gimp/paintshop pro/whatever, resize the image and upload it (again) but that is annoying.
• I can alter the img tags in my html to include a width attribute. I see a lot of this on other websites, as the page downloads you see lots of little images slowly filled in as huge 1000x1000 jpg's are downloaded and resized in the browser to 100x100 because the page designer couldn't be bothered to resize the image in gimp/paintshop pro/whatever.

Now there is another way. Here is a rails helper that allows the width of an image to be specified in a view, automatically resizing the image as appropriate:

   1  # Methods added to this helper will be available to all templates in the application.
   2  module ApplicationHelper
   3  
   4     #
   5     # Takes the url of an image and a desired width and ensures a cached resized
   6     # version of the image is available and returns the url of said image.
   7     #
   8     # the url should not be a relative url as it is assumed work relative to RAILS_ROOT/public
   9     # This also means the image must be on this server.
  10     #
  11     def cacheimage( strUrl, nWidth)
  12       # Find original file in file system
  13       strFile = "#{RAILS_ROOT}/public" + strUrl
  14  
  15       # create name for cached and resized image
  16       #
  17       # /blah/blah.jpg --> /blah/blah-123.jpg
  18       #
  19       strBits = strUrl.split( '.')
  20       strBits[-2] += "-#{nWidth}"
  21       strCacheName = strBits.join( '.')
  22  
  23       strCacheUrl  = "/imagecache#{strCacheName}"
  24       strCacheFileName = "#{RAILS_ROOT}/public#{strCacheUrl}"
  25  
  26       if not File.exists?( strCacheFileName)
  27         require 'image_science'
  28  
  29         FileUtils.mkpath( File.dirname( strCacheFileName))
  30  
  31         ImageScience.with_image( strFile) do |oImg|
  32           oImg.resize( nWidth, nWidth * oImg.height / oImg.width) do |oResized|
  33             oResized.save strCacheFileName
  34           end
  35         end
  36       end
  37  
  38       return strCacheUrl
  39     end
  40  end
Toggle Line Numbers

That wasn't too difficult. You need to create an 'imagecache' directory under /public to store a cache of resized images and make sure mongrel or whatever can write to it.

In a view one would put:

<%= image_tag cacheimage( "/images/prettypicture.jpg", 500) %>

This will cause a version of prettypicture.jpg to be created that is resized to 500 pixels wide with a proportional change in height (in a typical html page layout the width of things is often more important than the height except maybe for headers and footers). The resized version is stored in a cache and will be displayed from then on.

If the original version of prettypicture.jpg is changed then one must remember to erase the resized versions from the cache as this isn't automatic.

Another helper is:

   def applyimagewidth( strText)
      strText.gsub!( /([^"!<>{}]*)(jpg|JPG|gif|GIF|png|PNG)x(\d+)/) do |oMatch|
         cacheimage( "#{$1}#{$2}", $3.to_i)
      end

      return strText
   end

(UPDATE: warning- the gsub above is VERY SLOW, it can take minutes to scan a few k of text, even if there are no matches).

which allows the width to be appended to the image file name. This allows a view such as

<%= textilize applyimagewidth( @page.body) %>

and thus one's textile source could be:

!/imagecache/images/DSC_4064-300.JPG!

to show an image with a width of 300 pixels. I could hack the redcloth source directly to add this feature but I'd rather keep out of there.

I've done this in a simplistic hacky way but it should suffice (like all my stuff).

Netbeans and cygwin ruby don't have a happy relationship. To use the cygwin ruby interpreter it is necessary to launch Netbeans (6.1) with the arguments:

-J-Druby.no.sync-stdio=true

as otherwise Netbeans will pass a windows-style file name to the cygwin ruby that the latter cannot recognise (C:\Projects\Blah instead of /cygdrive/c/Projects/Blah). Unfortunately the ruby interpreter configuration in Netbeans tries to be clever and insists you point it to a ruby executable, you can't give it a batch file that will hack things into working shape.

rspec on cygwin has some similar problems. Here are some hacks to the rspec plugin to make it work from netbeans:

In vendor/plugins/rspec/bin/bin, change:

$LOAD_PATH.unshift(File.expand_path(File.dirname(__FILE__) + "/../lib"))

to

$LOAD_PATH.unshift(File.dirname(__FILE__).sub( /C:\//, '/cygdrive/c/') + "/../lib")

In vendor/plugins/rspec/lib/spec/runner/example_group_runner.rb, change:

load file

to

load File.expand_path(file).gsub(/C:/, '/cygdrive/c')

Tip: to diagnose problems with 'require' not finding files, put this:

p $:

before the failing line to see what directories are in the load path. If you see 'C:\' and it's cygwin then fix it.

Unfortunately it takes a good 20 seconds for cygwin/rspec to run a trivial test script. The rspec server starts and runs but netbeans/rspec seem reluctant to use it (it runs tests ok from the command line).

Why am I still bothering with cygwin? Because I still want to test gcc C code by compiling it into ruby extensions and using ruby test codes (rspec or unit testing).

In tracing these problems I came across more reasons that Netbeans is worth the hassle:

• ctrl-tab to switch to last open tab. ctrl and hold tab brings up a handy little context menu to choose a tab from (kinda like windows task switching with tab).
• press ctrl and mouse over an identifier to bring up rdoc info about it. Ctrl-click to go to it's definition.
• alt-b to go back to where you came from.
• click on an 'end' and the matching begin/do/if is highlighted and vice versa, which is damn useful.
• select an identifier and all matching identifiers in your file are automatically highlighted.

I've been trying out rspec, a new testing framework for ruby that allows one to better follow the principles of Behaviour Driven Design (BDD). BDD is a step on from Test Driven Design (TDD) approach exemplified in the standard ruby unit test framework. TDD is the process of starting ones development by writing the tests that your new module should pass. As you write each test you then implement the code to pass it and hence, voila, you have nicely designed and robust software. In BDD you approach testing the code from how it is supposed to behave. When liasing with clients this is a better approach as everyone can talk in the same language, essentially in terms of requirements so one can try to get less bogged down in implementation details.

Here are some rspec tests for a class I have been working on:

   1  describe ReportHelper do
   2    before(:each) do
   3      #
   4      # Create a new clean Report class for each individual test
   5      #
   6      @oReport = ReportHelper::Report.new
   7    end
   8  
   9    it "should allow a column to be added" do
  10      @oReport.AddColumn( 'First Column')
  11    end
  12  
  13    it "should allow multiple columns to be added" do
  14      @oReport.AddColumn( 'First Column')
  15      @oReport.AddColumn( 'Second Column')
  16    end
  17  
  18    it "should allow rows to be added to the report" do
  19      @oReport.AddColumn( 'First Column')
  20      @oReport.AddColumn( 'Second Column')
  21      @oReport.AddRow do
  22        # pass
  23      end
  24    end
  25  
  26    it "should only allow rows to be added if columns exist" do
  27      lambda{ @oReport.AddRow}.should raise_error(RuntimeError, 'No columns defined')
  28    end
  29  
  30    it "should return simple cell values" do
  31      @oReport.AddColumn( 'First Column')
  32      @oReport.AddColumn( 'Second Column')
  33  
  34      @oReport.AddRow do |oRow|
  35        oRow[ 'First Column'] = 98
  36        oRow[ 'Second Column'] = 198
  37      end
  38  
  39      @oReport[0]['First Column'].should == 98
  40      @oReport[0]['Second Column'].should == 198
  41    end
  42  
  43  end
Toggle Line Numbers

Each test has a descriptive string that can be interpreted in two ways: as a test that must be passed or as a requirement for the software. The rspec toolset includes something that will pull out all these strings and give a summary of what testing has been done/what the requirements for the software are. If an auditor asks how the software is tested, bang, there is a full report of the tests. At a practical level (i.e. makes life easier for me) I prefer this to the ruby unit test system where one has to think up meaningful_method_names for each test. The strings are easier to write and read and reduce the need for comments.

Example report:

ReportHelper

- should allow a column to be added
- should allow multiple columns to be added
- should allow rows to be added to the report
- should only allow rows to be added if columns exist
- should return simple cell values

Rspec contains sweeter syntactic sugar than the ruby unit test module. For example, rspec overloads every object in the system with a 'should' method for doing the equivalent of assert tests:

    @oReport[0]['First Column'].should == 98

Compare this to test/unit:

    assert_equal 98, @oReport[0]['First Column']

apart from more typing, assert_equal has the expectation/actual things in what is intuitively to me the wrong order and I'm always having to correct these lines because the error messages have the terms swapped.

Rspec can test assertions more cleanly:

    lambda{ @oReport.AddRow}.should raise_error(RuntimeError, 'No columns defined')

although it's a pity that the lambda is necessary (and probably no 1 noob trap) but with test/unit the following always seemed like a lot of typing:

    oError = assert_raises RuntimeError do
      @oReport.AddRow
    end
    assert_equal 'No columns defined', oError.message

rspec is nicely integrated into rails, plugins being available to generate rspec tested controllers and models and suchlike, rake tasks to generate documents (woohoo) etc. It is integrated into netbeans and runs ok on Windows although when the tests are run (CTRL-F6) there is a tedious few seconds wasted in startup time. It is possible to run an rspec server in the background to eliminate this startup but it doesn't work on windows native ruby, it gives an error lamenting Microsofts glorious omission of a fork api.

There is much more to Rspec than this. One aim seems to be to write psuedo testing languages and have clients use these to write stories about what the software will do, these stories turning into a test framework. As the slashdot tag goes, 'goodluckwiththat'. At a practical level, I'm liking rspec even if only as a unit test framework with added syntactic sugar.

Much as I hate flash, it is a nice way to embed a sound or video player in a page. Here's how I did it in rails using the JW FLV Media Player. I added the following to my view:

   1  <script type="text/javascript" src="/javascripts/swfobject.js"></script>
   2  
   3  <% for oRssItem in @rssitems %>
   4    <hr/>
   5    <h3><%= oRssItem.title %></h3>
   6    <p><%= oRssItem.description %></p>
   7    <table width="470">
   8      <tr>
   9         <td wIdth="50%"><%= oRssItem.created.strftime( "%m %B %y") %></td>
  10         <td align="right"><%= link_to "Download", "/#{ oRssItem.filename}" %></td>
  11      </tr>
  12    </table>
  13  
  14    <div id="player<%=oRssItem.id%>">This text will be replaced</div>
  15  
  16    <script type="text/javascript">
  17      var so = new SWFObject('/mediaplayer.swf','mpl','470','20','8');
  18      so.addParam('allowscriptaccess','always');
  19      so.addParam('allowfullscreen','true');
  20      so.addVariable('width','470');
  21      so.addVariable('height','20');
  22      so.addVariable('file','/<%= oRssItem.filename%>');
  23      so.write('player<%=oRssItem.id%>');
  24    </script>
  25  <% end %>
Toggle Line Numbers

This generates a page full of podcast entries, each with a little media player thing so you can listen to them easily. This does mp3 files but it can handle video too.

I am aware that the swfobject.js would be better loaded in the html header.

Here's a demo: I don't know if this will work in the rss feed (UPDATE: no!) so bear with me and please be gentle on my bandwidth, it's quite a catchy tune:

The main reason I am blogging this is because it only needs a change to a view file and installation of two files in the /public directory (the player and a javascript file). Since view files are awkward to comment legibly I decided to document it all here so I can find it again.

In a mammoth programming session I implemented proper smileys

[[':-)', 'smile'],
  [':-(', 'sad'],
  ['8-)', 'cool'],
  [';-)', 'wink']].each do |strCode, strFile|
  strText.gsub!( strCode, "<img alt=\"#{strFile}\" src=\"/images/smileys/#{strFile}.png\" />")
end

My fingers were bleeding from all the typing Worked first time though

There is even alt text for blind people

I figured out how to scrape the Wow Armory for up-to-date information about my characters using ruby. This means I can do this directly in PetersBlogger without using third party signature generators that disappear overnight or get absorbed into wow gold sites.

   1    require 'net/http'
   2  
   3    #
   4    # Scrape wow character info.
   5    #
   6    def WoWInfo
   7      begin
   8        strRealm = 'Eonar'
   9  
  10        oInfo = []
  11  
  12        ['Pookypoo', 'Maevyn', 'Maezyn', 'Maexyn'].each do |strCharacter|
  13          oCharInfo = []
  14  
  15          #
  16          # Open url.
  17          # Need to specify firefox as user agent as this makes the server return an XML
  18          # file.
  19          # Look the data up in the european armory. Change this for US.
  20          # If this is not done we get html.
  21          oResp = Net::HTTP.start( "armory.wow-europe.com", 80) do |http|
  22                http.get( "/character-sheet.xml?r=#{strRealm.tr( ' ', '+')}&n=#{strCharacter}",
  23                            { 'user-agent' => 'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-GB; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4'})
  24          end
  25  
  26          oDoc = REXML::Document.new oResp.body
  27          oDoc.elements[1].elements.each( 'characterInfo/character') do |oElement|
  28            oCharInfo << {:race => oElement.attributes['race']}
  29            oCharInfo << {:class => oElement.attributes['class']}
  30            oCharInfo << {:level => oElement.attributes['level']}
  31          end
  32  
  33          oDoc.elements[1].elements.each( 'characterInfo/characterTab/talentSpec') do |oElement|
  34            oCharInfo << {:spec => "#{oElement.attributes['treeOne']}/#{oElement.attributes['treeTwo']}/#{oElement.attributes['treeThree']}"}
  35          end
  36  
  37          oDoc.elements[1].elements.each( 'characterInfo/characterTab/professions/skill') do |oElement|
  38            oCharInfo << {oElement.attributes['key'].to_sym => oElement.attributes['value']}
  39          end
  40  
  41          oInfo << {strCharacter => oCharInfo}
  42        end
  43  
  44        return oInfo
  45      rescue
  46        return []
  47      end
  48    end
Toggle Line Numbers

This lot generates the info in this page which is linked to under 'My Characters' over to the right. It should be fairly up-to-date as the cache will flush once a day causing the armory to be requeried. Currently the output is presented in a rails view thus:

   1  <% cache( :part => :wow) do %>
   2    <table>
   3    <% WoWInfo().each do |oCharInfo| %>
   4      <% oCharInfo.each_pair do |strChar, oInfo| %>
   5        <tr><td colspan="2"><b><%= strChar %></b></td></tr>
   6        <% oInfo.each do |oItem| %>
   7          <% oItem.each_pair do |strItem, strValue| %>
   8            <tr>
   9              <td><%= strItem %>:</td><td><%= strValue %></td>
  10            </tr>
  11          <% end %>
  12        <% end %>
  13      <% end %>
  14    <% end %>
  15    </table>
  16  <% end %>
Toggle Line Numbers

All the work is done by the WoWInfo method which I put in the application helper. I didn't use a controller method for this.

I'm going to think about a nicer presentation with some pictures of the actual guys rather than generic pictures of bald redhead dwarves.

UPDATE: I've added my old Aerie Peak toons to the output but the code above still stands. Lugulas was my old auction alt who I've never named here before, hence level 10 with level 75 (dis)enchanting. I find myself missing them...

I've described before how I've been testing Firmware code written in C by compiling it into Ruby Extensions and using ruby's excellent unit testing framework.

Up till now I've been testing the 'middle tier' algorithmic code but how to go about testing the low level stuff, that which pokes the hardware?

The firmware I am developing is for the ST ARM750 microcontroller and I am using ST's own standard library which is very useful driver code for all the peripherals embedded on the chip. It is a very useful resource as the peripherals on this chip are very complex and the user manual is written in hardware engineer gobbledygook that you have to study like a legal document, looking for the small print (e.g. running it at 60MHz it kept crashing until I found the small print that advised me to enable 'burst mode').

Scanning through the source to this library I saw some #ifdef DEBUG statements that implied that it could be compiled into an emulation library. The code was written very cleanly so it took me very little time to write something in (of course) ruby to scan the header files and generate 5250 lines of C code to emulate this library. Here is an example of one of the generated library functions:

   1  /*
   2   * Emulate the function TIM_ClearFlag
   3   */
   4  void TIM_ClearFlag(TIM_TypeDef* TIMx, u16 TIM_FLAG)
   5  {
   6    VALUE nRet;
   7    VALUE oArgs = rb_ary_new();
   8    VALUE strFuncName = rb_str_new2( "TIM_ClearFlag");
   9    ID nId = rb_intern( "Function");
  10  
  11    rb_ary_push( oArgs, PassStructureArg( (void *)TIMx, sizeof( TIM_TypeDef)));
  12    rb_ary_push( oArgs, LONG2NUM( TIM_FLAG));
  13  
  14    nRet = rb_funcall( g_oCallBack, nId, 2, strFuncName, oArgs);
  15  
  16    /* nRet is not used */
  17  }
Toggle Line Numbers

TIM_ClearFlag is the name of the function in the ST standard library. The emulation code takes the name of the emulated function and the two arguments to the function, converts them into ruby objects and passes them to a method called 'Function' in a ruby object. In the testing case, this object is something that will make sure the functions are being called in the correct sequence and with the correct parameters and will return the appropriate return value. Structure arguments are passed as binary objects which the ruby can decipher from the structure definitions in the ST library header files (also parsed using ruby).

Note that my system has a few limitations that are acceptable because of the simplicity of the ST library and the way it was coded:

• The parser I wrote just about handles the coding style of this library
• The calls to ruby do not support the case where the library may poke the values within a structure passed to it by pointer.
• It doesn't handle passing structures by value.

Now I have my low level testing library I will be able to do my testing using something I call signature analysis (I'm not sure there is a better software engineering term). I write the code, test it on the real hardware, then I can record the sequence of function calls made including the values of function arguments. Later on, in leiu of real hardware to test it on I can run the test code and compare it against the previously recorded signatures. Et voila, the reassurance that I haven't broken anything and I can sleep at night.

UPDATE:

My test library now includes a YamlFaker that can be used in two modes:

1. it will record a sequence of function calls
2. it will ensure a sequence of function calls matches a previous recording

The yaml script looks like this:

#
# This file is maintained by the YamlFaker module DO NOT EDIT
#
---
- test_CommandPWMSet Demand -200:
  - :Comment: |-

      Looking for demand to be correct according to the requested range and the
      scaling factors. Also ensure that for the fifth channel, the period is
      large for small demand as the output of this channel is inverted
  - TIM_SetPulse:
      args:
      - TIMx: TIM0
      - TIM_Channel: "0x2"
      - Pulse: "0x0"
  - TIM_SetPulse:
      args:
      - TIMx: TIM1
      - TIM_Channel: "0x2"
      - Pulse: "0x0"
  - TIM_SetPulse:
      args:
      - TIMx: TIM2
      - TIM_Channel: "0x2"
      - Pulse: "0x0"
  - PWM_SetPulse:
      args:
      - PWM_Channel: "0x2"
      - Pulse: "0x0"
  - PWM_SetPulse:
      args:
      - PWM_Channel: "0x4"
      - Pulse: "0x12c"
- test_CommandPWMSet Demand -100:
  - TIM_SetPulse:
      args:
      - TIMx: TIM0
      - TIM_Channel: "0x2"
      - Pulse: "0x0"

Most of the entries in the script describe the expected sequence of function calls, complete with arguments. I can ensure that the demand being sent to my PWM channels is what I expect. These recorded scripts could be quite useful, it is a complete log of the interaction with the hardware. If I ever had to change the code that poked the hardware I can compare the script recorded from the new code against my old script and make sure that any changes are what I would expect. This way I can be far more certain that my changes are correct before they hit real hardware.

I've been testing my C libraries by compiling them into ruby extensions and then using ruby's unit testing to test them. Under Windows I had it all running nicely and I could run a main test script that would invoke all the other test scripts. Sweet.

Then it came to getting it running under linux (the target system) and while the unit tests ran fine individually, when I tried my main test script I was getting weird errors from my test extensions, including segment violations (!).

It took me a while to track down the problem but it turned out to be due to the behaviour of gcc and the linux program loader. All my test libraries were loaded into one ruby process and because they all contained functions with similar names it became a bit of a lottery as to which function in which library was called. I am used to the behaviour of windows dll's where all functions are by default hidden from the outside world unless you explicity export them. gcc/elf/whatever seems to export everything from shared libraries (linux equivalent of a dll) by default with hilarious results.

In version 4 of gcc they have added a facility for hiding these internal functions. I solved my problem as follows:

• Add '-fvisibility=hidden' to the gcc arguments to cause all functions to be hidden by default.
• Add this code to my stddefs.h:

     1  #ifdef _MSC_VER
     2    #ifdef BUILDING_DLL
     3      #define DLLEXPORT __declspec(dllexport)
     4    #else
     5      #define DLLEXPORT __declspec(dllimport)
     6    #endif
     7    #define DLLLOCAL
     8  #else
     9    #define DLLEXPORT __attribute__ ((visibility("default")))
    10    #define DLLLOCAL __attribute__ ((visibility("hidden")))
    11  #endif
  Toggle Line Numbers

• Change the functions that ruby calls to load the ruby extension to the following form (i.e. add DLLEXPORT):

     1  /*
     2   * Initialise the test class.
     3   */
     4  DLLEXPORT void Init_testtimer( void)
     5  {
     6      VALUE oMyClass = rb_define_class( "TestTimer", rb_cObject);
     7      rb_define_method( oMyClass, "initialize", test_initialise, 1);
     8      rb_define_method( oMyClass, "TimerSet", test_TimerSet, 1);
     9      rb_define_method( oMyClass, "TimerUpdate", test_TimerUpdate, 1);
    10      rb_define_method( oMyClass, "TimerStart", test_TimerStart, 1);
    11      rb_define_method( oMyClass, "TimerTest", test_TimerTest, 0);
    12  }
  Toggle Line Numbers

These functions are the only ones that need to be visible outside the library.

Results:

I forgot to mention the best ruby tip of all, the biggest time saver in the world ever.

p whatever

Simply prints the value of a variable on a new line. It is equivalent to:

print "#{whatever.inspect}\n"

or

puts "#{whatever.inspect}"

Who needs debuggers? Well I do occasionally but a well aimed print can shed light into the darkest gloom.

Moral: read more books.

I want to use rails as the front end for an application that I am developing. The application requires that I maintain a persistant object, i.e. one that lives on between multiple page requests. Given that my application would live in a single mongrel instance, is there an easy way to do this?

Well, I studied the code in the mongrel/rails stack (open source ftw) and it turns out that it is easy. Once the rails framework is loaded by mongrel, the dispatcher will handle any request that comes in and once the request is completed it will reload rails own reloadable classes (controllers, models etc) so that they are 'clean' and ready for the next request.

It is very easy to have an object that survives between individual page requests.

Here is an object:

   1  class SillyCounter
   2    def initialize
   3      @count = 0
   4    end
   5  
   6    def Inc
   7      return @count += 1
   8    end
   9  end
  10  
  11  $oSillyCounter = SillyCounter.new
Toggle Line Numbers

I put this in lib/SillyCounter.rb since I like CamelCase.

I load it in config/environment.rb, right at the bottom:

require 'lib/SillyCounter'

and in a view I added:

<p><%= $oSillyCounter.Inc %> pages served since last reboot</p>

Relaunch mongrel and it works!

CAVEAT: I'll emphasise that this will only work for a single process server! But my application is single user and will never need to serve 1000 requests/second! I would use dRb (distributed ruby) if I needed something scalable.

Todo: find a rails problem that takes more than 20 lines to get around.

PHP programmers out there may be interested to know that this is the first time I have used a $ in a variable name in ruby.

I've missed blogging while I've been working on PetersBlogger. My old drupal blog was limping along: while administering the site it kept logging me out. I can admit now that I was running an old version of drupal (4.7?) which I couldn't upgrade due to module dependencies (mainly awtags). The old code wasn't running that smoothly under php5 despite initial appearancies.

I'm happy with my new blog platform, it's running really nicely on my slicehost slice and I've learnt an awful lot about rails while developing it. I will try to do a braindump of rails experiencies in forthcoming posts (cross fingers).

Drupal is a fine CMS and I'm using Drupal 5 on my company intranet. My thoughts on a drupal vs ruby on rails debate would be:

• drupal/php/apache is easier to deploy than a rails/ruby/mongrel/nginx deploy: I moved my drupal site between about four different hosts and did it each time in less than an hour. A rails setup is more convoluted (especially mongrel_cluster). However, this site is running much faster under rails than it was under drupal. The difference is probably down to php and apache being more mainstream and slightly more refined and hence easy to set up. I'm not saying rails is a total pain, just that if I only had ten minutes in which to deploy a site I would be reaching for drupal. If I only wanted to install the source and a few modules and never touch any coding then I would be happy with drupal.
• I love ruby and rails development. There is no comparison, php seems to me as much a bastard language as visual basic 6. Ruby was cleanly developed as an object orientated language, php is having object orientation grafted on as an afterthough.
• If I look through my drupal source I see no unit testing. Ruby/Rails has unit testing built in and I'm completely sold on it. I don't think I will ever trust any code (especially code I write) if it hasn't been unit tested. Some interesting aspects of unit testing:
  • I understand now the 'test driven' approach to development where you design a module's api by first roughing out how the tests will work: the unit test is your first experience of using the new api. It's during testing that you learn how nice an api will be.
  • If the documentation of a new ruby/rails tool is dubious, look in the unit tests to see how the author intended it to be used. If there are no tests then run.
  • If I run into a tricky bug, it's better to first reproduce it in unit tests, fix it there and then be happy that the bug will never recur. It's easier to debug code in a unit test (essentially a command line application) than on a live website.
• all ruby on rails applications follow basically the same architecture. It is fairly easy to figure out how a new application works. Every php application is different. I know I'm comparing a web framework to a programming language but none of the php applications (drupal, phpmyadmin) or am familiar with (wordpress) share a common architecture. New php application to maintain? New learning curve.
• my new site is being hammered with attempts to post to /comments/reply, even though it doesn't use that url for comment submission. The comment spammers know drupal and know where to test the locks. If any of them ever bother to try to find my comment submission url they will only find the same captcha that protected my drupal site (which was much easier to implement in rails, simply as a validation on the model).

Sometimes software jargon gets in the way of understanding what practical use something is. For example, you have a problem and you want to know how to solve it so where do you look it up in the ruby book?

What I am going to describe is based on the concept of 'metaclasses' which are classes for creating classes. If that jargon fried your brain then lets have a problem and an example.

I want to create classes for implementing Commands. Each Command class has a command number and I want to define this Command Number in the class definition but I don't want to wear my fingers out typing them in. What nice syntactic sugar can ruby offer to help me with this?

   1  #
   2  # Class for building command classes
   3  #
   4  class Command
   5    class << self
   6      attr :nCommandNumber
   7  
   8      def CommandNumber( nNumber)
   9        @nCommandNumber = nNumber
  10      end
  11    end
  12  end
Toggle Line Numbers

Ok, horrible syntax so what on earth is going on here? Well the effect of 'class << <object>' is to add new methods to an object so in this case we are adding a new attribute 'nCommandNumber' to hold our command number and a new method 'CommandNumber' so set its value to an object referred to by 'self'.

So what is 'self' equal to? Well it turns out that when you use this class (Command) to create another class (ACommand) then 'self' will hold the value of the 'ACommand' class object, i.e. the object that creates instances of ACommand objects.

#
# Declare a specific command
#
class ACommand < Command
  CommandNumber 123
end

Hum, that looks easy. When ACommand is being defined the line 'CommandNumber 123' will result in the CommandNumber method in Command being executed on the ACommand class object so 'ACommand' will have an attribute called nCommandNumber added to it. We can see whether this happened with:

print ACommand.nCommandNumber
=>123

and indeed it works sweetly (which is what you would expect from syntactic sugar). It is important to note that it is the ACommand class itself that has these attributes, NOT objects CREATED by ACommand:

oAC = ACommand.new
print oAC.nCommandNumber
NoMethodError: undefined method `nCommandNumber' for #<ACommand:0x345e180>
        from (irb):15

so how can an object created by ACommand find the value of this attribute?

oAC.class.nCommandNumber
=> 123

one of the attributes of the object is the class that created it.

How would this be done in a lesser object orientated language? Probably using techniques such as:

   1  class Command
   2    def GetCommandNumber
   3       raise RuntimeError, "Pure base function called"
   4    end
   5  end
   6  
   7  class ACommand < Command
   8    def GetCommandNumber
   9       return 123
  10    end
  11  end
Toggle Line Numbers

or

   1  class Command
   2    def initialize( nCommandNumber)
   3      @nCommandNumber = nCommandNumber
   4    end
   5  end
   6  
   7  class ACommand < Command
   8    def initialize
   9      super( 123)
  10    end
  11  end
Toggle Line Numbers

or something similar. Like all syntactic sugar, it doesn't make the impossible possible (anything is possible in hand-coded assembler) it just makes for less typing and clearer code.

Found another ruby gotcha when calling methods declared with default argument values:

irb(main):004:0> def a( b=1,c=2,d=3)
irb(main):005:1> print "#{b} #{c} #{d}\n"
irb(main):006:1> end
=> nil
irb(main):007:0> a(c=23)
23 2 3
=> nil

A python programmer would have expected this to print

2 23 3

as when calling the function we are saying we want c to be 23 and the other arguments to be left at their default values. Ruby appears to be ignoring the 'c=' bit when the function is being called so the first argument 'b' gets the value 23.

Pity, this python trick simplifies calling functions with lots of default parameters: you don't have to get the order of the parameters right or specify the correct default values of arguments before the one you are having to specify a value for. I should be calling this with:

irb(main):007:0> a(b=1, c=23)

i.e passing 'b=1' even though the declaration should tell ruby what the default for that should be. The 'b=' and 'c=' are only serving to help me selfdocument my code.

Ruby's behaviour is C/C++ish i.e. primitive.

UPDATE:

O'Reilly Ruby Cookbook:

   1  def fun_with_text(text, args={})
   2    text = text.upcase if args[:upcase]
   3    text = text.downcase if args[:downcase]
   4    if args[:find] and args[:replace]
   5      text = text.gsub(args[:find], args[:replace])
   6    end
   7    text = text.slice(0, args[:truncate_at]) if args[:truncate_at]
   8    return text
   9  end
  10  
  11  fun_with_text("Foobar", {:upcase => true, :truncate_at => 5})
  12  # => "FOOBA"
  13  fun_with_text("Foobar", :upcase => true, :truncate_at => 5)
  14  # => "FOOBA"
  15  fun_with_text("Foobar", :find => /(o+)/, :replace => '\1d', :downcase => true)
  16  # => "foodbar"
Toggle Line Numbers

I expressed a desire to be able to test my C code by compiling it up as a Ruby extension. However I was unsure about the capabilities of such a C extension. I think in retrospect I had been confused by the complexity of the wrappers that swig generates, things are a lot easier than the 2000 lines of swig wrapper code would indicate.

So here's a howto covering creating a simple Ruby C extension running under cygwin on windows. I'm building cygwin extensions because I want to use the gpp compiler as I am also using a variant of this to generate source for an ARM microcontroller. The cygwin installation seems to coexist nicely with a standard windows installation and the cygwin build seems to be up to date. One has to be careful that the correct ruby is being invoked in the right circumstances: my setup is finding the standard windows version first on the path so I have to invoke the cygwin ruby and irb explicitly, e.g.

From Windows CMD prompt:

\cygwin\bin\ruby

From bash:

/bin/ruby

I have no idea how good a version of ruby the cygwin one is and I'm not inclined to test it out (UPDATE: 24-1-08 I've found nothing wrong with the cygwin ruby, it can run rails and everything else I've thrown at it). When I'm developing rails apps I would rather stick with the standard Windows version which is where I install all my gems. My cygwin is a vanilla ruby standard library (UPDATE: you can install gems into the cygwin ruby using /bin/gem at a bash prompt and they install just fine unless they require libraries that are not available in the cygwin repository).

Install the Tools

You will need to install Cygwin and, in particular, the following modules:

• gcc
• make
• ruby

The windows distribution of Ruby is compiled in Visual C++ 6 and so you cannot compile extensions for it using gcc. This howto creates extensions for the cygwin build of ruby.

Create a makefile

Ruby contains the tools to build the makefile for you. Create a file called 'extconf.rb' and in it put the following:

# Loads mkmf which is used to make makefiles for Ruby extensions
require 'mkmf'

# Do the work
create_makefile('try')

here 'try' is the name of my new extension. extconf.rb HAS to be run using the cygwin ruby, not the windows ruby. The easiest way to get it all working is to go into bash and run it with:

/bin/ruby extconf.rb

If you get a silly error like

/usr/bin/ruby: no such file to load -- ubygems (LoadError)

this can be fixed by installing ruby gems in your cygwin ruby installation.

Create the Source File

The source to the extension goes into 'try.c' which looks like this:

   1  /*
   2   * Just try
   3   */
   4  
   5  #include <stdio.h>
   6  #include "ruby.h"
   7  
   8  VALUE tryclass_initialise( VALUE self)
   9  {
  10    return self;
  11  }
  12  
  13  VALUE tryclass_doit( VALUE self)
  14  {
  15    printf( "doing it\n");
  16    return Qnil;
  17  }
  18  
  19  VALUE tryclass_whoops( VALUE self)
  20  {
  21    rb_raise( rb_eRuntimeError, "Whoopsie Daisy");
  22    return Qnil;
  23  }
  24  
  25  void Init_try( void)
  26  {
  27    VALUE oMyClass = rb_define_class( "TryClass", rb_cObject);
  28    rb_define_method( oMyClass, "initialize", tryclass_initialise, 0);
  29    rb_define_method( oMyClass, "Doit", tryclass_doit, 0);
  30    rb_define_method( oMyClass, "Whoops", tryclass_whoops, 0);
  31  }
Toggle Line Numbers

This is creates a class with just three methods:

initialize

initialization (sic) method that is called when the object is created.

Doit

silly method that just prints something

Whoops

the meat, a method that will raise a runtime error.

This should build sweetly:

make

The extconf should have guessed that the source code was in a file called 'try.c' and filled the makefile out accordingly.

Testing It

the whole point of this exercise is to be able to test my C using Ruby's unit test framework so this next bit better be good.

   1  require 'test/unit'
   2  require 'try.so'
   3  
   4  class TestTry < Test::Unit::TestCase
   5    def test_it
   6      o = TryClass.new
   7      o.Doit
   8    end
   9  
  10    def test_whoops
  11      #
  12      # Raise an error in the extension.
  13      #
  14      o = TryClass.new
  15      oException = assert_raises RuntimeError do
  16        o.Whoops
  17      end
  18      assert_equal "Whoopsie Daisy", oException.message
  19    end
  20  end
Toggle Line Numbers

A thing of beauty, call the C and expect an exception to be raised.

Run the unit test:

C:\Projects\757\try2>\cygwin\bin\ruby.exe test_try.rb
Loaded suite test_try
Started
doing it
..
Finished in 0.0 seconds.

2 tests, 2 assertions, 0 failures, 0 errors

Sweet.

In summary, taking out swig makes writing try.c more tricky but overall it clears up the picture for me.

The book programming Ruby has a whole chapter on writing Ruby Extensions.

I've really got into Unit Testing now. The Ruby Unit Testing framework makes it very easy. I've tried unit testing in the past on C++ projects but the lack of a good framework there meant that coding the tests was more laborious than coding the code it was testing.

So far I've written a Bootloader than downloads Hex files to the target firmware. It consists of the following modules:

• low level rs232 extension written in C++ and swig
• higher level rs232 port abstraction written in ruby
• Boot Loader modules to interface with the boot loader firmware embedded in the ARM7 microcontroller
• Intel Hex file decoder

I have been able to test these as follows:

• a test harness to test the low level RS232 extensions by using an RS232 loopback connection
• a test harness to test the higher level RS232 abstration using an RS232 loopback connection
• a test harness to test the Boot Loader module against the real microcontroller
• a test harness to test the Boot Loader module against a 'fake' rs232 port
• a test harness to test the parsing of Intel Hex files
• a test harness to download a 'known good' intel hex file to the microcontroller

I am able to test automatically just about anything here in one of three modes:

1. with the real microcontroller, real rs232
2. rs2332 loopback
3. 'fake'

The 'faker' is what a call a class that is able to emulate a port. Initially I wrote it to emulate an RS232 port in the following way:

• the 'faker' would receive what was being output to the port and check that it was correct
• the 'faker' was able to simulate data being returned from the port

I created a faker that parsed the input and output data from a IO class and hence could test the Bootstrap commands with a fake rs232 port like this:

   1    #
   2    # Test the Get command
   3    #
   4    def test_get
   5      strConversation = <<EOF
   6  # connect
   7  < 0x7f
   8  > 0x75
   9  # command
  10  < 0x01
  11  # data
  12  > 0x11 0x00 0x00 0x75
  13  EOF
  14      oFd = StringIO.new( strConversation)
  15      oPort = RS232Faker.new
  16  
  17      oPort.Run( oFd) do
  18        oBoot = BootLoader.new
  19        oBoot.Connect( oPort)
  20        oResults = oBoot.CommandGet
  21  
  22        assert_equal [0x11,0x00,0x00], oResults
  23      end
  24    end
Toggle Line Numbers

Lines in the fake script that start with a '<' is data that should be transmitted and lines that start with a '>' should be received.

The Bootloader module itself is written such that it can connect to either a real rs232 port or this fake one.

Within the Bootloader modules was a particularly complex method that converted memory address ranges into an array of sector numbers that needed erasing. I exposed this method so that I was able to test it specifically in the Unit Test.

The faker can be generalised beyond rs232 to handle anything that inputs and outputs. It doesn't even need to run a fixed script, it could contain heuristics to emulate hardware such as a temperature control channel and test whether the code does actually control temperatures.

I'm toying with the idea of testing my embedded C code by compiling it into a Ruby extension and calling it from Ruby Test Harnesses. The code would be compiled by gpp both for the ruby extension and the real hardware. This should provide thorough testing of the logical aspects of the code (e.g. servo control calculations) and may only be lacking in the following areas:

• it would not test the 'real time' aspects of the C code (interraction with interrupts etc)
• it would not test for problems in the code generation for the real firmware
• it would not test for differences between the processor architectures, e.g. number of bits in an integer, although the ARM is a 32 bit microcontroller.

However, this should provide a satisfying amount of testing.

Another problem with this is that I would have ruby calling a ruby extension in C which would in turn call 'faker' objects written in ruby. I am unsure how to go about doing this and I am unsure whether the faker objects would be able to raise exception back up to the main ruby script. It may be able to do this in a flaky and memory-leaking way but that isn't too much of a problem in a test harness: I am testing the algorithms, not the memory leaks in the test harness.

Conclusion: Unit Testing FTW

Some rubyisms to learn:

Convert String to Array of Characters

irb(main):004:0> "123".split(//)
=> ["1", "2", "3"]

Determine if an Object has a certain method

irb(main):006:0> "123".respond_to?( :split)
=> true

This is useful for working in terms of what objects can do rather than what objects are.

   1    def ReadFile( fd)
   2      #
   3      # If fd does not work with readlines (File or StringIO) then
   4      # assume it is the name of a file to open
   5      #
   6      if not fd.respond_to?( :readlines)
   7        fd = File.open( fd)
   8      end
   9  
  10      fd.readlines.each do |strLine|
  11        print strLine
  12      end
  13    end
Toggle Line Numbers

If a File object or StringIO object or anything else with a 'readlines' method that returns a collection of strings is passed to a function then it will process those lines. If the thing passed has no such method it is assumed to be a file name and the file is opened and the strings read from it.

Read Strings from Memory rather than a File

If you are sending test data to a function it is easier to create a StringIO object to pass the test strings into the function than writing the strings to a temporary file and asking the function to read the file:

strLines = << EOF
these are some test strings
they are declared in the source
without having to write to a temporary file
EOF

o = StringIO( strLines)

ReadFile( o)

Destructors?

I read that ruby has no destructors. If you want to control a resource and make sure it gets freed you can use the following paradigm:

   1  
   2  class MyThing
   3    def Run
   4       begin
   5         #
   6         # Allocate a thing
   7         #
   8         o = AllocateAThing()
   9         #
  10         # Pass it to the block that is passed to this function
  11         #
  12         yield o
  13       ensure
  14         #
  15         # Whatever happens, errors, returns, make sure the
  16         # thing is freed.
  17         #
  18         FreeThing( o)
  19       end
  20    end
  21  end
  22  
  23  #
  24  # Create thing wrapper
  25  #
  26  m = MyThing.new
  27  
  28  #
  29  # Run thing and pass block
  30  #
  31  m.Run do |oThing|
  32     #
  33     # The block is called with a reference to the thing
  34     #
  35     oThing.DoStuff()
  36  end
Toggle Line Numbers

It's an interesting way of doing things but it gets messy if you want a number of Things:

   1  #
   2  # Create thing wrapper
   3  #
   4  m = MyThing.new
   5  
   6  #
   7  # Run thing and pass block
   8  #
   9  m1.Run do |oThing1|
  10     m2 = MyThing.new
  11     m2.Run do |oThing2|
  12       oThing1.DoStuff()
  13       oThing2.DoStuff()
  14     end
  15  end
Toggle Line Numbers

Exceptions

I'm hooked on the RuntimeError exception:

irb(main):012:0> begin
irb(main):013:1*   raise RuntimeError, "Ouch that  hurt"
irb(main):014:1> rescue RuntimeError => err
irb(main):015:1>   print "Sh!t Happens:\n"
irb(main):016:1>   print err
irb(main):017:1> end
Sh!t Happens
Ouch that  hurt=> nil

RuntimeError is suitably catchall, everything happens at runtime. You pass it a string that explains the problem and can display/log/ignore it in the exception handler.

Argh

To extract a letter from a string:

irb(main):033:0> "123"[0,1]
=> "1"

The more intuitive way gives you the code of the letter instead:

irb(main):034:0> "123"[0]
=> 49

This is what you would expect in C but not from a modern scripting language.

Decided to try getting the Ruby Debugger working in my Eclipse install. I installed the ruby-debug-ide gem as stated in the debugger setup page but the debugger would not work: it kept coming up with an error about not being able to find rdebug-ide.

Googling gave mention of a bug where the path to the rdebug-ide had too many 'bin's in it: instead of looking in c:\ruby\bin it was looking in c:\ruby\bin\bin. I fixed the problem by creating a copy of c:\ruby\bin\rdebug-ide.cmd and placing it in c:\ruby\bin\bin.

I'm not sure if this problem has been fixed properly: the guy who wrote RDT is now working for Aptana and Aptana seem to be trying to set up a paid business model and their update site doesn't have RDT in it. I don't know if my RDT is the latest version or if it is available or from where.

Anyway, the debugger is running. It looks ok but no interactive prompt Also seems to baulk at routine exception handlers inside code you want to skip over. It has some usability issues:

Not a wonderful debugger, thank heavens for print statements.

I was trying to use blocks in ruby and had code of this form:

   1  
   2  def wangle
   3      while true
   4          nValue = rand(10)
   5          if yield( nValue)
   6              return
   7          end
   8          print "THIS IS NEVER PRINTED"
   9      end
  10  end
  11  
  12  def wibble
  13      wangle do |nValue|
  14          return nValue == rand(10)
  15      end
  16  end
  17  
  18  wibble()
Toggle Line Numbers

The method 'wibble' calls 'wangle' passing it a block of code to execute. Wangle is supposed to keep calling this block until two random numbers match. if the numbers match it will return true and cause the loop to terminate. However, the loop will always terminate immediately, the string "THIS IS NEVER PRINTED" is never printed.

The reason for this is blindingly obvious in hindsight and after analysing the small print describing the yield command. This says "the value of the last expression evaluated in the block is passed back to the method as the value of the yield". Notice how it doesn't mention the 'return' keyword? This is because the return keyword is being applied to the outer method 'wibble' and that is returning to whatever called it, completely bypassing 'wangle' on it's way up the call tree. If 'wibble' is changed to:

def wibble
    wangle do |nValue|
        nValue == rand(10)
    end
end

then it will pass the results of the comparison "nValue == rand(10)" back to wangle to deal with as I originally intended.

I can imagine that this subtlety was permitted as there may be occasions when you want wibble to be able to return directly to it's caller. In my ruby noobyness I'm getting blocks mixed up with methods. Unfortunately I like using the return keyword, it makes code easier to skim read, it waves a big flag to me that says THIS IS THE RETURN VALUE.

I have been working on my new project, using Eclipse as an IDE and ruby as a scripting language. Eclipse itself has it's quirks (such as having to create workspaces and projects before you can do anything) but once these are set up it is quite nice to use. I have discovered that the "Bitstream Vera Sans Mono" font is a very nice programming font to use in it, very crisp and clean and it doesn't have weird problems so I can use bold in syntax highlighting. I am also now, for the first time ever, sold on editors that display line numbers on each line, saves a lot of time when going through build/run errors. Then again, I am using a 22" Widescreen monitor with horizontal screen real-estate aplenty. UPDATE: on my 24" monitor at home in 1920x1200 mode Bitstream Vera Sans Mono looked very fuzzy until I enabled cleartype. That PC is running XP, maybe cleartype is on by default in Vista?

I have installed the RDT (Ruby Development Tools) eclipse plugin and I have found some very nice features:

• "Run as Ruby" or "Run as Test::Unit Test" options. I am creating Unit Tests in my new project and it is very easy to run the tests from inside eclipse. Eclipse includes a browser than displays the test results.
• RI documentation browser. A bit like fxri but it works a lot faster, although it raises an nasty looking error if it cannot find any search matches.

Since my Eclipse has both C development tools and Ruby development tools I am able to work nicely on both, which is useful since I had to write a ruby extension to handle serial communications. The extension is written in C++ but Eclipse is able to run a makefile to build it and then I can run Ruby Unit Tests to test the extension.

Eclipse has a wizard that is supposed to allow me to create new unit test files but it is totally confusing to use, always puts the file in the wrong directory and doesn't seem to give any huge advantages over simple 'New File' and 'Save As': it stills gives me the 'Run as Ruby' option.

SWIG and Ruby

I had tried using the ruby-serialport extension module for my serial comms but it was very painful to try building, requiring installation of Visual C++ version 6 (cygwin gpp wouldn't do) and Microsoft nmake rather than gnu make which in it's latest version doesn't work with extconf.rb: the makefiles it generates give syntax errors. Once I did get it to build it gave obscure gem errors when I tried to use it. It seems that it doesn't support the latest ruby libraries or something.

I decided to write my own simple ruby extension to handle rs232 so I went back to swig and this did it's usual great job of generating ugly boilerplate interface code for me. The biggest problem I found here was getting my module to generate ruby exceptions on errors but I solved that thusly. In my cpp code I put this:

   1  void RaiseError( const char *strFormat, ...)
   2  {
   3    static char strErrorBuffer[255];
   4    va_list args;
   5  
   6    va_start( args, strFormat);
   7    vsprintf( strErrorBuffer, strFormat, args);
   8    va_end( args);
   9  
  10    throw strErrorBuffer;
  11  }
Toggle Line Numbers

This formats up an error message and throws a C++ exception. It doesn't support __FILE__ and __LINE__ information but my extension is very simple and doesn't need it. This function is called wherever in the extension I want to raise errors, e.g.

   1  *
   2   * Read a byte from a serial port. Returns either the integer value of the byte received or -1
   3   * if no bytes are available.
   4   */
   5  int ReadByte( int nPort)
   6  {
   7    int nData = 0;
   8    ULONG nRxLength;
   9  
  10    HANDLE hPort = (HANDLE)nPort;
  11  
  12    if( ReadFile(hPort,  				// handle of file to read
  13    	&nData,               			// handle of file to read
  14        1,              				// number of bytes to read
  15        &nRxLength,                 	// pointer to number of bytes read
  16        NULL) == 0)              		// pointer to structure for data
  17    {
  18    	RaiseError("Read Serial port failed: 0x%08x", GetLastError());
  19    }
  20  
  21    if( nRxLength > 1) {
  22    	RaiseError("Read too many bytes?");
  23    }
  24  
  25    if( nRxLength > 0) {
  26    	return nData;
  27    } else {
  28    	return -1;
  29    }
  30  }
Toggle Line Numbers

In the swig interface definition file I put this which wraps each interface function with an exception handler that will catch the C++ expection and raise a ruby exception from it:

   1  %include exception.i
   2  
   3  %exception {
   4   try {
   5     $action
   6   }
   7   catch (char *strErr) {
   8     SWIG_exception( SWIG_RuntimeError, strErr);
   9   }
  10  }
Toggle Line Numbers

These exceptions can be caught just beautifully in Ruby Unit Tests so I know my error handling is working:

  #
  # Make sure an error is generated when attempting to read from a port that is not open.
  #
  def test_read_port_not_open
    oException = assert_raise RuntimeError do
      Rs232.ReadByte( 123)
    end
    assert_match( oException.message, 'Read Serial port failed: 0x00000006')
  end

Documentation Woes

In an ideal world I would be using either Doxygen or ruby's own RDoc to document my code. Both are designed to scan my source code and generate copious documentation for me. However, while Doxygen would do a great job with my C/C++ source it does yet handle Ruby. Conversely, RDoc does a great job on ruby source but it's support for C only extends as far as the C source for extension modules and this isn't working for my SWIG generated project. It seems as if the Rdoc C support has been written ONLY for C ruby extensions with little thought given to making it general purpose C documentation tool.

The ruby stuff I am writing will ultimately provide a programming API and the kind of documentation either of these tools would generate would be lovely. I don't really want to have to use two tools with different capabilities and output formats but it is looking like I will have to. Fortunately the C stuff should be pretty much embedded, the Ruby stuff is what other people must be able to use.

Another backing up setup is under construction. This time I'm backing up my project developed on a windows box to a linux server using rsync. It is using ruby as a scripting language. If this script is invoked thus:

ruby Backup.rb

then this script will merely update a simple backup of the source directory on the server. rsync is very fast and will only spend time uploading files that have actually changed and also it will only upload the changes to those files. I am using this on a 30M source archive and it takes less than a minute to update the backup. The backup files are just copies of the original files so easy to browse, diff, restore.

If the script is invoked thus:

ruby Backup.rb --backup

then it will use rsync's magic link-dest option to create historical archive of backups. What the script will do is this:

• Move the current main backup to an archive directory and give that archive directory a name that is date/time stamped.
• Create a new backup. Where files have not changed from the previous backup, the new backup will contain a link to the existing copy of the file in the archive, rather than consume more file space. The new backup directory will only contain files that are new or have been changed.

There is nothing in the script to limit the number of backups but they can be manually pruned every few months as required.

This is the script:

   1  #
   2  # This script is used to backup the source code to an offsite server
   3  #
   4  # If the parameter '--backup' is provided then create a backup directory
   5  #
   6  strMeAtMine = "me@myserver.org"
   7  strLocalFolder = "/cygdrive/c/projects/757/"
   8  strTargetFolder = "/home/pcw/757"
   9  strBackupFolder = "/home/pcw/Backup/757"
  10  
  11  #
  12  # If asked to backup then move previous backup to a backup directory
  13  # given a name related to date/time.
  14  #
  15  if ARGV.index( '--backup')
  16    bBackup = true
  17  
  18    strTimeStamp = Time.now.strftime( '%Y-%m-%d-%H-%M')
  19  
  20    strBackupDir = "#{strBackupFolder}/#{strTimeStamp}"
  21  
  22    strCommand = "ssh #{strMeAtMine} mv #{strTargetFolder} #{strBackupDir}"
  23  
  24    system( strCommand)
  25  else
  26    bBackup = false
  27  end
  28  
  29  #
  30  # Determine location of rsync exluded file list
  31  #
  32  strDir = File.dirname( __FILE__)
  33  strExcludeFile = File.join( strDir, "rsync-exclude.txt")
  34  
  35  #
  36  # Build command to invoke in an array of strings since this allows me to comment what
  37  # each parameter does.
  38  #
  39  strCommand = [ "rsync",
  40  #  	"-n",                                 # -n = dry run
  41    	"-v",                                 # verbose
  42                  "-a",                                 # archiving options
  43                  "--delete",                           # delete files no longer used from target
  44                  "--chmod=u=rwX",                      # set target file permissions
  45                  "--exclude-from=#{strExcludeFile}",   # exclude rubbish
  46                  "--delete-excluded",                  # delete excluded files from target
  47                  "-e ssh",                             # use ssh tunnel
  48                  strLocalFolder,                       # source
  49                  "#{strMeAtMine}:\"#{strTargetFolder}\"" # target
  50              ]
  51  
  52  if bBackup
  53    #
  54    # Backing up so instead of uploading everything again, link to files in the
  55    # backup directory where there are no changes.
  56    #
  57    strCommand.insert( -3, "--link-dest=#{strBackupDir}")
  58  end
  59  
  60  system( strCommand.join(" "))
Toggle Line Numbers

Since this is a backup of files from a windows box to linux I used the "--chmod=u=rwX" to specify simple file permissions on the linux end. Without this rsync was tending to create files with no access permissions for anyone.

The 'rsync-exclude.txt' file lives in the same directory as the script and the script uses some magic to find it. This is a list of stiff that doesn't need backing up:

/Downloads
*.map
Backup/*
Backups/*
*~

This uses the cygwin version of rsync which uses ssh to talk to the remote server. I have ssh set up with key files so I don't need to enter passwords.

I'm doing some Embedded Systems work on a new project and rather than splurge £2500 on the Keil ARM development tools I'm trying to run with gnu gpp and various open source options. The Keil stuff is very nice and includes emulators, debuggers and suchlike but these aren't must-haves. Also the £2500 gives you a single developer license. With the gnu tools we will have much more development flexibility.

As a toolchain I'm using gnuarm and I'm also trying the freertos embedded operating system. I managed to put together something that builds with the ST STR7 Standard Library which is effectively a set of device drivers for the on-chip peripherals. Freertos comes with many permutations of Arm7 derivatives, target development boards and development environments but not one that suited my setup so I had to munge one together myself.

Aptana has given me a taste for Eclipse based systems so I'm trying Eclipse itself as a general purpose IDE. I installed a version of Eclipse tailored to C development (CDE) and soon got it to launch a make file and build my project. The IDE is able to parse most (not all) of the error output from the build tools and give me a list of errors I can click on to go to the source line... all good basic IDE stuff I had in Turbo C twenty years ago (argh is it that long?).

Having got my Embedded code compiled the next step is to download it to the microcontroller. To do this I need to knock up a ruby script to read an intel hex file and download it via rs232 to the bootstrap loader built into the microcontroller (damn cool modern microcontrollers). I have a Keil development board with a cool JTAG widget that plugs into the USB and is effectively an in-circuit emulator but cost under £200. Unfortunately it is locked to the Keil tools so I can't use it...

I managed to add an RDT perspective for developing Ruby applications to my Eclipse setup. This means I can run and debug my ruby scripts from within my C development environment.. totally cool.

I came a cross a weird problem with a non-obvious solution that I'll record here to help other unfortunates. When I tried to run my ruby script I was getting the error:

"the specified jre installation does not exist"

which had me puzzled as eclipse was running fine with the JRE I knew about.

On a hunch I went through the ruby runtime options and discovered I didn't have a ruby VM set up. I created one, pointed it at my ruby installation and, joy of joys, the problem went away and I can run ruby from within Eclipse.

Similarly the RI Ruby Documentation viewer kept timing out with a 'Invalid thread access' error until I went into the ruby settings and pointed the rdoc and ri handlers at the 'c:\ruby\bin\rdoc.bat' and 'c:\ruby\bin\ri.bat' files.

Eclipse bullet point review:

• lots of features/over overcomplex, whichever way you look at it
• menu's with so many options you have to search for what you want every time you open one.
• there isn't a simple 'run' button, you have a menu full of run options to choose from
• I like the way you can install plugins for new features and there are 973 to choose from. I've mentioned Subclipse before but I've now got plugins to play with regular expressions and 'Wicked Shell' which gives me an integrated line command prompt where I can also run my makefile and see all the output.
• Misleading error messages

There's a lot to learn here, lots of workarounds and things I am discovering.

Fun times.

I wanted access to the help facility in Vista for reasons I forget now. I like the integrated search thing and I'm getting into the habit of just pressing the windows button and typing my query. I did this and typed 'help' and enter and what did I get? Windows Help? No. I got something called 'fxri'.

fxri turns out to be an interesting Ruby documentation browser with a built in Ruby Interactive Command Prompt. The documentation browser takes a few seconds to load but gives you very fast keyword filtering on search terms. The documentation displayed is a slightly crude monospaced text display (no hyperlinks) but can't beat the speed of access or the fact that there is a command prompt there to try things out immediately. Crude as it is, I think I prefer it to the approach taken by the web based ruby documentation: Frames? Web 0.5 anybody?

fxri gets the 'fx' part of its name from a gui package called Fox that may be worth investigating one day.

I must try typing more random words into the search and seeing what goodies I can find.

One other Vista note, it seems as if Microsoft are trying to disguise the annoyingly long time it takes for Windows to shut down by claiming to install an update every time.

Must say I'm still liking Vista, but still not enough to pay to upgrade my home pc's.