Peter's Blog - Nodes for testing

rspec

I've been trying out rspec, a new testing framework for ruby that allows one to better follow the principles of Behaviour Driven Design (BDD). BDD is a step on from Test Driven Design (TDD) approach exemplified in the standard ruby unit test framework. TDD is the process of starting ones development by writing the tests that your new module should pass. As you write each test you then implement the code to pass it and hence, voila, you have nicely designed and robust software. In BDD you approach testing the code from how it is supposed to behave. When liasing with clients this is a better approach as everyone can talk in the same language, essentially in terms of requirements so one can try to get less bogged down in implementation details.

Here are some rspec tests for a class I have been working on:

   1  describe ReportHelper do
   2    before(:each) do
   3      #
   4      # Create a new clean Report class for each individual test
   5      #
   6      @oReport = ReportHelper::Report.new
   7    end
   8  
   9    it "should allow a column to be added" do
  10      @oReport.AddColumn( 'First Column')
  11    end
  12  
  13    it "should allow multiple columns to be added" do
  14      @oReport.AddColumn( 'First Column')
  15      @oReport.AddColumn( 'Second Column')
  16    end
  17  
  18    it "should allow rows to be added to the report" do
  19      @oReport.AddColumn( 'First Column')
  20      @oReport.AddColumn( 'Second Column')
  21      @oReport.AddRow do
  22        # pass
  23      end
  24    end
  25  
  26    it "should only allow rows to be added if columns exist" do
  27      lambda{ @oReport.AddRow}.should raise_error(RuntimeError, 'No columns defined')
  28    end
  29  
  30    it "should return simple cell values" do
  31      @oReport.AddColumn( 'First Column')
  32      @oReport.AddColumn( 'Second Column')
  33  
  34      @oReport.AddRow do |oRow|
  35        oRow[ 'First Column'] = 98
  36        oRow[ 'Second Column'] = 198
  37      end
  38  
  39      @oReport[0]['First Column'].should == 98
  40      @oReport[0]['Second Column'].should == 198
  41    end
  42  
  43  end

Each test has a descriptive string that can be interpreted in two ways: as a test that must be passed or as a requirement for the software. The rspec toolset includes something that will pull out all these strings and give a summary of what testing has been done/what the requirements for the software are. If an auditor asks how the software is tested, bang, there is a full report of the tests. At a practical level (i.e. makes life easier for me) I prefer this to the ruby unit test system where one has to think up meaningful_method_names for each test. The strings are easier to write and read and reduce the need for comments.

Example report:

ReportHelper

- should allow a column to be added
- should allow multiple columns to be added
- should allow rows to be added to the report
- should only allow rows to be added if columns exist
- should return simple cell values

Rspec contains sweeter syntactic sugar than the ruby unit test module. For example, rspec overloads every object in the system with a 'should' method for doing the equivalent of assert tests:

    @oReport[0]['First Column'].should == 98

Compare this to test/unit:

    assert_equal 98, @oReport[0]['First Column']

apart from more typing, assert_equal has the expectation/actual things in what is intuitively to me the wrong order and I'm always having to correct these lines because the error messages have the terms swapped.

Rspec can test assertions more cleanly:

    lambda{ @oReport.AddRow}.should raise_error(RuntimeError, 'No columns defined')

although it's a pity that the lambda is necessary (and probably no 1 noob trap) but with test/unit the following always seemed like a lot of typing:

    oError = assert_raises RuntimeError do
      @oReport.AddRow
    end
    assert_equal 'No columns defined', oError.message

rspec is nicely integrated into rails, plugins being available to generate rspec tested controllers and models and suchlike, rake tasks to generate documents (woohoo) etc. It is integrated into netbeans and runs ok on Windows although when the tests are run (CTRL-F6) there is a tedious few seconds wasted in startup time. It is possible to run an rspec server in the background to eliminate this startup but it doesn't work on windows native ruby, it gives an error lamenting Microsofts glorious omission of a fork api.

There is much more to Rspec than this. One aim seems to be to write psuedo testing languages and have clients use these to write stories about what the software will do, these stories turning into a test framework. As the slashdot tag goes, 'goodluckwiththat'. At a practical level, I'm liking rspec even if only as a unit test framework with added syntactic sugar.

Related Posts: netbeans rails rspec ruby testing

Testing Firmware Code

I've described before how I've been testing Firmware code written in C by compiling it into Ruby Extensions and using ruby's excellent unit testing framework.

Up till now I've been testing the 'middle tier' algorithmic code but how to go about testing the low level stuff, that which pokes the hardware?

The firmware I am developing is for the ST ARM750 microcontroller and I am using ST's own standard library which is very useful driver code for all the peripherals embedded on the chip. It is a very useful resource as the peripherals on this chip are very complex and the user manual is written in hardware engineer gobbledygook that you have to study like a legal document, looking for the small print (e.g. running it at 60MHz it kept crashing until I found the small print that advised me to enable 'burst mode').

Scanning through the source to this library I saw some #ifdef DEBUG statements that implied that it could be compiled into an emulation library. The code was written very cleanly so it took me very little time to write something in (of course) ruby to scan the header files and generate 5250 lines of C code to emulate this library. Here is an example of one of the generated library functions:

   1  /*
   2   * Emulate the function TIM_ClearFlag
   3   */
   4  void TIM_ClearFlag(TIM_TypeDef* TIMx, u16 TIM_FLAG)
   5  {
   6    VALUE nRet;
   7    VALUE oArgs = rb_ary_new();
   8    VALUE strFuncName = rb_str_new2( "TIM_ClearFlag");
   9    ID nId = rb_intern( "Function");
  10  
  11    rb_ary_push( oArgs, PassStructureArg( (void *)TIMx, sizeof( TIM_TypeDef)));
  12    rb_ary_push( oArgs, LONG2NUM( TIM_FLAG));
  13  
  14    nRet = rb_funcall( g_oCallBack, nId, 2, strFuncName, oArgs);
  15  
  16    /* nRet is not used */
  17  }

TIM_ClearFlag is the name of the function in the ST standard library. The emulation code takes the name of the emulated function and the two arguments to the function, converts them into ruby objects and passes them to a method called 'Function' in a ruby object. In the testing case, this object is something that will make sure the functions are being called in the correct sequence and with the correct parameters and will return the appropriate return value. Structure arguments are passed as binary objects which the ruby can decipher from the structure definitions in the ST library header files (also parsed using ruby).

Note that my system has a few limitations that are acceptable because of the simplicity of the ST library and the way it was coded:

The parser I wrote just about handles the coding style of this library
The calls to ruby do not support the case where the library may poke the values within a structure passed to it by pointer.
It doesn't handle passing structures by value.

Now I have my low level testing library I will be able to do my testing using something I call signature analysis (I'm not sure there is a better software engineering term). I write the code, test it on the real hardware, then I can record the sequence of function calls made including the values of function arguments. Later on, in leiu of real hardware to test it on I can run the test code and compare it against the previously recorded signatures. Et voila, the reassurance that I haven't broken anything and I can sleep at night.

UPDATE:

My test library now includes a YamlFaker that can be used in two modes:

it will record a sequence of function calls
it will ensure a sequence of function calls matches a previous recording

The yaml script looks like this:

#
# This file is maintained by the YamlFaker module DO NOT EDIT
#
---
- test_CommandPWMSet Demand -200:
  - :Comment: |-

      Looking for demand to be correct according to the requested range and the
      scaling factors. Also ensure that for the fifth channel, the period is
      large for small demand as the output of this channel is inverted
  - TIM_SetPulse:
      args:
      - TIMx: TIM0
      - TIM_Channel: "0x2"
      - Pulse: "0x0"
  - TIM_SetPulse:
      args:
      - TIMx: TIM1
      - TIM_Channel: "0x2"
      - Pulse: "0x0"
  - TIM_SetPulse:
      args:
      - TIMx: TIM2
      - TIM_Channel: "0x2"
      - Pulse: "0x0"
  - PWM_SetPulse:
      args:
      - PWM_Channel: "0x2"
      - Pulse: "0x0"
  - PWM_SetPulse:
      args:
      - PWM_Channel: "0x4"
      - Pulse: "0x12c"
- test_CommandPWMSet Demand -100:
  - TIM_SetPulse:
      args:
      - TIMx: TIM0
      - TIM_Channel: "0x2"
      - Pulse: "0x0"

Most of the entries in the script describe the expected sequence of function calls, complete with arguments. I can ensure that the demand being sent to my PWM channels is what I expect. These recorded scripts could be quite useful, it is a complete log of the interaction with the hardware. If I ever had to change the code that poked the hardware I can compare the script recorded from the new code against my old script and make sure that any changes are what I would expect. This way I can be far more certain that my changes are correct before they hit real hardware.

Testing 123

I've really got into Unit Testing now. The Ruby Unit Testing framework makes it very easy. I've tried unit testing in the past on C++ projects but the lack of a good framework there meant that coding the tests was more laborious than coding the code it was testing.

So far I've written a Bootloader than downloads Hex files to the target firmware. It consists of the following modules:

low level rs232 extension written in C++ and swig
higher level rs232 port abstraction written in ruby
Boot Loader modules to interface with the boot loader firmware embedded in the ARM7 microcontroller
Intel Hex file decoder

I have been able to test these as follows:

a test harness to test the low level RS232 extensions by using an RS232 loopback connection
a test harness to test the higher level RS232 abstration using an RS232 loopback connection
a test harness to test the Boot Loader module against the real microcontroller
a test harness to test the Boot Loader module against a 'fake' rs232 port
a test harness to test the parsing of Intel Hex files
a test harness to download a 'known good' intel hex file to the microcontroller

I am able to test automatically just about anything here in one of three modes:

with the real microcontroller, real rs232
rs2332 loopback
'fake'

The 'faker' is what a call a class that is able to emulate a port. Initially I wrote it to emulate an RS232 port in the following way:

the 'faker' would receive what was being output to the port and check that it was correct
the 'faker' was able to simulate data being returned from the port

I created a faker that parsed the input and output data from a IO class and hence could test the Bootstrap commands with a fake rs232 port like this:

   1    #
   2    # Test the Get command
   3    #
   4    def test_get
   5      strConversation = <<EOF
   6  # connect
   7  < 0x7f
   8  > 0x75
   9  # command
  10  < 0x01
  11  # data
  12  > 0x11 0x00 0x00 0x75
  13  EOF
  14      oFd = StringIO.new( strConversation)
  15      oPort = RS232Faker.new
  16  
  17      oPort.Run( oFd) do
  18        oBoot = BootLoader.new
  19        oBoot.Connect( oPort)
  20        oResults = oBoot.CommandGet
  21  
  22        assert_equal [0x11,0x00,0x00], oResults
  23      end
  24    end

Lines in the fake script that start with a '<' is data that should be transmitted and lines that start with a '>' should be received.

The Bootloader module itself is written such that it can connect to either a real rs232 port or this fake one.

Within the Bootloader modules was a particularly complex method that converted memory address ranges into an array of sector numbers that needed erasing. I exposed this method so that I was able to test it specifically in the Unit Test.

The faker can be generalised beyond rs232 to handle anything that inputs and outputs. It doesn't even need to run a fixed script, it could contain heuristics to emulate hardware such as a temperature control channel and test whether the code does actually control temperatures.

I'm toying with the idea of testing my embedded C code by compiling it into a Ruby extension and calling it from Ruby Test Harnesses. The code would be compiled by gpp both for the ruby extension and the real hardware. This should provide thorough testing of the logical aspects of the code (e.g. servo control calculations) and may only be lacking in the following areas:

it would not test the 'real time' aspects of the C code (interraction with interrupts etc)
it would not test for problems in the code generation for the real firmware
it would not test for differences between the processor architectures, e.g. number of bits in an integer, although the ARM is a 32 bit microcontroller.

However, this should provide a satisfying amount of testing.

Another problem with this is that I would have ruby calling a ruby extension in C which would in turn call 'faker' objects written in ruby. I am unsure how to go about doing this and I am unsure whether the faker objects would be able to raise exception back up to the main ruby script. It may be able to do this in a flaky and memory-leaking way but that isn't too much of a problem in a test harness: I am testing the algorithms, not the memory leaks in the test harness.

Conclusion: Unit Testing FTW