Peter's Blog

Redefining the Impossible

Items filed under ruby


I was working on this site and noticed how slow it was, particularly on rss feeds. This was just after managing to completely screw up my ruby gem installation (apt-get remove rubygems/apt-get install rubygems didn't reinstall rubygems) which I only cured by upgrading the server to ubuntu hardy. I wasn't sure whether I had blown something in the upgrade (the server did a fsck when I rebooted it but maybe only because it had been 'up' for 100 days) or whether the code was just slow.

How to pinpoint the cause of the slowness? Profiling. Some research revealed I can profile ruby code as follows:

  • install a ruby profiler:
    gen install ruby-prof
    
  • insert some profiling code around your suspect slow code (in my case around the rss formatting):
       1  gem install ruby-prof
       2  
       3     require 'ruby-prof'
       4  
       5     RubyProf.start
       6  
       7     {code to profile}
       8  
       9   result = RubyProf.stop
      10     printer = RubyProf::FlatPrinter.new(result)
      11     printer.print(File.open( '/tmp/prof.txt', "w"), 0)
    
  • run script/server and try downloading rss
  • wait because it is very slow
  • examine the profile
    vim /tmp/prof.txt
    
  • lament

This told me that 70% of my time was being spent in a gsub in my wilki code that I added last week. gsub was being called 1800 times (!) and when I commented out the new gsub call the rss was rendered much faster.

The thing is, my wilki formatting is supposed to be cached, it should only be done when a posting is modified, not every time the post is rendered. Turns out to be because updating a field in an ActiveRecord doesn't automatically cause it to be saved because it is dirty, an explicit save helped. Problem solved, now downloading the rss feed doesn't max out my cpu for 20 seconds smile

NB (clarification): the rss doesn't go through the rails fragment cache.


Filed under: nooby rails ruby

Add a comment

My websites are all migrating to a CMS kinda vibe where pages are edited through the web browser. Easy for me, easy for anyone else. I've already got the technology to upload pictures but it has an annoying problem and it seems to be built into the attachment_fu plugin. The problem is that one must choose how big one wants the pictures to be when they are first uploaded and the size is defined in the database model. attachment_fu will then resize the image accordingly. That is all very well if one is organised and prepared and knows in advance how big they want their pictures or are running a simple gallery where all pictures are the same size.

But what about where one is trying to do some nice layouts on a page and wants to adjust the sizes of the pictures? Say I decide I want a particular picture on a page to be 100 pixels wide? I have two options:

  • I can go into gimp/paintshop pro/whatever, resize the image and upload it (again) but that is annoying.
  • I can alter the img tags in my html to include a width attribute. I see a lot of this on other websites, as the page downloads you see lots of little images slowly filled in as huge 1000x1000 jpg's are downloaded and resized in the browser to 100x100 because the page designer couldn't be bothered to resize the image in gimp/paintshop pro/whatever.

Now there is another way. Here is a rails helper that allows the width of an image to be specified in a view, automatically resizing the image as appropriate:

   1  # Methods added to this helper will be available to all templates in the application.
   2  module ApplicationHelper
   3  
   4     #
   5     # Takes the url of an image and a desired width and ensures a cached resized
   6     # version of the image is available and returns the url of said image.
   7     #
   8     # the url should not be a relative url as it is assumed work relative to RAILS_ROOT/public
   9     # This also means the image must be on this server.
  10     #
  11     def cacheimage( strUrl, nWidth)
  12       # Find original file in file system
  13       strFile = "#{RAILS_ROOT}/public" + strUrl
  14  
  15       # create name for cached and resized image
  16       #
  17       # /blah/blah.jpg --> /blah/blah-123.jpg
  18       #
  19       strBits = strUrl.split( '.')
  20       strBits[-2] += "-#{nWidth}"
  21       strCacheName = strBits.join( '.')
  22  
  23       strCacheUrl  = "/imagecache#{strCacheName}"
  24       strCacheFileName = "#{RAILS_ROOT}/public#{strCacheUrl}"
  25  
  26       if not File.exists?( strCacheFileName)
  27         require 'image_science'
  28  
  29         FileUtils.mkpath( File.dirname( strCacheFileName))
  30  
  31         ImageScience.with_image( strFile) do |oImg|
  32           oImg.resize( nWidth, nWidth * oImg.height / oImg.width) do |oResized|
  33             oResized.save strCacheFileName
  34           end
  35         end
  36       end
  37  
  38       return strCacheUrl
  39     end
  40  end

That wasn't too difficult. You need to create an 'imagecache' directory under /public to store a cache of resized images and make sure mongrel or whatever can write to it.

In a view one would put:

<%= image_tag cacheimage( "/images/prettypicture.jpg", 500) %>

This will cause a version of prettypicture.jpg to be created that is resized to 500 pixels wide with a proportional change in height (in a typical html page layout the width of things is often more important than the height except maybe for headers and footers). The resized version is stored in a cache and will be displayed from then on.

If the original version of prettypicture.jpg is changed then one must remember to erase the resized versions from the cache as this isn't automatic.

Another helper is:

   def applyimagewidth( strText)
      strText.gsub!( /([^"!<>{}]*)(jpg|JPG|gif|GIF|png|PNG)x(\d+)/) do |oMatch|
         cacheimage( "#{$1}#{$2}", $3.to_i)
      end

      return strText
   end

(UPDATE: warning- the gsub above is VERY SLOW, it can take minutes to scan a few k of text, even if there are no matches).

which allows the width to be appended to the image file name. This allows a view such as

<%= textilize applyimagewidth( @page.body) %>

and thus one's textile source could be:

!/imagecache/images/DSC_4064-300.JPG!

to show an image with a width of 300 pixels. I could hack the redcloth source directly to add this feature but I'd rather keep out of there.

I've done this in a simplistic hacky way but it should suffice (like all my stuff).


Filed under: rails ruby

Add a comment

Netbeans and cygwin ruby don't have a happy relationship. To use the cygwin ruby interpreter it is necessary to launch Netbeans (6.1) with the arguments:

-J-Druby.no.sync-stdio=true

as otherwise Netbeans will pass a windows-style file name to the cygwin ruby that the latter cannot recognise (C:\Projects\Blah instead of /cygdrive/c/Projects/Blah). Unfortunately the ruby interpreter configuration in Netbeans tries to be clever and insists you point it to a ruby executable, you can't give it a batch file that will hack things into working shape.

rspec on cygwin has some similar problems. Here are some hacks to the rspec plugin to make it work from netbeans:

In vendor/plugins/rspec/bin/bin, change:

$LOAD_PATH.unshift(File.expand_path(File.dirname(__FILE__) + "/../lib"))

to

$LOAD_PATH.unshift(File.dirname(__FILE__).sub( /C:\//, '/cygdrive/c/') + "/../lib")

In vendor/plugins/rspec/lib/spec/runner/example_group_runner.rb, change:

load file

to

load File.expand_path(file).gsub(/C:/, '/cygdrive/c')

Tip: to diagnose problems with 'require' not finding files, put this:

p $:

before the failing line to see what directories are in the load path. If you see 'C:\' and it's cygwin then fix it.

Unfortunately it takes a good 20 seconds for cygwin/rspec to run a trivial test script. The rspec server starts and runs but netbeans/rspec seem reluctant to use it (it runs tests ok from the command line).

Why am I still bothering with cygwin? Because I still want to test gcc C code by compiling it into ruby extensions and using ruby test codes (rspec or unit testing).

In tracing these problems I came across more reasons that Netbeans is worth the hassle:

  • ctrl-tab to switch to last open tab. ctrl and hold tab brings up a handy little context menu to choose a tab from (kinda like windows task switching with tab).
  • press ctrl and mouse over an identifier to bring up rdoc info about it. Ctrl-click to go to it's definition.
  • alt-b to go back to where you came from.
  • click on an 'end' and the matching begin/do/if is highlighted and vice versa, which is damn useful.
  • select an identifier and all matching identifiers in your file are automatically highlighted.

Filed under: cygwin netbeans rspec ruby

Add a comment

I've been trying out rspec, a new testing framework for ruby that allows one to better follow the principles of Behaviour Driven Design (BDD). BDD is a step on from Test Driven Design (TDD) approach exemplified in the standard ruby unit test framework. TDD is the process of starting ones development by writing the tests that your new module should pass. As you write each test you then implement the code to pass it and hence, voila, you have nicely designed and robust software. In BDD you approach testing the code from how it is supposed to behave. When liasing with clients this is a better approach as everyone can talk in the same language, essentially in terms of requirements so one can try to get less bogged down in implementation details.

Here are some rspec tests for a class I have been working on:

   1  describe ReportHelper do
   2    before(:each) do
   3      #
   4      # Create a new clean Report class for each individual test
   5      #
   6      @oReport = ReportHelper::Report.new
   7    end
   8  
   9    it "should allow a column to be added" do
  10      @oReport.AddColumn( 'First Column')
  11    end
  12  
  13    it "should allow multiple columns to be added" do
  14      @oReport.AddColumn( 'First Column')
  15      @oReport.AddColumn( 'Second Column')
  16    end
  17  
  18    it "should allow rows to be added to the report" do
  19      @oReport.AddColumn( 'First Column')
  20      @oReport.AddColumn( 'Second Column')
  21      @oReport.AddRow do
  22        # pass
  23      end
  24    end
  25  
  26    it "should only allow rows to be added if columns exist" do
  27      lambda{ @oReport.AddRow}.should raise_error(RuntimeError, 'No columns defined')
  28    end
  29  
  30    it "should return simple cell values" do
  31      @oReport.AddColumn( 'First Column')
  32      @oReport.AddColumn( 'Second Column')
  33  
  34      @oReport.AddRow do |oRow|
  35        oRow[ 'First Column'] = 98
  36        oRow[ 'Second Column'] = 198
  37      end
  38  
  39      @oReport[0]['First Column'].should == 98
  40      @oReport[0]['Second Column'].should == 198
  41    end
  42  
  43  end

Each test has a descriptive string that can be interpreted in two ways: as a test that must be passed or as a requirement for the software. The rspec toolset includes something that will pull out all these strings and give a summary of what testing has been done/what the requirements for the software are. If an auditor asks how the software is tested, bang, there is a full report of the tests. At a practical level (i.e. makes life easier for me) I prefer this to the ruby unit test system where one has to think up meaningful_method_names for each test. The strings are easier to write and read and reduce the need for comments.

Example report:

ReportHelper

- should allow a column to be added
- should allow multiple columns to be added
- should allow rows to be added to the report
- should only allow rows to be added if columns exist
- should return simple cell values

Rspec contains sweeter syntactic sugar than the ruby unit test module. For example, rspec overloads every object in the system with a 'should' method for doing the equivalent of assert tests:

    @oReport[0]['First Column'].should == 98

Compare this to test/unit:

    assert_equal 98, @oReport[0]['First Column']

apart from more typing, assert_equal has the expectation/actual things in what is intuitively to me the wrong order and I'm always having to correct these lines because the error messages have the terms swapped.

Rspec can test assertions more cleanly:

    lambda{ @oReport.AddRow}.should raise_error(RuntimeError, 'No columns defined')

although it's a pity that the lambda is necessary (and probably no 1 noob trap) but with test/unit the following always seemed like a lot of typing:

    oError = assert_raises RuntimeError do
      @oReport.AddRow
    end
    assert_equal 'No columns defined', oError.message

rspec is nicely integrated into rails, plugins being available to generate rspec tested controllers and models and suchlike, rake tasks to generate documents (woohoo) etc. It is integrated into netbeans and runs ok on Windows although when the tests are run (CTRL-F6) there is a tedious few seconds wasted in startup time. It is possible to run an rspec server in the background to eliminate this startup but it doesn't work on windows native ruby, it gives an error lamenting Microsofts glorious omission of a fork api.

There is much more to Rspec than this. One aim seems to be to write psuedo testing languages and have clients use these to write stories about what the software will do, these stories turning into a test framework. As the slashdot tag goes, 'goodluckwiththat'. At a practical level, I'm liking rspec even if only as a unit test framework with added syntactic sugar.


Add a comment

Much as I hate flash, it is a nice way to embed a sound or video player in a page. Here's how I did it in rails using the JW FLV Media Player. I added the following to my view:

   1  <script type="text/javascript" src="/javascripts/swfobject.js"></script>
   2  
   3  <% for oRssItem in @rssitems %>
   4    <hr/>
   5    <h3><%= oRssItem.title %></h3>
   6    <p><%= oRssItem.description %></p>
   7    <table width="470">
   8      <tr>
   9         <td wIdth="50%"><%= oRssItem.created.strftime( "%m %B %y") %></td>
  10         <td align="right"><%= link_to "Download", "/#{ oRssItem.filename}" %></td>
  11      </tr>
  12    </table>
  13  
  14    <div id="player<%=oRssItem.id%>">This text will be replaced</div>
  15  
  16    <script type="text/javascript">
  17      var so = new SWFObject('/mediaplayer.swf','mpl','470','20','8');
  18      so.addParam('allowscriptaccess','always');
  19      so.addParam('allowfullscreen','true');
  20      so.addVariable('width','470');
  21      so.addVariable('height','20');
  22      so.addVariable('file','/<%= oRssItem.filename%>');
  23      so.write('player<%=oRssItem.id%>');
  24    </script>
  25  <% end %>

This generates a page full of podcast entries, each with a little media player thing so you can listen to them easily. This does mp3 files but it can handle video too.

I am aware that the swfobject.js would be better loaded in the html header.

Here's a demo: I don't know if this will work in the rss feed (UPDATE: no!) so bear with me and please be gentle on my bandwidth, it's quite a catchy tune:

Oops, no flash? Javascript broken? Works for me.

The main reason I am blogging this is because it only needs a change to a view file and installation of two files in the /public directory (the player and a javascript file). Since view files are awkward to comment legibly I decided to document it all here so I can find it again.


Filed under: mp3 rails ruby

2 Comments

In a mammoth programming session I implemented proper smileys wink

[[':-)', 'smile'],
  [':-(', 'sad'],
  ['8-)', 'cool'],
  [';-)', 'wink']].each do |strCode, strFile|
  strText.gsub!( strCode, "<img alt=\"#{strFile}\" src=\"/images/smileys/#{strFile}.png\" />")
end

My fingers were bleeding from all the typing sad Worked first time though smile

There is even alt text for blind people cool


Filed under: petersblogger rails ruby

Add a comment

I figured out how to scrape the Wow Armory for up-to-date information about my characters using ruby. This means I can do this directly in PetersBlogger without using third party signature generators that disappear overnight or get absorbed into wow gold sites.

   1    require 'net/http'
   2  
   3    #
   4    # Scrape wow character info.
   5    #
   6    def WoWInfo
   7      begin
   8        strRealm = 'Eonar'
   9  
  10        oInfo = []
  11  
  12        ['Pookypoo', 'Maevyn', 'Maezyn', 'Maexyn'].each do |strCharacter|
  13          oCharInfo = []
  14  
  15          #
  16          # Open url.
  17          # Need to specify firefox as user agent as this makes the server return an XML
  18          # file.
  19          # Look the data up in the european armory. Change this for US.
  20          # If this is not done we get html.
  21          oResp = Net::HTTP.start( "armory.wow-europe.com", 80) do |http|
  22                http.get( "/character-sheet.xml?r=#{strRealm.tr( ' ', '+')}&n=#{strCharacter}",
  23                            { 'user-agent' => 'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-GB; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4'})
  24          end
  25  
  26          oDoc = REXML::Document.new oResp.body
  27          oDoc.elements[1].elements.each( 'characterInfo/character') do |oElement|
  28            oCharInfo << {:race => oElement.attributes['race']}
  29            oCharInfo << {:class => oElement.attributes['class']}
  30            oCharInfo << {:level => oElement.attributes['level']}
  31          end
  32  
  33          oDoc.elements[1].elements.each( 'characterInfo/characterTab/talentSpec') do |oElement|
  34            oCharInfo << {:spec => "#{oElement.attributes['treeOne']}/#{oElement.attributes['treeTwo']}/#{oElement.attributes['treeThree']}"}
  35          end
  36  
  37          oDoc.elements[1].elements.each( 'characterInfo/characterTab/professions/skill') do |oElement|
  38            oCharInfo << {oElement.attributes['key'].to_sym => oElement.attributes['value']}
  39          end
  40  
  41          oInfo << {strCharacter => oCharInfo}
  42        end
  43  
  44        return oInfo
  45      rescue
  46        return []
  47      end
  48    end

This lot generates the info in this page which is linked to under 'My Characters' over to the right. It should be fairly up-to-date as the cache will flush once a day causing the armory to be requeried. Currently the output is presented in a rails view thus:

   1  <% cache( :part => :wow) do %>
   2    <table>
   3    <% WoWInfo().each do |oCharInfo| %>
   4      <% oCharInfo.each_pair do |strChar, oInfo| %>
   5        <tr><td colspan="2"><b><%= strChar %></b></td></tr>
   6        <% oInfo.each do |oItem| %>
   7          <% oItem.each_pair do |strItem, strValue| %>
   8            <tr>
   9              <td><%= strItem %>:</td><td><%= strValue %></td>
  10            </tr>
  11          <% end %>
  12        <% end %>
  13      <% end %>
  14    <% end %>
  15    </table>
  16  <% end %>

All the work is done by the WoWInfo method which I put in the application helper. I didn't use a controller method for this.

I'm going to think about a nicer presentation with some pictures of the actual guys rather than generic pictures of bald redhead dwarves.

UPDATE: I've added my old Aerie Peak toons to the output but the code above still stands. Lugulas was my old auction alt who I've never named here before, hence level 10 with level 75 (dis)enchanting. I find myself missing them...


4 Comments

I've described before how I've been testing Firmware code written in C by compiling it into Ruby Extensions and using ruby's excellent unit testing framework.

Up till now I've been testing the 'middle tier' algorithmic code but how to go about testing the low level stuff, that which pokes the hardware?

The firmware I am developing is for the ST ARM750 microcontroller and I am using ST's own standard library which is very useful driver code for all the peripherals embedded on the chip. It is a very useful resource as the peripherals on this chip are very complex and the user manual is written in hardware engineer gobbledygook that you have to study like a legal document, looking for the small print (e.g. running it at 60MHz it kept crashing until I found the small print that advised me to enable 'burst mode').

Scanning through the source to this library I saw some #ifdef DEBUG statements that implied that it could be compiled into an emulation library. The code was written very cleanly so it took me very little time to write something in (of course) ruby to scan the header files and generate 5250 lines of C code to emulate this library. Here is an example of one of the generated library functions:

   1  /*
   2   * Emulate the function TIM_ClearFlag
   3   */
   4  void TIM_ClearFlag(TIM_TypeDef* TIMx, u16 TIM_FLAG)
   5  {
   6    VALUE nRet;
   7    VALUE oArgs = rb_ary_new();
   8    VALUE strFuncName = rb_str_new2( "TIM_ClearFlag");
   9    ID nId = rb_intern( "Function");
  10  
  11    rb_ary_push( oArgs, PassStructureArg( (void *)TIMx, sizeof( TIM_TypeDef)));
  12    rb_ary_push( oArgs, LONG2NUM( TIM_FLAG));
  13  
  14    nRet = rb_funcall( g_oCallBack, nId, 2, strFuncName, oArgs);
  15  
  16    /* nRet is not used */
  17  }

TIM_ClearFlag is the name of the function in the ST standard library. The emulation code takes the name of the emulated function and the two arguments to the function, converts them into ruby objects and passes them to a method called 'Function' in a ruby object. In the testing case, this object is something that will make sure the functions are being called in the correct sequence and with the correct parameters and will return the appropriate return value. Structure arguments are passed as binary objects which the ruby can decipher from the structure definitions in the ST library header files (also parsed using ruby).

Note that my system has a few limitations that are acceptable because of the simplicity of the ST library and the way it was coded:

  • The parser I wrote just about handles the coding style of this library
  • The calls to ruby do not support the case where the library may poke the values within a structure passed to it by pointer.
  • It doesn't handle passing structures by value.

Now I have my low level testing library I will be able to do my testing using something I call signature analysis (I'm not sure there is a better software engineering term). I write the code, test it on the real hardware, then I can record the sequence of function calls made including the values of function arguments. Later on, in leiu of real hardware to test it on I can run the test code and compare it against the previously recorded signatures. Et voila, the reassurance that I haven't broken anything and I can sleep at night.

UPDATE:

My test library now includes a YamlFaker that can be used in two modes:

  1. it will record a sequence of function calls
  2. it will ensure a sequence of function calls matches a previous recording

The yaml script looks like this:

#
# This file is maintained by the YamlFaker module DO NOT EDIT
#
---
- test_CommandPWMSet Demand -200:
  - :Comment: |-

      Looking for demand to be correct according to the requested range and the
      scaling factors. Also ensure that for the fifth channel, the period is
      large for small demand as the output of this channel is inverted
  - TIM_SetPulse:
      args:
      - TIMx: TIM0
      - TIM_Channel: "0x2"
      - Pulse: "0x0"
  - TIM_SetPulse:
      args:
      - TIMx: TIM1
      - TIM_Channel: "0x2"
      - Pulse: "0x0"
  - TIM_SetPulse:
      args:
      - TIMx: TIM2
      - TIM_Channel: "0x2"
      - Pulse: "0x0"
  - PWM_SetPulse:
      args:
      - PWM_Channel: "0x2"
      - Pulse: "0x0"
  - PWM_SetPulse:
      args:
      - PWM_Channel: "0x4"
      - Pulse: "0x12c"
- test_CommandPWMSet Demand -100:
  - TIM_SetPulse:
      args:
      - TIMx: TIM0
      - TIM_Channel: "0x2"
      - Pulse: "0x0"

Most of the entries in the script describe the expected sequence of function calls, complete with arguments. I can ensure that the demand being sent to my PWM channels is what I expect. These recorded scripts could be quite useful, it is a complete log of the interaction with the hardware. If I ever had to change the code that poked the hardware I can compare the script recorded from the new code against my old script and make sure that any changes are what I would expect. This way I can be far more certain that my changes are correct before they hit real hardware.


Filed under: c ruby testing

1 Comment

I've been testing my C libraries by compiling them into ruby extensions and then using ruby's unit testing to test them. Under Windows I had it all running nicely and I could run a main test script that would invoke all the other test scripts. Sweet.

Then it came to getting it running under linux (the target system) and while the unit tests ran fine individually, when I tried my main test script I was getting weird errors from my test extensions, including segment violations (!).

It took me a while to track down the problem but it turned out to be due to the behaviour of gcc and the linux program loader. All my test libraries were loaded into one ruby process and because they all contained functions with similar names it became a bit of a lottery as to which function in which library was called. I am used to the behaviour of windows dll's where all functions are by default hidden from the outside world unless you explicity export them. gcc/elf/whatever seems to export everything from shared libraries (linux equivalent of a dll) by default with hilarious results.

In version 4 of gcc they have added a facility for hiding these internal functions. I solved my problem as follows:

  • Add '-fvisibility=hidden' to the gcc arguments to cause all functions to be hidden by default.
  • Add this code to my stddefs.h:
       1  #ifdef _MSC_VER
       2    #ifdef BUILDING_DLL
       3      #define DLLEXPORT __declspec(dllexport)
       4    #else
       5      #define DLLEXPORT __declspec(dllimport)
       6    #endif
       7    #define DLLLOCAL
       8  #else
       9    #define DLLEXPORT __attribute__ ((visibility("default")))
      10    #define DLLLOCAL __attribute__ ((visibility("hidden")))
      11  #endif
    
  • Change the functions that ruby calls to load the ruby extension to the following form (i.e. add DLLEXPORT):
       1  /*
       2   * Initialise the test class.
       3   */
       4  DLLEXPORT void Init_testtimer( void)
       5  {
       6      VALUE oMyClass = rb_define_class( "TestTimer", rb_cObject);
       7      rb_define_method( oMyClass, "initialize", test_initialise, 1);
       8      rb_define_method( oMyClass, "TimerSet", test_TimerSet, 1);
       9      rb_define_method( oMyClass, "TimerUpdate", test_TimerUpdate, 1);
      10      rb_define_method( oMyClass, "TimerStart", test_TimerStart, 1);
      11      rb_define_method( oMyClass, "TimerTest", test_TimerTest, 0);
      12  }
    

These functions are the only ones that need to be visible outside the library.

Results:

images/TestsDone.gif

Filed under: c gcc linux ruby

Add a comment

I forgot to mention the best ruby tip of all, the biggest time saver in the world ever.

p whatever

Simply prints the value of a variable on a new line. It is equivalent to:

print "#{whatever.inspect}\n"

or

puts "#{whatever.inspect}"

Who needs debuggers? Well I do occasionally but a well aimed print can shed light into the darkest gloom.

Moral: read more books.


Filed under: ruby

Add a comment