Peter's Blog

Another fusion of WoW and rails.

While planning Maexyn's conversion to alchemy I thought it would be nice to be able to get a listing of my characters bag and bank items so I could look through my stocks and see what was what. It occurred to me that a nice way to do this would be to scan through the lua files that are dumped by the TBags mod since that holds a database of contents of all my characters bags and bank. Doing this in ruby turned out to be very easy:

• Install ruby lua support:

  sudo aptitude install liblua5.1-0-dev
  sudo gem install rua
  

• Ruby code to convert lua data file into ruby data:

  oRua = Rua.new(:all)
  oRua.eval( open( File.dirname( __FILE__) + "/../../db/TBag.lua").read)
  

• oRua now has an array of items that maps onto the data stored in the lua file.

  oRua['TBnkItm'] has a list of items in the bank
  oRua['TInvItm'] has a list of items in the inventory
  

The end result is this page where I can see what stocks I have. Auctioneer also stores its data in lua so in a similar way I added a lookup so I see the values to these stocks. I can also use the item numbers to add links to the items to thottbot for reference. It isn't perfect, it should show vendor values, especially for soulbound items.

Anyway with this I can plan some sales, particularly of Maexyn's enchanting materials.

Currently the auctioneer data is way out of date, I need to get it running again and do a few scans.

This isn't here as a way of boasting about my vast wealth, I guess a week's karak farming would be better that this junk pile (minions scourgestones?!?).

N.B: an alternative implementation would be simply to write something in the lua language that reads the two lua databases and then dumps out the html. For me it is easier to do it in ruby than learn lua.

I've been trying out rspec, a new testing framework for ruby that allows one to better follow the principles of Behaviour Driven Design (BDD). BDD is a step on from Test Driven Design (TDD) approach exemplified in the standard ruby unit test framework. TDD is the process of starting ones development by writing the tests that your new module should pass. As you write each test you then implement the code to pass it and hence, voila, you have nicely designed and robust software. In BDD you approach testing the code from how it is supposed to behave. When liasing with clients this is a better approach as everyone can talk in the same language, essentially in terms of requirements so one can try to get less bogged down in implementation details.

Here are some rspec tests for a class I have been working on:

   1  describe ReportHelper do
   2    before(:each) do
   3      #
   4      # Create a new clean Report class for each individual test
   5      #
   6      @oReport = ReportHelper::Report.new
   7    end
   8  
   9    it "should allow a column to be added" do
  10      @oReport.AddColumn( 'First Column')
  11    end
  12  
  13    it "should allow multiple columns to be added" do
  14      @oReport.AddColumn( 'First Column')
  15      @oReport.AddColumn( 'Second Column')
  16    end
  17  
  18    it "should allow rows to be added to the report" do
  19      @oReport.AddColumn( 'First Column')
  20      @oReport.AddColumn( 'Second Column')
  21      @oReport.AddRow do
  22        # pass
  23      end
  24    end
  25  
  26    it "should only allow rows to be added if columns exist" do
  27      lambda{ @oReport.AddRow}.should raise_error(RuntimeError, 'No columns defined')
  28    end
  29  
  30    it "should return simple cell values" do
  31      @oReport.AddColumn( 'First Column')
  32      @oReport.AddColumn( 'Second Column')
  33  
  34      @oReport.AddRow do |oRow|
  35        oRow[ 'First Column'] = 98
  36        oRow[ 'Second Column'] = 198
  37      end
  38  
  39      @oReport[0]['First Column'].should == 98
  40      @oReport[0]['Second Column'].should == 198
  41    end
  42  
  43  end

Each test has a descriptive string that can be interpreted in two ways: as a test that must be passed or as a requirement for the software. The rspec toolset includes something that will pull out all these strings and give a summary of what testing has been done/what the requirements for the software are. If an auditor asks how the software is tested, bang, there is a full report of the tests. At a practical level (i.e. makes life easier for me) I prefer this to the ruby unit test system where one has to think up meaningful_method_names for each test. The strings are easier to write and read and reduce the need for comments.

Example report:

ReportHelper

- should allow a column to be added
- should allow multiple columns to be added
- should allow rows to be added to the report
- should only allow rows to be added if columns exist
- should return simple cell values

Rspec contains sweeter syntactic sugar than the ruby unit test module. For example, rspec overloads every object in the system with a 'should' method for doing the equivalent of assert tests:

    @oReport[0]['First Column'].should == 98

Compare this to test/unit:

    assert_equal 98, @oReport[0]['First Column']

apart from more typing, assert_equal has the expectation/actual things in what is intuitively to me the wrong order and I'm always having to correct these lines because the error messages have the terms swapped.

Rspec can test assertions more cleanly:

    lambda{ @oReport.AddRow}.should raise_error(RuntimeError, 'No columns defined')

although it's a pity that the lambda is necessary (and probably no 1 noob trap) but with test/unit the following always seemed like a lot of typing:

    oError = assert_raises RuntimeError do
      @oReport.AddRow
    end
    assert_equal 'No columns defined', oError.message

rspec is nicely integrated into rails, plugins being available to generate rspec tested controllers and models and suchlike, rake tasks to generate documents (woohoo) etc. It is integrated into netbeans and runs ok on Windows although when the tests are run (CTRL-F6) there is a tedious few seconds wasted in startup time. It is possible to run an rspec server in the background to eliminate this startup but it doesn't work on windows native ruby, it gives an error lamenting Microsofts glorious omission of a fork api.

There is much more to Rspec than this. One aim seems to be to write psuedo testing languages and have clients use these to write stories about what the software will do, these stories turning into a test framework. As the slashdot tag goes, 'goodluckwiththat'. At a practical level, I'm liking rspec even if only as a unit test framework with added syntactic sugar.

Up till now to put pictures in this blog I have had to:

• go into an image editor to resize the file to suit the blog (about 450-500 pixels wide).
• fire up an sftp program to upload the file to /var/www/PetersBlogger/public/images
• add a tag such as

  [/images/TheNewPicture.jpg]
  

  to the wilki.

It was a bit laborious, especially the SFTP bit which is why I didn't do it very often.

I've now added better support for pictures to PetersBlogger. Here is an example:

Girls Aloud

if you click on this you should be treated to a larger version of the image, thanks to lightbox a javascript library.

PetersBlogger now supports a database of pictures so I can add a new picture, give it a name and upload it through the administration pages. The code uses the name as a caption under the photo and as alt text. I can use the name directly in my wilki text to choose which pictures to display.

The database of pictures is handled by the attachmentfu rails plugin. This article gives the steps required to get attachmentfu up and running.

The setup automatically handles resizing images to three sizes: thumbnail for listing in my picture admin page, a version that is no more than 500 pixels in either dimension for showing in the blog and a version that is no more than 800x600 for showing zoomed when you click on a picture. Attachmentfu has the sense to keep the aspect ratio correct automatically so I don't have to worry about portrait/landscape. I had to hack the imagescience library that does the resizing as it was hard-wired to save jpegs using JPEG_QUALITYSUPERB mode so the blog-sized images were taking up 150k. I changed this to JPEG_QUALITYAVERAGE and the size dropped to 15k or so, but it was a bit noisy so I've settled on JPEG_QUALITYGOOD and 38k images. You are on broadband?

I was slightly worried about how to change pictures, i.e. to upload a different version of a picture to replace an existing one, since the attachmentfu article I mentioned does not cover this. The answer is to use the exact same view code in the 'edit' page as in the 'new' page. Combined with rails scaffold code it Just Works! It even handles deleting the three old pictures and creating the new files.

My main motivation for doing this was so I can do it on the websites I host for other people and make it easier for them to upload pictures. PetersCMS is still in the planning stages but one problem has been cracked.

Much as I hate flash, it is a nice way to embed a sound or video player in a page. Here's how I did it in rails using the JW FLV Media Player. I added the following to my view:

   1  <script type="text/javascript" src="/javascripts/swfobject.js"></script>
   2  
   3  <% for oRssItem in @rssitems %>
   4    <hr/>
   5    <h3><%= oRssItem.title %></h3>
   6    <p><%= oRssItem.description %></p>
   7    <table width="470">
   8      <tr>
   9         <td wIdth="50%"><%= oRssItem.created.strftime( "%m %B %y") %></td>
  10         <td align="right"><%= link_to "Download", "/#{ oRssItem.filename}" %></td>
  11      </tr>
  12    </table>
  13  
  14    <div id="player<%=oRssItem.id%>">This text will be replaced</div>
  15  
  16    <script type="text/javascript">
  17      var so = new SWFObject('/mediaplayer.swf','mpl','470','20','8');
  18      so.addParam('allowscriptaccess','always');
  19      so.addParam('allowfullscreen','true');
  20      so.addVariable('width','470');
  21      so.addVariable('height','20');
  22      so.addVariable('file','/<%= oRssItem.filename%>');
  23      so.write('player<%=oRssItem.id%>');
  24    </script>
  25  <% end %>

This generates a page full of podcast entries, each with a little media player thing so you can listen to them easily. This does mp3 files but it can handle video too.

I am aware that the swfobject.js would be better loaded in the html header.

Here's a demo: I don't know if this will work in the rss feed (UPDATE: no!) so bear with me and please be gentle on my bandwidth, it's quite a catchy tune:

The main reason I am blogging this is because it only needs a change to a view file and installation of two files in the /public directory (the player and a javascript file). Since view files are awkward to comment legibly I decided to document it all here so I can find it again.

In a mammoth programming session I implemented proper smileys

[[':-)', 'smile'],
  [':-(', 'sad'],
  ['8-)', 'cool'],
  [';-)', 'wink']].each do |strCode, strFile|
  strText.gsub!( strCode, "<img alt=\"#{strFile}\" src=\"/images/smileys/#{strFile}.png\" />")
end

My fingers were bleeding from all the typing Worked first time though

There is even alt text for blind people

I figured out how to scrape the Wow Armory for up-to-date information about my characters using ruby. This means I can do this directly in PetersBlogger without using third party signature generators that disappear overnight or get absorbed into wow gold sites.

   1    require 'net/http'
   2  
   3    #
   4    # Scrape wow character info.
   5    #
   6    def WoWInfo
   7      begin
   8        strRealm = 'Eonar'
   9  
  10        oInfo = []
  11  
  12        ['Pookypoo', 'Maevyn', 'Maezyn', 'Maexyn'].each do |strCharacter|
  13          oCharInfo = []
  14  
  15          #
  16          # Open url.
  17          # Need to specify firefox as user agent as this makes the server return an XML
  18          # file.
  19          # Look the data up in the european armory. Change this for US.
  20          # If this is not done we get html.
  21          oResp = Net::HTTP.start( "armory.wow-europe.com", 80) do |http|
  22                http.get( "/character-sheet.xml?r=#{strRealm.tr( ' ', '+')}&n=#{strCharacter}",
  23                            { 'user-agent' => 'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-GB; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4'})
  24          end
  25  
  26          oDoc = REXML::Document.new oResp.body
  27          oDoc.elements[1].elements.each( 'characterInfo/character') do |oElement|
  28            oCharInfo << {:race => oElement.attributes['race']}
  29            oCharInfo << {:class => oElement.attributes['class']}
  30            oCharInfo << {:level => oElement.attributes['level']}
  31          end
  32  
  33          oDoc.elements[1].elements.each( 'characterInfo/characterTab/talentSpec') do |oElement|
  34            oCharInfo << {:spec => "#{oElement.attributes['treeOne']}/#{oElement.attributes['treeTwo']}/#{oElement.attributes['treeThree']}"}
  35          end
  36  
  37          oDoc.elements[1].elements.each( 'characterInfo/characterTab/professions/skill') do |oElement|
  38            oCharInfo << {oElement.attributes['key'].to_sym => oElement.attributes['value']}
  39          end
  40  
  41          oInfo << {strCharacter => oCharInfo}
  42        end
  43  
  44        return oInfo
  45      rescue
  46        return []
  47      end
  48    end

This lot generates the info in this page which is linked to under 'My Characters' over to the right. It should be fairly up-to-date as the cache will flush once a day causing the armory to be requeried. Currently the output is presented in a rails view thus:

   1  <% cache( :part => :wow) do %>
   2    <table>
   3    <% WoWInfo().each do |oCharInfo| %>
   4      <% oCharInfo.each_pair do |strChar, oInfo| %>
   5        <tr><td colspan="2"><b><%= strChar %></b></td></tr>
   6        <% oInfo.each do |oItem| %>
   7          <% oItem.each_pair do |strItem, strValue| %>
   8            <tr>
   9              <td><%= strItem %>:</td><td><%= strValue %></td>
  10            </tr>
  11          <% end %>
  12        <% end %>
  13      <% end %>
  14    <% end %>
  15    </table>
  16  <% end %>

All the work is done by the WoWInfo method which I put in the application helper. I didn't use a controller method for this.

I'm going to think about a nicer presentation with some pictures of the actual guys rather than generic pictures of bald redhead dwarves.

UPDATE: I've added my old Aerie Peak toons to the output but the code above still stands. Lugulas was my old auction alt who I've never named here before, hence level 10 with level 75 (dis)enchanting. I find myself missing them...

One of the things that is offputting about learning any of the web application stacks that I have tried (rails, turbogears, django) is the command line operations that need to be performed. When you start using them they seem like magic but after a while they become old friends. Hence this article is a summary of the rails command line operations that I use routinely.

All these commands except for the first one should be executed in the root directory of your application. Note that for the commands that start 'script' you are running ruby scripts and under windows you will need to type 'ruby script/blah'.

rails {app name}

The basic one, creates a new rails application. It will create a subdirectory with the given name in the current directory. This subdirectory is the root directory of your application and will be filled with all the appropriate files and subdirectories to start developing. It is a big code generator.

script/server

This could be considered the next step after creating your application. You cd to the application's root directory and run this command. It will start a development web server and you will be able to connect to localhost:3000 in a web browser and admire your new creation. This is the way to debug your application. Any print statements will be dumped out here.

script/generate model {model name}

This command will generate a new model in your application. A model is roughly equivalent to a table in the database.

script/generate controller {controller name}

This command will generate a new controller in your application in the file 'app/controllers/{controller name}_controller.rb'. A controller is essentially the logic that takes a web page request and decides what to do with it. It will also create a directory called 'app/views/{controller name}' to store the views for this controller, views being the templates that determine how the web pages will look.

rake db:migrate

This runs the migrations that will either define the database schema, modify it or undo those modifications. It defaults to updating the development database but can take the argument 'RAILS_ENV=production' to update the production database.

script/console

script/console is useful for debugging. It gives you a ruby irb command line prompt where you can play with the inner workings of your code, primarily you have direct access to your models and hence the database. It can be an easier way of manipulating the database from a command line than typing SQL into the mysql or sqlite command line clients.

rake --tasks

Lists what rake tasks are available.

rake db:test:prepare

Prepares the test database for running unit tests.

rake test

Runs all unit, functional and integration tests on your model.

I think that covers it. There are many more options than these are the ones that I have found to be most useful.

Rake is the ruby equivalent of make except it is much better in that you are scripting in ruby and not the nasty primeval make language. Rails uses rake a lot as a simple place to gather together a load of administration tasks such as running tests, running database migrations etc.

When I moved all my drupal blog posts from drupal over to PetersBlogger I decided to use a nice organised rake script rather than do my usual practise of having a directory full of utility scripts.

In my rails installation I created a file called lib/tasks/jobs.rake and in it I put my import code:

   1  task (:import_drupal => :environment) do
   2    print "Importing Drupal\n"
   3    oTagTable = {}
   4    #
   5    # Import all the articles.
   6    #
   7    Drupal::Article.find(:all, :conditions => "type <> 'image'").each do |oArticle|
   8      print "#{oArticle.title}\n"
   9      oTime = Time.at( oArticle.created)
  10      oPost = Post.create( :nid => oArticle.nid,
  11                          :title => oArticle.title,
  12        :body => oArticle.body,
  13        :created => oTime)
  14  
  15      #
  16      # import the tags related to the article.
  17      #
  18      oArticle.tags.each do |oTag|
  19        if oTagTable[oTag.tag]
  20          oNewTag = oTagTable[oTag.tag]
  21        else
  22          oNewTag = Tag.new( :tag=>oTag.tag)
  23          oTagTable[oTag.tag] = oNewTag
  24        end
  25  
  26        oPost.tags << oNewTag
  27      end
  28  
  29      #
  30      # Import the comments for the article.
  31      #
  32      oArticle.comments.each do |oComment|
  33        strWho = oComment.name
  34        strComment = oComment.comment
  35        if strWho == nil or strWho == ""
  36          strWho = "Peter"
  37        end
  38        if strComment == nil or strComment.strip == ""
  39          strComment = "{nothing to say}"
  40        end
  41        oPost.comments << Comment.create( :who => strWho,
  42            :body => strComment,
  43            :created => Time.at( oComment.timestamp), :published => 1)
  44        if not oPost.comments[-1].valid?
  45          p oPost.comments[-1]
  46          p oPost.comments[-1].errors
  47        end
  48      end
  49  
  50      oPost.save!
  51    end
  52  end

This creates a rake task called 'import_drupal' which I can invoke simply by running

rake import_drupal

The task runs with the appropriate rails environment so all my models are in place. I had to create a model to access the records in the drupal database thus:

   1  module Drupal
   2    class Comment < ActiveRecord::Base
   3      establish_connection configurations['drupal']
   4      set_table_name 'comments'
   5    end
   6    class Tag < ActiveRecord::Base
   7      establish_connection configurations['drupal']
   8      set_primary_key 'tid'
   9      has_and_belongs_to_many :articles,
  10        :class_name => 'Drupal::Article',
  11        :join_table => 'awtags_node',
  12        :foreign_key => 'tid',
  13        :association_foreign_key => 'nid'
  14      set_table_name 'awtags'
  15    end
  16    class Article < ActiveRecord::Base
  17      establish_connection configurations['drupal']
  18      set_primary_key 'nid'
  19      self.inheritance_column = 'poopy'
  20      has_many :comments, :table_name => 'comments', :foreign_key => 'nid', :class_name => 'Drupal::Comment'
  21      has_and_belongs_to_many :tags,
  22        :class_name => 'Drupal::Tag',
  23        :join_table => 'awtags_node',
  24        :foreign_key => 'nid',
  25        :association_foreign_key => 'tid'
  26      set_table_name 'node'
  27    end
  28  end

This model handles all the relationships between the records, including the has_and_belongs_to_many relationship between posts and tags (provided by awtags).

I needed an entry in config/database.yml to define the connection to the drupal database:

drupal:
  adapter: mysql
  database: petersblog
  username: secret
  password: doyouthinkidpostithere

and it all works nicely. If I invoke it thusly:

rake import_drupal RAILS_ENV=production

then it will import the posts into the rails production database.

I'm giving away a few of my schema secrets here which pedantic rails developers might leap on, specifically I have used my own fields called 'nid' and 'created' to perform roles that rails already provides 'id' and 'created_on' for. The reason for the duplication was that during development I was having problems setting these fields to the values from the drupal database, rails was insisting on giving them it's own values. By having my own fields I had total control. There are no doubt ways around this but I didn't have time to find them (although I did find other people moaning about the same problem).

And that's how I managed to save 1500 posts from the hell that is php.

If you are really bored, you might like to check out my new Archive Page.

I thought about putting it in the sidebar but I didn't for a few reasons:

• it's ugly
• it takes time to generate the page and I didn't want the rendering of every url that wasn't already cached to require this hit.
• who cares anyway? It's mainly there to open up the older posts to search bots without them having to go through page/next 100 times.

On a more technical level, it uses the following route to give 'nice urls' for the year/month combinations:

map.connect 'post/on/:year/:month', :controller => 'post', :action => 'listbyyearmonth'
map.connect 'post/on/:year', :controller => 'post', :action => 'listbyyearmonth'

so March 2005 would be '/post/on/2005/3' which is cool.

Since I added the page count thing yesterday (that only I can see since I don't want everyone to know how often I tweek the site and reboot) I got this:

I think I've managed to map all the urls from my old drupal blog to this new blog: according to statcounter (which is a bit slow loading today) I had 1,103 page loads yesterday compared to 1,167 last week on the old blog. Any missed urls would lead to the 404 page which doesn't have the hit counter or statcounter on it (hum, could catch the 404's and redirect them to main page with a flash notice..).

Statcounter works under javascript and so does not count bots and paranoid people with javascript disabled. It is interesting therefore that the latter outnumber the javascript users by 2.5:1. And none of this includes rss/atom feeds.

For the record, here are the routes I used to map my drupal urls to PetersBlogger urls. This is from config/routes.rb:

   1  map.root :controller => 'post', :action => 'list'
   2  
   3  #
   4  # map the various rss feeds
   5  #
   6  map.connect 'tag/:tagname/feed', :controller => 'tag', :action => 'rss'
   7  map.connect 'tags/:tagname/feed', :controller => 'tag', :action => 'rss'
   8  map.connect 'tag/feed/:tagname', :controller => 'tag', :action => 'rss'
   9  map.connect 'blog/feed/1', :controller => 'post', :action => 'rss'
  10  map.connect 'blog/feed', :controller => 'post', :action => 'rss'
  11  map.connect 'blog/1/feed', :controller => 'post', :action => 'rss'
  12  map.connect 'rss.xml', :controller => 'post', :action => 'rss'
  13  
  14  #
  15  # map the atom feeds
  16  #
  17  map.connect 'tags/:tagname/atom.xml', :controller => 'tag', :action => 'atom'
  18  map.connect 'node/:id/atom.xml', :controller => 'post', :action => 'atom'
  19  map.connect 'atom/feed', :controller => 'post', :action => 'atom'
  20  
  21  #
  22  # Map pages
  23  #
  24  map.connect 'node/view/:id', :controller => 'post', :action => 'show'
  25  map.connect 'node/:id', :controller => 'post', :action => 'show'
  26  # not sure what below was for
  27  map.connect 'node/:id/tag/function.require', :controller => 'post', :action => 'show'
  28  map.connect 'node/:id/function.require', :controller => 'post', :action => 'show'
  29  map.connect 'tag/:tagname', :controller => 'post', :action => 'listbytag'
  30  map.connect 'tags/:tagname', :controller => 'post', :action => 'listbytag'
  31  map.connect 'tags/:tagnamex/tags/:tagname', :controller => 'post', :action => 'listbytag'
  32  map.connect 'blog/tags/:tagname', :controller => 'post', :action => 'listbytag'
  33  map.connect 'tag', :controller => 'tag', :action => 'list'
  34  
  35  # Install the default routes as the lowest priority.
  36  map.connect ':controller/:action/:id'
  37  map.connect ':controller/:action/:id.:format'

I fixed most of the above by searching the access logs for 404's (page not found) and looking for patterns. I'm not sure why drupal felt the need to support so many variants on urls for the same page/feed or how people managed to find them. Note the silly 'tags/:tagnamex/tags/:tagname': I had old urls of the form '/tag/213/tag/34' because of sloppy relative linking.

Rails routing is sublimely simple and flexible. It would have taken me days to get all the above working with mod_rewrite rules.