Peter's Blog - Nodes for rails

Getting back up to speed

I was working on this site and noticed how slow it was, particularly on rss feeds. This was just after managing to completely screw up my ruby gem installation (apt-get remove rubygems/apt-get install rubygems didn't reinstall rubygems) which I only cured by upgrading the server to ubuntu hardy. I wasn't sure whether I had blown something in the upgrade (the server did a fsck when I rebooted it but maybe only because it had been 'up' for 100 days) or whether the code was just slow.

How to pinpoint the cause of the slowness? Profiling. Some research revealed I can profile ruby code as follows:

install a ruby profiler:
```
gen install ruby-prof
```

insert some profiling code around your suspect slow code (in my case around the rss formatting):

   1  gem install ruby-prof
   2  
   3     require 'ruby-prof'
   4  
   5     RubyProf.start
   6  
   7     {code to profile}
   8  
   9   result = RubyProf.stop
  10     printer = RubyProf::FlatPrinter.new(result)
  11     printer.print(File.open( '/tmp/prof.txt', "w"), 0)

run script/server and try downloading rss
wait because it is very slow
examine the profile
```
vim /tmp/prof.txt
```
lament

This told me that 70% of my time was being spent in a gsub in my wilki code that I added last week. gsub was being called 1800 times (!) and when I commented out the new gsub call the rss was rendered much faster.

The thing is, my wilki formatting is supposed to be cached, it should only be done when a posting is modified, not every time the post is rendered. Turns out to be because updating a field in an ActiveRecord doesn't automatically cause it to be saved because it is dirty, an explicit save helped. Problem solved, now downloading the rss feed doesn't max out my cpu for 20 seconds

NB (clarification): the rss doesn't go through the rails fragment cache.

How big would you like that?

My websites are all migrating to a CMS kinda vibe where pages are edited through the web browser. Easy for me, easy for anyone else. I've already got the technology to upload pictures but it has an annoying problem and it seems to be built into the attachment_fu plugin. The problem is that one must choose how big one wants the pictures to be when they are first uploaded and the size is defined in the database model. attachment_fu will then resize the image accordingly. That is all very well if one is organised and prepared and knows in advance how big they want their pictures or are running a simple gallery where all pictures are the same size.

But what about where one is trying to do some nice layouts on a page and wants to adjust the sizes of the pictures? Say I decide I want a particular picture on a page to be 100 pixels wide? I have two options:

I can go into gimp/paintshop pro/whatever, resize the image and upload it (again) but that is annoying.
I can alter the img tags in my html to include a width attribute. I see a lot of this on other websites, as the page downloads you see lots of little images slowly filled in as huge 1000x1000 jpg's are downloaded and resized in the browser to 100x100 because the page designer couldn't be bothered to resize the image in gimp/paintshop pro/whatever.

Now there is another way. Here is a rails helper that allows the width of an image to be specified in a view, automatically resizing the image as appropriate:

   1  # Methods added to this helper will be available to all templates in the application.
   2  module ApplicationHelper
   3  
   4     #
   5     # Takes the url of an image and a desired width and ensures a cached resized
   6     # version of the image is available and returns the url of said image.
   7     #
   8     # the url should not be a relative url as it is assumed work relative to RAILS_ROOT/public
   9     # This also means the image must be on this server.
  10     #
  11     def cacheimage( strUrl, nWidth)
  12       # Find original file in file system
  13       strFile = "#{RAILS_ROOT}/public" + strUrl
  14  
  15       # create name for cached and resized image
  16       #
  17       # /blah/blah.jpg --> /blah/blah-123.jpg
  18       #
  19       strBits = strUrl.split( '.')
  20       strBits[-2] += "-#{nWidth}"
  21       strCacheName = strBits.join( '.')
  22  
  23       strCacheUrl  = "/imagecache#{strCacheName}"
  24       strCacheFileName = "#{RAILS_ROOT}/public#{strCacheUrl}"
  25  
  26       if not File.exists?( strCacheFileName)
  27         require 'image_science'
  28  
  29         FileUtils.mkpath( File.dirname( strCacheFileName))
  30  
  31         ImageScience.with_image( strFile) do |oImg|
  32           oImg.resize( nWidth, nWidth * oImg.height / oImg.width) do |oResized|
  33             oResized.save strCacheFileName
  34           end
  35         end
  36       end
  37  
  38       return strCacheUrl
  39     end
  40  end

That wasn't too difficult. You need to create an 'imagecache' directory under /public to store a cache of resized images and make sure mongrel or whatever can write to it.

In a view one would put:

<%= image_tag cacheimage( "/images/prettypicture.jpg", 500) %>

This will cause a version of prettypicture.jpg to be created that is resized to 500 pixels wide with a proportional change in height (in a typical html page layout the width of things is often more important than the height except maybe for headers and footers). The resized version is stored in a cache and will be displayed from then on.

If the original version of prettypicture.jpg is changed then one must remember to erase the resized versions from the cache as this isn't automatic.

Another helper is:

   def applyimagewidth( strText)
      strText.gsub!( /([^"!<>{}]*)(jpg|JPG|gif|GIF|png|PNG)x(\d+)/) do |oMatch|
         cacheimage( "#{$1}#{$2}", $3.to_i)
      end

      return strText
   end

(UPDATE: warning- the gsub above is VERY SLOW, it can take minutes to scan a few k of text, even if there are no matches).

which allows the width to be appended to the image file name. This allows a view such as

<%= textilize applyimagewidth( @page.body) %>

and thus one's textile source could be:

!/imagecache/images/DSC_4064-300.JPG!

to show an image with a width of 300 pixels. I could hack the redcloth source directly to add this feature but I'd rather keep out of there.

I've done this in a simplistic hacky way but it should suffice (like all my stuff).

What's in my Pockets?

Another fusion of WoW and rails.

While planning Maexyn's conversion to alchemy I thought it would be nice to be able to get a listing of my characters bag and bank items so I could look through my stocks and see what was what. It occurred to me that a nice way to do this would be to scan through the lua files that are dumped by the TBags mod since that holds a database of contents of all my characters bags and bank. Doing this in ruby turned out to be very easy:

Install ruby lua support:

sudo aptitude install liblua5.1-0-dev
sudo gem install rua

Ruby code to convert lua data file into ruby data:

oRua = Rua.new(:all)
oRua.eval( open( File.dirname( __FILE__) + "/../../db/TBag.lua").read)

oRua now has an array of items that maps onto the data stored in the lua file.

oRua['TBnkItm'] has a list of items in the bank
oRua['TInvItm'] has a list of items in the inventory

The end result is this page where I can see what stocks I have. Auctioneer also stores its data in lua so in a similar way I added a lookup so I see the values to these stocks. I can also use the item numbers to add links to the items to thottbot for reference. It isn't perfect, it should show vendor values, especially for soulbound items.

Anyway with this I can plan some sales, particularly of Maexyn's enchanting materials.

Currently the auctioneer data is way out of date, I need to get it running again and do a few scans.

This isn't here as a way of boasting about my vast wealth, I guess a week's karak farming would be better that this junk pile (minions scourgestones?!?).

N.B: an alternative implementation would be simply to write something in the lua language that reads the two lua databases and then dumps out the html. For me it is easier to do it in ruby than learn lua.

rspec

I've been trying out rspec, a new testing framework for ruby that allows one to better follow the principles of Behaviour Driven Design (BDD). BDD is a step on from Test Driven Design (TDD) approach exemplified in the standard ruby unit test framework. TDD is the process of starting ones development by writing the tests that your new module should pass. As you write each test you then implement the code to pass it and hence, voila, you have nicely designed and robust software. In BDD you approach testing the code from how it is supposed to behave. When liasing with clients this is a better approach as everyone can talk in the same language, essentially in terms of requirements so one can try to get less bogged down in implementation details.

Here are some rspec tests for a class I have been working on:

   1  describe ReportHelper do
   2    before(:each) do
   3      #
   4      # Create a new clean Report class for each individual test
   5      #
   6      @oReport = ReportHelper::Report.new
   7    end
   8  
   9    it "should allow a column to be added" do
  10      @oReport.AddColumn( 'First Column')
  11    end
  12  
  13    it "should allow multiple columns to be added" do
  14      @oReport.AddColumn( 'First Column')
  15      @oReport.AddColumn( 'Second Column')
  16    end
  17  
  18    it "should allow rows to be added to the report" do
  19      @oReport.AddColumn( 'First Column')
  20      @oReport.AddColumn( 'Second Column')
  21      @oReport.AddRow do
  22        # pass
  23      end
  24    end
  25  
  26    it "should only allow rows to be added if columns exist" do
  27      lambda{ @oReport.AddRow}.should raise_error(RuntimeError, 'No columns defined')
  28    end
  29  
  30    it "should return simple cell values" do
  31      @oReport.AddColumn( 'First Column')
  32      @oReport.AddColumn( 'Second Column')
  33  
  34      @oReport.AddRow do |oRow|
  35        oRow[ 'First Column'] = 98
  36        oRow[ 'Second Column'] = 198
  37      end
  38  
  39      @oReport[0]['First Column'].should == 98
  40      @oReport[0]['Second Column'].should == 198
  41    end
  42  
  43  end

Each test has a descriptive string that can be interpreted in two ways: as a test that must be passed or as a requirement for the software. The rspec toolset includes something that will pull out all these strings and give a summary of what testing has been done/what the requirements for the software are. If an auditor asks how the software is tested, bang, there is a full report of the tests. At a practical level (i.e. makes life easier for me) I prefer this to the ruby unit test system where one has to think up meaningful_method_names for each test. The strings are easier to write and read and reduce the need for comments.

Example report:

ReportHelper

- should allow a column to be added
- should allow multiple columns to be added
- should allow rows to be added to the report
- should only allow rows to be added if columns exist
- should return simple cell values

Rspec contains sweeter syntactic sugar than the ruby unit test module. For example, rspec overloads every object in the system with a 'should' method for doing the equivalent of assert tests:

    @oReport[0]['First Column'].should == 98

Compare this to test/unit:

    assert_equal 98, @oReport[0]['First Column']

apart from more typing, assert_equal has the expectation/actual things in what is intuitively to me the wrong order and I'm always having to correct these lines because the error messages have the terms swapped.

Rspec can test assertions more cleanly:

    lambda{ @oReport.AddRow}.should raise_error(RuntimeError, 'No columns defined')

although it's a pity that the lambda is necessary (and probably no 1 noob trap) but with test/unit the following always seemed like a lot of typing:

    oError = assert_raises RuntimeError do
      @oReport.AddRow
    end
    assert_equal 'No columns defined', oError.message

rspec is nicely integrated into rails, plugins being available to generate rspec tested controllers and models and suchlike, rake tasks to generate documents (woohoo) etc. It is integrated into netbeans and runs ok on Windows although when the tests are run (CTRL-F6) there is a tedious few seconds wasted in startup time. It is possible to run an rspec server in the background to eliminate this startup but it doesn't work on windows native ruby, it gives an error lamenting Microsofts glorious omission of a fork api.

There is much more to Rspec than this. One aim seems to be to write psuedo testing languages and have clients use these to write stories about what the software will do, these stories turning into a test framework. As the slashdot tag goes, 'goodluckwiththat'. At a practical level, I'm liking rspec even if only as a unit test framework with added syntactic sugar.

Related Posts: netbeans rails rspec ruby testing

Illustrated Blog

Up till now to put pictures in this blog I have had to:

go into an image editor to resize the file to suit the blog (about 450-500 pixels wide).
fire up an sftp program to upload the file to /var/www/PetersBlogger/public/images
add a tag such as
```
[/images/TheNewPicture.jpg]
```
to the wilki.

It was a bit laborious, especially the SFTP bit which is why I didn't do it very often.

I've now added better support for pictures to PetersBlogger. Here is an example:

Girls Aloud

if you click on this you should be treated to a larger version of the image, thanks to lightbox a javascript library.

PetersBlogger now supports a database of pictures so I can add a new picture, give it a name and upload it through the administration pages. The code uses the name as a caption under the photo and as alt text. I can use the name directly in my wilki text to choose which pictures to display.

The database of pictures is handled by the attachmentfu rails plugin. This article gives the steps required to get attachmentfu up and running.

The setup automatically handles resizing images to three sizes: thumbnail for listing in my picture admin page, a version that is no more than 500 pixels in either dimension for showing in the blog and a version that is no more than 800x600 for showing zoomed when you click on a picture. Attachmentfu has the sense to keep the aspect ratio correct automatically so I don't have to worry about portrait/landscape. I had to hack the imagescience library that does the resizing as it was hard-wired to save jpegs using JPEG_QUALITYSUPERB mode so the blog-sized images were taking up 150k. I changed this to JPEG_QUALITYAVERAGE and the size dropped to 15k or so, but it was a bit noisy so I've settled on JPEG_QUALITYGOOD and 38k images. You are on broadband?

I was slightly worried about how to change pictures, i.e. to upload a different version of a picture to replace an existing one, since the attachmentfu article I mentioned does not cover this. The answer is to use the exact same view code in the 'edit' page as in the 'new' page. Combined with rails scaffold code it Just Works! It even handles deleting the three old pictures and creating the new files.

My main motivation for doing this was so I can do it on the websites I host for other people and make it easier for them to upload pictures. PetersCMS is still in the planning stages but one problem has been cracked.

Embedding Flash Player

Much as I hate flash, it is a nice way to embed a sound or video player in a page. Here's how I did it in rails using the JW FLV Media Player. I added the following to my view:

   1  <script type="text/javascript" src="/javascripts/swfobject.js"></script>
   2  
   3  <% for oRssItem in @rssitems %>
   4    <hr/>
   5    <h3><%= oRssItem.title %></h3>
   6    <p><%= oRssItem.description %></p>
   7    <table width="470">
   8      <tr>
   9         <td wIdth="50%"><%= oRssItem.created.strftime( "%m %B %y") %></td>
  10         <td align="right"><%= link_to "Download", "/#{ oRssItem.filename}" %></td>
  11      </tr>
  12    </table>
  13  
  14    <div id="player<%=oRssItem.id%>">This text will be replaced</div>
  15  
  16    <script type="text/javascript">
  17      var so = new SWFObject('/mediaplayer.swf','mpl','470','20','8');
  18      so.addParam('allowscriptaccess','always');
  19      so.addParam('allowfullscreen','true');
  20      so.addVariable('width','470');
  21      so.addVariable('height','20');
  22      so.addVariable('file','/<%= oRssItem.filename%>');
  23      so.write('player<%=oRssItem.id%>');
  24    </script>
  25  <% end %>

This generates a page full of podcast entries, each with a little media player thing so you can listen to them easily. This does mp3 files but it can handle video too.

I am aware that the swfobject.js would be better loaded in the html header.

Here's a demo: I don't know if this will work in the rss feed (UPDATE: no!) so bear with me and please be gentle on my bandwidth, it's quite a catchy tune:

Oops, no flash? Javascript broken? Works for me.

The main reason I am blogging this is because it only needs a change to a view file and installation of two files in the /public directory (the player and a javascript file). Since view files are awkward to comment legibly I decided to document it all here so I can find it again.

Smile

In a mammoth programming session I implemented proper smileys

[[':-)', 'smile'],
  [':-(', 'sad'],
  ['8-)', 'cool'],
  [';-)', 'wink']].each do |strCode, strFile|
  strText.gsub!( strCode, "<img alt=\"#{strFile}\" src=\"/images/smileys/#{strFile}.png\" />")
end

My fingers were bleeding from all the typing Worked first time though

There is even alt text for blind people

Scraping WoW Armory in Ruby

I figured out how to scrape the Wow Armory for up-to-date information about my characters using ruby. This means I can do this directly in PetersBlogger without using third party signature generators that disappear overnight or get absorbed into wow gold sites.

   1    require 'net/http'
   2  
   3    #
   4    # Scrape wow character info.
   5    #
   6    def WoWInfo
   7      begin
   8        strRealm = 'Eonar'
   9  
  10        oInfo = []
  11  
  12        ['Pookypoo', 'Maevyn', 'Maezyn', 'Maexyn'].each do |strCharacter|
  13          oCharInfo = []
  14  
  15          #
  16          # Open url.
  17          # Need to specify firefox as user agent as this makes the server return an XML
  18          # file.
  19          # Look the data up in the european armory. Change this for US.
  20          # If this is not done we get html.
  21          oResp = Net::HTTP.start( "armory.wow-europe.com", 80) do |http|
  22                http.get( "/character-sheet.xml?r=#{strRealm.tr( ' ', '+')}&n=#{strCharacter}",
  23                            { 'user-agent' => 'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-GB; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4'})
  24          end
  25  
  26          oDoc = REXML::Document.new oResp.body
  27          oDoc.elements[1].elements.each( 'characterInfo/character') do |oElement|
  28            oCharInfo << {:race => oElement.attributes['race']}
  29            oCharInfo << {:class => oElement.attributes['class']}
  30            oCharInfo << {:level => oElement.attributes['level']}
  31          end
  32  
  33          oDoc.elements[1].elements.each( 'characterInfo/characterTab/talentSpec') do |oElement|
  34            oCharInfo << {:spec => "#{oElement.attributes['treeOne']}/#{oElement.attributes['treeTwo']}/#{oElement.attributes['treeThree']}"}
  35          end
  36  
  37          oDoc.elements[1].elements.each( 'characterInfo/characterTab/professions/skill') do |oElement|
  38            oCharInfo << {oElement.attributes['key'].to_sym => oElement.attributes['value']}
  39          end
  40  
  41          oInfo << {strCharacter => oCharInfo}
  42        end
  43  
  44        return oInfo
  45      rescue
  46        return []
  47      end
  48    end

This lot generates the info in this page which is linked to under 'My Characters' over to the right. It should be fairly up-to-date as the cache will flush once a day causing the armory to be requeried. Currently the output is presented in a rails view thus:

   1  <% cache( :part => :wow) do %>
   2    <table>
   3    <% WoWInfo().each do |oCharInfo| %>
   4      <% oCharInfo.each_pair do |strChar, oInfo| %>
   5        <tr><td colspan="2"><b><%= strChar %></b></td></tr>
   6        <% oInfo.each do |oItem| %>
   7          <% oItem.each_pair do |strItem, strValue| %>
   8            <tr>
   9              <td><%= strItem %>:</td><td><%= strValue %></td>
  10            </tr>
  11          <% end %>
  12        <% end %>
  13      <% end %>
  14    <% end %>
  15    </table>
  16  <% end %>

All the work is done by the WoWInfo method which I put in the application helper. I didn't use a controller method for this.

I'm going to think about a nicer presentation with some pictures of the actual guys rather than generic pictures of bald redhead dwarves.

UPDATE: I've added my old Aerie Peak toons to the output but the code above still stands. Lugulas was my old auction alt who I've never named here before, hence level 10 with level 75 (dis)enchanting. I find myself missing them...

Related Posts: games petersblogger rails ruby warcraft wow

Noob guide to the Rails Command Line

One of the things that is offputting about learning any of the web application stacks that I have tried (rails, turbogears, django) is the command line operations that need to be performed. When you start using them they seem like magic but after a while they become old friends. Hence this article is a summary of the rails command line operations that I use routinely.

All these commands except for the first one should be executed in the root directory of your application. Note that for the commands that start 'script' you are running ruby scripts and under windows you will need to type 'ruby script/blah'.

rails {app name}

The basic one, creates a new rails application. It will create a subdirectory with the given name in the current directory. This subdirectory is the root directory of your application and will be filled with all the appropriate files and subdirectories to start developing. It is a big code generator.

script/server

This could be considered the next step after creating your application. You cd to the application's root directory and run this command. It will start a development web server and you will be able to connect to localhost:3000 in a web browser and admire your new creation. This is the way to debug your application. Any print statements will be dumped out here.

script/generate model {model name}

This command will generate a new model in your application. A model is roughly equivalent to a table in the database.

script/generate controller {controller name}

This command will generate a new controller in your application in the file 'app/controllers/{controller name}_controller.rb'. A controller is essentially the logic that takes a web page request and decides what to do with it. It will also create a directory called 'app/views/{controller name}' to store the views for this controller, views being the templates that determine how the web pages will look.

rake db:migrate

This runs the migrations that will either define the database schema, modify it or undo those modifications. It defaults to updating the development database but can take the argument 'RAILS_ENV=production' to update the production database.

script/console

script/console is useful for debugging. It gives you a ruby irb command line prompt where you can play with the inner workings of your code, primarily you have direct access to your models and hence the database. It can be an easier way of manipulating the database from a command line than typing SQL into the mysql or sqlite command line clients.

rake --tasks

Lists what rake tasks are available.

rake db:test:prepare

Prepares the test database for running unit tests.

rake test

Runs all unit, functional and integration tests on your model.

I think that covers it. There are many more options than these are the ones that I have found to be most useful.

Raking in Drupal posts

Rake is the ruby equivalent of make except it is much better in that you are scripting in ruby and not the nasty primeval make language. Rails uses rake a lot as a simple place to gather together a load of administration tasks such as running tests, running database migrations etc.

When I moved all my drupal blog posts from drupal over to PetersBlogger I decided to use a nice organised rake script rather than do my usual practise of having a directory full of utility scripts.

In my rails installation I created a file called lib/tasks/jobs.rake and in it I put my import code:

   1  task (:import_drupal => :environment) do
   2    print "Importing Drupal\n"
   3    oTagTable = {}
   4    #
   5    # Import all the articles.
   6    #
   7    Drupal::Article.find(:all, :conditions => "type <> 'image'").each do |oArticle|
   8      print "#{oArticle.title}\n"
   9      oTime = Time.at( oArticle.created)
  10      oPost = Post.create( :nid => oArticle.nid,
  11                          :title => oArticle.title,
  12        :body => oArticle.body,
  13        :created => oTime)
  14  
  15      #
  16      # import the tags related to the article.
  17      #
  18      oArticle.tags.each do |oTag|
  19        if oTagTable[oTag.tag]
  20          oNewTag = oTagTable[oTag.tag]
  21        else
  22          oNewTag = Tag.new( :tag=>oTag.tag)
  23          oTagTable[oTag.tag] = oNewTag
  24        end
  25  
  26        oPost.tags << oNewTag
  27      end
  28  
  29      #
  30      # Import the comments for the article.
  31      #
  32      oArticle.comments.each do |oComment|
  33        strWho = oComment.name
  34        strComment = oComment.comment
  35        if strWho == nil or strWho == ""
  36          strWho = "Peter"
  37        end
  38        if strComment == nil or strComment.strip == ""
  39          strComment = "{nothing to say}"
  40        end
  41        oPost.comments << Comment.create( :who => strWho,
  42            :body => strComment,
  43            :created => Time.at( oComment.timestamp), :published => 1)
  44        if not oPost.comments[-1].valid?
  45          p oPost.comments[-1]
  46          p oPost.comments[-1].errors
  47        end
  48      end
  49  
  50      oPost.save!
  51    end
  52  end

This creates a rake task called 'import_drupal' which I can invoke simply by running

rake import_drupal

The task runs with the appropriate rails environment so all my models are in place. I had to create a model to access the records in the drupal database thus:

   1  module Drupal
   2    class Comment < ActiveRecord::Base
   3      establish_connection configurations['drupal']
   4      set_table_name 'comments'
   5    end
   6    class Tag < ActiveRecord::Base
   7      establish_connection configurations['drupal']
   8      set_primary_key 'tid'
   9      has_and_belongs_to_many :articles,
  10        :class_name => 'Drupal::Article',
  11        :join_table => 'awtags_node',
  12        :foreign_key => 'tid',
  13        :association_foreign_key => 'nid'
  14      set_table_name 'awtags'
  15    end
  16    class Article < ActiveRecord::Base
  17      establish_connection configurations['drupal']
  18      set_primary_key 'nid'
  19      self.inheritance_column = 'poopy'
  20      has_many :comments, :table_name => 'comments', :foreign_key => 'nid', :class_name => 'Drupal::Comment'
  21      has_and_belongs_to_many :tags,
  22        :class_name => 'Drupal::Tag',
  23        :join_table => 'awtags_node',
  24        :foreign_key => 'nid',
  25        :association_foreign_key => 'tid'
  26      set_table_name 'node'
  27    end
  28  end

This model handles all the relationships between the records, including the has_and_belongs_to_many relationship between posts and tags (provided by awtags).

I needed an entry in config/database.yml to define the connection to the drupal database:

drupal:
  adapter: mysql
  database: petersblog
  username: secret
  password: doyouthinkidpostithere

and it all works nicely. If I invoke it thusly:

rake import_drupal RAILS_ENV=production

then it will import the posts into the rails production database.

I'm giving away a few of my schema secrets here which pedantic rails developers might leap on, specifically I have used my own fields called 'nid' and 'created' to perform roles that rails already provides 'id' and 'created_on' for. The reason for the duplication was that during development I was having problems setting these fields to the values from the drupal database, rails was insisting on giving them it's own values. By having my own fields I had total control. There are no doubt ways around this but I didn't have time to find them (although I did find other people moaning about the same problem).

And that's how I managed to save 1500 posts from the hell that is php.