Peter's Blog

Redefining the Impossible

Items filed under pyds


Decided upon Bloglines for my news aggregating needs. It has lots of features and is fast enough. It even allows me to add a blog roll box here.

It is comprehensive, almost overwhealming in the options that are available. My tip: subscribe to a blog just to get a username. That way you don't have to use your email address as an identifer that can be scraped by spambots.

Previously I was using the Python Desktop Server news aggregator that works very nicely but has two disadvantages:

  1. it needs a server to run on and now I am shutting down my home server I won't have one.
  2. I can access Bloglines from work without setting up an ssh tunnel.

Filed under: blog bloglines pyds python ssh


Some parasite posted a comment to an entry in this blog that was just a list of links to other websites, probably to boost google rankings. I'm fuming at these parasites. I've deleted the comment. It was only up for a day so hopefully the googlebot didn't spot it.

Python Desktop Server (or the version I am still using) seems to have a problem generating the archive calendar, some days do not get hyperlinks to archived entries. This makes it awkward to find them, I have to guess dates in the hyperlinks.

Also, if I have an archive page open and I type in a new posting, PyDS creates the new posting with the archived page date rather than today. It just happened, you may see two similar posts.


Filed under: blog google pyds python

1 Comment

I have been fiddling for days on getting decent wiki formatting working in Drupal. My requirements are modest (ahem):

  • Bullet points entered typing a simple asterisk and a space: no toolbars, nothing more elaborate.
  • New style hyperlinks, far easier than restructuredtext e.g:
  • Facility to put in raw html, specificly source code snippets that have been syntax highlighted using vim.
  • No need to edit snippets, such as inserting indents in every line, just adding a start and end tag.

Drupal's problem, acknowledged in the forums, is that the system of filtering that is used to convert the raw text to the formatted html does not really work very well: the filters step on each other's toes. My specific problem was that code snippets I put in would be altered by later filters, specificly text in square brackets would be converted to hyperlinks that didn't work. The wiki module also managed to interfere with itself: every line was transformed unless an exclamation mark was put at the start of each line: a tedious amount of extra editing.

I tried a number of options:

  • try to get the latest phpwiki source to work with Drupal. I got this to a state where it was kinda working but it continually messed up external links. The code was hard work, complex object model, no useful comments (only FixMes). I gave up on this as my head was hurting from banging on a brick wall.
  • try using textile. I didn't like this ones syntax.
  • try using bbcode but the syntax is worse, virtually html using [] instead of <>.
  • try using the markdown module but this is not free for commercial use.
  • hack on the existing wiki module, adding the concept of blocks that are not processed by other forms of markup within the wiki module itself.

I've gone for the last option, I've added two forms of block tags:

<verbatim> tag
html entities are escaped so < > & etc can all be used:
  $text = str_replace("<p>%::%</p>";,"",$text);
  $text = str_replace("%::%</p>","",$text);
  $text = str_replace("%::%","",$text);
  $text .= $DontScrewWithSquareBrackets[9];
  $two = 1 & 1;

<rawhtml> tag
insert raw html, e.g. pre syntax-highlighted code

<?php
// $Id: wiki.module,v 1.8 2004/02/08 22:39:38 tdobes Exp $
/**
 * wiki.module - mostly taken from <http://phpwiki.sourceforge.net/>, v1.2.2
 */

function wiki_link($type) {
  if ($type == "system") {
    menu("wiki", t("Wiki Text Formatting Rules"), "wiki_page", 0, MENU_HIDE);
  }
}

function wiki_page()
{
  if (!user_access("access content")) {
    print theme("page", message_access(), t("Access Denied"));
    return;
  }

I'm not entirely happy with this as a solution, beating on the phpwiki code would have been better as that is a far more complete language. Maybe I will give that another try?

Update: I beat some more and did fix my problem. However, I came across other problems:

  • The vertabim tag is equivalent to my code tag, it does not allow raw html to pass through
  • phpwiki allows certain raw html tags to pass through, for example b, small, etc but not the font tag. The regexp matching that it uses is restrictive and does not allow tags with attributes past. This could not be fixed without hacking on the phpwiki code itself. I don't want to use a library I have to hack on directly. I could work around it by subclassing but what if the base classes get changed (ref broken PyDS problems).
  • phpwiki has a 'RawHTML' plugin but that applies to the whole page, not sections thereof. Also this did not work as it relied upon the larger phpwiki structure rather then the bit that converted a string to html.
  • phpwiki folk are paranoid about javascript security holes in raw html. I want this for intranet so anyone who breaks things gets sacked.
  • there are also purists who think the markup should be enough and html is a cop out. They are probably not people who like to push boundaries, they prefer to erect them.

I'll stick with my hacked wiki module until something better comes along.


Filed under: drupal php pyds theme vim

1 Comment

Some parasite posted a comment to an entry in this blog that was just a list of links to other websites, probably to boost google rankings. I'm fuming at these parasites. I've deleted the comment. It was only up for a day so hopefully the googlebot didn't spot it.

Python Desktop Server (or the version I am still using) seems to have a problem generating the archive calendar, some days do not get hyperlinks to archived entries. This makes it awkward to find them, I have to guess dates in the hyperlinks.


Filed under: blog google pyds python


Sometimes I am Smiley

Sometimes I am Unsmiley

Sometimes I am Cool

Sometimes I am Puzzled

Sometimes I am Barf

Munged things so drupal->pyds smileys are consistant. I used a quick hack, replacing drupal format smileys with PyDS shortcut smileys before posting to PyDS.

Drupal has a system of filters for processing things like smiley->img conversion and in this one case of cross posting to PyDS it should not be done when formatting a posting. Fortunately the drupal smiley shortcuts do not have quotes around them so converting them to PyDS shortcuts before running them through the conversion filter sidesteps the problem neatly.


Filed under: drupal pyds


The following can be used in PyDS to wrap text around pictures:

$macros.imageTag( 'images/picture.jpg', align='left')

This makes me happy.


Filed under: pyds


At work I am using Drupal as a CMS for blogging, organising project related information etc. I feel it is more suited to a collaborative role than PyDS which is essentially a private tool for publishing to a community server. The community servers available do not appear to be strong CMS systems.

Drupal is very nicely written and even I, a PHP n00b, have managed to write a Drupal module that enables me to email a posting from the work server to my home PyDS installation where it can be posted to my personal/public blog. The email->PyDS gateway that I set up for photo blogging does the posting of the email to the blog.

Main stubling block encountered was when random pieces of the email message received were missing. This turned out to be because the message was stored in the database with \r line delimiters rather than \n. When the message was transmitted this looked to the email stuff like VERY long lines so it proceeded to screw up. I'm not sure where these came from, whether the web browser uses these in text fields or what. This script turns them to \n's which everybody loves (except dos/mac zealots).

Here is my Drupal 'sendhome.module' which does the posting.

   1  <?php
   2  /*
   3   * Add option to send postings to home.
   4   * Inspired by the emailpage module but simplified.
   5   * No admin options, edit this module.
   6   */
   7  
   8  function sendhome_help($section = 'admin/help#sendhome') {
   9    $output = "";
  10  
  11    switch ($section) {
  12      case 'admin/modules#description':
  13        $output = t("Send page home");
  14        break;
  15    }
  16    return $output;
  17  }
  18  
  19  function sendhome_link($type, $node=0, $main) {
  20    global $user;
  21  
  22    $links = array();
  23  
  24    /*
  25     * If logged in user is moi, give me the option of sending this home.
  26     */
  27    if( $user->name == 'my drupal login name') {
  28          $links[] = l(t("Send Page Home"), "sendhome/confirm/$node->nid", array("title" => t("Send Page Home."), "callback" => "sendhome_page"));
  29    }
  30    return $links;
  31  }
  32  
  33  /**
  34   * Implementation of hook_menu().
  35   */
  36  function sendhome_menu($may_cache) {
  37      global $user;
  38      $items = array();
  39  
  40  //    if ($may_cache) {
  41          /*
  42           * Register mapping of path to callback function.
  43           */
  44          $items[] = array('path' => 'sendhome/confirm',
  45          'callback' => 'sendhome_page',
  46          'access' => user_access('access content'),
  47          'type' => MENU_CALLBACK);
  48   //   }
  49  
  50      return $items;
  51  }
  52  
  53  
  54  function sendhome_page( $uid = 0) {
  55    /*
  56     * Get function parameters.
  57     */
  58    $op = $_POST["op"];
  59    $edit = $_POST["edit"];
  60  
  61    if( empty($op)) {
  62    	/*
  63    	 * In initial link nid is in the url
  64    	 */
  65    	if( arg(1) == 'confirm') {
  66    		$nid = arg(2);
  67    	}
  68    } else {
  69    	/*
  70    	 * From the send button the nid is in a hidden field.
  71    	 */
  72    	$nid = $edit["nid"];
  73    }
  74  
  75    /*
  76     * Search db for nodes.
  77     */
  78    $nodes = db_query("SELECT nid, title, teaser, created FROM {node} WHERE nid = %d", $nid);
  79  
  80      drupal_set_breadcrumb(array(l(t('Home'), NULL), l(t('View'), "node/view/$nid")));
  81  
  82    while ($node = db_fetch_object($nodes)) {
  83    	/*
  84    	** Load the specified node:
  85    	*/
  86    	$item = node_load(array('nid' => $node->nid));
  87  
  88    	/*
  89    	 * Add the title and the body to the output under construction.
  90    	 */
  91    	$output .= $item->title;
  92    	$output .= "\n";
  93    	$body = str_replace( '<!--break-->', '', $item->body);
  94    	/* Pass body through filters to create html */
  95  //  	$output .= check_output( $body);
  96    	$output .= $body;
  97    }
  98  
  99    /*
 100     * Maybe it's my fault for using windows but \r line delimiters lead to VERY
 101     * long lines that don't make it through email without random bits missing.
 102     */
 103    $output = str_replace( "\r\n", "\n", $output);
 104    $output = str_replace( "\r", "\n", $output);
 105  
 106    if ($op != t("Send")) {
 107    	/*
 108    	 * Display the posting that is to be sent and a send button.
 109    	 */
 110    	$output = "<div class=\"node\">$output</div>";
 111    	$output .= form( form_submit(t("Send")) . form_hidden("nid", $nid));
 112    	print theme( "page", $output, "Post Message Home?");
 113    }
 114    else {
 115    	/*
 116    	 * Set email parameters.
 117    	 */
 118    	$strTo = "censored@microsoft.com";
 119    	$strFrom = "me@here";
 120    	$strMagicSubjectLine = "hush, it's a secret";
 121  
 122    	/*
 123    	 * Send operation so send the post and show a 'Done' page.
 124    	 */
 125    	mail( $strTo, $strMagicSubjectLine, $output, "From: $strFrom\r\n", "-f$strFrom");
 126    	print theme( "page", '', "Posted Message");
 127    }
 128  }
 129  
 130  ?>

Filed under: blog drupal php pyds theme windows


Python Desktop Server has been repeatedly upstreamed two files this evening, wiki/Index.html and wiki/index.html and has failed to upstream a long blog post I made earlier this evening. Earlier it raised a few internal errors while trying to upload postings that had already been deleted:

 Background thread exception exceptions.ValueError: Invalid Posting id: P393Exception instance __dict__: {'args': ('Invalid Posting id: P93',)} 

/usr/lib/python2.3/site-packages/PyDS/Tool.pyWoW Daily News Issue 30 in thunk /usr/lib/python2.3/site-packages/PyDS/MirrorTool.pyNeed form D1 from post office to apply for driving.. in _initthread /usr/lib/python2.3/site-packages/PyDS/MirrorTool.pyIsilo looks good, it can format monospaced text.. in _addItem /usr/lib/python2.3/site-packages/PyDS/WeblogTool.pyStylish Stylus in getRSSItem /usr/lib/python2.3/site-packages/PyDS/WeblogTool.pyMy Vision in getPost

The way upstreaming works has a feel of something beyond my control where I have to cross my fingers and hope and wait patiently and keep refreshing the cloud until my changes appear.


Filed under: blog pyds python


As promised, how to do photo blogging from mobile phone. Actually this should support submitting plain text blog posts from any email source as I want to use the same mechanism to post blog entries from work that are for public display, bypassing the firewall. This recipe works with UK O2 MMS messages.

The first line of text in the email message becomes the title of the post, the following lines become the body.

Requirements:

  • Exim (or any MTA that supports forward files)

  • Python Desktop Server (or any server that supports MetaWeblogAPI or Blogger API, although you have to hack the image upload yourself, depending on the server).

  • Python (or you can rewrite it all in some other language)

Exim has to be set up to allow .forward files to process email. The .forward file should look like this:

# Exim filter
logfile ~/.forward.log

if $header_subject is "Multimedia message"
then
    pipe "/home/me/MailBot.py"
    seen finish
endif

This will direct emails with this subject line (i.e. UK O2 MMS messages) to the python script that does the hard work. This script is as follows and has to be made executable by the exim process:

#!/usr/bin/python
#
# Take an email message from stdin and post it to PyDS Blog.
# This works with UK O2 media messaging.
#
import smtplib
import sys
import email
import xmlrpclib

strMobileNumber = "0123456"
strXMLRPCUser = "fred"
strXMLRPCPassword = "xxx"

#
# Read email message piped in by exim filter from stdin.
#
strMsg = ""

for strLine in sys.stdin.readlines():
    
strMsg += strLine

oEmail = email.message_from_string( strMsg)

#
# Mobile number is in from address on O2.
# Stop people posting pr0n on my web site
# Hack this to add extra senders or security.
#
if not( oEmail'from'">'from'.find( strMobileNumber) >= 0:
    
#
    
# Does not appear to be from me.
    
#
    
sys.exit()

#
# Run email through parser, collecting all text/plain parts.
#
body = ""
strImages = []

for oPart in oEmail.walk():
    
#
    
# Look for plain text
    
#
    
if oPart.get_content_type() == 'text/plain':
        
if oPart.is_multipart() == False:
            
strPart = oPart.get_payload( decode=True)

            
#
            
# Ignore blurb added to message by o2.
            
#
            
if strPart.find( "This is a Media Message from O2") >= 0:
                
continue

            
body += strPart

    
#
    
# Look for images
    
#
    
if oPart.get_content_type() == 'image/jpeg':
        
strImage = oPart.get_payload( decode=True)
        
strFileName = oPart'Content-Location'">'Content-Location'
        
#
        
# Hack to pyds dir
        
#
        
open( '/home/me/.PyDS/www/images/' + strFileName, 'wb').write( strImage)
        
strImages.append( "$macros.imageTag( 'images/%s')\n" % strFileName)

#
# The first line is the title. The rest is the body.
#
strBody = body.split( '\n')
strTitle = strBody0">0
strBody = "".join( strImages) + "\n".join( strBody1:">1:)

#
# Open xmlrpc link to pyds.
#
oPyDS = xmlrpclib.ServerProxy( 'http://localhost:4334/RPC2')

#
# Get recent posts and build a map of title->postid.
#
oPosts = oPyDS.metaWeblog.getRecentPosts( '', strXMLRPCUser, strXMLRPCPassword, 1000)

oExistingPosts = {}

for oPost in oPosts:
    
oExistingPostsoPost['title'">oPost['title'] = oPost'postid'">'postid'

#
# Build parameters for post using unicode to pass weird characters.
#
oPost'title'">'title' = unicode( strTitle, 'ISO-8859-1')
oPost'description'">'description' = unicode( strBody, 'ISO-8859-1')

#
# See if this is a new post from it's unique title.
# If it is unique then post it as a new article.
#
if not oExistingPosts.has_key( strTitle):
    
oPyDS.metaWeblog.newPost( '', strXMLRPCUser, strXMLRPCPassword, oPost, 1)
else:
    
#
    
# Edit existing post
    
#
    
oPyDS.metaWeblog.editPost( oExistingPostsstrTitle">strTitle, strXMLRPCUser, strXMLRPCPassword, oPost, 1)

I had to make a small hack in Python Desktop Server. Posted articles default to either raw html or strict structured text but the image link requires structured text + cheetah. I had to edit the file MetaWebWeblogAPI.py to change the two lines that said:

structured = 2

to

structured = 1

ToDo:

  • Try to get text to wrap round image

  • Save up to pay my mobile bill



I've been looking around the options for dynamic web content recently for various projects. One of them is to be hosted under IIS, the other I can do with as I please. The IIS based one required a CMS so that many people can add content. The original request was for something that used ASP, VBScript and Access databases and was not .NET. I searched around and found nothing suitable. Widening the search I came across Drupal which seemed to fit the bill except that it was PHP and MySQL.

At about the same time I was considering this I came across the PHP EasyWindows Installer which installs PHP as a CGI engine which is cool enough, I don't think I have to worry about high volume traffic.

So I installed it and MySQL on Windows XP and it runs a treat. The pages are server up quite swiftly and the package looks quite powerful. It looks nicely designed: a clear segragation between content (all stored in the database) layout (a single, nicely laid out template file that builds all the pages on the site, easily edited in HTML-Kit) and formatting (.CSS).

I'm still exploring it but it is looking suitable for my needs and the person I am doing it for has agreed to install PHP and MySQL on his windows box. I'm not reluctant to learn PHP although I find that after python all the $s in the code look like noise: not as bad as perl (from which it is distantly related) but nearly so. The drupal source that I have looked through is quite clean. There is not a great deal of documentation but that is par for the course. I might not even need to write any PHP, I've found modules to do most of the things I want.

One nice thing about Drupal that Python Desktop Server is lacking: a button to preview messages before submitting them.

For my other project I am looking into a way to present email archives on a web site. I've looked through various options, none of which appealed:

HyperLink

Crude presentation, no Gentoo ebuild.

MHonarch

Gentoo package failed to emerge

Macho

Nice presentation but an embarrasing name to search for on the net and also written in lisp (((ugh))). I don't really want to install yet another language and I certainly don't want to learn lisp.

Mailman

GNU list manager thing which includes a modern version of PiperMail which was what I wanted (python email->web code). I installed it but was boggled by the complexity so I uninstalled it quick. Life is too short. There are various other list managers I could hack on but it's a matter of finding the small part for presenting the archives amongst the rest of it. Plus I need to be sure they can handle multiple addresses, CC's, attachments, html mail etc.

Mod_Python

So I gave mod_python a whirl. It installed easily enough but I soon ran into a problem whereby if I generated too much HTML (two or three pages) then I'd get a segmentation error and no response from Apache. It's probably another Gentoo version mismatch problem but I didn't want to get involved so I ditched mod_python.

I finally decided upon python CGI. This was easy to set up and gives me total control. No learning curve beyond the cgi module and I can use the IMAP library to access my email.

I've been looking around the options for dynamic web content recently for various projects. One of them is to be hosted under IIS, the other I can do with as I please. The IIS based one required a CMS so that many people can add content. The original request was for something that used ASP, VBScript and Access databases and was not .NET. I searched around and found nothing suitable. Widening the search I came across Drupal which seemed to fit the bill except that it was PHP and MySQL.

At about the same time I was considering this I came across the PHP EasyWindows Installer which installs PHP as a CGI engine which is cool enough, I don't think I have to worry about high volume traffic.

So I installed it and MySQL on Windows XP and it runs a treat. The pages are server up quite swiftly and the package looks quite powerful. It looks nicely designed: a clear segragation between content (all stored in the database) layout (a single, nicely laid out template file that builds all the pages on the site, easily edited in HTML-Kit) and formatting (.CSS).

I'm still exploring it but it is looking suitable for my needs and the person I am doing it for has agreed to install PHP and MySQL on his windows box. I'm not reluctant to learn PHP although I find that after python all the $s in the code look like noise: not as bad as perl (from which it is distantly related) but nearly so. The drupal source that I have looked through is quite clean. There is not a great deal of documentation but that is par for the course. I might not even need to write any PHP, I've found modules to do most of the things I want.

One nice thing about Drupal that Python Desktop Server is lacking: a button to preview messages before submitting them.

For my other project I am looking into a way to present email archives on a web site. I've looked through various options, none of which appealed:

HyperLink

Crude presentation, no Gentoo ebuild.

MHonarch

Gentoo package failed to emerge

Macho

Nice presentation but an embarrasing name to search for on the net and also written in lisp (((ugh))). I don't really want to install yet another language and I certainly don't want to learn lisp.

Mailman

GNU list manager thing which includes a modern version of PiperMail which was what I wanted (python email->web code). I installed it but was boggled by the complexity so I uninstalled it quick. Life is too short. There are various other list managers I could hack on but it's a matter of finding the small part for presenting the archives amongst the rest of it. Plus I need to be sure they can handle multiple addresses, CC's, attachments, html mail etc.

Mod_Python

So I gave mod_python a whirl. It installed easily enough but I soon ran into a problem whereby if I generated too much HTML (two or three pages) then I'd get a segmentation error and no response from Apache. It's probably another Gentoo version mismatch problem but I didn't want to get involved so I ditched mod_python.

I finally decided upon python CGI. This was easy to set up and gives me total control. No learning curve beyond the cgi module and I can use the IMAP library to access my email.