Peter's Blog - Nodes for bloglines

Net Abuse

Extracts from awstats logs, bandwidth used by various visitors to this site:

crawler.bloglines.com	353.02 MB
MSIECrawler	567.41 MB
Inktomi Slurp	135.79 MB
Googlebot	59.86 MB

Apparently MSIECrawler is IE sucking the entire contents of the site. A couple of people seem to have done this, what is the point? Is this site that interesting? Are the spam blogs (copies of legitimate blogs full of links to p0ker sites) using IE for their scraping technology? My attitude to the spammers turns from annoyance to pity.

Bloglines is getting a bit carried away, 353M just downloading RSS feeds.

InkTomi Slurp is still slurping and not returning any visitors from search results.

Googlebot drives 95% of my traffic so 59M is acceptable.

Here is my latest crack at apache log file analysis in python:

   1  #
   2  # Apache log file analysis.
   3  #
   4  import re
   5  import datetime
   6  
   7  #
   8  # Regular expression for parsing apache log file.
   9  #
  10  oLogRE = re.compile( r'''([\d.]+).*\s+                # host
  11                    [^\s]+\s+                        # ?
  12                    [^\s]+\s+                        # ?
  13                    \[(.*?)\]\s+                     # when
  14                    "(.*?)\s+(.*)\s+(.*)"\s+         # method, path, protocol
  15                    (\d+)\s+                         # Error code
  16                    ([^\s]+)\s+                      # Size ?
  17                    "(.*?)"\s+                       # Referrer
  18                    "(.*?)"                          # Agent
  19  ''', re.VERBOSE)
  20  
  21  LOG_Who = 0
  22  LOG_When = 1
  23  LOG_How = 2
  24  LOG_What = 3
  25  LOG_Protocol = 4
  26  LOG_Error = 5
  27  LOG_Size = 6
  28  LOG_Referrer = 7
  29  LOG_Agent = 8
  30  
  31  def ScanFile( strFile):
  32      """
  33      Scan apache log file and return hits.
  34      """
  35      for strLine in open( 'c:\\Desktop\\access.log').readlines():
  36          oMatch = oLogRE.match( strLine)
  37          if oMatch:
  38              yield( oMatch.groups())
  39          else:
  40              print 'Reject: %s' % strLine
  41  
  42  def GatherBy( oHits, nField):
  43      """
  44      Gather hits from list of hits into a dictionary keyed
  45      by unique values of a specific field.
  46      """
  47      oDict = {}
  48  
  49      for oHit in oHits:
  50          oKey = oHit[nField]
  51          if oKey in oDict:
  52              oDict[oKey].append( oHit)
  53          else:
  54              oDict[oKey] = [oHit]
  55  
  56      return oDict
  57  
  58  def FilterBy( oHits, nField, strFilter):
  59      """
  60      Filter hits from list of hits by unique values of a specific field.
  61      """
  62      oRE = re.compile( strFilter)
  63  
  64      for oHit in oHits:
  65          if oRE.search( oHit[nField]):
  66              yield( oHit)
  67  
  68  def FilterByDate( oHits,
  69                    oStartDate,
  70                    oEndDate = datetime.date.today() + datetime.timedelta(1)):
  71      """
  72      Filter hits >= Start Date and < End Date
  73      """
  74      oRE = re.compile( r'(\d+)/(\w+)/(\d+).*')
  75  
  76      for oHit in oHits:
  77          strDate = oHit[LOG_When]
  78          strDay, strMonth, strYear = oRE.match( strDate).groups()
  79  
  80          nDay = int( strDay)
  81          nMonth = ['Jan', 'Feb', 'Mar',
  82                    'Apr', 'May', 'Jun',
  83                    'Jul', 'Aug', 'Sep',
  84                    'Oct', 'Nov', 'Dec'].index( strMonth) + 1
  85          nYear = int( strYear)
  86  
  87          oDate = datetime.date( nYear, nMonth, nDay)
  88  
  89          if oDate >= oStartDate and oDate < oEndDate:
  90              yield( oHit)
  91  
  92  def AnalyseBy( oHits, nField, bJustSummary = False):
  93      """
  94      Print hits by unique values of a specific field
  95      and generate counts and bytes for each unique value.
  96      """
  97      oDict = GatherBy( oHits, nField)
  98  
  99      oKeys = oDict.keys()
 100  
 101      oKeys.sort()
 102  
 103      nGrandTotalCounts = 0
 104      nGrandTotalBytes = 0
 105  
 106      for oKey in oKeys:
 107          nCount = len( oDict[oKey])
 108  
 109          nTotal = 0
 110  
 111          for oHit in oDict[oKey]:
 112              strSize = oHit[LOG_Size]
 113              if strSize != '-':
 114                  nTotal += int(strSize)
 115  
 116          if not bJustSummary:
 117              print oKey, nCount, nTotal
 118  
 119          nGrandTotalCounts += nCount
 120          nGrandTotalBytes += nTotal
 121  
 122      print "Unique items: %d, Total Hits: %d, Total Bytes: %d" % (len(oKeys),
 123                                                                   nGrandTotalCounts,
 124                                                                   nGrandTotalBytes)
 125  
 126  oStartDate = datetime.date.today() - datetime.timedelta( 8 ) # week yesterday
 127  oEndDate = datetime.date.today() - datetime.timedelta( 1 )  # yesterday
 128  
 129  oAllHits = list( FilterByDate( ScanFile( 'c:\\Desktop\\access.log'),
 130                                 oStartDate, oEndDate))
 131  oAllHits.extend( list( FilterByDate( ScanFile( 'c:\\Desktop\\access.log.1'),
 132                                       oStartDate, oEndDate)))
 133  
 134  print "User Agents"
 135  AnalyseBy( oAllHits, LOG_Agent, True)
 136  
 137  print "All Hosts (hence all usage)"
 138  AnalyseBy( oAllHits, LOG_Who, True)
 139  
 140  print "Hits from bloglines"
 141  #
 142  # Determine different bloglines feeds and analyse each one
 143  #
 144  for strFeed, oFeedHits in GatherBy( FilterBy( oAllHits,
 145                                                LOG_Agent,
 146                                                'Bloglines'),
 147                                      LOG_What).items():
 148      #
 149      # Now analyse by agent:  agent includes number of subscribers
 150      # so we see subscribers per feed.
 151      print "Bloglines feed %s" % strFeed
 152      AnalyseBy( oFeedHits, LOG_Agent)
 153  
 154  print "Hits from MSIECrawler by host"
 155  AnalyseBy( FilterBy( oAllHits, LOG_Agent, 'MSIECrawler'), LOG_Who)
 156  
 157  print "Hits from Inktomi/yahoo slurp"
 158  AnalyseBy( FilterBy( oAllHits, LOG_Agent, 'Slurp'), LOG_Agent)

Example output for week ending yesterday. Notes:

659,628,604 bytes served in a week!
Slurp took 45,786,926 bytes
MSIECrawl user took 49,980,154 bytes
I've got more bloglines subscribers than I thought.
Just how many rss feed urls does drupal provide?

User Agents
Unique items: 497, Total Hits: 76052, Total Bytes: 659628604

All Hosts (hence all usage)
Unique items: 3301, Total Hits: 76052, Total Bytes: 659628604

Hits from bloglines
Bloglines feed /blog/1/feed
Bloglines/3.0-rho (http://www.bloglines.com; 1 subscriber) 252 13730570
Bloglines/3.0-rho (http://www.bloglines.com; 3 subscribers) 256 13947306
Bloglines/3.0-rho (http://www.bloglines.com; 5 subscribers) 252 13730570
Bloglines/3.0-rho (http://www.bloglines.com; 7 subscribers) 256 13947306
Unique items: 4, Total Hits: 1016, Total Bytes: 55355752
Bloglines feed /blog/feed
Bloglines/3.0-rho (http://www.bloglines.com; 1 subscriber) 242 13189698
Unique items: 1, Total Hits: 242, Total Bytes: 13189698
Bloglines feed /atom/feed
Bloglines/3.0-rho (http://www.bloglines.com; 3 subscribers) 256 1449434
Unique items: 1, Total Hits: 256, Total Bytes: 1449434
Bloglines feed /tags/18/feed
Bloglines/3.0-rho (http://www.bloglines.com; 1 subscriber) 256 12427520
Unique items: 1, Total Hits: 256, Total Bytes: 12427520
Bloglines feed /blog/feed/1
Bloglines/3.0-rho (http://www.bloglines.com; 5 subscribers) 252 0
Unique items: 1, Total Hits: 252, Total Bytes: 0
Bloglines feed /taxonomy/term/5/0/feed
Bloglines/3.0-rho (http://www.bloglines.com; 1 subscriber) 242 3832796
Unique items: 1, Total Hits: 242, Total Bytes: 3832796
Bloglines feed /node/feed
Bloglines/3.0-rho (http://www.bloglines.com; 3 subscribers) 256 13959338
Unique items: 1, Total Hits: 256, Total Bytes: 13959338
Bloglines feed /rss.xml
Bloglines/3.0-rho (http://www.bloglines.com; 3 subscribers) 256 0
Bloglines/3.0-rho (http://www.bloglines.com; 7 subscribers) 256 0
Unique items: 2, Total Hits: 512, Total Bytes: 0
Bloglines feed /tags/3/feed
Bloglines/3.0-rho (http://www.bloglines.com; 1 subscriber) 250 12999000
Unique items: 1, Total Hits: 250, Total Bytes: 12999000

Hits from MSIECrawler by host
81.159.46.223 1784 49980154
Unique items: 1, Total Hits: 1784, Total Bytes: 49980154

Hits from Inktomi/yahoo slurp
Mozilla/5.0 (compatible; Yahoo! Slurp China;) 58 800642
Mozilla/5.0 (compatible; Yahoo! Slurp;) 2066 44986284
Unique items: 2, Total Hits: 2124, Total Bytes: 45786926

Related Posts: apache awstats bloglines log python

Gregarius RSS Aggregator

I like using a web base aggregator so that I can access it from whereever I may be and it will remember my subscrptions and what articles I have already read. For some time I've been toying with the idea of installing an RSS aggregator on my dedicated server. This would be something to replace bloglines but which I could tweak to my liking. I haven't been in a big hurry to do this as bloglines has been ok thus far. However, the other day I was using it and I realised how slow it can be, waiting for it to download new articles. I thought this was an excellent reason to seek alternatives.

I had a look around for web based aggragators and found the wikipedia list of news aggregators and from this I found Gregarius. Installation was very straightforward:

Create mysql database, user and password for it
untar Gregarius into domain directory
edit config file to tell it details of the database
access it through web server.

It's pretty standard LAMP stuff.

It's nice to use, very configurable, much more so than bloglines. You can control when the feeds get aggregated (for now I plan to press the refresh button to manually trigger aggregation), how many articles per page, you can read as a 'river of news' (articles from different sources mixed togther by creation date) or read them grouped by feed. Other aggregators only support the 'river of news' format which I don't really like as I don't like to mentally context switch between unrelated topics between each posting: for example, first post is a dilbert cartoon, followed by bbc news about untimely deaths, followed by an engadget piece about a new mp3 player, it's all over the place. I prefer to go through my feeds in the order that is interesting to me at the time (cherry pick, Dilbert first normally).

Gregarius supports a few themes and plugins and so far these have fixed the minor quibbles I had. For example, I wanted to read postings in order of creation rather than newest first which means reading down the page. However, the 'mark all as read and get some more' button was at the top of the page some once I got to the bottom I had to scroll to the top to get more articles. Someone had already had this problem and there was a plugin available to put an extra button at the bottom of the page.

One little hack I had to do was with the 'lilina' theme which supports a button to collapse the left sidebar which will be useful on my Dell d410 with a 12" screen. However, this theme also collapses the articles down to titles by default which I did not want. I fixed this be editing the themes/lilina/item.php file as follows:

    <!--pcw <div class="content" id="c<?php echo rss_item_id(); ?>" style="display:none">-->
    <div class="content" id="c<?php echo rss_item_id(); ?>" style="display:block"

i.e. by default the article contents is display:block rather than display:none.

What I most miss from bloglines is being able to press 'j' to go to the next article.

What I dislike about Gregarius most is that it isn't spelt Gregarious.

ToDo: figure out how to get cron to prune old articles from the database. It can be done explicitly through the admin pages but I don't want more chores.

Google RSS Reader

I had a play with googles online rss reader. It is very fancy but I don't think it would tempt me away from bloglines:

You only see one article at a time. Bloglines shows you all the articles from a feed in one page which I think is better for spotting the interesting stuff. You can step through the articles with the j button in both bloglines and the google reader (coincidence? Group homage to vi?)
I don't want to go through a mix of articles from all feeds in chronological order. I like to go through feed by feed. If I open the list of feeds in the google reader it occupies half the screen on my d410.
It is a bit buggy: at one point the 'loading' splash thing got stuck on. But like most google software it is beta.
It carefully strips the colour coding and indentation out of syntax highlighted code. Bloglines strips the indentation. I am not sure why they do this, is it the presentation police?

What I am looking for is the mysthical rss aggregator with bayesian filtering that will learn what interests you and only show that. I'd write one if there were only nine days in every week.

Conclusion: google see rss as the future and want to slap adverts all over my blatherings.

bloglines

bloglines is a web-based news aggregator.

Bloglines article duplication

I use bloglines for my rss aggregation. On it I subscribe to my own rss feeds to reassure myself that they are reaching the outside world. Since I installed the awtags module my postings have all contained links to their tag entries. This doesn't really bother me, it may encourage people to visit my site if they see useful links in the rss feed.

Bloglines tries to deliver articles that have changed and it appears to do this by comparing the contents of the rss file with its previous contents. If there are any small changes bloglines displays the article in the same way as it does for new articles.

I don't like to hastle people when I edit articles to correct spelling mistakes or whatever so I have altered the drupal ping module to only ping (i.e. tell the outside world) if a new posting is created, not if it is modified. However, bloglines appears to poll my feed and so the slightest change will result in articles appearing as if the are new. Today I assigned some tags to some older articles using awtags and this was suficient for the them to be displayed by blogines as new articles as they were still in the rss feed.

This is the long way of apologising to anyone who thinks I a winding them up by republishing old articles with no noticable changes.

I see I have a new subscribed on bloglines. Welcome, hope you find my whitterings interesting. I am getting more visitors since I started using awtags, especially from technorati.

Related Posts: awtags bloglines drupal rss tagging

Ask Jeeves buys Bloglines

Apparently Ask Jeeves has bought Bloglines.

On friday Ask Jeeves spider was hammering my site for most of the day, thoroughly indexing it. It has been there before but this was quite intense. Could it be that Ask got a list of blogs from bloglines and were busy indexing them all? Or was it a glitch in their bot?

RSS Aggregation

I was thinking today how cool and hip I was using Bloglines as an RSS aggregator compared to people who just go through a list of favourite web sites every day. So 1995. Go RSS, join the 21st century.

Then I thought about it some more and this is essentially what I am doing with bloglines: I have a list of feeds that I go through one by one, looking through the articles. No real difference to going through a sidebar full of bookmarks, looking at websites.

If I am going to evangelise RSS aggregation, what am I going to say are the advantages? Why bother?

Uniform presentation
No adverts (for now: I cannot imagine this will forever be true)
Um

I'm going to continue with bloglines as the reasons above are enough for me, I just won't try to evangelise. As it happens I am not a good evangelist: I have yet to convert anyone to FireFox for example which should be pretty easy. Even after I removed the adware from my boss's PC he showed no interest in FireFox.

Maybe I should examine my RSSing habits and figure out if I am missing the point in some way.

The only people I know who are at all interested in gmail are those of a geeky persuasion.

Related Posts: bloglines email firefox gmail rss

Bloglines new aggregator

Decided upon Bloglines for my news aggregating needs. It has lots of features and is fast enough. It even allows me to add a blog roll box here.

It is comprehensive, almost overwhealming in the options that are available. My tip: subscribe to a blog just to get a username. That way you don't have to use your email address as an identifer that can be scraped by spambots.

Previously I was using the Python Desktop Server news aggregator that works very nicely but has two disadvantages:

it needs a server to run on and now I am shutting down my home server I won't have one.
I can access Bloglines from work without setting up an ssh tunnel.

Related Posts: blog bloglines pyds python ssh