Useful .profile scripts!

Well, I've only written one so far, but it's ultra helpful and saves you remembering all the options you need to pass to standard commands to do what you want. If anyone else feels like commenting with their time-saving commands, maybe I'll build up a library of them.

Here goes :-

files_with - Find files containing a string inside the current directory, recursively

files_with() { grep -H -n "$*" ./* -R | cut -d: -f1 -f2; }

$ files_with hello
./blah.txt:4
./sub/blah.txt:6

Rails 3 partial views

I got caught out today trying to use a Rails 3 partial view.. I was passing a local in to it but it just wasn't getting picked up.

=render "shared/cat_box", :locals => {:cat => @cat}

ActionView::Template::Error (undefined local variable or method `cat' for #<#<Class:0x1062afad8>:0x1062aa240>)

Scratching my beard for a while until I realised that maybe render only looks for locals in the first parameter hash.. e.g. using the full way of rendering a partial:

=render :partial => "shared/cat_box", :locals => {:cat => @cat}

All working fine now!

Hack days rock

I'd just like to say..

We had a hack day recently and the coolest thing has come out of it. It won't make us any money so it would never have been developed if not for a hack day but it's one of the most useful tools for finding people in our ever expanding office.

So, all the devs have macbooks (thanks Forward!). We roam around the office, moving rooms, desks, sitting in comfy chairs or on beanbags in multiple rooms around the office - It's great, but people have a hard time finding you.

Some of the devs came up with an awesome idea - Tail the logs on our router, find all the logins and logouts for each access point throughout the office, take down all the locations of the access points. Now we have a nice page that tells you what area of the office each person was last seen in! You can also get email notification when someone who hasn't been seen in a while logs in to the network.

Anyway.. Thought I'd share that, because it's awesome.

A short story about OpenStruct and a performance issue

Once upon a time, in a office far far away, I sat all day, wondering why my report query was taking 0.54 seconds to get the data, then 49 minutes (yes, 49 minutes) to process and render it. Ok, drop the story telling now.

It was only 13000 records! I ran ruby-prof (for the first time ever) and spend a while looking at the results wondering what they meant :)

One thing I noticed was that there were quite a few bold lines in the result that were in OpenStruct (thanks for making this easy for me ruby-prof).

I concluded that calling OpenStruct in a loop was defining a brand new class and defining all the attr_accessors for each of the keys in the hash... in my reporting case, each row is converting to a hash, and each row has exactl the same keys.

I decided to run a small benchmark with OpenStruct against a class I built that lets you create a Class from a hash, which just adds all the attr_accessors then returns the type, which you can new up with the hash many times over.

class DynamicRow
  def self.with_keys(keys)
    result = Class.new
    result.class_eval do
      keys.each { |k| attr_accessor k }
      
      def initialize(hash)
        hash.each { |k, v| instance_variable_set :"@#{k}", v }
      end
    end
    result
  end
  
  def self.benchmark
    require 'benchmark'
    
    sample_hash = {:a => "hello", :b => 3}
    
    Benchmark.bm do|b|
      b.report("OpenStruct ") do
        50_000.times { OpenStruct.new(sample_hash) }
      end

      b.report("DynamicRow ") do
        row_type = DynamicRow.with_keys(sample_hash.keys)
        50_000.times { row_type.new(sample_hash) }
      end
    end
  end
end

Running the benchmark gave me some convincing results

            user     system      total        real
OpenStruct  13.400000   0.080000  13.480000 ( 13.572973)
DynamicRow   0.570000   0.010000   0.580000 (  0.584201)

Well, thats a nice improvement. But the difference there isn't going to bring the processing down that much is it? The other factor I thought might be playing a role was the number of keys, so I increased this from 2 as shown above in sample_hash to 4.

            user     system      total        real
OpenStruct  25.750000   0.160000  25.910000 ( 26.117587)
DynamicRow   0.790000   0.010000   0.800000 (  0.809647)

So, with OpenStruct, the time taken increases equal to the number of keys.. with DynamicRow it's not so bad.

My report now runs in about 1 minute, instead of most part of an hour.

Hope the experience I've gained today helps someone else :)

Formatting dates and times in Javascript

So I needed to format a Date object in javascript today and couldn't find a decent clean solution. What I really wanted was Ruby's strftime.. So I wrote it.

format dates in javascript with Date.format()

Single line rescues

I've just found that you can use rescue like you can with if and unless

x = (do_something_that_might_raise rescue nil)

x will be nil if do_something_that_might_raise raises an exception... Obviously catching and swallowing all exceptions from isn't great but it does have it's uses :)

Writing an math expression evaluator: Lexical Analysis

Introduction

I've always been interested by compilers and specifically the effort that goes in to processing a string of code in to something a computer can understand. The most complete but simple example is a math expression parser.

At first thought, it doesn't sound all that complicated, but giving it a bit more thought, you can't just split an expression like "1 + 2 * (6 / 2)" on spaces and then work out what each part is, what if they enter "5+5"?

Then you think of actually evaluating the expression... operator precedence, processing the deepest expression parts first, etc.. also, how do you even know it's valid? you may be able to enter "+ + 2 4 ( hello 1" but you would have trouble evaluating that!

While writing these posts, I've learnt the basic methods for translating and evaluating mathematical expressions and implemented it as a ruby application. Please let me know if there are any improvements I can make, as I'm always looking to improve things :)

Lexical Analysis

To make this easier and cleaner, evaluation is split in to 2 phases, Lexical analysis is the first phase, taking the string and converting it in to a stream of symbols which are fed in to the second phase, the parser. The lexer should also raise errors indicating invalid characters and anything it can't find a symbol for.

As an example "1 + 1 * 3.14159", it would create a stream of symbols like: IntegerLiteral(1) AdditionOperator IntegerLiteral(1) MultiplicationOperator FloatLiteral(3.14159)

The method I chose was to hold a list of "matchers" that basically match a symbol at the current position in the input string.

So it'd start out with "1 + 1 * 3.14159" and the first request for a token would return IntegerLiteral(1) using a simple regular expression like: /^[0-9]*/. return IntegerLiteral and advance the string the number of characters that were matched.

now we are at " + 1 * 3.14159" a regular expression could match /\s+/ and just skip it because we aren't interested in that. it'll still advance the string the number of whitespace characters it matched.

We get to the plus, a simple literal check will match this so if code[position] is "+" then add one to the position and and return a AdditionOperator.

And so on

I have implemented this in a generic a way as possible to allow for other symbols and matchers to be lexed without changing the code code.

class Lexer
 def initialize(str, matchers)
  @str = str
  @position = 0
  @matchers = matchers
 end

 def consume
  return Symbol.new(nil, 0, :end) if @position >= @str.length
  part = @str[@position..-1]
  @matchers.each do |m|
   result = m.match(part)
   if result
    @position += result.length
    return self.send :next if m.ignore?
    return result
   end
  end
  raise "Unable to lex: #{part}"
 end
end

To create a Lexer class that would give us the correct symbols for a math expression, you just create a matcher for each type of symbol. This code isn't very pretty, but some nice DSL will be written later!

matchers = []
matchers << RegexMatcher.new(/(\s)+/, :whitespace, :ignore => true)
matchers << LiteralMatcher.new('+', :add)
matchers << LiteralMatcher.new('-', :subtract)
matchers << LiteralMatcher.new('*', :multiply)
matchers << LiteralMatcher.new('/', :divide)
matchers << LiteralMatcher.new('(', :left_paren)
matchers << LiteralMatcher.new(')', :right_paren)
matchers << RegexMatcher.new(/(\s)+/, :whitespace, :ignore => true, :)

lexer = Lexer.new("1 + 1 * 3.14159", matchers)

Well - I forgot about this post!

Since I started writing this I've now written a gem for lexical analysis with the pretty DSL I said could be done later.. now it looks like this:

ExpressionLexer = Lexr.that {
 ignores /\s+/ => :whitespace
 matches /[-+]?[0-9]*\.?[0-9]+/ => :number, :convert_with => lambda { |v| Float(v) }
 matches "+" => :addition
 matches "-" => :subtraction
 matches "*" => :multiplication
 matches "/" => :division
 matches "(" => :left_parenthesis
 matches ")" => :right_parenthesis
}

then can be used like this:

lexer = ExpressionLexer.new("1 * 12.5 / (55 + 2 - 56)")
until lexer.end?
 puts lexer.next
end

giving you an output of:

number(1.0)
multiplication(*)
number(12.5)
division(/)
left_parenthesis(()
number(55.0)
addition(+)
number(2.0)
subtraction(-)
number(56.0)
right_parenthesis())
end()

Ok, so this has kind of turned in to a shameless plug for my own gems.. Here is the details :)

Lexr - A simple but powerful lexical analyser for Ruby

There is also another gem I wrote that uses this called math_engine. It's a recursive decent parser which parses math expressions and gives you the result. It was a great experience writing these and I learnt a lot. Hopefully you can learn from them too.