Ruby Best Practices

Complexity

shortcutter@googlemail.com (Robert Klemme) — Sun, 06 Jun 2010 20:37:00 +0000

If I tell you a program has 1,000,000 lines of code you’ll immediately understand that this must be a complex piece of software. Why is that? You implicitly know that nobody copies the same 5 lines 200,000 times to create a long program which is full of redundancy. This is another term we will have to consider while trying to improve our understanding of complexity. As a first approximation we can say that a complex piece of software is free of redundancy. This is of course an oversimplification and for various reasons that we will have to talk about in a minute we will not usually reach that mark in practice.

This oversimplification however helps us to understand one important – if not the most important – task of software engineering: reduction of redundancy. Assume that someone in an application you are writing you need to output all elements contained in an Array in a special way (say, as an HTML unordered list). You would probably write something like this:

  puts "<ul>"
  
  arr.each do |x|
    puts "  <li>#{x}</li>"
  end
  
  puts "</ul>"

I know this is not the best of examples since you normally would be doing this with ERB or your favorite web framework. Picking the right tools for the job is of course another important task in software engineering but we want to focus on something different which can be nicely demonstrated with this simple toy example. (Also, you can easily follow by copying and pasting the code to IRB and play with it.)

What do you do once you discover that you also have to apply the same formatting to a Set in a different location of your program? Right, you make it a function and write something like:

def format_list(enum)
  puts "<ul>"
  
  enum.each do |x|
    puts "  <li>#{x}</li>"
  end
  
  puts "</ul>"
end

In other words, instead of increasing redundancy of your program you refactor a part of it so you can use this functionality in different locations. Now you discover that you also need to be able to write to a file. Instead of writing a second function that has a second argument for the file you want to write to you rather add an argument to this function potentially making it look like this:

def format_list(enum, out = $stdout)
  out.puts "<ul>"
  
  enum.each do |x|
    out.puts "  <li>#{x}</li>"
  end
  
  out.puts "</ul>"
end

Again you avoided increasing redundancy of the application by adding complexity to this function which now is capable of applying the formatting in an even broader range of situations. Let’s push this one step further: now we also want to be able to do custom conversions of elements in the Array instead of just always using #to_s.

TO_S = lambda {|o| o.to_s}

def format_list(enum, out = $stdout, &convert)
  convert ||= TO_S
  
  out.puts "<ul>"
  
  enum.each do |x|
    out.puts "  <li>#{convert[x]}</li>"
  end
  
  out.puts "</ul>"
end

Now we can call it for a list of Floats like this:

  floats = [1.2345, 4.567, 64.3]
  
  format_list floats do |f|
    "%010.3fcm" % f
  end

Again, complexity of this function increased but we gained versatility. In this case we are particularly lucky because all calls of the initial version of the function still work and produce the exact same result. We might have to take a bit of runtime overhead because the original string interpolation is likely faster for the general case where only #to_s is needed. (Btw, if this program runs on some form of virtual runtime environment (such as the JVM) the situation might be different because the JVM’s runtime optimization might produce better results if there is just a single function which is called more often than multiple methods that each are called infrequently.)

I’d say what we have seen above is a fairly typical evolution of a piece of software. Instead of increasing redundancy of the program we increased complexity of this function. It may seem that having different functions for various variants might be better, for example, these functions are certainly easier documented. So why did we do this and strive to stick with a single function?

Humans write software and while a piece of software might be bug free humans are not. More importantly the world keeps changing all the time and so do requirements for software. Either we did not read the spec properly or someone changed his mind and now all of a sudden we need to ouput ordered lists. That’s an easy change if we just have a single implementation:

def format_list(enum, out = $stdout, &convert)
  convert ||= TO_S
  
  out.puts "<ol>"
  
  enum.each do |x|
    out.puts "  <li>#{convert[x]}</li>"
  end
  
  out.puts "</ol>"
end

The change is so minimal one barely sees it. Yet, if we have to apply that change to multiple versions of this function potential for new bugs is much higher. We might forget one of them (remember that all these do not necessarily be located in the same file) or we might misspell in some of them etc. Also the effort is higher: for this particular change it might not be dramatic but just think about having to check in multiple files into your favourite source control system, having to update documentation for multiple functions, having to adjust unit tests or other test suites etc.

All in all I think it is fair to say that we gained more than we lost. We gained developer productivity and paid only a small runtime penalty and complexity of this function. Up to this point we can say that increased complexity has brought about a better piece of software. However you might have a premonition of degradation which will start if we add even more features to this function. Here we have the first reason why we do not have redundancy free applications in software: humans can only handle a certain level of complexity efficiently and so we must avoid too complex systems if we want to be able to maintain the code.

Another reason why we may end up with different versions of format_list is the sheer size of an application. Large applications need multiple authors and it is not too unrealistic that people independently invent similar functions in different parts. While this would increase redundancy it might actually be desirable to do it: if there is just the one implementation of the function all components that need it must depend on the component that contains it. Depending on the application and programming language used the price of an additional component dependency may actually be higher than the benefit.

Summary

Today we looked at software complexity caused by adding features to a piece of software to avoid redundancy. In our daily work we continuously have to make decisions that affect this dimension of software complexity. We have also seen how we might want to retain some level of redundancy to keep software maintainable. Next we will look at other dimensions of software complexity. Hopefully we will eventually come up with a classification of factors that lead to software complexity – and how to tame them.

Code Massage

shortcutter@googlemail.com (Robert Klemme) — Sun, 28 Feb 2010 13:32:00 +0000

This article started out as a mental experiment and led to a surprising result. I post this mostly for the fun of it. But of course you can take something away from it. With that I do mean not only technical solutions. I believe firmly that a certain level of playfulness actually helps finding better solutions. The other ingredient you need is a certain eagerness for improvement which means to not be be content too early. OK, let’s start.

The scenario I started out was this: suppose you want to anonymize email addresses because you want to publish an email but not expose addresses to spam harvesting. Yet, you want to make sure that every address is always represented with the same replacement address in order to not change the meaning. You might immediately answer “We’ll need a Hash so we can efficiently find addresses that have been replaced already” – and so did I.

Java Style

If you are familiar with the Java standar library you know that java.util.Map has methods to check whether a key is present, to set values and to retrieve values. So after switching to Ruby you might be tempted to do it like this:

subst = {}

puts email.gsub(ADDR) {|match|
  if subst.has_key? match
    subst[match]
  else
    subst[match] = "<<MAIL #{subst.size}>>"
  end
}

Whenever we encounter an email address we must first check whether we have generated a replacement string for this address already. If not, we create a new one. No rocket science.

A little more sophisticated

You might find documumentation of method Hash#fetch when reading the library documentation which comes in handy because the block is invoked if the key is not present in the Hash. The code now looks a little shorter already:

subst = {}

puts email.gsub(ADDR) {|match|
  subst.fetch(match) {|k| subst[k] = "<<MAIL #{subst.size}>>" }
}

O||=erator

A similar thing can be achieved with the ubiquituous operator ||= which allows for conditional execution. In case you are not yet familiar with it you’ll find plenty of discussions in ruby-talk that revolve around this. The short summary is that a ||= b is equivalent to a || a = b and not a = a || b as you might be tempted to believe.

subst = {}

puts email.gsub(ADDR) {|match|
  subst[match] ||= "<<MAIL #{subst.size}>>"
}

The code has become even more shorter. But we are not finished yet!

Outsourcing

If you need that replacement in multiple places of your code you’ll likely put it into a method. However, if you want to replace different things (i.e. you need different regular expressions) which can match the same string you might want to outsource the generation of the replacement string so you can use it with different calls of gsub. You can of course do it with an additional method but there is a more elegant way to do it:

subst = Hash.new {|h,k| h[k] = "<<MAIL #{h.size}>>"}

puts email.gsub(ADDR) {|match| subst[match]}

We simply use Hash's default proc functionality for this. This is basically the same the fetch block does but now the code is attached to the Hash instance and not to the #fetch call.

You might wonder, how much further can we get? And indeed, this solution is probably the most idiomatic one and the one you see most frequent in seasoned Ruby developers’ code. It turns out though, that we can drive this further if we are prepared to use some newer Ruby features.

Getting tricky

Since Ruby 1.8.7 you can use anything as a block parameter to a method provided it implements a method to_proc which returns a Proc. Namely class Symbol implements this method in the following way: it returns a proc which needs at least one argument when called and invokes the given method with the remaining arguments on that instance. This allows for convenient operations like mapping data:

irb(main):001:0> (1..3).map &:to_s
=> ["1", "2", "3"]

One thing that bugged me was that the block handed to gsub above does nothing more than basically only forward the Hash lookup. With the new feature it should be possible to make the code a bit more concise. Luckily there are some core classes that implement to_proc already:

$ ruby -e 'ObjectSpace.each_object(Module) {|m| p m if m.instance_methods.include? "to_proc"}'
Method
Proc
Symbol

$ ruby19 -e 'ObjectSpace.each_object(Module) {|m| p m if m.instance_methods.include? :to_proc}'
Method
Proc
Symbol

We can exploit this fact and now we can write the code like this:

subst = Hash.new {|h,k| h[k] = "<<MAIL #{h.size}>>"}

puts email.gsub(ADDR, &subst.method(:[]))

Note: code changed after arthurschreiber’s comment.

Now, that looks ugly, doesn’t it? We should be able to do something about that, because after all we love Ruby for its elegance and clear syntax. Yes, we can!

Even shorter with a general solution

Since we can use any object why not provide a general mechanism for this case? Not only Hash but also Array and a lot more classes provide method [] as a general hook for lookup or exeution:

$ ruby19 -e 'ObjectSpace.each_object(Module) {|m| p m if m.instance_methods.include? :[]}'
Thread
Method
Proc
Struct::Tms
MatchData
Struct
Hash
Array
Bignum
Fixnum
Symbol
String

Now, let’s allow all these to be simply used as block parameters!

class Object
  def to_proc(m = :[])
    method(m).to_proc
  end
end

subst = Hash.new {|h,k| h[k] = "<<MAIL #{h.size}>>"}

puts email.gsub(ADDR, &subst)

Now we can just pass any Hash instance to gsub. The working logic for calculating our replacement string is now completely restricted to the Hash creation. This is a really elegant solution!

Golf

We can reduce the number of characters to type a bit more by throwing out the variable declaration and effectively turn this into a one liner:

puts email.gsub(ADDR, &Hash.new {|h,k| h[k] = "<<MAIL #{h.size}>>"})

I don’t think this is an improvement over the last variant but sometimes it helps driving things as far as possible to find out where in the process we reached the optimum.

The fun begins

Some classes do also implement method [] – we should be able to make good use of that as well. We might be tempted to create a lot of Struct instances via this method. It can be done but we have to do some tweaking because Struct.[] does not splat a single Array argument so we have to redefine it a bit:

Name = Struct.new :forename, :surname

# unfortunately this does not work with the default Struct.[]
def Name.[](a)
  new(*a)
end

p [
  ["John", "Doe"],
  ["John", "Cleese"],
  ["Mickey", "Mouse"],
].map(&Name)

# maybe a bit better:
def Name.create(a)
  new(*a)
end

p [
  ["John", "Doe"],
  ["John", "Cleese"],
  ["Mickey", "Mouse"],
].map(&Name.to_proc(:create))

I hope you had some fun reading this and more importantly playing around yourself. Trying out all things will certainly help you discover new ways and improve your skills.

As usually I have placed the code at github . If you look at it, please don’t get yourself hung up on the regular expression for matching email addresses. This is a whole topic of its own and I just hacked something together to make the code work.

The Complete Numeric Class

shortcutter@googlemail.com (Robert Klemme) — Sat, 20 Feb 2010 14:25:00 +0000

As announced in the previous article we will look at a complete number class today. I will use the example of a integer number which, when printed, will show up as hex number (as opposed to the decimal presentation of Fixnum and relatives). As before the main point is not sophisticated logic or usefulness of the class. Instead I will keep the logic simple so we can focus on the aspects I try to convey with today’s article:

conversions into HexNum
conversions of HexNum to other types
equivalence vs. comparability
type coercion
math and operator overloading

For completeness reasons other aspects mentioned in the previous article will be implemented as well but I won’t discuss them in detail here.

Conversions into `HexNum`

The first step into the world of HexNum is creation of a new object of course. This part is not that much interesting and so I will only gloss over the implementation. I have provided a few ways:

constructor
conversion methods (to_hex),
explicit method (similar to Integer()).

These are shown in the code snippet below:

class HexNum < Numeric

  # Create a new instance from an int or String.
  def initialize(val)
    case val
    when String
      @i = parse_string(val)
      @s = val.frozen? ? val : val.dup.freeze
    when Numeric
      @i = val.to_i
    else
      raise ArgumentError, 'Cannot convert %p' % val
    end
  end

end

# more conversions

def HexNum(i)
  HexNum.new(Integer(i))
end

class Object
  def to_hex
    HexNum.new(to_i)
  end
end

As you can see, a HexNum consists of an integer value and optionally a String. The integer value is the mandatory part which will be used in most methods while the String is just an helper intended to make conversions to String more efficient (I did not make any measurements though – I mainly wanted to make the class a tad more interesting).

Method Object.to_hex is based on the presence of method to_i. This is debatable. Basing this conversion on method to_int is as reasonable IMHO. As you can see, the set of classes which implement to_i differs from those which implement to_int:

irb(main):002:0> s1 = []; s2 = []
=> []
irb(main):003:0> ObjectSpace.each_object(Module) do |m|
irb(main):004:1* s1 << m if m.instance_methods.include? :to_i
irb(main):005:1> s2 << m if m.instance_methods.include? :to_int
irb(main):006:1> end
=> 407
irb(main):007:0> s1
=> [Complex, Rational, Process::Status, Time, File, ARGF.class, IO, Bignum, Float, Fixnum, Integer, String, NilClass]
irb(main):008:0> s2
=> [Complex, Rational, Bignum, Float, Fixnum, Integer, Numeric]

The rationale behind this is that only types which can be used as integers should implement to_int. Method to_i is merely a conversion method which turns “something” into an int. I chose to base to_hex on to_i because this increases the number of cases where you can immediately use a HexNum. If you need more strict argument checks, you can use HexNum() (the method) which is modeled similar to Integer() and in fact uses it internally in order to benefit from its argument checking.

You might have expected HexNum to inherit Integer. Actually, that’s what I would have done, too. But, it turns out, if you make a class inherit Integer you cannot create instances of it any more:

irb(main):007:0> class X < Integer
irb(main):008:1> end
=> nil
irb(main):009:0> X.new
NoMethodError: undefined method `new' for X:Class
        from (irb):9
        from /usr/local/bin/irb19:12:in `<main>'
irb(main):010:0> class X;end
=> nil
irb(main):011:0> def X.new; allocate; end
=> nil
irb(main):012:0> X.new
TypeError: allocator undefined for X
        from (irb):11:in `allocate'
        from (irb):11:in `new'
        from (irb):12
        from /usr/local/bin/irb19:12:in `<main>'
irb(main):013:0> Integer.class
=> Class
irb(main):014:0>

This is a bit unfortunate since Integer would be the proper base class for our HexNum. Inheriting Numeric is the second best we can do.

Conversions to other types

These conversions are done by the typical set of to_xyz methods:

  # conversions

  def to_s
    @s ||= (@i < 0 ? '-0x%x' % -@i : '0x%x' % @i).freeze
  end

  def to_i
    @i
  end

  alias to_int to_i

  def to_hex
    self
  end

Please note that for efficiency reasons class HexNum also contains a method to_hex. Other than that there are really no surprises here. Conversion to string may look a bit complicated but it’s really just the different treatment of negative and positive values plus caching of the converted String. The #freeze just ensures that changes of the cached value outside the class cannot backfire. If we weren’t using #freeze here, we would have to create a new String instance for every to_s invocation which would defy the whole point of caching the value.

Mutability

An important point to note is that the class is immutable. While this is not mandatory creating a mutable class does not blend well with how Ruby handles arithmetic operators. If you implement operator + as in place modification you will get all the issues typically caused by aliasing, i.e. using the same instance from different places in code. As they say, “When in Rome do as the Romans do” – so we will stick with the convention used throughout Ruby’s core library and make our HexNum immutable too. That way we can use instances of HexNum where we would otherwise have used Fixnums or Bignums. And after all this was the aim of the exercise: to demonstrate how to create a class that seemlessly blends with all the other numeric types of Ruby’s core and standard library.

Equivalence vs. Comparability

Now we slowly get to the more interesting topics. Every class has comparison for equivalence through method #eql? and operator == (see also the discussion of the topic in the previous article). For seamless blending of HexNum with other numeric types comparison with == should only look at the numerical value so that HexNum(1) and 1.0 compare true. However since we do not have influence on Fixnum's and Float's implementation of === and since equivalence is a symmetric relation (i.e. a == b must return the same as b == a) we can only establish equivalence with other HexNum instances. I think this is unfortunate. I can only speculate about Matz’s reasons to not use #coerce in this situation: I assume he did it this way for performance reasons since these types of comparisons are very frequent in a program.

So, this is how equivalence checking code looks like:

  # equivalence

  def eql?(num)
    self.class.equal?(num.class) && @i == num.to_i
  end

  alias == eql?

No big surprises here. You’ll note the type check which is necessary because of the aforementioned implementation details of core classes.

Things stand dramatically different with respect to comparison operator <=>. The definition of this operator’s semantics was nicely presented by Marc in his comments to the last article. Basically, <=> must return nil if classes of compared instances are differnt. But what is this?

irb(main):001:0> 1 <=> 1.0
=> 0
irb(main):002:0> 1 <=> 1.5
=> -1
irb(main):003:0> 0.5 <=> 1
=> -1
irb(main):004:0> 0.5 <=> 1 << 40
=> -1
irb(main):005:0>

Numeric classes allow for comparison across a wide range ot types. Can we make HexNum blend in here? It turns out, that we can. This is the time to introduce a functionality that took me a while understand initially. But this is at the heart of Ruby’s operator overloading and probably the single most important point to consider when implementing numeric types.

What’s this `#coerce` thingy?

When talking about equivalence it might have occurred to you that the behavior of == should depend not on a single class but actually on the type of both arguments. The rule whether two objects are equivalent include’s both object’s class and statements about state of both instances (in case of HexNum for example, both must have the same integer value). Now, Ruby – like many object oriented languages – has ‘single dispatch’ which means only the type of the receiver (the object to the left of the dot in a method call or self) determines which method is actually used. This works remarkably well most of the time but there are cases where you rather want to make the dispatch (and thus the decision which code is executed) depend on the receiver and one or more of the method’s arguments. Binary operators are a typical case of this.

The usual solution in other languages is to overload a method based on argument types and in Ruby you would likely employ some form of type checking similar to what I have done in HexNum's method #initialize. However, this approach has a fundamental drawback: when writing class A the author must know all classes B, C and D that are used as argument types. This hurts extensibility seriously since for every new numeric type the code of older types needs adjustment. While this would be possible due to Ruby’s dynamic nature this is on the one hand tedious and it would have performance implications on the other hand because you would likely be implementing those “enahancements” in Ruby (and not C) and methods would have to check for more and more types.

Ruby’s solution to this is very elegant (as many other aspects of the language): whenever a methods receives a type different from its own type it asks that type to do a conversion so the operation can finally be handled. It does that via method #coerce which receives the caller as argument so the other (presumably newer) type can find an adequate conversion. That way, only types written later need to know about older types (e.g. everything in the core and standard library). Let’s look at how #coerce works by using it on some standard types:

irb(main):001:0> 1.coerce 2
=> [2, 1]
irb(main):002:0> 2.0.coerce 1
=> [1.0, 2.0]
irb(main):003:0> 1.coerce 2.0
=> [2.0, 1.0]

The result of invoking #coerce has two interesting properties:

The types of returned values are identical.
The return value is really an Array with the order reversed compared to the invocation (the receiver is the second object in the array).

While we immediately see the usefulness of 1 (every class should know how to handle instances of itself) the second property irritated me at first. But it does make sense if you consider that the argument to #coerce is really the original receiver of the method call.

I don’t want to hide some inconsistencies of the standard functionality from you:

irb(main):004:0> 1.coerce 1<<40
=> [1099511627776.0, 1.0]
irb(main):005:0> (1<<40).coerce 2
=> [2, 1099511627776]
irb(main):006:0> (1<<40).coerce 3.0
TypeError: can't coerce Float to Bignum
        from (irb):6:in `coerce'
        from (irb):6
        from /usr/local/bin/irb19:12:in `<main>'
irb(main):007:0> 3.0.coerce 1<<40
=> [1099511627776.0, 3.0]
irb(main):008:0> 1<<40
=> 1099511627776
irb(main):009:0> (1<<40) + 3.0
=> 1099511627779.0

You can add Bignum and Float but #coerce fails on one direction. If anybody can provide a sensible explanation of this please let us know. For the moment I am inclined to believe that it’s due to the fact that implementations of core types are done in C and there are probably some corners cut.

Now look at a simple example which demonstrates the mechanics of #coerce:

$ ruby19 <<CODE
> a = 1
> b = Object.new
> def b.coerce(x) [x,2] end
> p b
> set_trace_func lambda {|*a| p a}
> puts a + b
> CODE
#<Object:0x10028284>
["c-return", "-", 5, :set_trace_func, #<Binding:0x10027dd0>, Kernel]
["line", "-", 6, nil, #<Binding:0x10027adc>, nil]
["c-call", "-", 6, :+, #<Binding:0x10027724>, Fixnum]
["call", "-", 3, :coerce, #<Binding:0x1002721c>, #<Object:0x10028284>]
["line", "-", 3, :coerce, #<Binding:0x10026e48>, #<Object:0x10028284>]
["return", "-", 3, :coerce, #<Binding:0x10026924>, #<Object:0x10028284>]
["c-call", "-", 6, :+, #<Binding:0x100263e4>, Fixnum]
["c-return", "-", 6, :+, #<Binding:0x10025fbc>, Fixnum]
["c-return", "-", 6, :+, #<Binding:0x10025a28>, Fixnum]
["c-call", "-", 6, :puts, #<Binding:0x10025424>, Kernel]
["c-call", "-", 6, :puts, #<Binding:0x10024f54>, IO]
["c-call", "-", 6, :to_s, #<Binding:0x10024854>, Fixnum]
["c-return", "-", 6, :to_s, #<Binding:0x10023db8>, Fixnum]
["c-call", "-", 6, :write, #<Binding:0x10023878>, IO]
3["c-return", "-", 6, :write, #<Binding:0x10022f9c>, IO]
["c-call", "-", 6, :write, #<Binding:0x10022ae8>, IO]

["c-return", "-", 6, :write, #<Binding:0x10022768>, IO]
["c-return", "-", 6, :puts, #<Binding:0x1002227c>, IO]
["c-return", "-", 6, :puts, #<Binding:0x10021b0c>, Kernel]

As you can see in line 6 Fixnum's operator + is invoked. Since Fixnum does not know how to add an Object it invokes its #coerce. For simplicity reasons we always return a fixed value of 2 for the Object instance and as you can see Fixnum's operator + is invoked again and now returns immediately. The final result is properly calculated as 3.

Now let’s look at HexNum's method #coerce given the long introduction it looks surprisingly simple:

  # coercion

  def coerce(o)
    [HexNum.new(o.to_int), self]
  end

  # comparability

  include Comparable

  def <=>(o)
    case o
    when HexNum
      @i <=> o.to_i
    when Numeric
      @i <=> o
    else
      a, b = o.coerce(self)
      a <=> b
    end rescue nil
  end

We just try to create a new HexNum if the other type can be converted to an integer and go from there. In other cases you might have to check argument types to apply specific conversions but for our HexNum this is enough to ensure functionality.

Now you can also see how comparison works for HexNum: if the type is known we rely on comparison functionality of the standard library. And in order to make HexNum also work with classes written later #coerce is invoked on all other arguments. A last detail: if #coerce fails it is supposed to throw an exception. Since the contract of <=> prohibits that we add a rescue nil so we always end up with a proper comparison result.

Math and Operator Overloading

Now that we understand which role #coerce plays in Ruby’s operator land we can implement all the operators available in a way as to ensure we always get proper results. In our case this means, we always want to have a HexNum result with proper int math.

There is one thing that we need to ensure for all operators: they must handle their own type directly. If we fail to do that, we may end up in infinite recursion:

irb(main):001:0> class X
irb(main):002:1>   def +(o)
irb(main):003:2>     case o
irb(main):004:3>     when Integer
irb(main):005:3>       o + 1
irb(main):006:3>     else
irb(main):007:3*       a, b = o.coerce(self)
irb(main):008:3>       a + b
irb(main):009:3>     end
irb(main):010:2>   end
irb(main):011:1>
irb(main):012:1*   def coerce(o)
irb(main):013:2>     [X.new, self]
irb(main):014:2>   end
irb(main):015:1> end
=> nil
irb(main):016:0> x = X.new
=> #<X:0x10145c50>
irb(main):017:0> x + 1
=> 2
irb(main):018:0> 1 + x
SystemStackError: stack level too deep
        from (irb):7:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
... 7229 levels...
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):18:in `+'
        from (irb):18
        from /usr/local/bin/irb19:12:in `<main>'irb(main):019:0> x + x
SystemStackError: stack level too deep
        from (irb):7:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
... 7229 levels...
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):8:in `+'
        from (irb):19
        from /usr/local/bin/irb19:12:in `<main>'irb(main):020:0>
irb(main):021:0*

So let’s look at operator +:

  def +(o)
    case o
    when Integer
      HexNum.new(@i + o)
    when Numeric
      HexNum.new(@i + o.to_i)
    else
      a, b = o.coerce(self)
      a + b
    end
  end

We know that all Integer instances do return an Integer instance from their implementation of operator + so we can simply use that on our instance varible containing the numerical value. For all other Numeric instances (including HexNum itself) we can work with the instance value converted to int. All others are responsible for returning appropriate values for themself and the HexNum instance so the math can work properly.

Operator * works a bit differently because for proper results of multiplication we need to use the numeric value as isand convert later:

  def *(o)
    case o
    when HexNum
      HexNum.new(@i * o.to_i)
    when Numeric
      HexNum.new(@i * o)
    else
      a, b = o.coerce(self)
      a * b
    end
  end

Of course we could refactor common parts into an operator implementing method but I did not want to get too fancy here and rather demonstrate how particular math operations might need different treatment. For other operators we rely on the argument’s compatibility to an integer since other types do not really make sense here:

  # asymmetric binary operators

  def <<(o)
    HexNum.new(@i << o.to_int)
  end

  # bit operators

  def &(o)
    HexNum.new(@i & o.to_int)
  end

Summary

We have seen how a numeric class needs to be implemented differently from other classes that try to be complete. The major point is to provide #coerce with meaningful semantics and implement all operators. As always you can find the complete code at github. Have fun playing around with it!

The Complete Class

shortcutter@googlemail.com (Robert Klemme) — Sat, 24 Oct 2009 18:05:00 +0000

A remark: we enabled comment moderation because the blog was recently target of spam. You probably have not seen much of it because we were pretty quick in removing it manually. So if your comment does not show up please be patient.

There are some basic concepts (often called “aspects”) that need to be implemented for many classes although not all classes need all (or even any) of them:

initialization
conversion to a printable string
equivalence
hash code calculation
comparability
cloning (clone and dup)
freezing
customized persistence (Marshal and Yaml)
matching
math and operator overloading

Which of these is needed for a particular class depends of course completely on the circumstances. Classes which are never written to disk or used in a DRb context will not need customized persistence handling. For other classes there might not be a reasoable ordering.

We will look at these concepts individually in subsequent sections. For the sake of this presentation I will create a slightly artificial class which will have particular properties in order to be able to show all the concepts:

mutable fields
redundant fields, i.e. fields which carry cached values that can be derived from other fields
at least two fields for ordering priorities

Class Album implements a music album with a title, interpret, sequence of tracks and a fixed pause duration between tracks. I picked a slightly different approach than Eric in his article in two ways:

Eric has a stronger focus on collaboration with standard library classes while my guiding question was “what does a class need to be complete and consistent?”,
I present all the features in a single class to show how aspects play together.

You will likely never implement all these aspects in a single class. Even for those aspects you do use, you might not use the same implementations I will present. That’s OK. The implementations presented in this article are intended to cover aspects thoroughly even though you will not always have to do that in practice. For example, certain code is there in order to make the class work properly as part of an inheritance hierarchy. If you write a one off class for a script which is never intended for inheritance you can simplify many of the presented methods.

I left out the topic of math and operator overloading since that does not mix well with the concept of music album. Instead I will cover that in the next article in the blog which will present a class that shows how to override operators in Ruby and that plays well with Ruby’s built in numeric classes.

Initialization

Implementing method initialize is typically one of the first things I do when implementing a new class unless I can use the default implementation of Struct. There are a few things though that are worth considering.

First of all, who owns arguments to initialize? It is important to make clear what happens to arguments that are provided to the call to new. There are three cases

the value is immutable (like Fixnum) or
the caller retains ownership or
ownership is transferred to the new instance.

Case 1 is the simple one: basically you do not need care thinking about who owns the object as there are no bad effects which can be caused by aliasing. If the instance is mutable these effects can show up. If you want to make your code as robuts as possible you must ensure you got your own copy of the instance (typically via copying it, for example by using dup). The downside of this is of course that this costs performance as you’ll likely copy too many objects. In practice you will probably most of the time do nothing special and keep the code as simple as an assignment.

class Album

  def initialize(title, interpret, pause = 0)
    super()

    # main properties
    self.title = title
    self.interpret = interpret
    @tracks = []

    # additional properties
    self.pause = pause

    # redundant properties
    # left out
  end

  def title=(t)
    @title = t.dup.freeze
  end

  def interpret=(i)
    @interpret = i.dup.freeze
  end

  def pause=(time)
    @pause = time
    @duration = nil
  end

end

The other important aspect is inheritance. Most classes that are written probably do not have a super in their initialize method. If there are chances that you reopen the class and add mixin modules you should include super() right from the start because even if you only implicitly inherit Object and initialize in Object does nothing you may later reopen the class and add a mixin module which has an initialize method itself. If you inherit another class than Object you should explicitly mention arguments with super and not rely on having the same argument list as the super class initializer. That way you are making the call more explicit and are robust against changes in your initialize method.

class CdAlbum < Album
  attr_accessor :bar_code

  def initialize(t, i, code)
    super(t, i, 2)

    self.bar_code = code
  end
end

While we’re at it: if you write a module which is intended as mixin and needs initialization itself the initializer should simply pass on all arguments in order to be compatible with arbitrary inheritance chains:

module AnotherMixin
  # Prepare internal state and pass on all
  # arguments to the super class
  def initialize(*a, &b)
    super
    @list = []
  end
end

Conversion to a printable String

Often it is desirable to be able to convert an instance to a human readable string. Which representation is most appropriate depends on the uses of the class. One policy is to create a string representation so the instance can be later reconstructed given this string as is the case for all the numeric types from the standard library. In the case of our sample class we will provide all interesting information:

  def to_s
    "Album '#{title}' by '#{interpret}' (#{tracks.size} tracks)"
  end

If you want to reuse a string field as external representation you could copy it in order to avoid bad effects from aliasing. However, since typically to_s is invoked for printing and no references are held I would say most of the time it is safe to not explicitly copy the field in these cases.

Equivalence

There are two methods that deal with object equivalence eql? and ==. (Method equal? tests for object identity and should not be overridden.) Some core classes do have different equivalence relations implemented:

irb(main):003:0> 2 == 2.0
=> true
irb(main):004:0> 2.eql? 2.0
=> false

But most of the time both methods will implement the same equivalence relation. This also helps avoid confusion. Note that eql? is special because it is used by Hash and Set to test for instance equivalence. We will come to that in a minute when we look at hash code calculation.

Equivalence of instances must be tested against significant fields and should ignore redundant fields. Looking at derived field values does not add anything to equivalence and makes the process slower at best.

  def eql?(album)
    self.class.equal?(album.class) &&
      title == album.title &&
      interpret == album.interpret &&
      tracks == album.tracks
  end

  alias == eql?

Now here you see a test for class identity which you likely have not seen in other classes. Why is it there? Mathematical equivalence is a symmetric relation which means that if a.eql? b returns true so should b.eql? a. Implementing eql? and == that way also helps prevent strange effects when working with Hash instances. So if you are going to compare two instances of a class and a subclass then you might get true when called on the super class instance and false on the subclass instance. The same happens for two completely unrelated classes where the set of fields of one instance is a subset of those of the other instance. You may actually be tricked into thinking they are equivalent because the common fields are equivalent while the two instances represent completely different concepts. The only way to remedy this is to check for identity of the class.

The reason that omitting the class identity check does not cause issues most of the time is simple: usually you stuff uniform instances into a Hash or Set and even if you mix different classes most of the time they will have different fields and different field values. Still it is good to remember the point in case you experience unexpected effects with Hash keys.

Note that I omitted the test for self identity which you might be used to from Java classes. I believe this is a premature optimization because most of the time you are going to test different instances for equivalence so most of the time you pay the penalty of the failing identity check and win only in rare circumstances.

Hash Code Calculation

This topic is closely related to instance equivalence: classes Hash, Array, Set and other core classes rely on the fact that equivalent instances also return the same hash code. Note that this is not symmetric: instances wich have the same hash code may actually not be equivalent. But if the hash code differs they must not be equivalent.

An instance’s hash code should be based on the same fields that are used for determining equivalence. Our class Album has more fields and it is advisable to do some bit operations (often involving XOR) to combine all member hash codes into a single value in order to ensure better diversity of these values. If you base the hash code only on a single field you increase the likelyhood that non equivalent instances fall into the same bucket of the Hash which makes additional equivalence checks via eql? necessary. You can find more about how hash tables work on Wikipedia.

  def hash
    title.hash ^
      interpret.hash ^
      tracks.hash
  end

Comparability

Many classes have a natural ordering such as integers which are ordered by their numeric value. If your class does also have a natural order you can implement operator <=> and include module Comparable to get implementations of <, <= etc. for free.

  include Comparable

  def <=>(o)
    self.class == o.class ?
      (interpret <=> o.interpret).nonzero? ||
      (title <=> o.title).nonzero? ||
      (tracks <=> o.tracks).nonzero? ||
      (pause <=> o.pause) || 
      0 :
      nil
  end

Cloning

When cloning or duping an instance the default mechanism sets all fields of the new instance to refer to the same objects as the cloned instance. While this is not an issue with immutable instances (e.g. Fixnum or instances which are frozen) bad things can happen if you have a mutable instance (for example Array or String) which is suddenly referenced by two instances which both believe they are the sole owner of it. All bad effects can happen including violation of class invariants and the instance state likely becomes inconsistent. Do deal with such cases even for shallow copies such as #clone and #dup you need to copy a bit more. Fortunately Ruby provides a hook (#initialize_copy) which is invoked after the instance has been copied and which can make appropriate adjustments. In our case we only need top copy the Array of Track instances because all other fields are immutable:

  def initialize_copy(source)
    super
    @tracks = @tracks.dup
  end

There is a recent discussion on comp.lang.ruby that started out with the subject of deep cloning but uncovered some general aspects of cloning.

Freezing

For freezing similar reasoning applies as for cloning: immutable fields need no additional attention as you cannot change those objects anyway. For others you need to decide how deep you want the freeze to go. In case of class Album we certainly want to prevent addition of more tracks after an album has been frozen. We also explicitly trigger calculation of duration so to avoid errors when accessing the duration on the frozen instance; the value cannot change any more anyway. This also makes it important to invoke super as last method.

  def freeze
    # unless frozen?
    @tracks.freeze 
    duration
    # end

    super
  end

Note that the code is robust against multiple invocations of #freeze because duration is calculated only once (on first transition from unfrozen to frozen). If you have more complex calculations going on that involve state changes of the instance you should place unless frozen? and end around the custom freeze code.

Custom Persistence

If you want to serialize the complete state of your instance there is nothing more to do: you can simply use Marshal and YAML from scratch. However, if you have redundant data (such as field duration in our case) which you want to omit from the serialization you have to adjust the process yourself.

For Marshal the proper approach is to use the newer approach which invoves writing methods marshal_dump and marshal_load. The former is supposed to return something which is serialized instead of the current instance and the latter is invoked on a new empty object and handed the deserialized object so fields can be initialized properly.

  def marshal_dump
    a = (super rescue [])
    a.push(@title, @interpret, @pause, @tracks)
  end

  def marshal_load(dumped)
    super rescue nil
    @title, @interpret, @pause, @tracks = *dumped.shift(4)
  end

For YAML it is even simpler: you basically just need to ensure to_yaml_properties returns a list of symbols containing only those members that you want serialized.

  def to_yaml_properties
    a = super
    a.delete :@duration
    a
  end

Both approaches do have their own strengths: with Marshal it is easier to completely replace an object with something else. While that can be done with YAML as well it is not as simple and elegant as in the case of Marshal. YAML on the other hand shines because you need to override only a single method.

One word about the implementations: while these methods could look simpler I picked an approach which also works when inheritance comes into play. You can repeat the pattern of marshal_dump, marshal_load and to_yaml_properties throughout a class hierarchy and still have each method only deal with fields of the class in which it is defined. That makes it easier to deal with later additions or removals of fields in some class in the hierarchy. This is even more so important when dealing with mixin modules, which can happen on a per instance basis (via #extend).

Matching

I use the term “matching” for the functionality of the three equals operator (===) and for the matching operator (=~). The first is used in case statements and with Array#grep while the latter is usually used only explicitly.

The semantic is completely up to the class at hand and there are no general guidelines that could be given. Implement it when it is reasonable for your class. If you want to elegantly use instances of your class in case expressions you have to implement === doing something meaningful.

  # Check whether we have that track or track title by title.
  def ===(track_or_title)
    t = (track_or_title.title rescue track_or_title)
    tracks.find {|tr| t == tr.title}
  end

  # Match title against regular expression.
  def =~(rx)
    rx =~ title
  end

Summary

Today we looked at a class which implements various common aspects that you will meet over and over again when coding Ruby. Many of these are in fact present in other programming languages as well: Java has serialization, equals and hash code calculation. C++ also has an equivalence operator and other operators that can be overloaded to implement ordering etc. You can find the full code of the class at github.

Structs inside out

shortcutter@googlemail.com (Robert Klemme) — Mon, 21 Sep 2009 21:48:00 +0000

Today we’re back to normal blog mode, where each article stands for itself. Muppet Labs are closed and we will be continuing our journey across the Ruby universe starting with an indepth look at Ruby’s Struct class — Ruby’s Swiss army knife for structured data.

Struct can be used without any additional require statement — it’s just there. This means it comes with zero additional overhead during initial interpreter startup — one of the many advantage of using Struct. But first let’s look at the basics.

Data Container

Class Struct is a strange beast if you are confronted with it the first time: it has a method new like other classes. But that method does not return an instance of Struct but rather a class which is a subclass of class Struct:

irb(main):001:0> pnt = Struct.new :x, :y
=> #<Class:0x101c3d68>
irb(main):002:0> pnt.class
=> Class
irb(main):003:0> pnt.ancestors
=> [#<Class:0x101c3d68>, Struct, Enumerable, Object, Kernel, BasicObject]

Arguments to Struct.new are names of fields that the generated class will have. These names must be provided as Symbols which makes sense, because these field names are basically identifier you hardcode into the program. (I will explain later why this is an accurate assessment when I look at Struct vs. OpenStruct vs. Hash.) You must provide them as Symbols because of another peculiarity of Struct:

irb(main):001:0> Struct.constants
=> [:Tms]
irb(main):002:0> s = Struct.new "Point", :x, :y
=> Struct::Point
irb(main):003:0> Struct.constants
=> [:Tms, :Point]
irb(main):004:0> t = Struct.new "doesNotWork", :x, :y
NameError: identifier doesNotWork needs to be constant
        from (irb):5:in `new'
        from (irb):5
        from /usr/local/bin/irb19:12:in `<main>'

I have to say I never used this feature of Struct because

I don’t want all my custom classes in a single namespace because that leads to potential issues,
I prefer explicit assignment to constants which makes clearer what’s happening.

If someone knows advantages of this auto constantification which I overlooked please let us know via the comment function. Otherwise I suggest to rely on standard constant magic if you want to name your Struct:

irb(main):053:0> pnt.name
=> nil
irb(main):054:0> Point = pnt
=> Point
irb(main):055:0> pnt.name
=> "Point"

Behavior

A Struct generated Ruby class is more similar to a C++ struct than to a C struct because it does not only carry data but also functionality. Some of that is already predefined when Struct.new returns:

irb(main):001:0> pnt = Struct.new :x, :y
=> #<Class:0x101c2fa0>
irb(main):002:0> puts pnt.instance_methods(false)
x
x=
y
y=
=> nil
irb(main):003:0> puts Struct.instance_methods(false)
==
eql?
hash
to_s
inspect
to_a
values
size
length
each
each_pair
[]
[]=
select
values_at
members
=> nil

These can be devided into two categories:

methods which originally deal with struct members.
methods that allow an instance to mimic behavior of other classes.

Struct Member Behavior

First of all there are attribute accessors (those defined in the Struct generated class, irb listing position 2). Then there are those methods that make a Struct instance usable as Hash key:

eql?
hash

These are defined in such a way that two Struct instances are equivalent if all of their fields are:

irb(main):014:0> a = pnt.new 1,2
=> #<struct x=1, y=2>
irb(main):015:0> b = pnt.new 1,2
=> #<struct x=1, y=2>
irb(main):016:0> a.eql? b
=> true
irb(main):017:0> a.hash
=> -1066353017
irb(main):018:0> b.hash
=> -1066353017

Method == also implements a similar equivalence relation but with a twist:

irb(main):019:0> b = pnt.new 1.0,2
=> #<struct x=1.0, y=2>
irb(main):020:0> a.eql? b
=> false
irb(main):021:0> a == b
=> true

It uses method == of field values internally in the same way as eql? uses eql? of field values. By providing these methods Struct can save you a lot of tedious typing of pretty dull code that you would hack manually otherwise. As we all know, the less code we have to write the less errors we can do…

Mimicry

A Struct can — within certain limits — mimic an Array as well as a Hash:

# Array alike
irb(main):027:0> a.to_a
=> [1, 2]
irb(main):028:0> a[0]
=> 1
irb(main):029:0> a[1]
=> 2
irb(main):030:0> a.values_at 1,0
=> [2, 1]
irb(main):031:0> a.length
=> 2
irb(main):032:0> a.size
=> 2
irb(main):033:0> a.each {|e| p e}
1
2
=> #<struct x=1, y=2>
irb(main):034:0> a.select {|e| e % 2 == 0}
=> [2]
irb(main):035:0> a << "more" # of course not
NoMethodError: undefined method `<<' for #<struct x=1, y=2>
        from (irb):41
        from /usr/local/bin/irb19:12:in `<main>'

# Hash alike
irb(main):036:0> a[:x]
=> 1
irb(main):037:0> a["x"]
=> 1
irb(main):038:0> a.each_pair {|k,v| printf "%p => %p\n",k,v}
:x => 1
:y => 2
=> #<struct x=1, y=2>
irb(main):039:0> a.keys # not quite
NoMethodError: undefined method `keys' for #<struct x=1, y=2>
        from (irb):38
        from /usr/local/bin/irb19:12:in `<main>'
irb(main):040:0> a.members
=> [:x, :y]
irb(main):041:0> a[:y]=123
=> 123
irb(main):042:0> a
=> #<struct x=1, y=123>

Btw. the generated class also exposes some interesting methods, which do not really need additional explanation:

irb(main):049:0> pnt.members
=> [:x, :y]
irb(main):050:0> pnt[10,20] # for lazy typers
=> #<struct x=10, y=20>

Custom Behavior

From time to time you will want additional methods in your Struct class. Well, there is a “hidden” feature — the documentation does not mention it but you can pass a block to Struct.new which will be used as class body:

irb(main):001:0> Point = Struct.new :x, :y do
irb(main):002:1*   def distance(point)
irb(main):003:2>     Math.sqrt((point.x - self.x) ** 2 +
irb(main):004:3*       (point.y - self.y) ** 2)
irb(main):005:2>   end
irb(main):006:1> end
=> Point
irb(main):007:0> Point[3,4].distance Point[0,0]
=> 5.0

This obsoletes the frequently seen idiom which uses inheritance from a Struct generated class to add more methods:

irb(main):001:0> class Point < Struct.new(:x, :y)
irb(main):002:1>   def distance(point)
irb(main):003:2>     Math.sqrt((point.x - self.x) ** 2 +
irb(main):004:3*       (point.y - self.y) ** 2)
irb(main):005:2>   end
irb(main):006:1> end
=> nil
irb(main):007:0> Point[3,4].distance Point[0,0]
=> 5.0

When using inheritance like this you are wasting resources (a class in this case) and so far I see only one advantage: changing the constructor is a bit easier because the old implementation is available via super without having to resort to alias or reimplementing member initialization. If you want to have multiple subclasses of a Struct class then inheritance is OK of course. But in that case I would rather make the super class a named class by assigning it to a constant and use that later when subclassing.

Struct vs. OpenStruct vs. Hash

All three classes offer functionality with some similarities and overlap. Sometimes you can use them interchangeably. Still there are some guidelines you can use to decide when to use which of these. As always — take these recommendations with a grain of salt. If you have your own different rules of thumb let us know.

Use Struct if

you need a data container and fields are fixed and known beforehand (i.e. at time of writing of the code),
you need a structured Hash key,
you want to quickly define a class with a few fields,
you need to detect errors caused by misspelled field names.

Use OpenStruct if

the number of fields is fixed at runtime but varies frequently during development time (this is often the case for objects that should hold the result of command line option parsing),
you need a mock or a want to quickly have objects at hand which can be filled via usual attribute setters and getters. You might want to replaced with a proper class (or Struct) later — with explicit attribute declarations via attr_accessor and relatives.

Use Hash if

the number of fields is unknown at coding time,
there is a potentially unlimited number of fields (e.g. when reading key values from a file as is often the case for script based text processing).

Now it also becomes apparent why I indicated that Symbols are appropriate for naming Struct fields: when defining a Struct you are actually declaring attributes much the same way as with an ordinary class. Still, Struct is a bit inconsistent here when it allows Symbols and Strings as keys for the Hash like access via [] and []=. The convenience of being able to use both probably outweighs this inconsistency. Overall that Hash likelyness is probably one of the less important features of Struct. You are likely going to use it when refactoring from / to real Hashes of if you have an application that has to deal with several Structs in a uniform way and needs to use metadata (obtained via #members) to access fields.

Deficiencies, any?

My whishlist for Struct is pretty short. The only thing that I am missing is an equally quick and elegant way to add Comparable functionality to a generated Struct class. That could be achieved by having something like this:

class Struct
  def self.comparable
    define_method :<=> do |o|
      members.each do |m|
        c = self[m] <=> o[m]
        return c unless c == 0
      end
      0
    end
    include Comparable
  end
end

Now we can do

irb(main):014:0> Point = Struct.new(:x, :y).comparable
irb(main):015:0> puts points = (1..5).map {Point[rand(4), rand(4)]}
#<struct Point x=3, y=3>
#<struct Point x=1, y=3>
#<struct Point x=2, y=3>
#<struct Point x=1, y=0>
#<struct Point x=0, y=0>
=> nil
irb(main):016:0> puts points.sort
#<struct Point x=0, y=0>
#<struct Point x=1, y=0>
#<struct Point x=1, y=3>
#<struct Point x=2, y=3>
#<struct Point x=3, y=3>
=> nil

Summary

Today we looked at various aspects of Ruby’s built in class Struct. Struct allows you to create classes quickly (typically in a one liner) while providing a lot of useful functionality out of the box — most notably it is the fastest way to get a class suitable as a Hash key (apart from using one of the other built in classes like String, Symbol or even Array). If you haven’t been using Struct so far you might want to try it out in your next hack. Enjoy less typing with more Structs!

Muppet Labs closing for now

shortcutter@googlemail.com (Robert Klemme) — Mon, 07 Sep 2009 19:03:00 +0000

Looking back

It is time to look back at the Muppet Laboratory series. Here’s my list of noteworthy items — I will keep it rather short as interest in the series seems to have dwindled anyway:

The first version with LRU caching integrated was way too complex and had ridiculous performance. Continuous checking of interactions against the search criteria made the code complex and slow. In this case the premature optimization was worse than ineffective.
I am test lazy (I confess) but apparently this works remarkably well. Note, that this does not mean I don’t test. But for my scripts I usually do not write unit tests or RSpecs. (file under: Ruby Bad Practices — fortunately the acronym is the same)
Overall the LRU approach works remarkably well at limiting memory usage while processing huge amounts of input data so I would say the plan succeeded mostly.
I found the idea to use block passed to a method as a class body especially interesting. I will probably recycle this for other cases where I want to provide a framework (including command line parsing) which needs to be filled with user code.
Summertime is a bad time for a blog series — author and readers are occupied with other things, mainly enjoying sunny outsides.

What did you consider most valuable for you? What would you have done differently (software as well as blog series wise)? I would be curious to learn what your thoughts are.

What’s ahead?

The next article will be about the versatility of Struct. This is another swiss army knife of the lazy Ruby coder.

Completing the Animal

shortcutter@googlemail.com (Robert Klemme) — Mon, 24 Aug 2009 21:09:00 +0000

Leftovers

Summertime was a bit tough — but the end is near! Today I will cover two major parts that were needed to complete the “animal”. After that I will present lessons that I learned in this public blog software project. Muppet Laboratories will then close their doors but the code will still be there.

For a fully functioning Animal mainly two things were missing:

command line parsing,
creation of a proper filter mechanism based on command line arguments.

We will first look at command line parsing.

Command Line Parsing

There is actually no rocket since involved whatsoever. The most noteworthy piece of information is probably that you need to include ‘optparse/time’ in order to be able to allow OptionParser to parse time stamps passed as option arguments. (Funny thing is, OptionParser will also happily accept strings like “now” and parse them as timestamps.)

require 'ostruct'
require 'optparse'
require 'optparse/time'

  def self.parse_command_line(argv = ::ARGV) 
    o = OpenStruct.new(:output_dir => '.')

    # parse
    OptionParser.new do |opts|
      opts.on '-d', '--dir=DIRECTORY', 'Output directory ' do |v|
        o.output_dir = v
      end

      opts.on '-r', '--rx=REGEXP', ::Regexp, 'Regular expression matched',
      'against log line text' do |v|
        o.rx = v
      end

      opts.on '-t', '--time=TIME', ::Time, 'timestamp' do |v|
        o.ts = v
      end

      opts.on '-s', '--start=TIME', ::Time, 'start timestamp' do |v|
        o.start_ts = v
      end

      opts.on '-e', '--end=TIME', ::Time, 'end timestamp' do |v|
        o.end_ts = v
      end

      opts.on '--ids=ID_LIST', ::Array, 'Comma separated list of interaction ids' do |v|
        (o.ids ||= Set.new).merge v
      end

      opts.on '--id-file=FILE', 'File with ids one per line',
       '(empty lines are ignored)' do |v|
        s = o.ids ||= Set.new

        File.foreach v do |line|
          line.strip!
          s << line unless line == ''
        end
       end

      opts.on '--buffer=INTERACTIONS', ::Integer,
        'Max no. of interactions to keep in memory' do |v|
        o.max_size = v
        end

      opts.on_tail '-h', '--help', 'Print this help' do
        puts opts
        exit 0
      end
    end.parse! argv

    raise 'Only one of time or (start, end) allowed' if o.ts && (o.start_ts || o.end_ts)
    raise 'Missing end timestamp' if o.start_ts && !o.end_ts
    raise 'Missing start timestamp' if !o.start_ts && o.end_ts

    o
  end

I personally find OptionParser very elegant and complete but other seem to prefer other command line parsing packages, like GetoptLong and there are others around as well. Main advantages of OptionParser in my opinion:

Options, their documentation and their processing are all placed close to each other.
Built in support for conversion of argument strings to most basic types and even lists of values.
It is a part of the standard Ruby distribution, i.e you can safely assume that it is available.

Filter Creation

There are several approaches that can be taken on creating a bit of filter code from options taken from the command line — all have different strengths and weaknesses:

Interpreting the option set on each filter invocation (slow but easy to implement).
Combination of written filter code with filter criteria values stuffed in a closure (faster than the first option but equally easy to implement).
Generation of filter code (usually fastest, can be a bit complex to implement).

I picked option 2 because of the speed advantage and the avoidance of safety issues that come with using eval. I also find it a bit inelegant to generate code but that’s a purely aesthetic argument which I don’t claim any substance for. (But since we’re into Ruby at least partly for the fun, that argument is not too far off the mark.)

As interface I choose lambda's square brackets. So, here is the filter creation code:

    def create_filter(opts)
      # store in local variables to ensure
      # ensure values do not get changed
      # and a small speed improvement
      o_ids = opts.ids
      o_ts = opts.ts
      o_start_ts = opts.start_ts
      o_end_ts = opts.end_ts
      o_rx = opts.rx

      f = case
          when o_ids
            warn 'WARNING: Ignoring time filters with ids given' if
            o_ts || o_start_ts || o_end_ts
            lambda {|ip| o_ids.include? ip.id}
          when o_ts
            lambda {|ip|
              ip.entries.first.time_stamp <= o_ts &&
                ip.entries.last.time_stamp >= o_ts
            }
          when o_start_ts && o_end_ts
            lambda { |ip|
              ip.entries.last.time_stamp >= o_start_ts && 
                ip.entries.first.time_stamp <= o_end_ts
            }
          end

      case
      when f && o_rx
        lambda {|ip| f[ip] && ip.entries.any? {|e| o_rx =~ e.line}}
      when o_rx
        lambda {|ip| ip.entries.any? {|e| o_rx =~ e.line}}
      when f
        f
      else
        YES
      end
    end

As you can see there is a bit of prioritization going on: I choose to ignore time range options if also interaction ids are used as filter criterion. Reason is that interactions are usually short and additional time based filters would either have to add more interactions to the result or remove interactions whose id was selected (depending on whether “and” or “or” combination was chosen).

The text filter (regular expression really) on the other hand is “and” connected with the other filter, i.e. if the list of interactions selected by time or ids is large then it will be narrowed down through the text filter.

One word about time filters: when given a single point in time, the test is pretty straightforward — the first timestamp must be lower and the last timestamp must be larger than the test time, i.e. the interaction was active at the given time. The test for a given time interval (start and end time given separately) may look a bit odd at first. However, there is no typo involved. The test ensures that the test interval and the interaction’s time interval overlap. Or put differently: the test ensures that all interactions are included that were active during the test interval.

Things left to do

For now I am pretty satisfied with the results so I do not feel a lot pressure to continue working on it. However, there are a few things that could be done:

Cleanup of require and deletion of source files that were not used.
Adding of statistics output like lines read, lines written, interactions found etc.
Profiling and optimization.

Anything else?

A bit of Optimization

shortcutter@googlemail.com (Robert Klemme) — Sun, 09 Aug 2009 20:47:00 +0000

As suggested earlier I did a bit of benchmarking and found these timings of the first version which kept everything in memory and the next version with two LRU storages (for InteractionProcessor and IO objects):

v0.1

robert@fussel ~/Eigene Dateien/Projects/muppet-laboratories
$ time ruby19 bin/sample-animal.rb sample.log

real    11m53.226s
user    6m34.561s
sys     4m33.921s


v0.2.1

robert@fussel ~/Eigene Dateien/Projects/muppet-laboratories
$ time ruby19 bin/sample-animal.rb sample.log

real    25m40.842s
user    14m40.655s
sys     8m29.701s

I then went on and refactored the code in these ways:

removed LRU storage of IO objects
removed state pattern and replaced it with a single filter test again

The solution now resembles the first version quite a bit with the difference that the storage of InteractionProcessor instances is now an LRUHash. The InteractionProcessor now looks very simple — again:

  class InteractionProcessor
    
    # mode for writing files
    OPEN_MODE = IO::WRONLY | IO::CREAT | IO::TRUNC

    attr_reader :id, :coord, :entries

    def initialize(id, coordinator)
      @id = id
      @coord = coordinator
      @entries = []
    end

    # Process the first line
    def process_initial(time_stamp, line)
      process(time_stamp, line)
    end

    # Process an initial line
    def process(time_stamp, line)
      @entries << Entry.new(time_stamp, line)
    end

    # Append a continuation line to the last entry
    def append_line(line)
      @entries.last.line << line
    end

    def finish
      if ! @entries.empty? && @coord.filter[self]
        fn = file_name
        FileUtils.mkdir_p(File.dirname(fn))

        File.open(fn, OPEN_MODE) do |io|
          @entries.each {|e| io.puts(e.line)}
        end
      end
    end

    private

    # calculate the file name, this fails if
    # there are no entries!
    def file_name
      ts = @entries.first.time_stamp
      File.join(@coord.options.output_dir,
                ts.strftime('%Y-%m-%d'),
                ts.strftime('%H-%M'),
                ts.strftime('%S.%3N-') + id)
    end
  end

And now this is the timing we get:

v0.3

robert@fussel ~/Eigene Dateien/Projects/muppet-laboratories
$ time ruby19 bin/sample-animal.rb sample.log

real    7m29.546s
user    2m16.702s
sys     4m8.296s

Now, this is even faster than the first version! The difference seems to mainly be caused by a reduction in user time. I would have expected at most a small change in system time for less paging since we limit the memory usage. Maybe the difference is caused by the fact that the main Hash's size does not increase beyond a fixed limit.

I guess the LRU based storage of file handles qualified as premature optimization: apparently it wasn’t necessary to get rid of individual log lines as fast as possible. Instead, it makes more sense to treat an interaction as entity and wait with the processing until the interaction is complete (well, not really: until the LRU storage decides, that the interaction must be purged from memory). That’s a nice example how granularity of processing directly influences performance and application design. In fact, treating an interaction as entity is probably also the most straightforward approach. And it turns out to be more efficient than the much more complex state pattern logic.

LRU Integration explained

shortcutter@googlemail.com (Robert Klemme) — Fri, 31 Jul 2009 16:33:00 +0000

Today I will present my reasoning which lead me to the implementation of the LRUHash as well as how I integrated it into the main code. We will look at how LRUHash works and how it is integrated into the project. Then I will answer some questions that were actually asked — or only thought.

How `LRUHash` works

The interface should resemble a Hash as much as possible so that LRUHash can be easily substituted for a Hash instance without alteration of the code that uses it. As long as the instance remains below the maximum size it behaves much the same as a regular Hash. Only when the size limit defined via max_size is reached any subsequent store operations will remove the oldest entry.

Internally a LRUHash maintains a linked list of LRUHash::Node instances and a Hash. Nodes contain key and value of a hash entry as well as links to their predecessor and successor. The Hash is used for fast access of individual nodes while the linked list is used to maintain information about access order: every node which is accessed is moved to the head of the list so the least recently used element is always at the tail.

There are two nodes referenced as head and tail which are never changed. This makes it easier to move nodes around in the list because extraction and insertion are always operations on inner nodes and do not need to account for the first and last element on the list which is tedious because then you would also have to change the first and last pointer.

Additionally to the default_proc which works exactly the same way as in Hash there is a release_proc which is invoked whenever an item is removed from the LRUHash — either via explicit delete operations like delete, delete_if and clear or via the automated expiry which kicks in as soon as the LRUHash reaches its maximum size.

How `LRUHash` is used in the project

There are two LRUHash instances in class Coordinator:

A storage for interaction processors,
A storage for File objects.

The first instance is used to limit the overall number of interactions which are kept in memory. Once the LRUHash is filled and a new interaction needs to be stored the least recently used one is expired and removed.

The second one is needed because InteractionProcessor instances keep their File object open and the limit for open file handles per process is usually much lower than the reasonable limit for interactions kept in memory. So File objects are stored in the LRUHash and closed whenever necessary and reopened as well.

Why do you use `equal?`?

There are no “equivalent” nodes in a single LRUHash instance because they all have different keys. Actually the concept of equivalence is not needed here; rather I just needed to check for identity which is exactly what equal? does.

Why did you not use PUPA’s Ruby/Cache?

ged suggested in his comment the use of PUPA’s Ruby/Cache. I had looked at it before cooking my own little version and decided against using it for several reasons:

It did not include the default_proc mechanism of Ruby’s Hash although you can use fetch with a block the same way as in Hash.
It uses the notion of “object size” (explained here) as one of the limits of cache size. Besides the problems with defining “object size” which I pointed to here and here the more crucial reason was the problem with updating the cache’s idea of the current size of objects (either the overhead is significant, because every time the overall size limit is checked all objects need to be looked at or the cache cannot be aware of size changes if it caches object sizes).
It uses wall clock expiration time for cached objects — something I did not have use for in my implementation.
The fun of cooking my own.

While items 2 and 3 could be remedied by using arbitrary large limits the overhead of calculating values would remain (at least the documentation does not state that the overhead is saved when any of these limits are left out).

Performance Observations

While I did not actually measure timings the current version is significantly slower than the first draft version which kept everything in memory before writing out interactions to individual files. I suspect that this may be caused by these factors:

Criterion evaluation is done multiple times per interaction (although it’s still a dummy currently).
Frequent opening and closing of files caused by LRU handling of IO objects which is necessary because of the much lower limit of open file handles compared with interaction instances.

Both were introduced to get rid of log file lines as soon as possible. This also lead to a complex design (especially in the area of evaluation of the criterion). The effect may change though if real criteria are used. This is certainly something that I need to analyze.

An improved version would probably get rid of the immediate writing of lines as soon as possible and defer that to the point in time when the interaction is removed from the LRU cache of interactions. That way the criterion needs to be executed only once, can be made much simpler — especially for complex criteria which need to look at multiple entries such as the time range criterion. Also, we do not need to keep multiple files open.

Note: This shows how important it is to keep an open mind and to think again about the code you have written from time to time. I have discovered quite a bit of quirks I put into my code by this. Of course you can say that if you do it right the first time, this is never necessary — but who would claim to write optimal code in the first attempt?

Finally, a Best Practice

During my vacation I read Zen and the Art of Motorcycle Maintenance (you can browse the text but I’d rather read a real book). I had read it some 15 years ago and wanted to go back to this fascinating text which provides so many perspectives and crystallization points for thought.

Now, what does this have to do with Ruby? The best practice here is: once in a while do something completely different. It helps keep your mind flexible and will open it up to new approaches to old matters as well as inspire your creativity. You will also notice, that while you’re away from your everyday business your mind will actually continue to work on it which typically shows by suddenly having an idea that turns up when you least expect it.

LRU Integration

shortcutter@googlemail.com (Robert Klemme) — Fri, 10 Jul 2009 21:07:00 +0000

This will be just a short article since I am a bit tight on time right now.

For those of you who might not have heard the term “LRU” means “least recently used”. It refers to a replacement strategy used in caches. For every cache with an upper limit on the number of elements cached it must be decided which element to remove from the cache once the cache gets full and another element should be put in the cache. The decision can have dramatic impacts on the efficiency of the cache (its hit ratio).

LRU is easy to implement and works pretty well in many cases so I picked that one. “LRU” means, remove the element which has not been used for the longest time. This is typically implemented using a doubly linked list because with that elements can be moved to the head very quickly (O(1)). The algorithm works by moving every accessed element to the front of the list and deleting the last one when space is needed.

I created a class with a Hash like interface and an additional feature, a release_proc which gets called with the removed entry’s key and value. That way we can automate the cleanup process easily.

Internally the class has a Hash for fast access which has list nodes as values. These are the main methods for LRU mechanism during read access.

  def fetch(key, &b)
    n = @h[key]

    if n
      # hit -> move to front
      front(n).value
    else
      (b || FETCH)[key]
    end
  end

  # move node to front
  def front(node)
    node.insert_after(@head)
  end

Individual entries are of class Node:

  # A single node in the doubly linked LRU list of nodes
  Node = Struct.new :key, :value, :pred, :succ do
    def unlink
      pred.succ = succ if pred
      succ.pred = pred if succ
      self.succ = self.pred = nil
      self
    end

    def insert_after(node)
      raise 'Cannot insert after self' if equal? node
      return self if node.succ.equal? self

      unlink

      self.succ = node.succ
      self.pred = node

      node.succ.pred = self if node.succ
      node.succ = self

      self
    end
  end

And here’s the code for the removal of old entries:

  def store(key, value)
    # same optimization as in Hash
    key = key.dup.freeze if String === key && !key.frozen?

    n = @h[key]

    unless n
      if size == max_size
        # reuse node to optimize memory usage
        n = delete_oldest
        n.key = key
        n.value = value
      else
        n = Node.new key, value
      end

      @h[key] = n
    end

    front(n).value = value
  end

  # remove the node and invoke the cleanup proc
  # if set
  def remove_node(node)
    n = @h.delete(node.key)
    n.unlink
    release_proc and release_proc[n.key, n.value]
    n
  end

  # remove the oldest node returning the node
  def delete_oldest
    n = @tail.pred
    raise "Cannot delete from empty hash" if @head.equal? n
    remove_node n
  end

You can see the whole story in the git repo where the LRUHash is also integrated into the main animal code. Class InteractionProcessor has changed a bit as well as Coordinator. Some places are still a bit inelegant but that will have to wait until I have a bit more time.

Animal Interaction Processing

shortcutter@googlemail.com (Robert Klemme) — Tue, 30 Jun 2009 16:30:00 +0000

As stated I intend to exploit locality of interaction log messages by using LRU cache algorithm and handle log lines as efficiently as possible. The general concept looks like this:

Whenever a new interaction is seen in the log file, create a new InteractionProcessor for it and put it in some storage which allows for fast access.
Any further processing of log lines read for a particular interaction is dealt with by that InteractionProcessor instance.
The storage should have an upper limit on the number of entries and expire entries via a cache algorithm in order to avoid overusing memory.
To further reduce memory usage, write log lines of an interaction to the corresponding output file as soon as possible.

It should be immediately clear that this confronts us with a number of challenges we will now have to look at.

Challenge: Early Expiry

Considering that we do not have a means to detect the end of an interaction we can only rely on the expiry algorithmof our InteractionProcessor storage. This means, there is always the chance that an InteractionProcessor is evicted from the storage although there are more log records to come for this interaction. This situations becomes more likely if

the size limit of the storage is reduced,
the number of interactions which are active at one point in time increases.

The latter can be caused by high activity on the original system (many interactions are started in a time interval) or interactions having huge gaps (making them live longer).

Now, what does it mean for our processing? Since we can only detect the initial line of an interaction through the fact that there is not yet an interaction processor for this interaction, we may be in a situation that we believe this is a new interaction while in fact we have seen it already. This may lead to wrong filtering results (for example, if the beginning timestamp of this interaction needs to be evaluated for filtering and the difference between the real first timestamp and the second “first” timestamp makes a difference).

There are a few things that we can do:

Check the filesystem for this interaction and read previous log records into memory before continuing processing this interaction.
Remember all interaction ids along with their initial timestamps or with the file name to know whether an interaction has been seen already and to find it efficiently in the filesystem to read previous records (see 1).
Nothing.

Option 1 is very inefficient, because we do not know the initial timestamp and so must potentially search multiple directories. Option 2 is dangerous because it does not impose any limits on the storage needed for interaction ids with their timestamps. That leaves us with option 3 – a seemingly bad choice on first sight.

To comfort you let’s try to find out how dangerous it actually is to do nothing about it. Assuming for the moment that test-gen.rb has a realistic model of log files we will see in practice. Now let’s also hope I get the math right… Looking into the source code, you will see that the average time interval between two lines of a single interaction is 5 seconds:

Timer = Struct.new :t do
  def tic
    self.t += rand(10_000) / 1000.0
  end
end

Furthermore we see that there are 1000 new interactions per minute (variable commands) or, put differently, a new interaction starts every 60ms. Furthermore there are on average 5.5 (2 + 3.5) log records per interaction (note, I fixed the time distribution of start times which wasn’t uniform in the first version of the generator):

while t < end_time
  # create new entries
  commands.times do
    id = generate_id
    tx.t = t + rand(60_000) / 1000.0

    entries.add(Entry.create(tx.t, id, 'START'))

    (2 + rand(8)).times do
      pick = rand(10)
      msg = pick == 0 ? MESSAGES.last : MESSAGES[pick % 2]
      entries.add(Entry.create(tx.tic, id, msg))
    end

    entries.add(Entry.create(tx.tic, id, 'END'))
  end

  # write entries of this minute
  entries.print t
  t += 60
end

This means, the average life span of an interaction is 22.5 seconds (5 sec * (5.5 – 1)). So now, continuing with our average calculation, this means that there are roughly 375 new interactions in a 22.5 second interval (22.5 sec / 60ms). So, for the average case keeping 400 InteractionProcessors in memory is sufficient. Of course we need to take into accound that this is just an average calculation and also does not cover temporary load surges but if we apply factor 100 (i.e. 40,000 storage limit) this still doesn’t look like it will blast system memory and we should be safe for most situations (unfortunately my stochastic skills are totally rusty today, so if someone cares to calculate the likelyhood of failure that would be interesting to see). Assuming that we need 0.5 KB per InteractionProcessor because of log lines that need to be kept in memory we’re at about 20 MB which isn’t a lot of memory these days.

To sum it up: with the formula (avg entries - 1) * avg interval * new IA per minute / 60s we get the average number of interactions active at a time. It turns out that for our sample log generator this is 375 which leaves a lot of headroom for the interaction processor storage. Without diving into stochastics too much it is obvious that we are pretty safe if we do not implement a solution for dealing with interactions that were expired too early.

Challenge: How to keep as few log lines in memory as possible?

Since our concept includes getting rid of log lines from memory as soon as possible, we will have to look at how this might be achieved. Remember, we want to only output interactions which are included in our filter criteria, which can be quite different and need to look at

interaction id,
initial timestamp,
last timestamp,
a string matched somewhere in any of the log lines.

It should be immediately clear that not all of these criteria can be evaluated when the initial line of an interaction is seen. A simple matches / does not match logic won’t help here. We need a filter that can tell us “matches”, “does not match” and “maybe matches” where the first two answers must only be given if there is no new information (log lines) which can make it invalid. So we will use a filter that returns any of :yes, :no or :maybe. This is our dummy filter, which also has a changed interface:

    YES = Class.new do
      def first(iap, line, ts) :yes end
      def initial(iap, line, ts) :yes end
      def followup(iap, line) :yes end
    end.new

We do not use the complete InteractionProcessor as argument but add the current line and also the timestamp if needed. This also leads to three processing states in our InteractionProcessor: undecided, including, excluding. We handle this using the state pattern:

  class InteractionProcessor

    UNDECIDED = Class.new do
      def process_initial(iap, line, time_stamp)
        case iap.coord.filter.first(iap, line, time_stamp)
        when :yes
          # we'll improve this once LRU is there
          iap.entries << Entry.new(time_stamp, line)
          INCLUDE
        when :no
          iap.entries.clear
          EXCLUDE
        when :maybe
          iap.entries << Entry.new(time_stamp, line)
          self
        else
          raise 'Illegal return'
        end
      end

      def process(iap, line, time_stamp)
        case iap.coord.filter.initial(iap, line, time_stamp)
        when :yes
          # we'll improve this once LRU is there
          iap.entries << Entry.new(time_stamp, line)
          INCLUDE
        when :no
          iap.entries.clear
          EXCLUDE
        when :maybe
          iap.entries << Entry.new(time_stamp, line)
          self
        else
          raise 'Illegal return'
        end
      end

      def append_line(iap, line)
        case iap.coord.filter.followup(iap, line)
        when :yes
          # we'll improve this once LRU is there
          l = iap.entries.last and l.line << line
          INCLUDE
        when :no
          iap.entries.clear
          EXCLUDE
        when :maybe
          l = iap.entries.last and l.line << line
          self
        else
          raise 'Illegal return'
        end
      end
    end.new

    INCLUDE = Class.new do
      def process_initial(iap, line, time_stamp)
        iap.entries << Entry.new(time_stamp, line)
        self
      end

      def process(iap, line, time_stamp)
        iap.entries << Entry.new(time_stamp, line)
        self
      end

      def append_line(iap, line)
        l = iap.entries.last and l.line << line
        self
      end
    end.new

    EXCLUDE = Class.new do
      def process_initial(iap, line, ts)
        self
      end

      def process(iap, line, ts)
        self
      end

      def append_line(iap, line)
        self
      end
    end.new

  # ...

    # Process the first line
    def process_initial(time_stamp, line)
      @state = @state.process_initial(self, line, time_stamp)
    end

    # Process an initial line
    def process(time_stamp, line)
      @state = @state.process(self, line, time_stamp)
    end

    # Append a continuation line to the last entry
    def append_line(line)
      @state = @state.append_line(self, line)
    end

  # ...

  end

Challenge: Number of Open File Descriptors

If we store 40,000 InteractionProcessors in memory then the worst case with regard to file descriptors is that all have one open. That’s more than usually allowed for user processes. We’ll remedy this by using a LRU strategy here as well: we will have a set of open files and close the least recently used if we hit a fixed limit. I will cover this in one of the next versions, when we’ll have LRU handling integrated. (Maybe there is even a gem we can reuse.)

Summary and Outlook

This has grown into a rather largish article but I wanted to cover InteractionProcessor with reasonable completion. Note, that we still do not have the LRU mechanics and a couple of other things so this class will change again. Next articles will have to deal with LRU implementation if I do not find anything suitable and integration of that into the rest of the application. Furthermore, I expect the implementation of the filtering to be one more complex and thus interesting piece. Stay tuned.

The Animal raises its head

shortcutter@googlemail.com (Robert Klemme) — Mon, 22 Jun 2009 22:36:00 +0000

You can find the first sample-animal.rb over there at github. Deficits of this version:

No command line parsing,
No filtering,
Works only for moderately sized files,
No particular optimizations yet (well, apart from freezing of id Strings).

One remark about method main: this is the result of asking myself “What is the minimal interface I can provide for users of the Animal and who basically only need to define the parser class?” At first I found it a bit odd, but in terms of lines of code this is likely one of the leanest solutions.

I’ll have to call it a day now, but I invite you to have a look and comment. Do you consider method main a good idea? I’ll chime in with some more explanations later.

[Update:_] If you are looking for the entry point of the implementation for the sample log you need to look into file sample-animal.rb:

#!/usr/local/bin/ruby19 -w

# Implementation of the parser for the sample
# generator script test-gen.rb.
$: << File.join(File.dirname(File.dirname($0)), "lib")

require 'time'
require 'animal'

# main defines the custom parser class!
Animal.main do

  TIME_FORMAT = '%Y-%m-%d %H:%M:%S.%N'.freeze

  attr_accessor :year
  attr_reader :interaction_id, :time_stamp

  def parse(line)
    if %r{
       ^
       ( \d{4}-\d{2}-\d{2} \s \d{2}:\d{2}:\d{2}(?:\.\d+)? )
       \s+
       (\S+) # interaction_id
       \s+
       }x =~ line
      @time_stamp = Time.strptime $1, TIME_FORMAT
      @interaction_id = $2
    else
      @time_stamp = nil
      @interaction_id = nil
    end
  end

  def initial_line?
    time_stamp
  end
end

# EOF

The intersting thing is: this is complete already! As long as the file format does not change we won’t have to touch this file any more. Basically we only provide information about the log file format through the parser whose class is defined with the code block passed to main. The rest – command line argument parsing, creating of all necessary objects etc. is done in main which is not much longer either:

require 'ostruct'

# Namespace for all the Animal related classes
# Animal is the project on Ruby Best Practices
# blog which demonstrates the thought process
# of writing an application
module Animal

  # Parse the given command line and return an option instance.
  # Options are removed from the argument so what is left in
  # there must be file names.
  def self.parse_command_line(argv = ::ARGV) 
    o = OpenStruct.new(:output_dir => ".")
    # parse
    o
  end

  # This metho allows to write extremely short applications
  # because it accepts a block which is used to define the
  # parser class.  Alternatively users can provide a parser
  # instance.  The default command line is added implicitly
  # and will be option parsed and the whole processing will
  # start automatically.
  def self.main(parser = nil, argv = ::ARGV, &class_body)
    $stderr.puts 'WARNING: ignoring class body' if parser && class_body
    parser ||= Class.new(&class_body).new
    options = parse_command_line(argv)
    coord = Coordinator.new
    coord.parser = parser
    coord.options = options
    coord.process_files argv
  end

  # autoload init
  %w{
    Coordinator
    ProcessingStorage
    FileStatistics
    InteractionProcessor
  }.each do |cl|
    autoload cl, "animal/#{cl.gsub(/([a-z])([A-Z])/, '\\1_\\2').downcase}"
  end

end

There is really not much magic in main apart from (ab)using the method block as class body. (Btw, this is something you can do with Struct as well. I’ll write about that in a future post.) I can’t think of a leaner interface to get the job done but maybe someone out there has another idea.

I had thought of defining a single regular expression with capturing group indexes for timestamp and key. But then we might also need a way to specify timestamp conversion. And then there are also file formats which have timestamps separated in time and date and even binary formats… This looks like an area we should revisit after v1.0 – maybe there is some room for improvement. For now we’ll stick with the line oriented formats which should cover a pretty wide range – including the requirements.

I am also not convinced that the autoload part is really that great. Initially it looked like a good idea but now I am not sure any more: writing out class and file names is probably better readable and as easy as the automated class name to file name conversion hack.

Now, what do you guys say?

Shadow of the Animal

shortcutter@googlemail.com (Robert Klemme) — Sun, 21 Jun 2009 19:36:00 +0000

Let’s recapitulate where we have been for readers who are new to the blog and for the convenience of others. I started this a while ago as an experiment trying to give some insights into my way of reasoning. I choose processing of large log files as subject which should help me analyze problems on production systems. I do believe that this might also help others as log file analysis is a fairly common task. The key point here is to be able to efficiently sift through many large logfiles and give access to the relevant information (see also the list of requirements and the initial description of the scope).

I picked “Animal” as name of the logfile analyzer because that must sift through tons of log data in a similar way as the like named character of a TV show treated his instrument – ferociously.

In today’s article I will present some major design decisions. Some core components of the architecture have been mentioned in the previous article already so I won’t repeat them here.

Design Decisions

As I want to give insights into my way of reasoning about software and design in particular I will list some of the major design decisions along with my rationale.

No class for individual log entries

The reasoning has been presented in a previous article: I want to avoid the overhead of allocating short lived objects. You can play around with a toy Java project (Eclipse as well as ant) to see the effect I was talking about. I had up to 9% time difference between the straight version and the version which puts data in a special record class.

No meta data

I had pondered the option to use a two step approach with an initial analysis step which will generate meta data which then is used by another program to efficiently extract relevant information. The advantage clearly would have been that we can apply a lot of different filtering and selection criteria without having to go through the initial analysis phase. I have decided against for these reasons:

Easier propagation of relevant data: Others do not need the extraction program and the original log files to look at interactions,
Simpler software structure: Only one program is needed, no writing and reading of meta data needed,
More robust: A change in the location of the original logs does not affect analysis.

Efficient Processing

This is the part I personally find most interesting: how do we manage to stream process log data while still limiting memory usage? We cannot simply aggregate all interactions in a huge Hash and dump the interesting bits at the end because this will burn too much memory (which costs speed through paging) or even make the whole process fail completely. That would be especially bad since in a case of lacking memory we would loose all the work that has been done up to the point.

My idea for solving this involves two components:

A hash based storage with LRU semantics,
An interaction processor.

Basically the LRU storage is responsible for keeping only a limited number of interaction processors in memory while the interaction processor is responsible for dealing as efficiently as possible with a single interaction. This means, the interaction processor will have to efficiently decide whether an interaction is included in the output and write out log lines as soon as possible (!) to a file in a given location.

Output Storage

Every interaction will be stored in its own file. To avoid potential issues with file systems becoming inefficient with thousands of files we will use an output directory tree with these levels:

date, formatted YYYY-MM-DD (example ‘2006-06-21’),
hour and minute, two digit 24 hour clock and minute (example ‘19-11’).

Interaction file names will include the timestamp of the first record. The naming will be like this: “SS.SSS-” where “SS” are the seconds of the timestamp and “SSS” are milliseconds". A valid name might then look like this 2009-06-21/19-11/00.003-sdmyrsmmlbxodfd.

The advantage is that we will not face any filesystem issues because there will be only a few thousand entries per minute. Plus, because of the hierarchical naming of directories and files we can easily use textual sorting of names and get chronological output. If we discover that there are too many entries per minute we can still extend the schema pretty easily to include a directory level for seconds.

Structure

The structure of the application will look like this: there is an option parsing method which outputs options which contain a combination of all processing options. These options will be hand off to the Coordinator which is the main driver of the processing. The Coordinator also needs a Parser instance which is capable of parsing log lines and providing interaction_id as well as a time_stamp of the last parsed entry. The coordinator will create InteractionProcessors as needed and hold them in a structure keyed by interaction_id. Each InteractionProcessor will handle a single interaction only. It knows the coordinator through which it gains access to options and a filter evaluator which determines whether an interaction is included; this filter works on the InteractionProcessor.

Thought Process

Now you might wonder how I arrived at these decisions. Well, I wanted the resulting bit of software to get the job done simply and efficiently. Generating a whole lot of meta data which helps in finding interactions efficiently would have created additional complexity as I have pointed out. So it was logical to rather choose a grepish approach, i.e. read the input once and spit out everything that we are interested in.

Since volume of input data is large we cannot expect to hold all the input in memory at some point in time. So we would rather have to pick a streaming approach. This means, we can only hold as many entities in memory at a time. Luckily, log entries of a single interaction have high locality, i.e. whenever the first occurs follow up lines are not too far away and the interaction end is usually near as well.

Now, I considered two approaches for aging out interactions from the processor’s storage: timestamp based and access based. I picked access based because in reality there are some interactions which take a bit longer than the usual interaction which would make picking a high latency necessary. This in turn would keep short interactions longer in memory than necessary. Also, with this approach you cannot put a hard numeric limit on the number of interactions in memory which could lead to failures for some logs while other logs are processed ok. With a LRU based storage we can set a size limit and avoid this problem. Of course, long interactions which also have many log entries could still cause memory issues but since these are comparatively rare the likelyhood of issues is small.

I will now wander off hacking together a rough application which does not yet do much but should give you an idea of how all the pieces are supposed to work together.

First Design Considerations

shortcutter@googlemail.com (Robert Klemme) — Fri, 12 Jun 2009 20:36:00 +0000

Now the interesting part begins! In an early phase like this I like to look at the problem I want to solve from different angles to get a feeling for implementation options. The output will look like an unsorted collection of notes – and that’s what it is! I don’t claim that this is the best approach around but this helps me to get a better grip of the problem.

Side note: I believe that despite all the methodologies around the way you approach a new problem is highly influenced by your personal way of thinking and working. There is no one size fits all approach and “chaos” at the beginning is nothing to be afraid of. In fact it may well be that “chaos” gives birth to the best ideas.

Obtaining Information about individual Records

Requirement M3 makes it necessary that in some place we have a plug where we can insert something that does log file format specific things. Basically for parsing a single line a block would be sufficient. If the format parses ok, the interaction identifier is returned, if not, we have a continuation line. However, it seems this solution is too simplistic: for time filtering (M6) and time gap detection (S3) we certainly also need the timestamp of the record. M7, which I had forgotten initially, also makes it necessary that the year is somehow synthesized – or inserted into the line parser from surrounding code. So, we will likely end up with something that would be an interface in other programming languages, i.e. we need a parser class which speaks a certain protocol / has a certain API.

Here’s an alternative approach: we create a record class and write our parser in a way that it spits out instances of this record class all the time:

Record = Struct.new :time_stamp, :interaction_id, :message

parser = ...

parser.each do |rec|
  interaction = find_interaction(rec.interaction_id)
  time_gap = rec.time_stamp - interaction.last_time_stamp
  ...
end

From an object oriented point of view there’s much which speaks in favour of this:

A record is an entity in our description of the problem domain and as such it is naturally to have it represented in the application,
We can add record specific functionality to this class (e.g. get all potential keywords used for searching / filtering),
We do not have to care about multiline detection, this is completely encapsulated in the parser,
We can enforce class invariants (e.g. all fields must be non nil and time_stamp must be a Time instance).

However, there’s a price to pay in object allocations and collections. Remember, we are talking about files with tons of records; if all these records are short lived (i.e. not used for storing them somewhere) we’re generating a lot of overhead which the garbage collector has to get rid of again. So I rather lean towards the “parser interface” approach mentioned above:

parser = ...

parser.line = line

if parser.continuation_line?
  last_record << "\n" << line
else
  ...
  last_record = line
  diff = parser.time_stamp - last_time
  ...
end

Now, it’s debatable whether this falls into the category of premature optimisation, but since we know that we are going to crunch a lot of data we should probably avoid this kind of overhead.

Filtering

From the requirements it is clear that we would like to have quite flexible filtering (M6, S3, S4). We have two options:

We apply the filtering during processing.
Processing gathers meta data about the data and we can efficiently query later with whatever filters we like.

Hybrid approaches are also possible, e.g. time filtering during processing but searching for keywords later. From the user’s point of view option 2 is certainly more desirable because it gives greater flexibility. On the other hand this must be balanced with processing overhead and disk space. As usual, if we want great flexibility for queries after analysis we need more disk space for meta data (indexes). In another hybrid approach the types of desired filters are provided as input into the processor so that only indexes for criteria are created which are actually queried later. The “flexible query” approach certainly reminds me of relational database.

When we do online filtering (option 1 above) we need to keep some aspects in mind. For example, if we want to filter by text found in an interaction log message (the part after the interaction id in the sample generator output), that text might not appear in the first record of a particular interaction yet we would want to see all preceeding records as well (M1). Similar for time range filtering: an interaction might have started before the range start but end after it. It should be clear that if we choose this approach we need a way to keep some history of past records during processing. Since we can’t store everything we’ve seen so far we need to decide what we need to keep and what we can discard.

I need to ponder this a bit more but at the moment I would rather choose the online filtering approach for these reasons:

Less disk space used on output,
Potentially less write IO during processing, which would help performance since we are almost certainly IO bound,
Easier distribution to other staff – if we need meta data to get at information efficiently this also means that someone needs to have the software to extract the information he wants (plus the index data and the original data set which is large).

Statistics

Collecting statistics like no. of records and no. of lines per file isn’t too expensive so we probably create a statistics class which records just that.

Application Structure

If you do not know CRC Cards yet this is a good opportunity to look into this simple concept. It does not have the power of an UML collaboration diagram but it helps identify classes and how they might play together.

We can roughly identify some pieces already:

Parser: parse individual lines of the input, identify relevant information (timestamp, interaction id).
FileStatistics: collect various data points about a processed file (lines, time taken…).
OptionProcessor: process command line options.
Coordinator: This is the MCP which binds everything together and coordinates the processing.
ProcessingStorage: this is the foggiest entity so far; it will be responsible for storing information about processed interactions. Since I haven’t decided on the strategy yet, I’ll leave it at this for now. We’ll have to make this more concrete later. One thing is for sure: this will likely the most complex class – or rather set of classes.

So much for initial design considerations. I’ll stop here, to give us a break and allow for some discussion, which probably will turn up new aspects and new ideas.

Requirements Summary of the Laboratory Project

shortcutter@googlemail.com (Robert Klemme) — Mon, 08 Jun 2009 21:59:00 +0000

I will now take on the role of project secretary and put all requirements into a form suitable for easier reference. There will be the following groups:

must (M): requirement must be fulfilled
should (S): requirement should be fulfilled but is not a show stopper
nice to have (N): this one is purely optional
future (F): this might be a good idea for a later release but should not influence the current version

I will later reference requirements via letter and number, e.g. “S4” refers to “filter by text” requirement (and not a fast car from a German manufacturer).

Until now there was not much reasoning about software although there were some ideas (and even prototypes) mentioned in the discussion thread of the last article. With the next article I will start to flesh out the architecture (or even some alternatives) along with the reasoning that led to different decisions.

Must have

number	description
1	Provide efficient access to all entries of selected interactions
2	Retain ordering of input lines per interaction
3	Parsing needs to be able to identify multi line entries
4	Parsing of lines needs to be easily adjustable to different line formats
5	Analysis must not rely on any particular text to identify first and last lines of an interaction
6	Filter interactions by time range (between a and b, before a, after a)
7	If timestamps are parsed formats without year must be properly parsed

Should have

number	description
1	Input file names should be provided via command line arguments
2	Analysis should deal gracefully with partial interactions (at the beginning and end of a logfile)
3	Idenfity pauses in interactions with a configurable length and report them
4	Filter interactions by some text contained in messages, ideally via regexp matching
5	Make interactions available in their original formatting

Nice to have

number	description
1	Identification of the proper chronological order of files
2	Read from stdin
3	HTML output
4	Do not use more than 20% on top of a `cat >/dev/null`
5	Statistics about the analysis such as time range, files read, lines read, interactions found etc.

Future

number	description
1	Analysis data stored in a relational database for later analysis with SQL queries
2	Correlate data from different components with different log file naming schemes and formats

Please let me know if I missed something or there are other mistakes (such as contradictory requirements).

The Laboratory Project

shortcutter@googlemail.com (Robert Klemme) — Wed, 03 Jun 2009 21:56:00 +0000

The company I am with creates billing systems for telecom companies. More precisely we call them “convergent billing systems” because billing is not limited to phone calls or SMS. You can also bill multimedia content, internet traffic and whatnot. Basically everything that runs through your telephone or IT infrastructure. In our office we even have a solution where the billing system is used to charge money via our corporate id cards at the local vending machine.

These billing systems have a tremendous amount of interesting features. Listing them all would need an article of its own. For the sake of our project it is important to remember that these systems are quite complex and have a high throughput. We have a central component implemented as a JEE application which front ends to customer care and other systems that may be present at a customer site. (The term “customer” is really ambiguous: there are our customers, which are telco companies and their customers, i.e. everybody using their phone lines. I will talk about our customers only.)

Now, this central application writes out logfiles using usual mechanisms (you can look at log4j in case you do not know Java logging and are interested in more details). As you can imagine these log files grow from large to huge because they will contain several lines of per business interaction (basically this is a remote call from a client system). Sometimes these interactions fail and our support staff need to look at these log files to find out what’s wrong. When it’s a tricky issue we people from the development department need to do this as well. Simplyfying this task is the goal of this toy project.

Some things you need to know about these log files:

all log entries belonging to the same interaction share a common identifier,
they are huge,
multiple interactions are interspersed,
some log entries span multiple consecutive lines,
log files are rotated, i.e. there are some incomplete interactions at the beginning and end.

I have set up a git repository where I created a generator script which you can use to generate sample logs and get a rough idea how they look like. Of course, there is no real data but the basic properties from above are present. Try to run the script with test-gen.rb -h to see supported parameters. If you don’t the script will throw a lot of text at you. I included “START” and “END” messages per interaction only to simplify testing – in practice we cannot rely on the presence of lines which unambiguously identify the first and last line of an interaction.

What I like to have at the end of the project is a Ruby application which efficiently separates individual interactions thus they can easily be analyzed individually. In order to get there, we will start collecting requirements. Let’s assume for the moment that I am your customer and you try to get a list of requirements. I would like to invite you to query me via the comment system. I am sure, together we will come up with a better list than if I would write it down alone. In the next article of the series I will then present the summary of our efforts.

Enter the Muppet Laboratories

shortcutter@googlemail.com (Robert Klemme) — Fri, 29 May 2009 17:54:00 +0000

I have decided I will start an experiement. I haven’t done this before and I have no idea whether it will work as indended but I am sure we can pull this off together. Here’s the deal: I will try to let you participate in my thought process which hopefully will turn up some useful if not best practices. I don’t claim to be Don Knuth but hopefully you’ll find it interesting and worthwhile nevertheless. The tricky part will be to actually catch all the reasonings and record them in a format suitable for blogging.

In this article I will give a brief overview of how I intend to proceed. Of course a thing like this needs some form of project. We will come to that in a minute. The whole series of articles will roughly look like this:

Initial presentation (this article)
Description of the project goal
Summary of collected requirements
Several postings about actual code and decisions
Presentation of the final result

After I present the goal you are invited to help distill requirements via the blog’s comment system. I will be the customer for the moment and I am sure with your active support we will get a list together with requirements that the project can meet.

The whole thing is loosely related to my job but I think you can extract at least parts of it for your own purposes. Of course we’ll be doing everything nicely modularized so reuse won’t be an issue. :-) I any case the main benefit for us all should be the thought process and insight into decision making during software development.

The project will be about analysis of large log files which contain high numbers of small business transactions. In parallel with coming up with a more detailed description I will try to come up with a generator which we can use to create files for testing purposes. This should help you get a better idea of what I am talking about.

Tune in next week when you’ll hear Dr. Bob say…

Control flow features and readability

shortcutter@googlemail.com (Robert Klemme) — Tue, 19 May 2009 20:46:00 -0000

First of all I would like to thank our readers who participate in discussions so actively! These discussions provide interesting food for thought as well as inspirations for new blog entries. Today’s article was partly inspired by the question that surfaced recently: “Why is catch .. throw seen so infrequently?”

In this article I will explore how the choice of control flow constructs and their usage affects readability and maintainability of code. For the purpose of this investigation I will provide a definition, partly because the Wikipedia article seems a bit inconsistent and partly because it may change.

A control flow construct is a language feature which disrupts the normal progression to the next statement and conditionally or unconditionally branches to another location in source code.

This definition includes the usual if ... then but also method invocation and return.

Classes of control flow statements

Following a sequence of regular (i.e. non control flow) statements is easy. Things get only slightly more complicated for the reader in case of if ... then ... else ... end as long as proper indentation is used and we do not have to scroll multiple pages to see the other branch(es).

But it is a different story altogether if the stack frame changes and we as readers have to jump to a completely different location — maybe even in a different file! Hence I will classify control flow constructs depending on the distance measured in stack frames:

level-0: There is no change in stack frame at all.
level-1: Code moves one stack frame up or down.
level-n: An arbitrary amount of stack frames can be added or removed.
level-x: The stack is completely exchanged.
level-1n: This is a hybrid where “technical” and “visual” jumps differ.

I am sure those first three categories do not bear any surprises for you. The last two probably need a bit of explanation. Category level-x contains Ruby’s continuations. I won’t cover them here because I am not too familiar with them, they are rarely used in “ordinary” code and they are so complex that they probably deserve an article of their own. It should be obvious that this complexity does not really help understanding code.

level-0

This category includes

if ... then ... elsif ... else ... end as well as unless ...,
ternary operator ... ? ... : ...,
for, while and until loops,
both forms of case statements,
and and or,
all statement modifiers (if etc. at the end of a statement).

All these provide for easy following of code and can — when chosen wisely — even form Ruby code which almost reads like English. There are really only two things that can make them hard to follow

Not indenting code properly.
Putting too much lines of code between different keywords belonging to the same construct.

As long as you follow these basic rules, the reader of your code will be able to follow the flow easily.

level-1

In this category we have

method invocation,
method return.

Note, that I did not place return in this category — we will see it in level-1n below.

The readability of a method invocation depends mostly on how good it conveys what the method does. A Ruby programmer who reads x.to_s immediately knows that this expression returns a string representation of x. (Strictly speaking there is of course no guarantee that this method actually returns a String instance but for all practical purposes we can safely assume this.)

For unknown methods the name is crucial for our understanding — or at least rough idea — of what happens during the method call. This is important because only if we have this understanding we can continue reading and understand what the current method does. This shows how maintainability not only depends on proper modularization but also on well chosen method names. In fact it might be more important to get a method’s name right than to have documentation which covers all aspects of the method’s semantics. Don’t get me wrong, I do not want you to neglect your documentation! But a proper chosen name goes a long way in telling the reader of the code that uses this method what it does.

level-n

Again, there are two contenders:

raise ... rescue,
catch ... throw.

Although these two may seem to only differ in syntax on first sight, there are fundamental differences. I believe that these are ultimately the reason why we see exceptions quite frequently while we rarely discover a catch statement in code. Let’s first look at exceptions:

The power of exceptions comes from decoupling the signalling of an error and its handling. The meaning of the error condition is engraved in the exception class (via inheritance and documentation). We know that Errno::ENOENT denotes a non existing file and we can write an exception handler for this. When we raise this exception we do not know how many stack frames upwards there will be a handler for it — and we do not need to. If there is none ultimately the interpreter will exit with an error message and an exit code != 0.

Contrast this with catch ... throw — everything seems to be the opposite here:

This combination is for regular control flow and not for dealing with error situations.
You’ll first see the catch and then throw.
There is strong coupling between the catch location and the throw location in code via the symbol used; and both statements can be in different methods altogether.

Now, if you indeed place catch and throw in different methods you have established a strong link between the two: in order for the program to work properly both need to use the same symbol and both must be aware of the returned object (if any) so the result of catch can be processed in any meaningful way. You also need to take care about the last statement in the block attached to the catch in order to not accidentally return something which will be interpreted as thrown return value.

It’s fairly easy to complicate things even more by placing catch and throw methods in different classes or by nesting two or more catch constructs. It’s fairly safe to say that these obfuscating effects are best avoided by using catch ... throw in a single method only — and in that case there are other control flow constructs that we can usually use. In fact I am still searching for more convincing examples of catch ... throw usage; so far the best contender has been a jump out of multiple nested loops. Although, to me the “multiple nested loops” item has a slight code smell of its own. But, read on…

Addition: Actually, you can find a good application of catch ... throw in Ruby’s standard library. In this case it is elegant and catch ... throw has the advantage of not interfering with exception handling: assume more complex code in the block which is passed to Find.find which has a rescue internally then using exceptions behind the scene might have surprising effects.

level-1n

You might be curious why I came up with this category. Let’s first look at an example: assume there is a method that yields all Ruby files which are found below any number of directories and we use that to write another method which returns the first of those files that we own:

def find_my_ruby_file(*dirs)
  find_ruby_files *dirs do |f|
    return f if File.stat(f).owned?
  end
  nil
end

Now, if return is executed find_my_ruby_file exits and returns f, we go up one stack frame. Visually this is true, but in reality, when return is executed, the stack is several levels deeper than it appears to be. You can easily check that by inserting something like puts caller(0) before the return – here’s a complete version that you can use to experiment:

require 'find'

def find_ruby_files(*dirs)
  Find.find *dirs do |file|
    yield file if test ?f, file and /\.rb\z/i =~ file
  end
ensure
  puts 'exit find_ruby_files'
end

def find_my_ruby_file(*dirs)
  find_ruby_files *dirs do |f|
    if File.stat(f).owned?
      puts 'found!', 'stack <<<', caller(0), '>>> stack'
      return f
    end
  end
  nil
ensure
  puts 'exit find_my_ruby_file'
end

p find_my_ruby_file(*ARGV)

You’ll find some surprising entries there and as a side effect I now know one more example where catch ... throw does seem like the best tool for the job…

This property of being able to exit more stack frames than visible on first sight is shared by break which can be used to exit multiple stack levels in a similar way. It won’t leave the current method (i.e. the method in which it is lexically scoped) but via invoked methods with blocks the unnesting can include any number of stack frames. This reinforces what we have said before since we now clearly see that there are more ways that a block of code can be left “not normally” besides raise and throw.

What do we learn?

Looking at control flow constructs from an execution stack perspective has provided some interesting insights (at least to me). While general rules for readability (proper indentation, limit the amount someone has to read etc.) do apply here as well, some of those constructs can have surprising effects on program behavior and should be used carefully. This is especially true for the “far reaching” constructs which can actually make the stack unwind multiple levels — or even change completely as in the case of callcc.

The Universe between begin and end

shortcutter@googlemail.com (Robert Klemme) — Fri, 01 May 2009 10:05:00 -0000

This time we’ll explore the space between begin and end. Today’s article won’t be as much about individual best practices but rather I will try to explore various aspects of begin ... end blocks so you can decide how to make best use of this tool. In fact, there are so many aspects that this construct sometimes reminds me of a swiss army knife – not so much because you can do everything with it but rather because it has so many features that you can use. When it comes to control structuring elements begin ... end is probably the most complex thing found in Ruby.

One final introductory remark: although I am pretty sure that Ruby hasn’t changed between 1.8 and 1.9 with regard to begin ... end I did my tests with 1.9.1 only. So if you find something to be wrong, please comment!

Starting the Investigation

For easier reference here’s a block which contains all the options:

begin
  # do our work
rescue
  # standard oops!
rescue SomeException => e
  # oops!
rescue Exception => e
  # deal with other errors
else
  # good, no exception surfaced!
ensure
  # good or bad, this needs to be done
end

Note that you can replace begin with def meth... to define a method with “integrated” exception handling. That way you can avoid one level of nesting.

There are various interesting aspects to each section:

control flow (most particularly, how is the section left?),
result (if any),
documentation (what does it tell me that code is in a particular section?).

We’ll keep these in the back of our heads when exploring section after section.

`begin`

This is the main section where you place the code that does the work you want to get done. If evaluation reaches the end of this section normally the result of the last expression evaluated is also the result of this section. In the absence of rescue or else that value is propagated to the surrounding context.

If we look at control flow things start to get a bit more involved. First, you should note that there are these additional ways this section (actually any section of code) can be terminated:

return is executed,
break is executed in a do block invoked,
an exception is triggered via raise,
throw is invoked.

How many of those did you think of? (Did I miss another one?) All these have one thing in common: the end of the section is not reached normally and the code produces a different result.

`rescue`

This can have an optional ExceptionType => variable in which case exceptions of this class and subclasses (if there are no preceeding rescue clauses with them) are caught. If that optional part is missing only StandardError and subclasses are caught.

If there are multiple rescue clauses, order matters. You must rescue most specific errors first and less specificerrors later because otherwise a super class rescue clause will shadow a sub class clause which comes later. Only one rescue clause of a group is ever executed.

In case a rescue clause is executed the result of the whole block is that of the code in the rescue clause. In other words: when catching exceptions the exception code replaces the block’s result. Unless, that is, you invoke raise or retry.

The tricky part about rescue is which exceptions to catch. If you cannot handle an exception, you should not catch it because otherwise you will prevent other code that is capable of handling it to work. Actually, as long as you do not raise again, nobody outside will notice that there was an error in the first place. A special case is raise without arguments: sometimes it is reasonable to catch all exceptions, log the event and rethrow:

def f
  do_work
rescue Exception => e
  log.error "There was an error: #{e.message}"
  raise
end

`else`

This section does make sense only if there is a rescue as well. (You’ll get a warning if you use it without it.) If the main section completes regularly and an else section is present it is executed.

Now, some of the discussion revolved around the utility of this section and whether code placed here is equivalent to code placed in other places. While at first sight it may seem that you can just place it at the end of the main section a closer look reveals some subtleties, some of which have been mentioned already in the discussion. I’ll list them here anyway for completeness reasons:

If the code raises an exception, it might be rescued when placed in the main section but it won’t be rescued when placed in else section. This might also lead to unnecessary retries if there is a retry in the rescue clause.
Placing the code after end generally has a similar effect as putting it in an else section but if there is an ensure section order of execution between the two code bits is reversed. This can make a serious difference if the else code uses a resource which is cleaned up in ensure.
When using the version of begin ... end in a method definition there is no place “after end” which is part of the method invocation:

def do_the_work
  some_code_which_may_throw
rescue ArgumentError
  $stderr.puts "Ooops! Passed the wrong argument."
else
  puts "Job done."
end

`ensure`

This section is the last one before the final end. Code in this section will be executed regardless of how the main section is left, i.e. even in case of raise, throw, return and break! The result of the section is ignored unless you choose to explicitly return it or raise an exception. While raising exceptions might be reasonable in some cases, it should generally be avoided because those exceptions will shadow errors coming from the main section or from rescue clauses:

irb(main):007:0> def f
irb(main):008:1> raise "Foo"
irb(main):009:1> ensure
irb(main):010:1* raise "Bar"
irb(main):011:1> end
=> nil
irb(main):012:0> f
RuntimeError: Bar
        from (irb):10:in `ensure in f'
        from (irb):10:in `f'
        from (irb):12
        from /usr/local/bin/irb19:12:in `<main>'
irb(main):013:0>

Another thing you should definitively avoid is a return statement in ensure because this will shadow the result from the “business logic” code in the main section:

irb(main):013:0> def f
irb(main):014:1> 1
irb(main):015:1> ensure
irb(main):016:1* 2
irb(main):017:1> end
=> nil
irb(main):018:0> f
=> 1
irb(main):019:0> def g
irb(main):020:1> 1
irb(main):021:1> ensure
irb(main):022:1* return 2
irb(main):023:1> end
=> nil
irb(main):024:0> g
=> 2
irb(main):025:0>

There’s a reason why the result of executing ensure section is ignored: this code has nothing to do with the “business logic” which should go into the main section but is solely for cleaning up. Assume you open a file with io = File.open (which I hope you won’t do after reading a previous post), have search code in the main section and place io.close in an ensure section. Then you would not want to shadow the result of the search by the result from the close operation.

Although ehsanul questioned the utility of ensure I certainly use it more often than else. While else section code can be put somewhere else in some cases, there is no other place of code which can easily replace an ensure clause and guarantee the same robustness of the code at the same time. This is nicely demonstrated by his suggested alternative which uses only rescue without Exception — this does not catch all exceptions! “Robustness” in this case refers not only to runtime robustness but also robustness of the code against maintenance (i.e. changes).

Let’s assume you have caught all exceptions via rescue clauses which do not raise and placed your cleanup code after end as it was suggested. Code works as expected and everything is fine. You might have to use a local variable to make sure the proper value is returned from your method but this is just a nuisance. Now all these changes will put execution of your cleanup code at danger:

A rescue clause is removed.
A previously uncaught exception type is thrown from the main section (the change need not be in your piece of code).
A rescue clause is changed to raise an exception.
An else section is introduced and code might throw.
Code in else section is changed to raise an exception.

And lastly, do not underestimate readability! Placing code in an ensure section tells the reader immediately that he can ignore it when trying to understand what the main purpose of the code is. This is “only” cleanup which makes sure some resources that were used for the calculation are not kept longer than needed. Whereas, if you place that code after end it could belong to the normal flow of the core business logic.

Guidelines

So after all the detail here are some rules which I hope will guide you in making the most appropriate use of begin ... end blocks.

If you have cleanup code that must be executed under all circumstances, put it in ensure.
Do not place return or break in ensure sections and try to avoid throwing exceptions from them.
Place rescue clauses for more specific exceptions (sub classes) before those for less specific ones (super classes).
rescue the most specific exception you can handle.
Do not rescue exceptions that you cannot or do not want to handle.

Keep in mind that these are just guidelines and you have to check on a case by case basis which is the most appropriate solution. As Buddha says: verify the teaching with your own mind.

Writing Block Methods with automatic Resource Cleanup

shortcutter@googlemail.com (Robert Klemme) — Mon, 20 Apr 2009 19:47:00 -0000

After we have seen how File.open with a block is safer than without we will look into how such methods are created today.

Ingredients

We need two ingredients:

yield
A begin … end block with ensure

yield is a keyword and invokes the block which is passed to a method. If the caller did not provide a block he’ll earn a LocalJumpError. Result of evaluating yield is the value of the block (remember, we also called them “anonymous functions”).

As you will probably know begin … end blocks can be used to catch exceptions and handle them properly. But this construct has two more features apart from rescue; a full blown block might look like this:

begin
  # do our work
rescue SomeException => e
  # oops!
rescue Exception => e
  # deal with other errors
else
  # good, no exception surfaced!
ensure
  # good or bad, this needs to be done
end

Code in an else clause after rescue is executed when the block is left normally, i.e. without an exception being thrown. Note that in case of an exception it is irrelevant whether it is caught in this begin … end block or not — else will not be executed. Code after ensure is executed under all circumstances — regardless whether an exception is thrown or not. This is the feature that we’ll exploit for our cleanup.

An important thing to know is that the result of the “ensure” code does not affect the block’s result which is normally the value of the last expression evaluated between begin and the first rescue. So anything the cleanup code returns is invisible to the caller (which makes perfect sense if you think about it). Results of rescue and else clauses are retained when they are executed.

Cooking

Assume we have a class that distributes stdin to a set of files much the same as the ‘tee’ command line utility does. This class features a method open which opens files and another method close which closes all file descriptors. Then we can create a class method similar to File.open() which does the automatic cleanup via an ensure section:

class Tee
  def self.open(file_names, mode = "w")
    tee = new(file_names, mode)
    tee.open

    if block_given?
      begin
        yield tee
      ensure
        tee.close
      end
    else
      tee
    end
  end
end

This code does actually work similar to File.open because it acts differently depending on the presence of a block:

If there is no block, then the “tee” is created and opened. It is the return value of the method.
If there is a block, then that is called with the opened “tee” instance and after termination this is closed.

That’s really all there is to it. A final remark: in order to ensure that this pattern works properly you need to make sure the initialization is done before the begin. Otherwise any exceptions thrown during initialization will trigger the cleanup code which then will act on an incomplete or completely different object altogether (chances are that you will get NoMethodError from nil in this case).

You can also look at the full code if you want to see the complete story.

Using Blocks for Robustness

shortcutter@googlemail.com (Robert Klemme) — Thu, 09 Apr 2009 14:25:00 -0000

Ruby’s blocks can be used for many purposes — in fact, they might well be the most used feature of the language. Today we will start looking at a frequently-used idiom, analyze its implications for robustness and demonstrate how blocks can greatly improve it.

Robustness

When talking about robustness of software¹ we mean the degree to which it is able to function properly under changed circumstances. Those changes could be internal (i.e. introduced by a code change) or external (e.g. by changed input data). Sometimes achieving robustness is a hard task especially for complex software like operating systems. In this article we’ll look at robustness at a smaller scale.

A common file handling idiom

When you read through ruby-talk you will find code that looks like this (I just did some minimal adjustments to formatting and corrected obvious errors):

  # read file into an array
  f = File.open("mylogfile.log")
  saved = File.open("archive.log", "a")
  results = f.readlines
  # Is there a way to remove everything that's been read here?
  f.close

  results.each_line do |sending|
    conn_to_db.print("data #{sending}")
    # Make sure we keep a backup
    saved.puts("#{sending}")
  end

  # Close the archive file
  saved.close

The logic is straightforward: a file is opened, all its lines are read and then it is closed again. Similarly, the second file is opened for appending, then written to and finally closed. Sure, the second file could be opened later, but this is not where I want to draw your attention to.

Rather, please notice that File.open is used together with an explicit close. Closing the file IO object ensures that all pending output is written to the file and operating system resources associated with the underlying file handle are freed. While not closing a read only file usually does not have dramatic consequences, not closing a file opened for writing likely has dramatic consequences. You can see it by running this bit of code through Ruby 1.8.*:

  NAME = "foo"

  f1 = File.open NAME, "w"
  f1.puts "text 1"
  # forgot to close f1
  f2 = File.open NAME, "a"
  f2.puts "text 2"
  # forgot to close f2

On my system (CentOS 5.3, Ruby 1.8.5) I see only the first line in the file but the second puts has no effect. Even if you add a f2.close at the end of the script you won’t see any difference. You need to add the f1.close before you open f2 to actually see two lines in the output file.

File.close is the solution, or is it?

We just remember to always add a f.close whenever we have opened a file and be done. Everything written to the IO object will reach the file and we do not need to worry about anything any more. Or do we? Unfortunately, this is not the case. Especially in situations like the one shown above where exceptions can happen (we do not know what conn_to_db.print does, but it looks like another IO operation which could fail) we need to take additional measures to ensure the file is really closed.

We can use begin … rescue … end to do proper exception handling and make sure file descriptors are closed properly. But this tends to get verbose. Fortunately there is another option: File.open accepts a block and ensures that the file descriptor is closed regardless how the block is left.

Large files

When dealing with files as shown in the code example above there is another potential source of trouble: if files can grow arbitrarily large they will burn a lot of memory when read as a whole or even terminate the program because there is not enough memory available.

For reading files there is an even better alternative: File.foreach will read a file line by line (or using a different separator given as second argument) and hand each string to the block that is provided. This saves an additional io.each {|line| …} inside the File.open block — less typing and one level of indentation less. Also, as an added benefit we can efficiently process large files because we do not have to hold the complete file in memory when using iterative solutions.

Example rewritten

With these tools equipped we are now ready to rewrite the original bit to a more robust version:

  # read file and write to archive
  File.open("archive.log", "a") do |saved|
    File.foreach("mylogfile.log") do |sending|
      conn_to_db.print("data #{sending}")
      # Make sure we keep a backup
      saved.puts(sending) # removed superfluous string interpolation
    end
  end

Note, that if the timing of the original script is critical we might want to use File.readlines in order to read in all lines of “mylogfile.log” before starting to write to the “archive.log” as the original code does.

Conclusion

Using Ruby’s blocks goes a long way to making programs and scripts that deal with file IO more robust with regard to

exceptions occurring somewhere along the way,
large files.

So, whenever you do file IO in a Ruby program, remember you can use

the block form of File.open or
File.foreach

to make your code more robust without paying a significant price.

You can even resort to File.readlines if you have to read the whole file into memory and at least save a bit of typing. If you make it a habit to always use these idioms chances are that you’ll save a lot file IO related bug hunting in the future.

Next time I will look into how to create a method like File.open which uses a block to ensure automatic resource deallocation.

¹ Here is a definition of robust and a Wikipedia Article about Robustness Principle.

Ruby Best Practices

Complexity

Summary

Code Massage

Java Style

A little more sophisticated

O||=erator

Outsourcing

Getting tricky

Even shorter with a general solution

Golf

The fun begins

The Complete Numeric Class

Conversions into HexNum

Conversions to other types

Mutability

Equivalence vs. Comparability

What’s this #coerce thingy?

Math and Operator Overloading

Summary

The Complete Class

Initialization

Conversion to a printable String

Equivalence

Hash Code Calculation

Comparability

Cloning

Freezing

Custom Persistence

Matching

Summary

Structs inside out

Data Container

Behavior

Struct Member Behavior

Mimicry

Custom Behavior

Struct vs. OpenStruct vs. Hash

Deficiencies, any?

Summary

Muppet Labs closing for now

Looking back

What’s ahead?

Completing the Animal

Leftovers

Command Line Parsing

Filter Creation

Things left to do

A bit of Optimization

LRU Integration explained

How LRUHash works

How LRUHash is used in the project

Why do you use equal??

Why did you not use PUPA’s Ruby/Cache?

Performance Observations

Finally, a Best Practice

LRU Integration

Animal Interaction Processing

Challenge: Early Expiry

Challenge: How to keep as few log lines in memory as possible?

Challenge: Number of Open File Descriptors

Summary and Outlook

The Animal raises its head

Shadow of the Animal

Design Decisions

No class for individual log entries

No meta data

Efficient Processing

Output Storage

Structure

Thought Process

First Design Considerations

Obtaining Information about individual Records

Filtering

Statistics

Application Structure

Requirements Summary of the Laboratory Project

Must have

Should have

Nice to have

Conversions into `HexNum`

What’s this `#coerce` thingy?

How `LRUHash` works

How `LRUHash` is used in the project

Why do you use `equal?`?

`begin`

`rescue`

`else`

`ensure`