After a massive initial load of 155k gems, the size was about 45GB (currently, it grows pretty quick per week). The rubygem and gem mirror command is smart enough to just download just the deltas when you run it again:


$ gem mirror
Fetching: http://rubygems.org/specs.4.8.gz
Total gems: 170843
Fetching 16176 gems
................................................................


Then I wanted to know the size of all the latest gems only. If I had to do a lazy sneakernet, this might be one method of grabbing a whole bunch of dependencies (of course this would never work). Regardless of that, I still wanted to know what percentage of ruby gems space is old versions.

So I wrote a ruby program to find all the latest versions of the gem files and total up their size. I was not very happy about my experiments with #sort and #sort_by. The biggest problem is that it took 64 HOURS to run. I knew it had lots of problems but I didn’t want to kill it. I wanted to see how bad it really ran.

I’m not going to post the actual code. You can see the old version at this git commit url. The basic gist of the crappy algorithm was something like this:

Find all the files in the gem mirror off the filesystem.
Get the basename of the file name (ie: strip the path).  /tmp/foo-0.1.gem -> foo-0.1.gem
Go through all the basenames (gem names) find the gem family.

Here’s the problem. I had a massive list of 170k gems and then I’m trying to do a find_all right here to sort the gems into gem families. For example: there might be foo-0.1.gem, foo-0.2.gem and foo-async-0.1.gem. In this example, there are two gem families out of the three gems. Foo-async and foo are two different gems with their own versions. Later on, I would:

Do a version compare.
Push the latest version name to an array.
Delete the gem family name from the gem_names array.

Sounded good on paper. And then it took 65 hours to run (227305.19 seconds) and CPU was absolutely pegged the entire time. This algorithm was easy to come up with in IRB using a small test data set but scaling up in the real use case completely sucked. So I pushed it to github for versioning and rewrote the loop.

The latest version runs in 8.5 seconds and spits out a total size of all the latest ruby gems at 6.5GB. Of course, this information is useless since it’s not going to check compatibility or anything. I was just curious to know how much space is back versions.

The real key to the new version is the fact that I’m using a proper “grouped” data structure (Hash) instead of a massive flat Array. This allows the regexes and other operations to work on a smaller data set. The compound nature of the previous inefficiency is pretty amazing (hours to seconds).

So hopefully you see above that a huge array of a File glob is flat and makes regex’s or grouping operations very time consuming. Ruby’s magic group_by method sorts and groups the data structure once and then it’s much easier to regex out versions and do other things.

See below for the code inline or take a look at the github repo.

# refactored the 63 hour version to a much better 8.5 second version
 
require 'action_view'
include ActionView::Helpers
 
# change to location of rubygems mirror
GEM_DIR = "/opt/rubygems/gems"
 
gems = Dir.glob("#{GEM_DIR}/**/*.gem"); 1
gems = gems.collect {|g| g.split("/").last}; 
 
class Version
  include Comparable
  attr_reader :major, :feature_group, :feature, :bugfix, :version_string
 
  def initialize(version="")
    @version_string = version
    @major = "0"; @feature_group = "0"; @feature = "0"; @bugfix = "0"
 
    v = version.split(".")
    # puts v.join("|")
 
    if v[0]; @major = v[0]; else; raise "Major number blank."; end
    if v[1]; @feature_group = v[1]; end
    if v[2]; @feature = v[2]; end
    if v[3]; @bugfix = v[3]; end
  end
 
  # strangely enough .to_i works even for
  # >> "6-mswin32".to_i
  # => 6
  def <=>(other)
    return @major <=> other.major if ((@major.to_i <=> other.major.to_i) != 0)
    return @feature_group <=> other.feature_group if ((@feature_group.to_i <=> other.feature_group.to_i) != 0)
    return @feature <=> other.feature if ((@feature.to_i <=> other.feature.to_i) != 0)
    return @bugfix <=> other.bugfix if ((@bugfix.to_i <=> other.bugfix.to_i) != 0)
    # we probably have two things equal here
    return -1
    puts "FALLING THROUGH in <=>, not good"
  end
 
  def self.sort
    self.sort!{|a,b| a <=> b}
  end
 
  def to_s
    @version_string
  end
end
 
# temporary benchmarking
RubyProf.start
 
group_r = Regexp.new(/(.*)-(\d+\.\d+.*)\.gem$/)
gems_grouped = gems.group_by {|g| g.scan(group_r).flatten[0] }
# => {"firewool"=>["firewool-0.1.0.gem", "firewool-0.1.1.gem"}], ... }
 
latest_gems = []
 
gems_grouped.each do |g|
  versions = g[1].collect {|ver| ver.scan(group_r).flatten[1] }
  # => ["0.1.0", "0.1.1", "0.1.2"]
 
  begin
    latest = versions.collect {|v| Version.new(v)}.sort.reverse.first
    # => "0.1.2"
  rescue ArgumentError
    puts g
  rescue NoMethodError
    # somebody's got some crazy gem naming conventions
    # for example: chill-1.gem
    gems_grouped.delete g
  end
 
  latest_gems << "#{g[0]}-#{latest}.gem"
end
 
total = 0
latest_gems.each do |gem|
  begin
    total = total + File.size("#{GEM_DIR}/#{gem}")
  rescue Errno::ENOENT => e
    puts "WTF no #{gem}"
  end
 
end
 
puts "Total size of newest gems in #{GEM_DIR} is #{number_to_human_size(total)}"

Algorithm win. Rubygem mirror size curiosity complete. 6.5GB is current gems out of 45GB (right now).

A 20 minute Rails demo that I used as part of a “What is Ruby on Rails?” talk. The store was not designed or developed from scratch in 20 minutes but serves as a Cooking Show style demo of what is possible in a very short amount of time.

Quickie Mart on Github

• Set your Terminal theme to Basic. Make sure to re-set any preferences you might have (like no audible bell etc).
• Set your Terminal font to 13pt Inconsolata. This isn’t the exact font he uses but it’s as close as I could find.
• Set your ANSI Color for Normal White(7) to Tin (Crayons tab in color picker)
• Put this bash code from this gist at the end of your .profile file.
• Install rbenv so you don’t get the errors I got below because I’m still on RVM. :)

I covet, I steal.

Even though I don’t show it there, the bash script will do the git repo status magic for you on OSX. You need to brew install git and have the shell completion scripts in /usr/local (homebrew will do this).

Don Cheadle is the best version of Captain Planet there is. I hope you’ve already seen the video. If not, go now. I’ll wait.

As it typically happens, I was creating some nested data structure and was reminded of all the different combinations that there are. For example:

• An array of arrays.
  [ [1,2,3], [4,5,6] ]
• A hash of arrays.
  { :lucky => [77,42], :unlucky => [666,13] }
• An array of hashes.
  [ {:cat=>"meow"}, {:dog=>"ruff"} ]
• A hash of hashes.
  { :best_in_life => {:enemies => "crushed"}, :worst => { :meatloaf => "old"} }

And I realized that I hadn’t really played with a hash of hashes much. And now I realize why. It’s really pretty useless. It’s hard to work with and the additional key really isn’t all that useful. I found it much better to just denormalize the key into the data attributes. Anyway, you can see what I mean by reading and running what’s below.

We’re going to create Captain Planet and the planeteers in a 2D hash of hashes and do some searching, iterating and other simple things. This should illustrate also how an array of hashes is a bit better. You’ll see halfway through the program we redefine the planeteers.

# search a 2d hash
 
def spacer(msg)
  puts "\n"
  puts "-" * 50
  puts msg
  puts "-" * 50
end
 
# this is not really a good data structure but we'll use it anyway.
planeteers = {
  :kwame    => { :element => "earth", :from => "Ghana, Africa", :actor => "LeVar Burton" },
  :wheeler  => { :element => "fire", :from => "Brooklyn, NY", :actor => "Joey Dedio" },
  :linka    => { :element => "wind", :from => "Soviet Union", :actor => "Kath Soucie" },
  :gi       => { :element => "water", :from => "Thailand", :actor => "Janice Kawaye" },
  :ma_ti    => { :element => "heart", :from => "Brazil", :actor => "Scott Menville" }
}
 
spacer "Here are our planeteers and their elements:"
puts planeteers.keys.collect {|p| {p => planeteers[p][:element]} }
 
 
puts "\nFind the fire planeteer:\n"
fire = planeteers.keys.collect {|p| {p => planeteers[p][:element]=="fire"} }
# => [{:kwame=>false}, {:wheeler=>true}, {:linka=>false}, {:gi=>false}, {:ma_ti=>false}]
 
only_fire = fire.select{|h| h.values[0] == true}
# => {:wheeler=>true}
 
# print just one
puts only_fire.first.keys.first
 
 
spacer "Let's do this a bit cleaner with a better data structure."
planeteers = [
  { :name => "kwame", :element => "earth", :from => "Ghana, Africa", :actor => "LeVar Burton" },
  { :name => "wheeler", :element => "fire", :from => "Brooklyn, NY", :actor => "Joey Dedio" },
  { :name => "linka", :element => "wind", :from => "Soviet Union", :actor => "Kath Soucie" },
  { :name => "gi", :element => "water", :from => "Thailand", :actor => "Janice Kawaye" },
  { :name => "ma_ti", :element => "heart", :from => "Brazil", :actor => "Scott Menville" }
]
 
planeteers.max_by {|p| p[:name]}
# => {:name=>"ma_ti", :element=>"heart", :from=>"Brazil", :actor=>"Scott Menville"}
 
planeteers.max_by {|p| p[:element]}
# => {:name=>"linka", :element=>"wind", :from=>"Soviet Union", :actor=>"Kath Soucie"}
 
puts "Find the heart planeteer:"
puts planeteers.select {|p| p[:element] == "heart" }.first[:name]
 
# first we'll put a fake planeteer on the end for cpt planet
planeteers << { :name => "all", :element => "go planet" }
 
spacer "Let's summon Captain Planet!"
planeteers.each do |planeteer|
  puts "#{planeteer[:name].capitalize}: #{planeteer[:element].capitalize}!"
end
 
# pop off fake guy
planeteers.pop

Here’s what it spits out:

--------------------------------------------------
Here are our planeteers and their elements:
--------------------------------------------------
{:kwame=>"earth"}
{:wheeler=>"fire"}
{:linka=>"wind"}
{:gi=>"water"}
{:ma_ti=>"heart"}

Find the fire planeteer:
wheeler

--------------------------------------------------
Let's do this a bit cleaner with a better data structure.
--------------------------------------------------
Find the heart planeteer:
ma_ti

--------------------------------------------------
Let's summon Captain Planet!
--------------------------------------------------
Kwame: Earth!
Wheeler: Fire!
Linka: Wind!
Gi: Water!
Ma_ti: Heart!
All: Go planet!

First find a copy of wirble. If you don’t have it anywhere, download it from github:


cd ${hbase_home}/lib/ruby
wget https://raw.github.com/blackwinter/wirble/master/lib/wirble.rb


Now edit ${hbase_home}/bin/hirb.rb. Add to the end but above IRB.start

begin
  # load wirble
  require 'wirble'
 
  # start wirble (with color)
  Wirble.init
  Wirble.colorize
rescue LoadError => err
  warn "Couldn't load Wirble: #{err}"
end
 
IRB.conf[:HISTORY_FILE] = "#{ENV['HOME']}/.hbase-history"
 
# add right before end but above this line
IRB.start

Now when you start hbase shell, you’ll have lovely color output. Why would you want this? I don’t know. You probably don’t want it. But I was happy to understand how the hbase shell works. It’s just jruby irb that loads hirb automatically.

I’ve had this idea for schemaless data collection for a while now. It seems like everyone is trying to ETL data in. Inevitably, people start writing mappers programs and documentation trying to build a massive Rosetta stone. “They call person_name name? We’ll call everything name. Let’s write this all down. Person_name = name, so say we all.” What happens is a lot of investigation and work in determining their ENTIRE schema just to make a copy of it. In the case of XML parsing, sometimes I actually do need to know what the entire source looks like just so I can loop through it. What a pain. Not to mention if the source changes formats, I have to do all this work to re-understand their schema and change my mappers to reflect their change. Maybe I even have to do a mass migration on my end to bring everything up to date. Schemaless data collection will let you copy the data when the source changes and even be able to historically tell you when the schema changed. In an RDBMS, this is impossible without blobs or something else horrible.

What this example shows is simply the collection of the data. But the advantage here, I will show that any kind of querying can be done later very easily. What I won’t show is that normalization can be done in parallel and in batches later too and the whole thing lives in a horizontally scalable database. Of course, the catch is, you need keys and a structured data source to start with. This won’t work with CSV and simple formats.

For example, if we wanted to load XML files into a database. MongoDB is great for this because it can possibly make coding dead simple. For each attribute and children, create attributes and children in the database.

In a normal database, I would have to parse out the XML file and create normalized rows all over the place or use blobs. Of course blobs are useless for search. Let me show you an example.

First, let’s take a look at the XML returned by a Wikipedia exporter URL (trimmed the XSD line a bit for readability):

<mediawiki xmlns="http://www.mediawiki.org/xml/export-0.5/" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.5.xsd" version="0.5" xml:lang="en">
  <siteinfo>
    <sitename>Wikipedia</sitename>
    <base>http://en.wikipedia.org/wiki/Main_Page</base>
    <generator>MediaWiki 1.18wmf1</generator>
    <case>first-letter</case>
    <namespaces>
      <namespace key="-2" case="first-letter">Media</namespace>
      <namespace key="-1" case="first-letter">Special</namespace>
      <namespace key="0" case="first-letter" />
      <namespace key="1" case="first-letter">Talk</namespace>
      <namespace key="2" case="first-letter">User</namespace>
      <namespace key="3" case="first-letter">User talk</namespace>
      <namespace key="4" case="first-letter">Wikipedia</namespace>
      <namespace key="5" case="first-letter">Wikipedia talk</namespace>
      <namespace key="6" case="first-letter">File</namespace>
      <namespace key="7" case="first-letter">File talk</namespace>
      <namespace key="8" case="first-letter">MediaWiki</namespace>
      <namespace key="9" case="first-letter">MediaWiki talk</namespace>
      <namespace key="10" case="first-letter">Template</namespace>
      <namespace key="11" case="first-letter">Template talk</namespace>
      <namespace key="12" case="first-letter">Help</namespace>
      <namespace key="13" case="first-letter">Help talk</namespace>
      <namespace key="14" case="first-letter">Category</namespace>
      <namespace key="15" case="first-letter">Category talk</namespace>
      <namespace key="100" case="first-letter">Portal</namespace>
      <namespace key="101" case="first-letter">Portal talk</namespace>
      <namespace key="108" case="first-letter">Book</namespace>
      <namespace key="109" case="first-letter">Book talk</namespace>
    </namespaces>
  </siteinfo>
  <page>
    <title>Ford Motor</title>
    <id>255240</id>
    <redirect />
    <revision>
      <id>16130856</id>
      <timestamp>2003-06-30T02:32:03Z</timestamp>
      <contributor>
        <username>Infrogmation</username>
        <id>4444</id>
      </contributor>
      <minor/>
      <comment>redir</comment>
      <text xml:space="preserve" bytes="32">#REDIRECT [[Ford Motor Company]]</text>
    </revision>
  </page>
</mediawiki>

Even though this page is just a redirect (see the text element at the end in mediawiki markup), it’s still very long. For a multitude of posts or documents, creating a mapper and tightly handling the document might be very annoying. Even worse, we might delay sucking data in because there is so much mapping work to do. We might even start writing documents detailing the source format, what we will call the attributes internally and document schema changes as they occur.

What I propose is to forget all that mapping and just load the document as-is. We will use a document database (MongoDB) to make this magic happen.

# we are going to intentionally use the vanilla mongo driver
require 'mongo'
require 'nokogiri'
require 'open-uri'
require 'active_support/core_ext' # from rails
 
include Mongo
pages = Connection.new('localhost', 27017).db('loadtest').collection('pages')
 
wikipedia_page = "http://en.wikipedia.org/wiki/Special:Export/Ford_Motor"
 
# noblanks is magic here?  had problems without it
doc = Nokogiri::XML(open(wikipedia_page)) { |config| config.noblanks }
page = Hash.from_xml(doc.to_s)    # here's the magical method from rails
pages.insert page

Ok this is pretty cool. In 10 lines of ruby, I’m downloading an XML file and inserting it into a new collection called ‘pages’ in a database that hasn’t even been created (as long as mongodb is running). Great! But it quickly falls apart.

If you run it again, you now have two documents (rows) in Mongo. Boo. Not to mention, if you actually query mongo you see this (trimmed the XSD line a bit for readability):

> db.pages.find()
{
   "_id":ObjectId("4ec6c07e5a498d64a1000001"),
   "mediawiki":{
      "xmlns":"http://www.mediawiki.org/xml/export-0.5/",
      "xmlns:xsi":"http://www.w3.org/2001/XMLSchema-instance",
      "xsi:schemaLocation":"http://www.mediawiki.org/xml/export-0.5.xsd",
      "version":"0.5",
      "xml:lang":"en",
      "siteinfo":{
         "sitename":"Wikipedia",
         "base":"http://en.wikipedia.org/wiki/Main_Page",
         "generator":"MediaWiki 1.18wmf1",
         "case":"first-letter",
         "namespaces":{
            "namespace":[
               "Media",
               "Special",
               {
                  "key":"0",
                  "case":"first-letter"
               },
               "Talk",
               "User",
               "User talk",
               "Wikipedia",
               "Wikipedia talk",
               "File",
               "File talk",
               "MediaWiki",
               "MediaWiki talk",
               "Template",
               "Template talk",
               "Help",
               "Help talk",
               "Category",
               "Category talk",
               "Portal",
               "Portal talk",
               "Book",
               "Book talk"
            ]
         }
      },
      "page":{
         "title":"Ford Motor",
         "id":"255240",
         "redirect":null,
         "revision":{
            "id":"16130856",
            "timestamp":"2003-06-30T02:32:03Z",
            "contributor":{
               "username":"Infrogmation",
               "id":"4444"
            },
            "minor":null,
            "comment":"redir",
            "text":"#REDIRECT [[Ford Motor Company]]"
         }
      }
   }
}

There’s a whole lot of metadata in there and really all I care about is the content (maybe). So in some cases, you might want to filter incoming data. I have to actually look at my data and pick which attributes I want. Now I’m tightly bound to the source document and have to worry about it changing etc.

But let’s do it anyway. All we have to do is change one line:

pages.insert page["mediawiki"]["page"]

Now our inserted document looks like this:

> db.pages.find()
{
   "_id":ObjectId("4ec6c1775a498d64f2000001"),
   "title":"Ford Motor",
   "id":"255240",
   "redirect":null,
   "revision":{
      "id":"16130856",
      "timestamp":"2003-06-30T02:32:03Z",
      "contributor":{
         "username":"Infrogmation",
         "id":"4444"
      },
      "minor":null,
      "comment":"redir",
      "text":"#REDIRECT [[Ford Motor Company]]"
   }
}

Of course, we had to clear out the pages collection and re-run it. That’s just because we haven’t written any logic yet to check for existence yet. But let’s take a break here and talk about what we could do even with this piddly little bit of 10 lines of Ruby running. We can query:

> db.pages.find({}, {'title':1} )
{ "_id" : ObjectId("4ec6c1775a498d64f2000001"), "title" : "Ford Motor" }
{ "_id" : ObjectId("4ecafffd5a498d0136000001"), "title" : "Nissan" }

Here we are just showing that we have two pages from Wikipedia stored. The title:1 is like SELECT title FROM pages; in SQL. So if we wanted to search on attributes, it’s pretty easy:

> db.pages.find({ title:/^N/ }, {title:1})
{ "_id" : ObjectId("4ecafffd5a498d0136000001"), "title" : "Nissan" }

It’s pretty forgiving on the key quotes.

In the next part, we’ll dive into handling updates and other formats.

guard 'rspec', :version => 2, :cli => '--drb --color --format doc' do
  watch(%r{^spec/.+_spec\.rb$})
  ...
end

You can specify multiple formats with --format one --format two.

Anyway, here are some shots of what the output looks like:

--format doc

--format progress

--format nested --format progress

--format nested

--format html

HTML format is also the same as the Textmate format. I couldn’t get the output to go to a file like the documentation says. Maybe it hasn’t been updated for rspec2?

We’ll start with a simple person class that is initialized with a name and an age.

class Person
  attr_accessor :name, :age, :problems
 
  def initialize(name, age)
    @name = name
    @age = age
  end
end

Creating a person is as simple as passing “James” and 99 as arguments.

puts Person.new("James", 99).inspect
# => #<Person:0x007f9e2a136878 @name="James", @age=99>

Now I have a Person object as expected. The twist comes in when you look at this line by itself and realize that you can’t get what 99 is. Is it the age? Is it the problems? Of course, you might opt to simply get rid of the constructor and set the instance variables manually. But let’s try to do something more fancy.

First, we’ll try to invoke a method that doesn’t exist.

begin
  Person.create_with_name_and_problems("James", 99)
  # => undefined method `create_with_name_and_problems'
  #    for Person:Class (NoMethodError)
rescue NoMethodError
  # just continue
end

We will get an exception here. For the sake of our single source file, we’ll catch the exception and continue.

So when we try to call #create_with_name_and_problems so that 99 is clearly a problem and not an age argument, the method doesn’t exist. We could create that method but that’s not very scalable, we’d have to create every permutation of possible construction options.

Instead what we are going to do is use method_missing to handle calls to unknown methods and at the same time set the instance variables and return an object.

class Person
  def initialize
  end
 
  def self.method_missing(meth, *args, &block)
    puts "OH NO!  No method!"
    if meth.to_s =~ /^create_with_(.+)$/
      self.run_create_with_method($1, *args, &block)
    else
      super
    end
  end
 
  def respond_to?(meth, *args, &block)
    if self.meth.to_s =~ /^create_with_.*$/
      true
    else
      super
    end
  end
 
  def self.run_create_with_method(attrs, *args, &block)
    attrs = attrs.split('_and_')
    # #transpose will zip the two arrays together like so:
    #   [[:a, :b, :c], [1, 2, 3]].transpose
    #   # => [[:a, 1], [:b, 2], [:c, 3]]
    attrs_with_args = [attrs, args].transpose
    attributes = Hash[attrs_with_args]
    p = Person.new
    attributes.keys.each do |a|
      p.instance_variable_set "@#{a}", attributes[a]
    end
    return p
  end
 
end

First we reopen the Person class (monkey patch) and redefine a parameter-less initialize method. Next we create a method_missing method on the class object that looks for any method that starts with “create_with_”. If it does then it creates a new object with the correct instance variables set. Finally, the respond_to? method ensures that our Person class is advertising that #create_with_ methods are valid to outside calls.

Ok, so now our Person object is ready to be used. We can create James again this time with a name and a number of problems. We can even create a person with all three attributes and vary the order.

puts Person.create_with_name_and_problems("James",99).inspect
# <Person:0x007ffc8a835250 @name="James", @problems=99>
 
puts Person.create_with_age_and_problems_and_name(55, 99, "Jay-Z").inspect
# <Person:0x007ffc8b0ae990 @name="Jay-Z", @age=55, @problems=99>

So in actuality, this is a bit contrived. It’s cool to have these dynamic methods created for us but doing this way is a little too much work just to get parameterized constructors. The better way would be to use a hash for initialization. See below:

class Person
  attr_reader :name, :age, :problems
 
  def initialize args
    args.each do |k,v|
      instance_variable_set("@#{k}", v) unless v.nil?
    end
  end
end
 
p = Person.new(:name => "James", :age => 99, :problems => 99)
# <Person:0x007fda421044f0 @name="James", @age=99, @problems=99>
 
puts p.name       # James
puts p.age        # 99
puts p.problems   # 99

First let’s create a rails app and an rvm gemset to play in.

rvm use 1.9.2@memcache --create

If you don’t have rails in your global gemset, you can install rails now. I used the 3.1 RC here just to test out 3.1 and memcached at the same time.

gem install rails --pre
gem install dalli

Dalli is written by Mike Perham (awesome guy) and simply wires up Rails.cache to be our memcached server. This is convenient and more standard than creating your own global variable or other configuration.

Next, let’s install memcached if we haven’t already:

brew install memcached
memcached -v

You can also start memcached as a service under Mac using the homebrew instructions. I like to leave things in the foreground when I’m first setting them up or grok’ing.

Ok, now we can create our dummy app.

rails new memcache_test

Our dummy rails app is going to have a slow controller action in it. In this case, it could be a row count of a huge database. In your case it could be a slow network call or a result of an expensive SQL operation.

First, edit your Gemfile:

gem 'dalli'

Edit config/environments/development.rb

# Memcached
  config.perform_caching = true
  config.action_controller.perform_caching = true
  config.cache_store = :dalli_store, 'localhost:11211'

Rails c should work at this point and this line:

Rails.cache.read('foo')
=> nil

Now generate a controller:

rails g controller posts

class PostsController < ApplicationController
  def index
    @posts_count = Rails.cache.read 'posts_count'
    if @posts_count
    else
      # expensive operation
      sleep 3
      @posts_count = 321
      Rails.cache.write('posts_count', @posts_count, :expires_in => 3)
    end
  end
end

Create a route in routes.rb:

resources :posts

Create a simple view in app/views/posts/index.json.erb:

{ posts: <%= @posts_count %> }

Now when you hit the page it should be slow the first time but fast the second time. After 3 seconds, it’s slow again.

$ time wget -O - localhost:3000/posts.json
HTTP request sent, awaiting response... 200 OK
Length: 14 [application/json]
real	0m3.049s
user	0m0.001s
sys	0m0.003s

$ time wget -O - localhost:3000/posts.json
HTTP request sent, awaiting response... 200 OK
Length: 14 [application/json]
real	0m0.049s
user	0m0.001s
sys	0m0.003s

So you can see that memcached returned the cached object and avoided the sleep call. In the real world, this would equate to lower page rendering time or better application performance.

-------------------------------------
DJ Shadow       |  electronica
DJ Shadow       |  1996
DJ Shadow       |  Endtroducing
The Avalanches  |  Since I Left You
The Avalanches  |  2000
The Avalanches  |  electronica
-------------------------------------

There are really only two CDs here but the data is presented to us as artist:attribute pairs. We want the data to look like this:

Artist         | Genre       | Album            | Year
------------------------------------------------------
DJ Shadow      | electronica | Entroducing      | 1996
The Avalanches | electronica | Since I Left You | 2000

First Example

First of all, the starting data (the first key value pair listing) doesn’t have the data labeled so this creates a problem. We have to assume that the key is the artist and label that accordingly. After that, the attributes are not labeled so we’re not going to be able to create the column headings without more information. So I’ll show two examples here, one with unlabeled data and the second with proper labeling.

The first example makes an assumption that the order of the attributes is significant. IE: album, year, genre. We have to make this assumption for the pretty table printing. If this is not the case, the logical impact is minimal and you can ignore this.

# FIRST EXAMPLE where data is associated by unique strings but
# not explicit with keys or labels, for example :year => 1996 is not explicit
unpivoted_data = [
  { "DJ Shadow" => "Endtroducing" },
  { "DJ Shadow" => "1996" },
  { "DJ Shadow" => "electronica" },
  { "The Avalanches" => "Since I Left You" },
  { "The Avalanches" => "2000" },
  { "The Avalanches" => "electronica" }
]
 
# make new array
pivoted = Hash.new
 
unpivoted_data.each do |d|
  # key of each hash element should be only one element long
  key = d.keys.first.to_sym
 
  if pivoted[key] == nil
    pivoted[key] = Array.new
  end
 
  pivoted[key] << d.values.first
 
end
 
# The variable pivoted at this point looks like this:
# {:"DJ Shadow"=>["electronica", "1996", "Endtroducing"], 
# :"The Avalanches"=>["Since I Left You", "2000", "electronica"]}
 
 
# pretty table print
# first pass, determine biggest columns and store
 
# biggest size of key
max_lengths = []
pivoted.keys.each do |key|
  if max_lengths[0].nil? || key.length > max_lengths[0]
    max_lengths[0] = key.length
  end
end
 
# biggest size of values array inside key
pivoted.keys.each do |element|
  # we already have max_lengths[0] from key
  i = 1
  pivoted[element].each do |subelement|
   if max_lengths[i].nil? || subelement.length > max_lengths[i]
     max_lengths[i] = subelement.length
   end
   i += 1
  end
end
 
puts "Pivoted Unlabeled:"
line_size = max_lengths.inject{|sum,x| sum + x } + max_lengths.length * 3 - 1
puts "-" * line_size
 
# pad columns with spaces and bars from max_lengths
pivoted.keys.each do |key|
  string = key.to_s.ljust(max_lengths[0])
  string << " | "
 
  i = 1
  pivoted[key].each do |attribute|
    string << attribute.ljust(max_lengths[i]) << " | "
    i += 1
  end
 
  # string = key.to_s << " | " << pivoted[key].join(" | ")
  puts string
end
 
# spacer
puts ""

This code will output:

Pivoted Unlabeled:
--------------------------------------------------------
DJ Shadow      | Endtroducing     | 1996 | electronica |
The Avalanches | Since I Left You | 2000 | electronica |

Nice bit of pretty printing there. Completely unnecessary but gives a nice feel of a pivot table even if we are not working with a database or activerecord.

Second Example

The second example of pivoting data involves a data structured that is related by a primary key and each subelement has a field with a name. This could be a 2D data structure from a database or anything that is highly structured and organized. The ID field in this case is barely significant. It just acts as a label, in this example it is the ID of the album as pulled from freedb.org.

This code takes a different approach of creating a hash of hashes to pivot the data. After that, it creates a 2D array for tablular printing. The maximum column size is computed for pretty printing.

# SECOND EXAMPLE where data is labeled.  This is easier / cleaner.
# IDs represent CDs pulled from freedb.org
unpivoted_data = [
  { :freedb => "a70eb30d", :artist => "DJ Shadow" },
  { :freedb => "a70eb30d", :genre => "electronica" },
  { :freedb => "a70eb30d", :year => 1996 },
  { :freedb => "a70eb30d", :album => "Endtroducing" },
  { :freedb => "090e6012", :artist => "The Avalanches" },
  { :freedb => "090e6012", :genre => "electronica" },
  { :freedb => "090e6012", :year => 2000 },
  { :freedb => "090e6012", :album => "Since I Left You" }
]
 
# hash of hashes
pivoted = Hash.new
 
unpivoted_data.each do |d|
  # store the unique ID so we can remember if we've seen the CD before
  id = d.values.first.to_sym
 
  # attribute to merge
  attribute_key = d.keys[1]
  attribute = d[attribute_key]
 
  # if we haven't seen the CD, create an empty attribute hash
  if !pivoted.keys.include?(id)
    pivoted[id] = Hash.new
  end
 
  # add the new attribute
  pivoted[id][attribute_key] = attribute
end
 
# make data into 2d array
table = Array.new
pivoted.keys.each do |e|
  heading = Array.new
  heading << "ID"
  pivoted[e].keys.each do |sub_e|
    heading << sub_e.to_s.upcase
  end
  table << heading
  # only need first row of heading
  break
end
 
pivoted.keys.each do |e|
  row = Array.new
  row << e
  pivoted[e].values.each do |sub_e|
    row << sub_e
  end
  table << row
end
 
# biggest size of key
max_lengths = []
table.each_with_index do |x|
  i = 0
  x.each_with_index do |y|
    if max_lengths[i].nil? || y.to_s.length > max_lengths[i]
      max_lengths[i] = y.to_s.length
    end
    i += 1
  end
end
 
puts "Pivoted Labeled:"
line_size = max_lengths.inject{|sum,x| sum + x } + max_lengths.length * 3 - 1
puts "-" * line_size
 
# pad columns with spaces and bars from max_lengths
table.each_with_index do |x|
  string = ""
  i = 0
  x.each_with_index do |y|
    string << y.to_s.ljust(max_lengths[i]) << " | "
    i += 1
  end
 
  puts string
end
 
# spacer
puts ""

This code will produce this text:

Pivoted Labeled:
-------------------------------------------------------------------
ID       | ARTIST         | GENRE       | YEAR | ALBUM            |
a70eb30d | DJ Shadow      | electronica | 1996 | Endtroducing     |
090e6012 | The Avalanches | electronica | 2000 | Since I Left You |

Conclusion

Both examples are very procedural. There might be opportunities to make this code more OO or even functional by breaking up common tasks. However, the problem I ran into was of the structure itself. I don’t see a generic way of creating code reuse except for creating methods that can handle a “hash of hashes” or other pre-defined types. I wanted to post a copy and paste template that would help anyone out there trying to pivot data (or even find this name for a problem like this) but unfortunately the solutions are tightly bound to the data structure you are starting out with. In the case of database rows, example #1 will probably work for you with minimal changes.