After a massive initial load of 155k gems, the size was about 45GB (currently, it grows pretty quick per week). The rubygem and gem mirror command is smart enough to just download just the deltas when you run it again:


$ gem mirror
Fetching: http://rubygems.org/specs.4.8.gz
Total gems: 170843
Fetching 16176 gems
................................................................


Then I wanted to know the size of all the latest gems only. If I had to do a lazy sneakernet, this might be one method of grabbing a whole bunch of dependencies (of course this would never work). Regardless of that, I still wanted to know what percentage of ruby gems space is old versions.

So I wrote a ruby program to find all the latest versions of the gem files and total up their size. I was not very happy about my experiments with #sort and #sort_by. The biggest problem is that it took 64 HOURS to run. I knew it had lots of problems but I didn’t want to kill it. I wanted to see how bad it really ran.

I’m not going to post the actual code. You can see the old version at this git commit url. The basic gist of the crappy algorithm was something like this:

Find all the files in the gem mirror off the filesystem.
Get the basename of the file name (ie: strip the path).  /tmp/foo-0.1.gem -> foo-0.1.gem
Go through all the basenames (gem names) find the gem family.

Here’s the problem. I had a massive list of 170k gems and then I’m trying to do a find_all right here to sort the gems into gem families. For example: there might be foo-0.1.gem, foo-0.2.gem and foo-async-0.1.gem. In this example, there are two gem families out of the three gems. Foo-async and foo are two different gems with their own versions. Later on, I would:

Do a version compare.
Push the latest version name to an array.
Delete the gem family name from the gem_names array.

Sounded good on paper. And then it took 65 hours to run (227305.19 seconds) and CPU was absolutely pegged the entire time. This algorithm was easy to come up with in IRB using a small test data set but scaling up in the real use case completely sucked. So I pushed it to github for versioning and rewrote the loop.

The latest version runs in 8.5 seconds and spits out a total size of all the latest ruby gems at 6.5GB. Of course, this information is useless since it’s not going to check compatibility or anything. I was just curious to know how much space is back versions.

The real key to the new version is the fact that I’m using a proper “grouped” data structure (Hash) instead of a massive flat Array. This allows the regexes and other operations to work on a smaller data set. The compound nature of the previous inefficiency is pretty amazing (hours to seconds).

So hopefully you see above that a huge array of a File glob is flat and makes regex’s or grouping operations very time consuming. Ruby’s magic group_by method sorts and groups the data structure once and then it’s much easier to regex out versions and do other things.

See below for the code inline or take a look at the github repo.

# refactored the 63 hour version to a much better 8.5 second version
 
require 'action_view'
include ActionView::Helpers
 
# change to location of rubygems mirror
GEM_DIR = "/opt/rubygems/gems"
 
gems = Dir.glob("#{GEM_DIR}/**/*.gem"); 1
gems = gems.collect {|g| g.split("/").last}; 
 
class Version
  include Comparable
  attr_reader :major, :feature_group, :feature, :bugfix, :version_string
 
  def initialize(version="")
    @version_string = version
    @major = "0"; @feature_group = "0"; @feature = "0"; @bugfix = "0"
 
    v = version.split(".")
    # puts v.join("|")
 
    if v[0]; @major = v[0]; else; raise "Major number blank."; end
    if v[1]; @feature_group = v[1]; end
    if v[2]; @feature = v[2]; end
    if v[3]; @bugfix = v[3]; end
  end
 
  # strangely enough .to_i works even for
  # >> "6-mswin32".to_i
  # => 6
  def <=>(other)
    return @major <=> other.major if ((@major.to_i <=> other.major.to_i) != 0)
    return @feature_group <=> other.feature_group if ((@feature_group.to_i <=> other.feature_group.to_i) != 0)
    return @feature <=> other.feature if ((@feature.to_i <=> other.feature.to_i) != 0)
    return @bugfix <=> other.bugfix if ((@bugfix.to_i <=> other.bugfix.to_i) != 0)
    # we probably have two things equal here
    return -1
    puts "FALLING THROUGH in <=>, not good"
  end
 
  def self.sort
    self.sort!{|a,b| a <=> b}
  end
 
  def to_s
    @version_string
  end
end
 
# temporary benchmarking
RubyProf.start
 
group_r = Regexp.new(/(.*)-(\d+\.\d+.*)\.gem$/)
gems_grouped = gems.group_by {|g| g.scan(group_r).flatten[0] }
# => {"firewool"=>["firewool-0.1.0.gem", "firewool-0.1.1.gem"}], ... }
 
latest_gems = []
 
gems_grouped.each do |g|
  versions = g[1].collect {|ver| ver.scan(group_r).flatten[1] }
  # => ["0.1.0", "0.1.1", "0.1.2"]
 
  begin
    latest = versions.collect {|v| Version.new(v)}.sort.reverse.first
    # => "0.1.2"
  rescue ArgumentError
    puts g
  rescue NoMethodError
    # somebody's got some crazy gem naming conventions
    # for example: chill-1.gem
    gems_grouped.delete g
  end
 
  latest_gems << "#{g[0]}-#{latest}.gem"
end
 
total = 0
latest_gems.each do |gem|
  begin
    total = total + File.size("#{GEM_DIR}/#{gem}")
  rescue Errno::ENOENT => e
    puts "WTF no #{gem}"
  end
 
end
 
puts "Total size of newest gems in #{GEM_DIR} is #{number_to_human_size(total)}"

Algorithm win. Rubygem mirror size curiosity complete. 6.5GB is current gems out of 45GB (right now).

Anyway, here’s a screenshot to remind me of the captcha fun.

A 20 minute Rails demo that I used as part of a “What is Ruby on Rails?” talk. The store was not designed or developed from scratch in 20 minutes but serves as a Cooking Show style demo of what is possible in a very short amount of time.

Quickie Mart on Github

• Set your Terminal theme to Basic. Make sure to re-set any preferences you might have (like no audible bell etc).
• Set your Terminal font to 13pt Inconsolata. This isn’t the exact font he uses but it’s as close as I could find.
• Set your ANSI Color for Normal White(7) to Tin (Crayons tab in color picker)
• Put this bash code from this gist at the end of your .profile file.
• Install rbenv so you don’t get the errors I got below because I’m still on RVM. :)

I covet, I steal.

Even though I don’t show it there, the bash script will do the git repo status magic for you on OSX. You need to brew install git and have the shell completion scripts in /usr/local (homebrew will do this).

Don Cheadle is the best version of Captain Planet there is. I hope you’ve already seen the video. If not, go now. I’ll wait.

As it typically happens, I was creating some nested data structure and was reminded of all the different combinations that there are. For example:

• An array of arrays.
  [ [1,2,3], [4,5,6] ]
• A hash of arrays.
  { :lucky => [77,42], :unlucky => [666,13] }
• An array of hashes.
  [ {:cat=>"meow"}, {:dog=>"ruff"} ]
• A hash of hashes.
  { :best_in_life => {:enemies => "crushed"}, :worst => { :meatloaf => "old"} }

And I realized that I hadn’t really played with a hash of hashes much. And now I realize why. It’s really pretty useless. It’s hard to work with and the additional key really isn’t all that useful. I found it much better to just denormalize the key into the data attributes. Anyway, you can see what I mean by reading and running what’s below.

We’re going to create Captain Planet and the planeteers in a 2D hash of hashes and do some searching, iterating and other simple things. This should illustrate also how an array of hashes is a bit better. You’ll see halfway through the program we redefine the planeteers.

# search a 2d hash
 
def spacer(msg)
  puts "\n"
  puts "-" * 50
  puts msg
  puts "-" * 50
end
 
# this is not really a good data structure but we'll use it anyway.
planeteers = {
  :kwame    => { :element => "earth", :from => "Ghana, Africa", :actor => "LeVar Burton" },
  :wheeler  => { :element => "fire", :from => "Brooklyn, NY", :actor => "Joey Dedio" },
  :linka    => { :element => "wind", :from => "Soviet Union", :actor => "Kath Soucie" },
  :gi       => { :element => "water", :from => "Thailand", :actor => "Janice Kawaye" },
  :ma_ti    => { :element => "heart", :from => "Brazil", :actor => "Scott Menville" }
}
 
spacer "Here are our planeteers and their elements:"
puts planeteers.keys.collect {|p| {p => planeteers[p][:element]} }
 
 
puts "\nFind the fire planeteer:\n"
fire = planeteers.keys.collect {|p| {p => planeteers[p][:element]=="fire"} }
# => [{:kwame=>false}, {:wheeler=>true}, {:linka=>false}, {:gi=>false}, {:ma_ti=>false}]
 
only_fire = fire.select{|h| h.values[0] == true}
# => {:wheeler=>true}
 
# print just one
puts only_fire.first.keys.first
 
 
spacer "Let's do this a bit cleaner with a better data structure."
planeteers = [
  { :name => "kwame", :element => "earth", :from => "Ghana, Africa", :actor => "LeVar Burton" },
  { :name => "wheeler", :element => "fire", :from => "Brooklyn, NY", :actor => "Joey Dedio" },
  { :name => "linka", :element => "wind", :from => "Soviet Union", :actor => "Kath Soucie" },
  { :name => "gi", :element => "water", :from => "Thailand", :actor => "Janice Kawaye" },
  { :name => "ma_ti", :element => "heart", :from => "Brazil", :actor => "Scott Menville" }
]
 
planeteers.max_by {|p| p[:name]}
# => {:name=>"ma_ti", :element=>"heart", :from=>"Brazil", :actor=>"Scott Menville"}
 
planeteers.max_by {|p| p[:element]}
# => {:name=>"linka", :element=>"wind", :from=>"Soviet Union", :actor=>"Kath Soucie"}
 
puts "Find the heart planeteer:"
puts planeteers.select {|p| p[:element] == "heart" }.first[:name]
 
# first we'll put a fake planeteer on the end for cpt planet
planeteers << { :name => "all", :element => "go planet" }
 
spacer "Let's summon Captain Planet!"
planeteers.each do |planeteer|
  puts "#{planeteer[:name].capitalize}: #{planeteer[:element].capitalize}!"
end
 
# pop off fake guy
planeteers.pop

Here’s what it spits out:

--------------------------------------------------
Here are our planeteers and their elements:
--------------------------------------------------
{:kwame=>"earth"}
{:wheeler=>"fire"}
{:linka=>"wind"}
{:gi=>"water"}
{:ma_ti=>"heart"}

Find the fire planeteer:
wheeler

--------------------------------------------------
Let's do this a bit cleaner with a better data structure.
--------------------------------------------------
Find the heart planeteer:
ma_ti

--------------------------------------------------
Let's summon Captain Planet!
--------------------------------------------------
Kwame: Earth!
Wheeler: Fire!
Linka: Wind!
Gi: Water!
Ma_ti: Heart!
All: Go planet!

First find a copy of wirble. If you don’t have it anywhere, download it from github:


cd ${hbase_home}/lib/ruby
wget https://raw.github.com/blackwinter/wirble/master/lib/wirble.rb


Now edit ${hbase_home}/bin/hirb.rb. Add to the end but above IRB.start

begin
  # load wirble
  require 'wirble'
 
  # start wirble (with color)
  Wirble.init
  Wirble.colorize
rescue LoadError => err
  warn "Couldn't load Wirble: #{err}"
end
 
IRB.conf[:HISTORY_FILE] = "#{ENV['HOME']}/.hbase-history"
 
# add right before end but above this line
IRB.start

Now when you start hbase shell, you’ll have lovely color output. Why would you want this? I don’t know. You probably don’t want it. But I was happy to understand how the hbase shell works. It’s just jruby irb that loads hirb automatically.

I’ve had this idea for schemaless data collection for a while now. It seems like everyone is trying to ETL data in. Inevitably, people start writing mappers programs and documentation trying to build a massive Rosetta stone. “They call person_name name? We’ll call everything name. Let’s write this all down. Person_name = name, so say we all.” What happens is a lot of investigation and work in determining their ENTIRE schema just to make a copy of it. In the case of XML parsing, sometimes I actually do need to know what the entire source looks like just so I can loop through it. What a pain. Not to mention if the source changes formats, I have to do all this work to re-understand their schema and change my mappers to reflect their change. Maybe I even have to do a mass migration on my end to bring everything up to date. Schemaless data collection will let you copy the data when the source changes and even be able to historically tell you when the schema changed. In an RDBMS, this is impossible without blobs or something else horrible.

What this example shows is simply the collection of the data. But the advantage here, I will show that any kind of querying can be done later very easily. What I won’t show is that normalization can be done in parallel and in batches later too and the whole thing lives in a horizontally scalable database. Of course, the catch is, you need keys and a structured data source to start with. This won’t work with CSV and simple formats.

For example, if we wanted to load XML files into a database. MongoDB is great for this because it can possibly make coding dead simple. For each attribute and children, create attributes and children in the database.

In a normal database, I would have to parse out the XML file and create normalized rows all over the place or use blobs. Of course blobs are useless for search. Let me show you an example.

First, let’s take a look at the XML returned by a Wikipedia exporter URL (trimmed the XSD line a bit for readability):

<mediawiki xmlns="http://www.mediawiki.org/xml/export-0.5/" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.5.xsd" version="0.5" xml:lang="en">
  <siteinfo>
    <sitename>Wikipedia</sitename>
    <base>http://en.wikipedia.org/wiki/Main_Page</base>
    <generator>MediaWiki 1.18wmf1</generator>
    <case>first-letter</case>
    <namespaces>
      <namespace key="-2" case="first-letter">Media</namespace>
      <namespace key="-1" case="first-letter">Special</namespace>
      <namespace key="0" case="first-letter" />
      <namespace key="1" case="first-letter">Talk</namespace>
      <namespace key="2" case="first-letter">User</namespace>
      <namespace key="3" case="first-letter">User talk</namespace>
      <namespace key="4" case="first-letter">Wikipedia</namespace>
      <namespace key="5" case="first-letter">Wikipedia talk</namespace>
      <namespace key="6" case="first-letter">File</namespace>
      <namespace key="7" case="first-letter">File talk</namespace>
      <namespace key="8" case="first-letter">MediaWiki</namespace>
      <namespace key="9" case="first-letter">MediaWiki talk</namespace>
      <namespace key="10" case="first-letter">Template</namespace>
      <namespace key="11" case="first-letter">Template talk</namespace>
      <namespace key="12" case="first-letter">Help</namespace>
      <namespace key="13" case="first-letter">Help talk</namespace>
      <namespace key="14" case="first-letter">Category</namespace>
      <namespace key="15" case="first-letter">Category talk</namespace>
      <namespace key="100" case="first-letter">Portal</namespace>
      <namespace key="101" case="first-letter">Portal talk</namespace>
      <namespace key="108" case="first-letter">Book</namespace>
      <namespace key="109" case="first-letter">Book talk</namespace>
    </namespaces>
  </siteinfo>
  <page>
    <title>Ford Motor</title>
    <id>255240</id>
    <redirect />
    <revision>
      <id>16130856</id>
      <timestamp>2003-06-30T02:32:03Z</timestamp>
      <contributor>
        <username>Infrogmation</username>
        <id>4444</id>
      </contributor>
      <minor/>
      <comment>redir</comment>
      <text xml:space="preserve" bytes="32">#REDIRECT [[Ford Motor Company]]</text>
    </revision>
  </page>
</mediawiki>

Even though this page is just a redirect (see the text element at the end in mediawiki markup), it’s still very long. For a multitude of posts or documents, creating a mapper and tightly handling the document might be very annoying. Even worse, we might delay sucking data in because there is so much mapping work to do. We might even start writing documents detailing the source format, what we will call the attributes internally and document schema changes as they occur.

What I propose is to forget all that mapping and just load the document as-is. We will use a document database (MongoDB) to make this magic happen.

# we are going to intentionally use the vanilla mongo driver
require 'mongo'
require 'nokogiri'
require 'open-uri'
require 'active_support/core_ext' # from rails
 
include Mongo
pages = Connection.new('localhost', 27017).db('loadtest').collection('pages')
 
wikipedia_page = "http://en.wikipedia.org/wiki/Special:Export/Ford_Motor"
 
# noblanks is magic here?  had problems without it
doc = Nokogiri::XML(open(wikipedia_page)) { |config| config.noblanks }
page = Hash.from_xml(doc.to_s)    # here's the magical method from rails
pages.insert page

Ok this is pretty cool. In 10 lines of ruby, I’m downloading an XML file and inserting it into a new collection called ‘pages’ in a database that hasn’t even been created (as long as mongodb is running). Great! But it quickly falls apart.

If you run it again, you now have two documents (rows) in Mongo. Boo. Not to mention, if you actually query mongo you see this (trimmed the XSD line a bit for readability):

> db.pages.find()
{
   "_id":ObjectId("4ec6c07e5a498d64a1000001"),
   "mediawiki":{
      "xmlns":"http://www.mediawiki.org/xml/export-0.5/",
      "xmlns:xsi":"http://www.w3.org/2001/XMLSchema-instance",
      "xsi:schemaLocation":"http://www.mediawiki.org/xml/export-0.5.xsd",
      "version":"0.5",
      "xml:lang":"en",
      "siteinfo":{
         "sitename":"Wikipedia",
         "base":"http://en.wikipedia.org/wiki/Main_Page",
         "generator":"MediaWiki 1.18wmf1",
         "case":"first-letter",
         "namespaces":{
            "namespace":[
               "Media",
               "Special",
               {
                  "key":"0",
                  "case":"first-letter"
               },
               "Talk",
               "User",
               "User talk",
               "Wikipedia",
               "Wikipedia talk",
               "File",
               "File talk",
               "MediaWiki",
               "MediaWiki talk",
               "Template",
               "Template talk",
               "Help",
               "Help talk",
               "Category",
               "Category talk",
               "Portal",
               "Portal talk",
               "Book",
               "Book talk"
            ]
         }
      },
      "page":{
         "title":"Ford Motor",
         "id":"255240",
         "redirect":null,
         "revision":{
            "id":"16130856",
            "timestamp":"2003-06-30T02:32:03Z",
            "contributor":{
               "username":"Infrogmation",
               "id":"4444"
            },
            "minor":null,
            "comment":"redir",
            "text":"#REDIRECT [[Ford Motor Company]]"
         }
      }
   }
}

There’s a whole lot of metadata in there and really all I care about is the content (maybe). So in some cases, you might want to filter incoming data. I have to actually look at my data and pick which attributes I want. Now I’m tightly bound to the source document and have to worry about it changing etc.

But let’s do it anyway. All we have to do is change one line:

pages.insert page["mediawiki"]["page"]

Now our inserted document looks like this:

> db.pages.find()
{
   "_id":ObjectId("4ec6c1775a498d64f2000001"),
   "title":"Ford Motor",
   "id":"255240",
   "redirect":null,
   "revision":{
      "id":"16130856",
      "timestamp":"2003-06-30T02:32:03Z",
      "contributor":{
         "username":"Infrogmation",
         "id":"4444"
      },
      "minor":null,
      "comment":"redir",
      "text":"#REDIRECT [[Ford Motor Company]]"
   }
}

Of course, we had to clear out the pages collection and re-run it. That’s just because we haven’t written any logic yet to check for existence yet. But let’s take a break here and talk about what we could do even with this piddly little bit of 10 lines of Ruby running. We can query:

> db.pages.find({}, {'title':1} )
{ "_id" : ObjectId("4ec6c1775a498d64f2000001"), "title" : "Ford Motor" }
{ "_id" : ObjectId("4ecafffd5a498d0136000001"), "title" : "Nissan" }

Here we are just showing that we have two pages from Wikipedia stored. The title:1 is like SELECT title FROM pages; in SQL. So if we wanted to search on attributes, it’s pretty easy:

> db.pages.find({ title:/^N/ }, {title:1})
{ "_id" : ObjectId("4ecafffd5a498d0136000001"), "title" : "Nissan" }

It’s pretty forgiving on the key quotes.

In the next part, we’ll dive into handling updates and other formats.

guard 'rspec', :version => 2, :cli => '--drb --color --format doc' do
  watch(%r{^spec/.+_spec\.rb$})
  ...
end

You can specify multiple formats with --format one --format two.

Anyway, here are some shots of what the output looks like:

--format doc

--format progress

--format nested --format progress

--format nested

--format html

HTML format is also the same as the Textmate format. I couldn’t get the output to go to a file like the documentation says. Maybe it hasn’t been updated for rspec2?

We’ll start with a simple person class that is initialized with a name and an age.

class Person
  attr_accessor :name, :age, :problems
 
  def initialize(name, age)
    @name = name
    @age = age
  end
end

Creating a person is as simple as passing “James” and 99 as arguments.

puts Person.new("James", 99).inspect
# => #<Person:0x007f9e2a136878 @name="James", @age=99>

Now I have a Person object as expected. The twist comes in when you look at this line by itself and realize that you can’t get what 99 is. Is it the age? Is it the problems? Of course, you might opt to simply get rid of the constructor and set the instance variables manually. But let’s try to do something more fancy.

First, we’ll try to invoke a method that doesn’t exist.

begin
  Person.create_with_name_and_problems("James", 99)
  # => undefined method `create_with_name_and_problems'
  #    for Person:Class (NoMethodError)
rescue NoMethodError
  # just continue
end

We will get an exception here. For the sake of our single source file, we’ll catch the exception and continue.

So when we try to call #create_with_name_and_problems so that 99 is clearly a problem and not an age argument, the method doesn’t exist. We could create that method but that’s not very scalable, we’d have to create every permutation of possible construction options.

Instead what we are going to do is use method_missing to handle calls to unknown methods and at the same time set the instance variables and return an object.

class Person
  def initialize
  end
 
  def self.method_missing(meth, *args, &block)
    puts "OH NO!  No method!"
    if meth.to_s =~ /^create_with_(.+)$/
      self.run_create_with_method($1, *args, &block)
    else
      super
    end
  end
 
  def respond_to?(meth, *args, &block)
    if self.meth.to_s =~ /^create_with_.*$/
      true
    else
      super
    end
  end
 
  def self.run_create_with_method(attrs, *args, &block)
    attrs = attrs.split('_and_')
    # #transpose will zip the two arrays together like so:
    #   [[:a, :b, :c], [1, 2, 3]].transpose
    #   # => [[:a, 1], [:b, 2], [:c, 3]]
    attrs_with_args = [attrs, args].transpose
    attributes = Hash[attrs_with_args]
    p = Person.new
    attributes.keys.each do |a|
      p.instance_variable_set "@#{a}", attributes[a]
    end
    return p
  end
 
end

First we reopen the Person class (monkey patch) and redefine a parameter-less initialize method. Next we create a method_missing method on the class object that looks for any method that starts with “create_with_”. If it does then it creates a new object with the correct instance variables set. Finally, the respond_to? method ensures that our Person class is advertising that #create_with_ methods are valid to outside calls.

Ok, so now our Person object is ready to be used. We can create James again this time with a name and a number of problems. We can even create a person with all three attributes and vary the order.

puts Person.create_with_name_and_problems("James",99).inspect
# <Person:0x007ffc8a835250 @name="James", @problems=99>
 
puts Person.create_with_age_and_problems_and_name(55, 99, "Jay-Z").inspect
# <Person:0x007ffc8b0ae990 @name="Jay-Z", @age=55, @problems=99>

So in actuality, this is a bit contrived. It’s cool to have these dynamic methods created for us but doing this way is a little too much work just to get parameterized constructors. The better way would be to use a hash for initialization. See below:

class Person
  attr_reader :name, :age, :problems
 
  def initialize args
    args.each do |k,v|
      instance_variable_set("@#{k}", v) unless v.nil?
    end
  end
end
 
p = Person.new(:name => "James", :age => 99, :problems => 99)
# <Person:0x007fda421044f0 @name="James", @age=99, @problems=99>
 
puts p.name       # James
puts p.age        # 99
puts p.problems   # 99

So I built an Arduino frame using nothing but the parts from the Mindstorms kit (it’s all just Technics parts). It’s pretty sturdy and the little rubber ends hold the board down pretty good. I figure it’s good enough to put on wheels or a quad-ped bot. I don’t know if I’d send it up in the air. I’d probably want to add a few more cross bars in there. Anyway, I did a lot of refactoring (if you could call it that) and I think this design is pretty good. Here’s the money shot with the rest of the build shots so you can copy it. I’m fresh brand-new to the Lego scene so leave some comments if I did something stupid.

Top down without the board.

Bottom face without the board.

Room and support for plugs.

Doesn’t shake loose.