So let’s say I have some CLI I want to exist …
The concrete example I’m going to use is my previous blog post about Slop where I demonstrated how to use the slop gem. The code in that post is slightly contrived and certainly not clean but I think it demonstrates how to test CLI scripts which suffer from some testability problems (how do you capture STDOUT?). The thing that it does not demonstrate is long term maintenance problems that happen after it’s written once for a blog post.
Code review aside, this desire to have a binary CLI was inspired from a very real work situation where we had a CLI utility and not surprisingly it was damaged from some gem and dependency problems. Mainly, if I use (consume as a user) the slop gem it’s in my bundle. If my list of gems grows forever eventually I might want want to develop another gem that uses slop as a dev. So now I need to use RVM’s gemsets or gem_home or otherwise keep my gems and projects sandboxed. Because (as it did happen) pry uses slop and when pry stayed behind causing slop problems between projects. Distributing this gem to our team was problematic because different people used different gem isolation tools.
So … uh … what if I just want a CLI? Why can’t I just live and die in
/usr/local/bin like “normal” unix-y
Golang to the Rescue?
So for the past few years I’ve experimented with Go as a tool in the toolbelt for the above problem. It has fast compile times, can cross-compile to other cpu types and you can get a binary even for a web service. Shipping a binary for an api service sounds pretty neat! However, it lacks high-level density (usually called expressiveness). So without starting a language war, what if I want something in-between loose shell scripts and strict compiled C (not that I’m specifically talking about shell or C)?
Ruby is so close to shell script sometimes and then you can drop into the “real stuff” for the heavy lifting
and then just continue in happy script land. I feel like a lot of shell script problems align with this flow.
Looping over images and doing mass conversion for example. It’s just a little bit of heavy algorithm
surrounded by a lot of shell stuff, which is great. So Ruby has been fine in that way. But then not fine
for it to live in
Go as an experiment has been fine while I’ve sought a panacea for $PATH. Go has a lot of interesting things in it and I’m not giving up on it. But porting isn’t real. Rewriting is real. Porting Ruby to Go is a rewrite. You really need to go back to requirements / thinking and you will feel tempted to refactor. It’s closer to rewriting I mean. It works the other way too. I’ve seen “Java in Ruby” in a lot of libraries.
There’s no such thing as porting. Only rewriting.
I’ll show otherwise later.
What Sharing Ruby is Like
So if I make a hello world CLI in Ruby called
utility, how do I share it?
Here I list the dependencies that are implied in the top box. In ruby there are many.
Many times they aren’t listed or described. If you are a ruby dev, you just know that things start with
bundle exec, you probably have it aliased. If you aren’t, you are confused and probably don’t use the thing
README didn’t work.
Maybe this above in the middle is the source code I’m trying to share. Scripts can be commited with file
permissions so the chmod on the left isn’t entirely needed. What is definitely needed is some path setup
which may or may not already be configured. I suppose you could put
/usr/local/bin but then
it’s like an oddball exception.
brew list won’t show it and it’ll never be updated. You’ll just have to
remember you installed utility as a one-off? Uhh …
Basically it boils down to this:
“Please install a dev environment” vs “Please use a package manager.”
You can see that on the left I’m basically asking a user to install a dev environment for a Ruby program. And then as time progresses, what happens to that dev environment? Does it bit-rot? Does homebrew break it?
And maybe you might say “just gem publish”. Phusion used to do this for passenger. And logstash. But then they stopped. Using rubygems to distribute ruby code is sometimes done but then sometimes it’s frowned upon. I’m not sure exactly why and I don’t have a source although codegangsta kinda hints at it.
This isn’t a ruby problem. The same thing happens with node & python. But when I run into a utility written in Go, I breathe a sigh of relief.
It’s written in golang. Woo! This should be easy to install and run.
Worst case it’s a
go get. Sometimes it’s a
brew install. I think these mechanics keep people from
packaging ruby utilities into homebrew. I know there are packages that help with this like Phusion’s
tool and FPM but I just
don’t see that a lot. Most of the time the README just says
gem install but they skip all the context that
I diagrammed up there. Even my own projects blow up on me. Sometimes I have to reset bundler and ruby (OSX
upgrades). Then I’m missing a gem.
gem install slop. I had already done bundle before but cleaning out gems, upgrading homebrew,
upgrading to Sierra or switching from rbenv/chruby/rvm and back and forth can leave this script “broken”.
So, what to do? I just want a command in my path. Do I have to switch languages?
Porting Ruby to Crystal
Crystal is a (very) Ruby-like language that can be built to a binary. It’s an entire language so it’s hard to sum up but here are some interesting attributes of Crystal according to me:
- It sits on top of LLVM. This is how you get a binary.
- The stack traces are pretty verbose.
- It’s Ruby-like but not exactly. If you know Ruby, you know it already.
- Adding types aren’t that bad.
So let’s say that I wanted to port the
whatthefi utility to Crystal. To recap,
whatthefi is a command who’s
purpose is to be run when your wifi (really Internet) is acting up. The ruby version has a few modes.
So the usage help text is all generated from Slop which is really nice. Automatic proliferation of standards in the usage formatting and text.
Neat. It’s really a contrived utility created for a previous blog post but I do occasionally use it and I
expect it to be in my
$PATH so it’s very-first-world annoying when it breaks because of gem problems.
Starting a Crystal Project
It’s a script but it’s really a binary. Even the ruby project had a folder. We should make a folder. So let’s do that and set up some basic files and structure.
You need crystal installed.
brew install crystal-lang
What Did We Have?
We had a utility that basically takes command line arguments and then switches on what “mode” the user wants and then calls a method that does a HTTP call out to services that in reality do the heavy lifting (like checking if a website is up). The ruby version used slop to parse ARGV and then a stdlib HTTP get. Basically the code breaksdown into sections like this.
Then there were rspec tests that tested the flags. Here is the entire test suite, it’s very small.
You can see that we are testing the different CLI flags. We’ll do the same in the crystal version except we’ll actually capture STDOUT.
Starting the Crystal Version
We’ll make shard.yml file for our dependencies. I’ll explain the 3 shards we’re going to use.
cli is a shard (gem) for crystal that helps in handling flags and building a CLI. It’s equivalent to slop or
perhaps more like thor since it supports subcommands. The shard
spec2 is used to make testing a bit nicer.
Spec2 isn’t quite as flushed out as rspec (understandably). But that’s ok. Lastly,
stdio is a shard which is going to help
just for testing, the annoying problem of capturing STDOUT text when trying to test a main type script.
Note that these are compile type and developer only dependencies. In ruby, we’d require these to be available at run time because interpreted bro. Not hating on Ruby here! This is just how this works which has advanages for distribution when we compile this all down.
We just run
shards to install dependencies and this is the same as
bundle. If we want to update, run
shards update like
In the ruby version, we had some setup for the option parser where we basically define the flags for our
project, then later we have a main or
run and then the options are detected and methods are run. It’s very
simple. Basically a big case/when depending on the CLI flags that are passed just like the breakdown image
above shows. The methods for the actions are very small and they stand on their own:
--ip is passed then it fires the
what_is_my_ip method. There’s the case/when kinda behavior.
We can do the same with crystal just with a different library so the details change a little.
There’s some differences in error handling here because the stack traces in Crystal are much more machine formatted. The same basic flow exists though.
The repo for
whatthefi is tagged at the rewrite point.
Newer versions and improvements will continue as the crystal version 0.2.0+.
The Crystal Specs
Just like the Ruby version, we’ll test the CLI flags so we don’t have to manually test. I’ll just show one test so you can see the general look of a test for brevity.
Note how we can use a block to capture STDOUT. This isn’t super clean or reliable so watch out for terminal weirdness. But I’d rather really test a CLI than not.
We can run our test suite like this:
Note that the first 5.25ms time isn’t wall time. I don’t know what that’s from. The test suite runs in about 8 seconds. It’s not doing anything smart here. It really hits the internet. Sub-optimal. I’m not sure about VCR in crystal or mocking external calls. For a project this size, YAGNI.
Quick Note on “Porting”
I don’t believe porting exists. I think to port something between a lot of languages you actually need to rethink about it and almost rewrite it. It’s more of a rewrite process.
But in this exercise, I really did port. It’s a simple program, sure but I really did copy ruby methods over and translate them. Many lines remained unchanged and most of the changes came from static typing or the changes between gems to shards.
Overall this is the most port-y port I’ve ever done. So that’s cool.
Creating a Homebrew Formula for Crystal or Ruby
It is absolutely possible to create ruby projects without resorting to rubygems as a distribution mechanism. I’m not implying that Crystal is required in order to make custom homebrew taps. I am asserting that binaries and homebrew (in my mind) go together more naturally because the requirements are less to get a binary going. Granted, I’m avoiding any downsides of managing binaries and platforms. Not on purpose, that’s just going to take time and I’ll have to do a follow up blog post.
Steps to create homebrew for a ruby gem:
- Create a new repo called
username/homebrew-tap(not tap, homebrew-tap)
- Write a formula definition for your project in your homebrew-tap repo.
- Download your release zip and run
shasum -a 256 release.zipto get the sha256 for the formula.
- Download your release zip and run
- Push master to homebrew-tap to publish.
- Tag a release as a zip in the project you want to distribute.
brew tap you/tap(my tap is
brew install your_project(this will auto-update taps as of recent hombrew versions)
- Run your command out of
Now this skips things like bundler and dependencies. If you have gems, this will fail for people using your formula at runtime which is not nice. I’m going to skip this since my goal is to get to a binary CLI tool with Crystal.
I made a Makefile for running common tasks like building a binary. There may be a more Crystal specific tool out there but I didn’t look.
make by itself will make a binary in our project root. Just run with
We’re almost done! Just put this in your path and then it’s “installed” although this
is very manual and prone to error. We probably want to make this maintainable with homebrew.
Doing a Release
Let’s say you’ve make changes and you want to cut a new version.
- Git tag it.
- Build the binary with
- Zip it up.
Github will make a release automatically based on that tag. We’re going to attach files to that release tag in a second.
Attaching a Binary to a Release
Now that we have a project that builds to a binary, we can publish the binary so people don’t need a dev environment to run the project. This can be done with a git tag and then pushing that git tag.
NOTE: This example is a very manual process. For a larger and ongoing project, a better way would be to have Travis-CI (or other CI tool) build the binaries and cut github releases. For a good example of this in crystal check out crul and the repo’s config. Looks like they cross compile to Linux using Docker as a build server.
Ok, now we have a zip. Find the 0.2.0 release in your repo. Click edit. Then drag that zip within the release edit page like this.
Now homebrew has a zip file to download that’s attached to a release. You’ll want to cross-compile for other platforms you want to support. Then wire up the package manager for that platform. I’m only showing Mac here.
Update Your Homebrew Tap
Testing Homebrew Without Publishing
So let’s say you want to play with homebrew before you publish? Do you have to make people mad while you play? No. You can just work off your local Formula.
If you tap your own custom repo then it will live in a place that looks like this. You could either work in there or work out of your homebrew git repo and copy into here.
Then once you are happy do a commit to
you/homebrew-tap and push to origin.
After all this is setup, updates can be done to homebrew-tap and do your repo in the steps listed above. Going between Ruby and Crystal netted this fabulous change: