Information Is Late Game

17 Sep 2021


Many years ago I was at a meetup at a brewery and I noticed a bunch of IoT devices on the brewery equipment. I have no idea how the machines work or what they are measuring but I started thinking about this Information Is Late Game idea.

Someone at that brewery had to get the beer working. After they got the beer working, they sold the beer. As they are selling the beer, they are wondering “are we selling enough beer?” and “how is our beer doing?”. Ultimately these questions after physical things are established are informational questions. First came the thing and then came the information describing the thing.

pricing demand curve

The IoT sensors can measure temperature and liquid flows. The payment system can record revenue. The schedule can be a calendar. The people working can be in a database. All the stuff that happens after the real thing is information. The information is what doing all this stuff means. The review score, the money. Even if it’s not digital, it’s information. We were doing this almost immediately after inventing farming.

Physicists and mathematicians might say that all this is entirely information but that’s entirely too interesting for me to talk about.

Here another one using the same brewery. What should the beer’s price be? Not what the intial price should be. We’re already selling beer. I mean, what should we change the price to? The adjustment and the measuring of price’s impact on sales isn’t even digital. It’s just information. Information about the “real” business. You really need and want this stuff late-game the most. Optimization and measurement when there’s too much on the line and organization when there’s too much information. Automation when there’s too much to do, automation with rules and inputs that are information.

Electronics follow this same pattern. You have sensors and code. Sensors measure the real world. Your code works on the information of the world. Once I hook up the motors, sensors and displays; it’s mostly set physically. I’m working in information from there on out in development and especially when it’s running.

I understand this is annoyingly generalized. Just one more.

minecraft ae2

Here’s another one at the risk of exposing myself as a huge nerd. A friend and I are playing Minecraft. We make a hut, we get established. We start smelting things. We have a couple of iron bars. How many do we have? We want to know or we want to act on this. It’s a number! It’s a number in a chest. What is it? How much do we have? We did the “physical” work to get them, what does it mean to us?

Time progresses. Eventually we feel the need to have a digital inventory system to maintain stock levels or do reporting on how much stuff we have. Is this fake digital game any different than what we would do?

We had the same thing happen in Factorio. Money and finance is kind of an easy one but I think information systems and how those things work is probably more nuanced and interesting. I just think it’s late game, it’s not a unique perspective. The information age itself came later on, after the “real” thing. But we wanted to know what it all meant, or how to organize it.

Sciencing Out Updates

10 Sep 2021

During a project’s life, the minimum amount of change you will encounter is from security updates sources. There are many sources of change. If you are going to switch versions, bump a package, try a new framework version or port to a new language version that is a big lift; the least you could do for yourself is make it less stressful. I’m going to describe a nice workflow for the simpler versions of this situation.

I’m using python as an example just to highlight some tooling but this is not python specific although I hope this is a new take or maybe cross-pollinated from other tech circles than python.

Try Updates In A Branch

There’s a workflow around updates that I wish I could write on the moon. Given ~90% code coverage and lots of confidence in the test suite, here’s how I would update a project in a major way (say moving from like python2 to python3 or updating Django by a major version).

git checkout -b python3_update
asdf local python 3.9.6 # asdf is an every language language version manager
poetry run task test # invoke pytest
# note errors
# let's say requests throws an error
poetry update requests
poetry run task test # invoke pytest again
# keep iterating, you have a list of stacktraces at this point, work the list

# on test suite pass, commit the change for python
# remove .tool-versions if `asdf local` created one
# edit pyproject.toml (upcoming PEP that poetry already uses) to require 3.9.6
# commit everything (this should be package changes in your deps and lock file and the pyproject python version)

Now the effect of this is:

  • You can open an MR with all the code and all the change that represents moving from python2 -> python3
  • The person reviewing can replicate by doing poetry install and will receive an error message saying “you need 3.9.6” etc
  • The reviewer has a better time, new hires / project members have a better time
  • The best part of this is … it’s a branch for a major change. Minimal worry and then less stress. It’s an experiment and a commit. CI reuses it.
  • This same workflow can work to update django or flask or something huge and you have a list to work on

This is a great workflow to me. It’s very familiar to me from other languages. I’m not saying the upgrade is automatic, I’m saying this is more like an experiment where you measure and find out how hard the upgrade is going to be. Not doing at least this flow makes me think of hoping and praying and I think it’s enabled by tooling so that’s the next topic.

What We Are Really Doing

  1. Specifying dependencies we want to try
  2. Setting up an environment to measure things in
  3. Exercising as much of our code as we can
  4. Asserting things

When we measure our code for defects in the environment that includes updated packages or language runtime, we try to find out as much as we can.

The Tools in Play


Our project’s dependencies are managed by a human-editable project file and a machine usable lock file. The dependency file is pyproject.toml and the lock file is poetry.lock. There are other tools but they follow this pattern.

  1. You edit the dependency file with what you want
  2. You run an explicit update and the dependency tree is resolved (hopefully, this is not guaranteed)
  3. Your lock file is written with the solution
  4. Your update is complete and repeatable

Language Runtime

Dynamic languages need a runtime or really a development environment (unfortunately). Poetry has a section for this:

python = "^3.9.6"

Using requirements.txt won’t get you this. There’s also sort of a different take on this with .tool-versions and asdf which will switch versions for you automatically. You can drop a file in your project declaring what you want. Your shell or language manager might pick up on this file so you aren’t activating and deactivating.

The downsize of not setting your runtime or your packages in a more permanent way is that all commands you would run are prefixed with poetry run. Some people find this really annoying. They might find this so annoying that they bail on the entire idea. I have a few workarounds:

  1. Make an alias. In fish, when I type pr it autoexpands to poetry run.
  2. Use long running commands like test watches or http servers.
  3. Get used to it? : Many other languages have prefixes: yarn run foo and cargo foo and bundle exec foo and others
  4. If you have an app, make a bin dir full of helper scripts. You probably want these anyway instead of README blocks.
  5. If you embrace prefixes, you can use taskipy and get poetry run task foo, stay in python and avoid Makefiles.

Git Branches

By using all these files that are inside the project, nothing exists outside the project except for the install directory. Thusly, you can commit everything and your major change is able to be viewed as any old change.

Larger Upgrades

On a huge project, this simple approach doesn’t work. A large lift is a long running branch, those are no good. It’s going to be horrible to maintain parallel branches and deal with merge conflicts. Scaling this out is kind of beyond what I wanted to talk about but you can read about Github’s major upgrade which will show you some similar tricks. In their case, the workflow is very similar (but different). They ended up with a 2,000 item todo list off a branch and iterated. It’s a good listen about scaling this idea out.


This post is focused on python but it’s not about python. I’ve taken this workflow to many languages and if the tooling is in place, the flow stays familiar.

  1. Checkout a branch to play in
  2. Make changes
  3. Measure change with tests
  4. Commit changes for review

This is just the basics to me, failure is still possible. You’d want smoke tests and other tooling. I just wanted to describe how language tooling is an enabler.

A REPL Based Debugging Workflow

03 Sep 2021

Let’s say we have this code that isn’t working.

// some code we inherited, we don't get it, now what?
function findUsers(users, age) {
  var hits = new Array([]);

  for(var i=0; i<(users.length); i++){
    if (users[i].age < age) {

  return hits;

It’s causing some issues somewhere else and we have no idea what root cause is. In the real world, ending up at a tiny piece of code where we think the problem lies is very lucky. This is probably the result of lots of tracing and investigative work. But let’s just pretend that we ended up here.

We’re going to walk through our approach and challenge ourselves to think about why and how we are thinking. We ourselves are not going to turn into computer compilers or syntax interpreters. But before that, we’ll study the anti-patterns first.

When you are trying to figure out something that is broken or even develop something to make something new: manual debugging is annoying

Anti Pattern 1 - The Human Interpreter

If I ran into this bit of code, the first thing I’d not do is grab my chin and stare at each line, trying to catch the bug. I don’t want to do this because the bug is usually not as obvious as this example (it might not be obvious to you, that’s ok). So, staring at the code is healthy and normal but this is not the core of our process. What we’re going to do is get a feel for the code by exercising and playing with it.

Anti Pattern 2 - Print Statements

You might be tempted to start dropping console.log() statements all over the place. I do this too but I try to get away from this as fast as I can. Print statements are throwaway code and mostly a waste of time (in general). You can get something that works much better as a one-liner (breakpoint) or a mouse click (breakpoint).

Anti Pattern 3 - Opening a Shell

Node / Python / Ruby and other languages have an interactive shell called a REPL. You could copy and paste the above function and mess around it in a REPL. This is great for learning and is completely on the right track.

The problem is, what is users and age? Age is probably a number. But even if we know what users is, we have to type it in. And, if we want to experiment on the function itself, we have to type that in too to redefine it. In some REPLs this can be very annoying because of whitespace and syntax.

A Debugging Workflow

First, you need tests. If your project doesn’t have tests, this won’t work as well. You’ll have to pay the cost to make that happen or step back a bit. I’ll have a section about finding a workaround.

Every test follows a pattern:

  1. Setup
  2. Execute
  3. Assert or Test

What we do is create some dummy data to expose the bug for the Setup.

// setup
let john = { name: "John", age: 25 };
let pete = { name: "Pete", age: 30 };
let mary = { name: "Mary", age: 28 };

let users = [ pete, john, mary ];

And then we call the function as execute:

// execute
var result = findUsers(users, 29)

And then we can tell the computer to fail if it is not what we expect. First, let’s define what we expect. This function seems to want to find users under a certain age. So the answer should be John and Mary:

var expected = [ { name: 'John', age: 25 }, { name: 'Mary', age: 28 } ]

And then we assert in whatever testing library we are using. This test is going to fail at this point (that’s the bug). So, now we have a failing test. But more importantly, we don’t have to keep typing our function or our data over and over again in a REPL to play around with it.

But wait! We haven’t outperformed simple print statements yet!

First, let’s get our test created with some module and importing boilerplate.

var findUsers = require('./repl_debugging.js');

describe("finding users", () => {

  it("finds users under a certain age", () => {
    // setup
    let john = { name: "John", age: 25 };
    let pete = { name: "Pete", age: 30 };
    let mary = { name: "Mary", age: 28 };
    let users = [ pete, john, mary ];

    // execute
    var result = findUsers(users, 29);
    var expected = [ { name: 'John', age: 25 }, { name: 'Mary', age: 28 } ];

    // assert or test, jest uses expect


The main file looks like this:

// something we inherited, we don't get it
var findUsers = function findUsers(users, age) {
  var hits = new Array([]);

  for(var i=0; i<(users.length); i++){
    if (users[i].age < age) {

  return hits;

module.exports = findUsers;

We’re going to put a debugger; print statement in to break in our code.

Now unfortunately, some languages are easier to debug than others. You might be using javascript for a web frontend in which case you’re going to need to figure out chrome or firefox developer tools or wire up a VSCode or other editor/IDE config to enable breakpoints and interactive debugging.

For now, I’m going to show you NDB and this flow.

debugging in a test run

Run this command to watch tests and run our debugger

ndb $(npm bin)/jest --watchAll --no-cache --runInBand

First, our breakpoint breaks in this function and when we hover over variables we can see their values without console.log().

the NDB UI

Now we see that the element at 0 is wrong. We need to fix that.

We change the line to be

// something we inherited, we don't get it
var findUsers = function findUsers(users, age) {
  var hits = new Array();

And the test passes. Now you could refactor that for loop into a map or something. :)

I Don’t Have Tests!

Create a file that executes the function with setup data. You’re basically making a test suite yourself. This is close enough to the nice REPL + test workflow that you might be able to make it worth it. In this case at least you don’t have to type the test data over and over again. When I find myself doing this, I copy it into a test. Once I have it in a test, I can drop REPL breakpoints in.

The Big Lesson

  1. Put your setup and dummy data into a file
  2. This file can be a test.js file in a tests folder
  3. If you do this, your debugging becomes a test
  4. You can put breakpoints in code or in a test
  5. If you put breakpoints in the test, you can break and inspect outside of the code
  6. If you put breakpoints in the code, you get all your test data and setup in the REPL session, no typing over and over again
  7. You can add/remove breakpoints and have your test suite run all the time

Doing these things creates a steady feedback loop. In addition, if you develop an algorith in the REPL shell, just copy it into the code. If you find some useful test data in the REPL shell, copy it into the test.

Playing around in a REPL from inside in a test run is a great way to develop algorithms, methods, test data and expectations that your probably need to save in a file anyway

The anti-patterns aren’t “evil” or “wrong”. I just prefer the REPL + test flow for all the previous reasons listed and it sets us up for other things like refactoring and stability.

Commodities, Economic Scaling and Compute

27 Aug 2021

Photo by Andre Medvedev on Unsplash

I wanted to chat about economic engines, side effects and network effect type sources of change. These topics are very broad and so please bear with me. I hope you see an interesting pattern and don’t get too caught up in the specifics.

A Bookstore Makes The Cloud

This is probably the most recognizable one so I’m starting with this first.

AWS had a successful online bookstore. They eventually enhanced their site to include other types of products but the the initial push was books. As they scaled their site (fueled by success) they had extra compute capacity but also a shared platform for internal and external sites. Eventually they decided to open up the platform to everyone and sell the service. What was useful for them internally and a limited number of customers could be useful for everyone.

Eventually the scaling was so successful that it became a named compute paradigm. In certain workloads and architectures using AWS is so cheap and effective that it’s a bit of a defacto choice. The word Cloud before AWS would likely get you a rare network symbol search engine hit but now it generally means something else. Cloud Computing is advertised during NFL games and should be credited to AWS and its bookstore engine.

A bookstore creates a new compute paradigm

Video Games Created HPC GPUs

Video games are compute intensive simulations. Video game players demand high performance. Nvidia and other companies invented GPUs to bring that performance for these simulations. GPUs would increase in power every year, funded by gamers who want high performance and companies who wanted their games to pop and astonish. Eventually Nvidia would create a slightly more general purpose language called CUDA which could run specialized code on these GPU cards. Originally, CUDA was intended to run small programs called shaders to do specialized effects or work in a game.

Eventually, someone realized that you could use this general-ish language CUDA to do meaningful work on wide datasets. The width of the datasets were a close match for the wide nature of video game pixels. Doing this same work on CPUs would be too slow. This is already what video games had experienced for graphics and given birth to GPUs in the first place.

Eventually Nvidia would create a new business division around selling GPUs to companies doing High Performance Computing (HPC) and Machine Learning (ML). Many new and interesting innovations would come out of this progression. HPC was a concept before and there were many other products and companies involved but following just the Nvidia story illustrates the point.

Gamers funded modern HPC and compute

Mobile Phones and Processor Manufacturing

This one is really interesting to me. There are two parts to this. One, manufacturing. The other, architecture.

First, CPUs have a long history but let’s focus on recent events. ARM recently has been challenging x86 on a performance front but instead I want to focus on process nodes. You can’t really compare process nodes between manufacturers but let’s try anyway. Why is Apple on 5nm and Intel (king of kings for so long) on 14-10nm manufacturing process?

Making chips is hard. Apple doesn’t make their own chips. Who are they using and who are they? Today, TSMC is making Apple’s M1 chips on their 5nm node which has been fueled by mobile phones. TSMC has gained so much experience from making mobile phones that they are really good at it. Samsung makes phones, it’s the same and Samsung is also driving transitor density at scale.

Even if Apple wanted to work with Intel, they wouldn’t do it. TSMC is doing leading edge work and buying the majority of ASML’s EUV machines.

Mobile phones change transistor manufacturing

Processor Speeds Increase Network Bandwidth

This is really outside my lane but I talked to a low level network engineer about this (and so my understanding is simplified). He said that compute speed is speeding up network speed. As an example, two switches with a fiber cable connected between them will emit and multiplex light at a certain rate. This rate is CPU bound. As processor (or CPU speeds) increase, so does bandwidth. We weren’t talking about your ISP or specific products. What I mean is, why have progressed at all from 10mb to 100mb to gigabit and beyond over the same physical medium (copper)?

Or put another way, have we hit the theorical limit of fiber? Do we know what the practical limit is?

The answer to both of these is no. We’re doing more and more tricks with light divisions, noise and other things but the fiber itself is unchanged. We’ll find out the limits when the system is no longer CPU bound. It doesn’t really have a lot to do with fiber. Copper hasn’t changed either.

So whatever processor side effects I’ve mentioned above would also affect this thing here.

If Apple came out with an ARM chip that is one million times as fast as any specialized processor, ASIC or other chip; you’d use it for network switching and set a network world record (assuming the same amount of noise).

Processor speed improvements increase global network bandwidth

Commodity Processors

A long time ago, x86 was pretty bad compared to IBM (power) and Sun (sparc) chips. “Real” servers would run on these custom chips. But then commodities kicked in. Not just on the CPUs themselves, but even the motherboards. There were Dell server motherboards so cheap and derived that you could saturate the PCI bus by filling in all the I/O cards. This wasn’t the case with Sun boards but the Sun boards were much more expensive. This would be the equivalent of today of having a server chip without enough PCI-E lanes.

Eventually Intel chips would be faster and cheaper than the IBM/Sun chips. Even if the technology was better (or different) in many ways, it didn’t matter. In this way, commodities were one piece of it but the other was performance. Intel got all this economic fuel from desktops, gamers and cheap servers. The mass market was fueling the engine. Now the same is happening from mobile but it’s architecture. There’s the very real possibilities that we are at an inflection point and chip manufacturers are going to have to catch up. But Apple has 2 trillion dollars (mostly from mobile).

The same thing happened with the PS3 to PS4 transition. PowerPC was based on IBM’s power architecture. This was dropped for x86. In the future I wouldn’t be surprised if consoles were ARM or something that reflects economies of scale.

How did this happen? Why couldn’t x86 keep up? I think it’s mobile phones. Just look at Apple’s revenue and where it’s coming from.

Commoditity x86 replaced POWER and SPARC server chips

In the future, maybe commoditiy and server x86 chips are replaced by ARM which was fueled by mobile phones. When this happens, there will be another reductive quote here. :)

Wrap Up

I hope these reductive cause and effect topics aren’t too polarizing. I just wanted to highlight some second-order effect engines of S-curves and a few examples in one place.

Send Me Your Stuff

20 Aug 2021

do not send too much content

Send Me Your Stuff is when you send so much content over email (or any messaging) that the other person doesn’t know what to say. You explained it in overwhelming detail but there’s no intelligent response the other person can craft. You have done a great job of explaining every detail but it’s too much for them. They reached out to you because you have something or know something they don’t. That’s the gap. They asked a very open question and you gave an overwhelmingly large response. You were very clear and thorough but nothing happens. The conversation is dead. It’s crickets. The person has a huge email sitting in their inbox; they tried to read it but they can’t respond. If you are lucky, they’ll say “I’m sorry I haven’t gotten to this yet” or “this is interesting” but the conversation is actually over.

Instead, craft an email which starts a dialogue or a relationship. Don’t provide the answer, provide support. Don’t answer the question, acknowledge the problem and offer guidance. Yes, you can do that and if you’d like to talk further let’s set up a time to chat. No, you can’t directly answer their question right now but you’d like to hear more about their project so you can tailor your response to their needs.

We can pick any dense subject but as an example, someone asks you “What is the best programming language”. They are forming a team and they want to hire some developers. There are a million ways to answer this question. Maybe you’ve answered this question many times. Maybe you are tired of the explaining it. They don’t know that. They don’t know how complex the question is.

The wrong type of email to send:

This is a hard question to answer but I’ll try anyway. If you are doing a mobile app, you probably want to hire a few …
[ several pages of content ]

Instead of that, send this:

Let’s find a time to chat about hiring and the technical landscape. We can look at some industry surveys and talk about what the different sweet spots and communities are like. This will probably take about two hours depending on the detail. We can start with a short 30 minute meeting to talk about things at a high level but the additional conversations will probably get you primed for hiring a recruiter and making a job description.


Send Me Your Stuff is not my idea. I heard it explained by Sara Batterby who was doing business and fund-raising consulting. I loved the fact that her talk had nothing to do with my job and yet, I find this idea hauntingly applicable. I make this mistake constantly. I can give you a list of topics and replies I’ve messed up but that’s probably not interesting. I’ve even prefixed long email responses by saying “I don’t want to do Send Me Your Stuff” (and then explaining what this is) and then puting an inevitable but statement and doing it anyway. I get the email equivalent of the sweat smile every time.

Send Me Your Stuff can apply to requests for repos, code, project zips, brain-dumps, best practices, portfolios, design schematics or almost anything. Whatever they are asking for, don’t send exactly and completely what they want. Send a symlink.

Don't Test Code You Don't Own

13 Aug 2021

The Concept

I wanted to describe a concept of not testing code you don’t own but it also touches on other bits about pushing I/O to the edges. I think pushing I/O to the edges and the concept of messages is a much more important thing to learn here but it’s ok if you want to solve specific problems like “how do I test a thing without it really writing a file / charging a credit card?”.

These examples are in Python just to change things up a bit. These concepts apply to Ruby, Javascript, Go, Elixir, Java and almost any language. The key difference is that dynamic languages make stomping on real methods very easy through their dynamic nature. So how you mock in Go might be harder (maybe you need to use an interface or use a protocol) but the concept of pushing I/O to the edges is still the core here (to me).

When I use the term stomp on a method, I mean the test library overwrites the real method. In python this might be .mock in pytest or MagicMock. In Ruby, it’s expect or allow. In Javascript, it’s Jest’s syntax or Sinon’s (Jest is fine). Stomp just means overwrite at testing time.

Finally, this is a unit testing perspective. There’s a section that talks about mocking trade-offs which you should be covering by other levels of testing depending on criticality.

The Start

If I make a call out to python’s open(), you could write a test that tries to stomp on that at runtime. But instead, I try to name what it’s doing in a method like save_game() and then I would test that my save_game() method takes some game state. I don’t need to test python’s stdlib. So I don’t need to mock open(). I do however sometimes want to make sure the wiring between my code and the stdlib is ok. Sticking with the save_game example …

First, I test my save_game(game_state) takes some state and would call some file writing method.
In this case, I’d want another method called write_save_file("a_hard_coded_path"), just assuming the game has one save file and I didn’t have a parameter to save_game.

import json

class Game:
  # when you read this method, list its jobs and who owns the code
  def save_game(game_state):
    f = open("game.dat", "w+")       # opening a file, python stdlib
    f.write(json.dumps(game_state))  # game_state as JSON is our choice, write is stdlib
    f.close()                        # this is still part of opening a file, stdlib

# main, imagine the __name__ python trick here
game_state = { 'name': "Annoying Youtube Let's Play Series S24E49" }

game = Game()

So in these lines …

1  f = open("game.dat", "w+")
2  f.write(json.dumps(game_state))
3  f.close()

Line 2 is the weird one. Writing a file in python takes 3 lines to do. But our business rule of writing the game state as JSON is our own rule. We could split up the line with two jobs into two lines and make this 4 lines to make it more obvious:

1  f = open("game.dat", "w+")
2  formatted_game_data = json.dumps(game_state)
3  f.write(formatted_game_data)
4  f.close()

In this case now, line 2 is not I/O. It’s our business logic but it’s not I/O. You can tell because it’s in python memory and it stays in python memory. Lines 1, 3 & 4 are external to python (really). The actual I/O happens (I am guessing) on line 4 when it closes the file handle and lets the OS write the file, flush the buffer or whatever system call actually happens. The point here is, line 2 is not I/O. We won’t count memory as I/O for this discussion.

Another aspect to the original 3 lines is that the method has two jobs. One is opening a file (which is in python docs) and another is converting to JSON (which is also in the python docs). But we can split these jobs up. We can change how it looks from the outside (refactoring). But I wants tests before I start refactoring.

So here’s a test and code change all at once. We’re changing quite a bit all at once which really isn’t fair to the original. Mostly to enable testing we need to get rid of that main code at the bottom. Even if this was using the pythonic __name__ trick, the concept of pushing I/O around is still not realize. I’ll revisit this in a section about CLIs at the end.

The Refactor

# sorry this is all in one file for clarity

# test file imports
from unittest.mock import MagicMock

# code under test imports
import json

class Game:
    def save_game(self, game_state):
        save_data = json.dumps(game_state)

    def write_game_file(self, save_data):
        f = open("game.dat", "w+")

def test_save_game():
    # test setup
    game_state = { 'name': "Annoying Youtube Let's Play Series S24E49" }
    game = Game()
    game.write_game_file = MagicMock()

    # execute

    # verify
    expected_game_data_as_json = json.dumps(game_state)

Now you can see that the file writing job and the JSON formatting job are in one method each. The tradeoff is more code and more code tracing. The payoff is that I can mock the I/O. With the I/O on the edge, I can mock the seam.

There’s a tradeoff to mocking though. If you comment out the f.write(save_data) in write_game_file() then the test still passes. Stubbing and especially over-stubbing can create lies. It’s obvious when you are being lied to because tests pass but prod blows up. So, if this piece is important (game saving probably is), then I’d do some more design/refactoring and make a serializer class and add more tests.

Some tools disagree with the article and stub out an entire filesystem. I did that in Go. It worked fine. Probably overkill. In Go, options are more limited. You have to test through an interface. You’re changing a lot of code to accommodate testing. In other words, your app kind of knows it’s under test. It’s not as bad as having a flag of if test_mode == true but the design is changing. I don’t super love that. In general, I don’t want my app to know about tests. If I deleted my tests, my app would stay the same. Ideally. But with Go/types, this is harder. You probably didn’t make this part of your app so flexible.

In short:

  • Don’t test code you don’t own. Don’t test stdlib. That’s someone else’s job.
  • Push I/O problems and code to the outside edge. Then test everything except that tiny piece. That way, you have high code coverage but have a blindspot that is runtime/environmental. Imagine that disk permissions are wrong for the player of this game. What can you do about that anyway?
  • Write small methods.
  • If you don’t like this or if your tests lie to you, change the rule. Do whatever gives you confidence. Mock all file reading and writing, mock all network calls and sockets, go nuts. Some libraries do this. In Go, I haven’t found this.

Even in this example, stdlib writes a file. We could stub out the IO using the same mocking library. So that the write_game_file method doesn’t write to the disk. It’s a matter of taste if this is worth it. Or software design if you wanted to swap an “adapter” or some other abstraction. In this case, (since saving a game is a rare event and important) I would probably stub out as much as I could so my test suite never writes a file. If it does write a file, I better be using that as some test artifact or asserting something by reading the file. My code should be flexible enough in this case to accept a path to write the file to and then I mock out stdlib. So, I am still writing tests for python stdlib? No. I would simply assert that my method is calling stdlib methods and those methods are mocked so they don’t hit the disk.

About CLIs

In the original code, we had a main section with a literal main comment. Code like this is untestable because it fires when we load the file. We can’t load the file in a test environment because the code runs. This happens a lot with untested code because no one ever thought about loading the code in a test environment. So the first thing you can do is add an “if”, this might be python’s if __name__ == __main__ trick but the bigger idea is this.

Put Your Lib in Lib and Your Main in Bin

If you had a project named foo like this:

This is very hard to test even if you have tiny methods. Most CLIs read top to bottom like a script. So they are hard to test. They have inline variables that make it hard to mock. Additionally, the top-to-bottom script has argument concerns as well as “the app”. You don’t need to do this.

First make two jobs and split up these jobs into two files

  1. bin/foo - Read CLI arguments, invoke the foo module
  2. foo/ - whatever the foo module does, plain old coding time

This is nice because now you can load the foo package in test. Then make bin/foo as small as possible. This CLI example is also about pushing I/O to the edges. A CLI is simply an app that runs on the command line where some of the trickiest I/O is the arguments. If you are using a CLI library like cliq, you don’t need to test that library’s capability, you just need to make sure that your app is able to deal with the options dictionary that comes from cliq.

So here are the big ideas about CLI:

  • Your main is a thin script that handles arguments and then invokes your “lib” style code which is plain old code.
  • Your code doesn’t know about arguments as strings, it knows about options or switches or something that is parsed.
  • Your code is testable because it’s plain old code.
  • The I/O you push to the edges first is CLI arguments. You can continue to push I/O to the edges with files or network as described above with the save_game examples.
  • You don’t need to test a CLI argument library. You do need to test how your program accepts arguments.
  • You do not need to really invoke your program to test all arguments (permutations). You don’t need a simulator. You don’t need a fake filesystem. If you do need these things, they are rare and more like seldomly run integration tests. Your CLI would need to be very large to need this complexity. It’s probably not your situation.


Pushing I/O to the edges to me is the same as “don’t test code you don’t own”. But it can be applied to lots of things. I/O is everywhere and using this strategy can help you solve the following problems:

  • How can I test email code without emailing people?
  • How can I test a credit card charger without charging a credit card?
  • How can I fake a filesystem?
  • How can I test my AWS cloud manager widget without costing me money? (AWS runs on HTTP)
  • How can I mock a database?

Some of these problems are bigger than others. Mocking a database is a bad time, good idea though. Faking a filesystem also might be a bad time. But if you have a class/function whose job is to interact with files and it’s not horribly complicated then you can use the above example almost verbatim and not have a bad time. The rest of the examples are really about messages. The reason why HTTP and Email are similar in I/O mocking is because the message metaphor is very strong (because the technology backing these things are essentially messages). The database I/O is a different ball of wax. An RDBMS is not just a message metaphor. It’s more of an interactive language. It’s possible to do but the seam is messier. I might have another post about this later. My point for now is don’t go crazy with this concept.

Scenario Practice

06 Aug 2021

One of the hardest questions I’ve been asked is “how do I get production experience?”. Someone was looking for a job and they were seeing job requirements about production and operations. They didn’t have any. The landscape has changed quite a bit since I was a sysadmin but the Catch-22 is the same. The Catch-22 being: you can’t get ops experience without an ops job and you can’t get an ops job without ops experience. But I think the chicken-egg cycle can be broken. Let me explain what Scenario Practice is and offer some scenarios to practice.

For someone interested in in ops, the big idea I would communicate is: a server really isn’t that much different than your local shell. For early learning, this is really true. If you get better on a Linux shell through your raspberry pi / Mac / WSL terminal, it will directly translate to Linux server skills. If you make your main machine Linux, I promise you will learn Linux just from a purely survival standpoint. If you need to fullscreen a VM and do it with discipline instead of wiping your disk, do that. The point is to feel the necessity of fixing it yourself.

The difference between cat /proc/cpuinfo on a huge zillion-core server and a tiny raspberry pi is just the text output. The knowledge of what /proc is doesn’t change. But this sandboxing skill translation can’t scale to all problems. You probably don’t have access to expensive network switches, strange storage equipment, locked-in vendor tools or even OSS that has real-world complexity in it that you can’t recreate what would be at work without that knowledge already. Some areas (such as realistic production scenarios) are hard to sandbox on your laptop, home lab or in a personal space. But there’s still opportunity to learn things that do sandbox-translate.

draw the rest of the owl

So, making your main machine Linux, learning some shell skills and surviving might teach you sysadmin what about ops? This is trickier but I think it has to do with systems. The remainder of this post moves kind of fast and might sound a bit like Draw the Rest of the Owl because it’s encouraging building systems for the ops experience but skips over the OS/Linux/shell stuff. If you are stuck, you might have to research and work on that particular topic. Or maybe I’m not targetting the right audience. I’m sorry I don’t have repos and done demos to offer right now.

Scenario Practice

Scenario Practice is about setting up production scenarios to fight the catch-22 of ops experience. It’s probably useful for people already in ops too; ops folks that are curious, worried or want to practice these situations. I’m just going to describe some of the scenario ideas I’ve had, mostly ordered from simplest to most complicated. The situations mostly involve a developer/ops hybrid role. So if you are purely ops and I say to create an application, you might have to improvise or find an existing app (maybe todomvc but you need agency and control of this app to change it). The point here is to explain Scenario Practice, give it a name and perhaps inspire some things to practice.

The Mock Client

mock client overview

First, we need to have a mock client for simulating constant load and use. This client represents a user of the system. This user is a credit-card carrying, impatient type that would cost us money if we lose their request. This client is really important to be constantly running so we can experience failures and develop some feel. There are many ways to implement this mock client:

  1. You could write a shell while loop to call curl
  2. You could make an infinite looping HTTP client call in your language of choice
  3. You could use a load tester like k6 or siege (for the scenarios that just deal with availability, a benchmarking tool like siege isn’t going to expose JSON problems)

It doesn’t matter too much. The important thing here is to make the client sensitive to the failure you are trying to avoid. If you have a shell loop every second then your failure mitigation cannot cheat by typing faster than a second where you survived a failure by luck alone. So, for each scenario I’m going to assume you have a mock client already running in another terminal.

For example, the solution to the above scenario is this: mock client solution

Continue Reading →

The Database Ruins All Good Ideas

08 Jul 2021

Stop me if you’ve heard this one before …

a typical three tier setup

Here is a three tier web stack. It has lots of web and app servers but only one database box. You can substitute this with cloud things but the principles are the same. I bet your infrastructure looks really similar. For the remainder of the post, assume I mean a traditional RDMS when I say database.

Why is there always only db01? Your box might be called prod-db1 or mysql-01. You know the one I mean. Maybe you have a db02 but I bet it’s running in a special mode. Db02 is another exception to the rule (maybe a read-only replica for reporting). Whatever the case, I bet there’s only one db01 but there are so many of the other things. Why?

We can summarize scaling each tier in this whole stack like this:

  • clients: not our problem, there’s millions of them
  • web tier: easy-peasy, it’s a daemon plus a config file
  • app tier: it’s our code / stuff; it’s in a load balancer pool even if it has state we have tricks to scale it
  • database tier: don’t touch it! there can only be one!

Each tier is either easy to reason about scaling out horizontally except for the database. What is going on here? I’m going to go over a few good ideas and why they die on the database tier.

The Good Ideas

Let’s Load Balance

Load balancer pools work great for tiers without state. You can even use tricks like sticky sessions when you have some state. But the request is short. A database resists these ideas because connections are long and the entirety of state is local. You can’t put database volumes on a shared drive and expect it to work (well). So the problem is at least state but let’s keep chatting about some other ideas.

Let’s Dockerize

Docker works great for tiers without state. You can dockerize your database but you don’t get the scaling and uniformity advantages of the other tiers. Docker is great for production and deployment but you are not deploying your database that often without a lot of fancy uptime tooling. In addition, you have footguns with non-obvious behavior around volumes. You can do it but it’s the exception when the app and web tiers are so easy to explain and reason about.

There are few threads and debates about dockerizing the other tiers. Dockerizing the database layer can be debated and googled.

Let’s Go Active-Active

Horizontal scaling doesn’t work on the database tier. You can’t easily have a read/write (active) pair. There are alternate daemons and methods (NewSQL) but here I mean common relational SQL databases.

Let’s Do Immutable or Config Management

What about NixOS? Or some other hot and trendy new idea? My first concern and question when I heard about NixOS was about the database layer. I have asked this question about NixOS and apparently it’s ok to do so. However, I don’t completely grok this but I guess this is part of my point. The database tier is a special case again.

You definitely can’t do the cattle thing because you can’t have a load balancer. You can only do the cattle/pets thing in the app tier because you have a load balancer with a health check.

Let’s Mock IO Seams

During unit testing you might want your tests not to hit an API. You can mock out the HTTP interface and test against a mock response (or even better, ignore the response entirely). This is basically mocking out someone else’s (or your own) app server. So why don’t people do this with the database? Is it because the response is so important? It’s more of a language and state engine than a simple message passing metaphor?

You can find fakeredis adapters in Python, fake caches in Ruby and in-memory databases in C#. But it’s all surrounded by caveats. It’s just easier to create a test database because databases ruin all good ideas. At least database tech enables a workaround.

There is so much state and back-and-forth protocol in a relational database that treating it like client/server message passing is too simple. All the state and data lives in the database. Even triggers and internals would be too complicated to account for. It’s just easier to create a test database because database namespaces/collections are very easy to create. Databases also have the advantage of rolling back in a transaction which works great for unit testing.

So your project might have fake adapters but not for mysql/postgres. Or maybe you use sqlite in dev/tests and something bigger in prod. But you don’t change entire products for your caches/queues based on the environment do you? See what I mean?

Let’s Use The Cloud

Renting large boxes usually doesn’t make sense financially. You’d be better off just buying. The same is true for performance clusters and GPUs. The scaling and pooling problems from above don’t change. Even a SaaS has the same issue. In this case the singular db01 box just moves to the cloud.

Let’s Keep It Simple with a Microframework

I really like the syntax and style of Labstack’s Echo framework for Golang. But my experience changed when adding a database to my app. The simplicity fell apart in that putting a global variable makes it hard to test. Without state, I don’t have this problem. There are many microframeworks where this happens. You can almost predict it happening if you look at the table of contents for the documentation and see that they have no database story.

Let’s Deploy Often with a Chatbot

We had a chatbot that had two endpoints for deployment: /deploy and /deploy-with-migrations. It worked well, we did deploys almost every day. I’m not saying that the database is unusable but this illustrates my point. The happy path of /deploy is naive. It was probably written first. And then you say “oops, I forgot about the database” and you have to write a special or more careful version to do database migrations.

A Horrible Story

Very long ago, I worked on an Oracle cluster that required a ton of specialized software, hardware, admin and configuration. Almost the entire idea was about availability and performance. The CEO just couldn’t stand the fact that half of the system is wasting money being read-only. He wanted read-write on both nodes. Active active. This was a long time ago but the CAP theorum isn’t going to change. I learned a ton about splitbrain mostly through trauma.

At the time, you couldn’t just download a relational database that will do horizontal scaling. You had to buy all these vendor options and stuff. It was super expensive. I forget the price, probably $40k for the db license and $20k for the clustering addon. And then you needed specialized disk and volume software. The hardware was really pricey too because it was Sun at the time.

During cluster install it tells you to plug in a crossover cable to a dedicated NIC. Like, you had eth1 just sitting there free or you had to buy a NIC for it. I think we bought a NIC. The install isn’t going to work unless you do this crossover thing. In addition, you need to set up a quorum disk on your SAN to act as a tiebreaker (more on that later). All the traffic over this crossover cable is SSH. All it’s doing is doing relational database agreement over SSH. There’s no data sharding or splitting you have to do so it’s all or nothing. Full-on ACID agreement, all the time. This is why you have a dedicated NIC because of network load.

So you finally beat the CAP theorum. You got your active-active database and you didn’t have to change your app at all. Now comes the trade off, the the devil’s details. ACID means we have to 100% agree on every query. That means, all nodes, all the time. This is why scaling nodes was so bad. You got about 50% on the second node and then +25% on the third node. It stopped scaling after 4. Remember, each node is incredibly expensive. Also, your nervous system is this crossover cable (actually a pair). What happens if I take some scissors to it?

Well, db01 thinks it’s up. And db02 thinks it’s up. But db02 thinks db01 is gone. And db01 thinks db02 is gone. So, now what? What happens if a write comes in to both db01 and db02?

db01:  foo=bar
db02:  foo=ohno

What’s foo supposed to be? s p l i t b r a i n

So this is why you configured a quorum disk. When the cluster looses quorum, there’s a race to the quorum disk. It writes a magic number to the start of the disk sector (not even like in the normal part of the disk iirc) and whoever arrives 2nd, panics on purpose. Now you have survived split brain. But you needed crazy shared disk technology to even do this for arbitrary reasons.

It was a crazy time and I should share this as production horror chops sometime later. A lot of the technology in this story is super old. But some of it hasn’t changed. When I learned Mongo, I had a high degree of context from this horror and I didn’t have to ask “yeah but why” a lot.

Way back when, our CEO couldn’t stand to have half the hardware sitting around doing nothing. He wanted it involved. It’s not like it’s a “dumb idea”. It was a good idea! A lot of people have good ideas around the database. To me though, databases ruin all good ideas. This is how I chunk it. I know it’s cute but it keeps coming up.

Your 1% A/B Testing is A Lot To Me

07 Jul 2021

There’s this popup thing happening. I’m not sure it’s because I really don’t care about GDPR cookies is making me exhausted. I think there’s this business optimization thing that I want to talk about.

youtube red popup

Imagine we have a company. We’ve been in business for a while and we’re public. But along comes automation and analytics and we find if you pull this lever, you get a few more hits on the website. If we send an email, we make X. If we send an SMS, we make Y. If we put a banner on the site, we make Z. On and on. And us having scale, dashboards and reports; this is almost like a noise function through a filter. We’re tracking our lever pulls and our knob twists. This is what we wanted all this information for. We wanted to optimize and act.

So we make our site, our cart, our onboarding, our existing users’ experiences all have some options to randomly upsell or increase revenue. Not on purpose from the start but iteratively through many small changes. Why wouldn’t we? If someone finishes checking out, we send an email making sure that everything went fine and that email has more product links. When we do this, we notice that we make +X%. Just on random noise from sampling.

# sample all users as some_users
# send marketing to some_users

Flipper.enable_percentage_of_actors :youtube_red_popup, 1

But now in this (long established) digital world everything is like this. I get sampled so often that I get popups as not occasional crackles and pops but as constant noise. This is aggregate personalized noise across all the services I use. I get the random sampling so often that I approach the constant random noise that the feature flags were trying to avoid from their perspective. But this is the problem, it’s just one perspective.

If I am 1% sampled on the many services I use, I experience annoyance beyond what each service by themselves expected.

The particular numbers don’t matter. My point is, it’s not 1% sampling to me. I’m a part of many things but the single things think that they are everything.

This is what they think their sampling is like. From their perspective the annoyance, call to action, popups, upsells are rare. error boundary

But this is how it is from my perspective when I’m a user of many services. error boundary

In Fast and the Furious everything is cars. Cars solve all problems. There are no bikes. So A/B testing which car produces the most click-through makes all the sense in the world. But you can’t consider bikes. Bikes don’t exist and certainly not bike click-throughs or bike prompt exhaustion. “It’s only 1% car prompts, that’s not that annoying.”

Ok, back to the youtube red popup. Even if we could design a popup with memory (this absolutely could be a thing), no for-profit company will use it. We could absolutely design a popup component that has memory. “How many times has Chris dismissed me? Maybe I’m annoying!”. No one would use it. Certainly not at scale. At scale, 1% is amazing. It enables projects, it destroys worry.

error boundary

There’s this great talk from Velocity NY 2013 where Richard Cook explains that businesses never know where the failure line is. This isn’t really in the same domain as reliability but I think it applies. It’s a great talk, you should watch it.

You fiddle with these knobs and see the profits coming in but who is going to represent the users? It’s only until after you have negative revenue impact that you’d have ammunition to argue against money. The feature flags continue.

Git is Below Your Project

05 Jul 2021

Tricking Git is Tricking Yourself

I’ve sometimes seen people asking about dependency management, hooks, tracking bugs and other sort of higher level (to me) things than git provides. You can see this if you look at stackoverflow questions about submodules. What’s wrong with submodules? Well, compared to what exactly? When I do a clone of a project and run yarn install, it gives me a list of CVEs that match. When I do a bundle exec it loads my project and has an opportunity (with a very high level of context) to tell me that I’ve forgotten to run migrations or run yarn update in a while. You don’t get this with git. Maybe these examples are too web-tech specific. But I’d like to suggest that this pattern will probably apply to Go, Rust and whatever else. Git is below your project and your project is trying to get better stuff done. So stop trying to solve your problem with Git and listen to how a few other communities do their thing.

I’ll also say that every project is different and as much as I want there to be universal truths, project differences really put some of this stuff into a spin. A lot of this is “to me”. But, I’ve also seen people doing “weird stuff” with Git and when I probe, they haven’t seen or felt success and so they are turning to Git as the tool they already have in place.

Git is really dumb (I mean the cli utility, not the “wrapper frontends” like Github or Gitlab). It’s in the name. It’s in the slogan: “stupid content tracker”. It only really knows how to work with text. You can teach it to understand machine learning binaries and possibly image assets but this is fighting the default. Game developers know this well (I don’t). So it’s interesting when I see other people trying to do it anyway. What I see is a lack of tooling in the language they are using.

Let me give some concrete examples.

  1. Someone that is trying to do Git tricks for a game project. The answer is go inside your editor (in this case, Unity). They probably continued with git submodules, I’m not sure.
  2. Someone who was trying to track what code is in what place. They have parallel supported releases apparently. The answer here is probably a spreadsheet, automation software or feature flags (this whole project was a bit of an oddball).
  3. Someone who is trying to run code automatically on the repo (git hooks). The answer here is CI.
  4. Someone who wants to share code between projects so they want to use submodules. The answer here might be to look at your language’s packaging and produce a library just like the ones you are consuming from the Internet.

In each case, the details don’t matter too much. Someone is trying to trick Git into doing something. It’s almost like a challenge. “If I can sneak past the guard then I can …”. Just stop for a minute. Listen to other projects and how they are doing it. Explore other languages. You don’t have to learn the whole thing. If you are stuck in Java or C++, learn about Yarn/Cargo/Bundler. Look at what Go went through.

git is a water pipe

But most of all, move up a level. Instead of hatching a Git plot, move toward your language, your IDE, your framework or your engine. Git is this plumbing bringing you water and you need to add the Kool-Aid packet for your Kool-Aid. It’s so much closer to what you are trying to make.

Let me give you two more examples while sharing a couple of neat tips.

Getting Out of Yarn.lock Hell

You are working on a team and two people modify package.json at the same time. Your project is using yarn. This means the machine generated file yarn.lock is going to conflict. What should you do? Do some git cherry picking wizardry? If we follow our above rules, we will use Git eventually but we want to lean in to higher level tools, in this case yarn.

# dealing with a yarn conflict
git checkout origin/master -- yarn.lock
yarn install
git add yarn.lock
git rebase --continue

We’re keeping our package.json changes but letting yarn do the work of resolving the graph. Easy and it’s higher level than text.

Are We Breaking Anyone?

You have many clients on many versions. You have concurrent support. You want to make a change but you don’t know if you are going to break anyone. Should you create a complex system of tags, SHAs or feature flags? Maybe. But if you want to track where you’ve deployed your code and on what version, you could do this with a spreadsheet (maybe automating later) but what about this particular problem of “did I break someone?”. Using Git, the idea would be something like reacting. You have all these concurrent versions and you want to track each of them so that you can do this whole backporting and parallel support thing (which is expensive).

If you have a web app, you could use contract testing with pact do handle the “can-i-deploy” question (it even has this as a feature). But what if you have a CLI? Well, can’t we see the pattern here? Look beyond Git and see how Pact is approaching the problem. It’s parallel specs and you want to know if your change is going to break anything.

Of these things involved:

  • Contract Testing
  • Feature Flags
  • Tracking Deployments and Customers
  • Backporting Code

Only Backporting Code is related to Git and it’s really not that interesting.

Wrap Up

The thing with git is: it’s almost always better to move toward your language tooling. A lot of communities have different values and different strengths. What is obvious in one is not so obvious in another. Tour around a little bit and sample. Bring back what you’ve learned.

No Advice

31 Mar 2020

My new year’s resolution was not to do advice for one fiscal quarter. That ends tomorrow. I thought I’d share what I’ve learned and what happened generally.

Why? Advice to me is very uninvolved. It’s fire and forget.

“I won the lottery, use the numbers I did.”

This is advice. Give me a break. In addition to this, advice is exhausting to me because I worry the most after delivering advice. I continue to think about what we both said. Advice is inaccurate because I actually don’t know everything and your project or your life really is unique. I’ve learned this the most when answering the question “what’s the best git workflow?”. Advice is in demand, there is so much information and tooling available but very little filtering and recommendation engines. People want advice because they want to save themselves the time to find out. Advice is almost always sought ahead of time when we haven’t tried anything. Every once and a while, we want advice for when we are stuck but this just doesn’t seem to happen that much.

So my new year’s resolution was to not do advice of any kind for 3 months. How did it go? I failed miserably. I think I broke the rule 5 times. Even though I said no advice and explained it very plainly to people, I ended up telling stories as plain facts but would wander into summaries that were advice actions. Advice is easy to avoid. Just never say “you should”. For example, an intern asked me if they should switch to CS in college. There’s no answer here but “you should” advice. I avoided it. I told them my early career stories and how I had very bad jobs in the early Internet / IT age. But ultimately this conversation and conversations like this would have me slip up with “you should study CS because it sounds like you’re into it”. Whoops.

But it was nice having this goal. I chimed in a lot less on Slack. I have essentially quit twitter so that’s that. I ignored flamewar bait of any kind, even pre-flame-war bait. “It’s going to turn into advice”. It was good practice. My opinion doesn’t matter was the general mindset and that was good.

Unfortunately, in the abstract world of software and sometimes being in a senior position, advice is what it comes down to. People are looking for optimization. They want lessons learned and stories. You have something to say and they know it. You can’t just be quiet. They want advice because they want to pre-learn. So do I! I want advice. I want to pre-learn instead of bumbling my way through first-hand experiences. But I also know that these questions are so insanely hard to answer without a few paragraphs of context and history.

  • “Should I learn Rust?”
  • “What’s the best tech stack for rapid web development?”
  • “What’s wrong with pipenv?”
  • “Is Ruby dead?”
  • “You are critical of Flask, what’s so bad about it?” (this is a trap that leads to “you should”)
  • (a project underwater) “What do you think we should do?”

These are questions that Quora will accept but Stackoverflow won’t. People want to know these things. But they are valid and terrible questions to me. Mostly because the answer is so long it would be a book. Not because the asker is stupid/evil but because software is too abstract and too rapid. If we could measure anything then there wouldn’t be bias, guesses and we all wouldn’t have to give advice that ultimately, changes with time and perspective.

Advice bit-rots and it wasn’t very good content to begin with. A real mystery is if this experiment taught me anything. Will I chime in less? Will I avoid opinion threads? Will I try to die on hills / represent? I don’t know. It was a good experiment either way. More mindfulness.

Sprinkle Time on That Thing

04 Jun 2019

Take any decision, question, concept, idea, team, project or plan and sprinkle the concept of time on it.

world changes

It might seem obvious or silly but I’ll explain with some concrete tech/project examples in a bit. Almost everything exists within the concept of time. Only a few things don’t. Constants sometimes don’t. PI is a constant. It’s not obvious how time relates to PI. Except when you consider that no one has yet found its end. The final value of PI (if there is one) is constrained by time. The answer of what PI actually is (if it can have a complete identity) is constrained by time. The speed of light is another constant. It might seem like it is unrelated to time. But speed is a function of time. Speed is meaningless without time. You can sprinkle time on the speed of light but this isn’t really what I mean.

What I mean is, you should consider time if you are building something, asking questions, trying to figure out what is best or doing what I’d consider everyday engineering things. Sprinkle time on the thing. It’s always there. So add it back.

Let’s look at some common questions:

Someone: What programming language should I learn?

The question is too open-ended in many ways. Let’s put time related things back into the original question. We’ll ask the person for more detail.

Sprinkle Time on it:

  • How experienced are you?
  • What year is it right now?
  • What has happened recently that is influencing the possible future
  • When do you need a job if that’s what you mean?

All those time details should always be packaged with that question. Without it, we are missing a dimension of the question and it barely makes sense. You can see this context/setup in well-phrased and experienced question-askers on stackoverflow. They setup the situation (present time) and maybe frame their constructed reality (their past).

Someone: What’s the point of a CI/CD pipeline? Doesn’t that slow down everything?

Maybe the person asking doesn’t even understand what CI is about. Add time to what they are doing now.

Sprinkle Time on it:

  • What’s your employee turnover rate?
  • How long does it take to onboard someone and have them deploy to prod?
  • How many deploys do you do per day?
  • How would you guarantee and feel drift of tests to code over time?

We could do the same thing with discussions about waterfall, big design up front, feedback loops, bit rot and just change in general. Change is time. Time is change. Nothing exists outside of time. It’s a dimension in our very universe. And sometimes/somehow it is excluded or forgetten about in decisions, thinking and conversations. It’s easy to add back in. You just sprinkle it like a salt shaker. It’s like a seasoning. Sprinkle time on that thing!

Someone: I hate javascript.

I know what they mean to say but … what if it changes? Software can change. Software will change. Culture is forever but software usually changes.

Sprinkle Time on it:

  • When’s the last time you used javascript?
  • How long did you work at it. We tend to like things we understand (experience/time).
  • What version are you talking about, software can change.

So should you hate javascript forever? Should I bake my criticisms of any language into my brain forever? I understand chunking. But you need to tag these chunks as possibly out of date or have an expiration date. That memory chunk or bias you have is probably going to expire. Especially in software. Software’s whole advantage is that it can change. And that’s going to be change over time. I have a few niggles with Go 1.x (right now) but I know it could change (in the future).

Even this blog post needs time sprinkled on it. This very post will go out of date and bit rot. I have a blog post titled “Why Aren’t You Running Gentoo?”. A very low empathy start of a post from 2004. It’s not a relevant question to me anymore (time) and the tech landscape has changed (time). I have blog posts with the date of the post and no other information for this reason.

you cannot rollback forever

But to me, the biggest problem mistake I see is not considering time and ignoring that things can change. Take git history. You have 15 years of git commits on a web app going back to 2004. You have archived releases saved just in case you need to rollback. But why? Sprinkle time on that idea. Heartbleed came out in 2014. All your commits are useless beyond 2014. What are you going to do? Boot an unpatched system, compile against vulnerable openssl and deploy it? The world has changed. You can’t go back. This is true of most security patches. It’s not that security flaws are being found. It’s that the world is changing. The state of the world does not stand still even for your core technologies. It’s not standing still for your business or features either. You can’t rollback your database that far. Feature flags, sure. But deleting whole columns or tables? Breaking the user experience? It’s forward-only. And that’s how time works.

Ok, that’s enough about that (time). I think you get the point (time).

Testing is a Gradient Spectrum

08 Nov 2018

I’m reading the n-th article where someone mentions TDD (test-driven development) as a magic word that means “doing testing” or something else and I thought I’d write down a few things as a note. There have been nearly infinity plus one articles and discussions about testing and TDD. I don’t mean to pile on the old, dead and old-dead horse here but I just keep hearing language that makes me want to pull out a tobacco pipe near a fireplace and puff Well Actually … and that’s not great without helpful context and more detail.

So, let me TLDR this real quick. There is no right or wrong way to test if you’ve tried many different types and flavors and have your own personal preference. There probably is a wrong way to test your project if you have never had time, don’t care that much or someone sent you this link and this is just another opinion man.

The TLDR here is:

  • Every project is different and has different testing needs and opportunities.
  • Testing increases confidence and reduces “whoops” moments.
  • There’s no such thing as no testing. When you say you don’t have time for tests, you mean automated tests. Everyone tests, you are just doing manual and slow (over the project) testing.
  • Testing is a spectrum and TDD does not mean “I am doing testing”.
  • There are many flavors of testing and even though TDD is usually the most dogmatic form of testing, it’s not “best”.

The Spectrum

a testing spectrum

No Testing

Me: Do you have tests?
Someone: Hehe, I know. We didn’t really have time for this project. Look, I joined late and …

No one does “No Testing” but people think they do. This is typing the code, never running it and shipping it to production or scheduling it to run for real. You never observe it running. When people say they have no tests they really mean they only only have manual tests.

Think about this with “hello world”. Someone would type out hello world, save it and put it somewhere as production and never run it. They would dust off your hands, say “another job well done!” and go home. No one does this. From here on out, this isn’t what I mean by testing vs no testing. By testing, I mean automated testing and that includes your local machine.


  • You are infallable and you type the correct solution perfectly
  • You don’t waste any time by checking what you did at all
  • You are a physical manifestation of Ship It!


  • As a perfect being, you feel alienated from your fellow man
  • You are an impossible construct

No one does No Testing. What they mean is Manual Testing. And this is the point about time. They didn’t have time then for automated tests and they are running manual tests now. Do they have time for manual tests now? Maybe. I’m fine with it as long as it’s an informed decision and it’s not causing bugs/outages/delays.

Manual Testing

This is what most people do when they don’t have a test suite. You type hello world, run it, look at the output.

$ ./some_program
  Hellow orld

"Whoops!  Let me fix that real quick.  Ok.  I think I fixed it, let me see …"

$ ./some_program
  Hello world

Great. It’s “done”. It worked right?


  • Saves time short term
  • You won’t need more tools


  • You have no idea if your change is going to break everything
  • You will wonder and hope each time you change anything
  • The only way to verify correctness is with your fragile and tired human eyes
  • You are definitely going to be running ./some_program a lot

Some Partial Testing

This is when there are tests but maybe only small coverage or one type (like unit tests only).

There’s a huge inflection between partial testing and manual testing. The manual testing project has never had time, doesn’t deeply care (or deeply know) and has had little positive experience (if any) with testing. There is a huge gap here in culture and minds. It could be developer minds, it could be boss minds, who knows. This is the mind-gap where you have to explain what testing is. This is the mind-gap where you try to tell stories and impart wisdom. This is where you try to explain the feelings of having tests at your back when you are trying to refactor or understand legacy code. Sometimes, you might sound crazy or religious.

Cutting the team some slack, maybe there are constraints. Usually there are constraints. Constraints can keep a project or team from making their tests better. Maybe the domain, language, history or some other constraint is keeping the tests from becoming better. But there is a test suite and someone cares or understands at least some aspect of testing.

Maybe people are trying their best. But I would argue that partial test teams haven’t deeply felt tests helping them stay on track and ship quality projects. If they can explain this blog post then I believe them. If they can’t, they haven’t had time yet and maybe they will. It’s not their fault but they also aren’t requiring that testing is a required tool of the trade.


  • At least you tried to write tests
  • Less developer time (in the short term)
  • Maybe you don’t waste effort trying to automate testing some horrible legacy GUI with a hack
  • Minimal tests run fast


  • Major aspects of the system aren’t covered
  • Confidence isn’t really there
  • Maybe it’s testing theater, maybe you give up on your tests
  • If you have a UI or web frontend, that bit probably breaks a lot with unit tests only
  • Lack of testing ecosystem understanding and strategies to enable more coverage

Excuses can live here too. And some products are hard to test. But have these options been tried?

  • I/O and Third-Party APIs (can be faked or the seams designed away with dependency injection)
  • GUIs and Mobile device specifics (could be mocked?)
  • The product is actually on the moon, how can we simulate the moon? (decrease gravity?)

I would argue some of these things can be mitigated and you really need to reach for a lot of tooling and language features to fake some of this stuff. And maybe it’s hard to test everything But that’s why you don’t need 100% test coverage. But yes, some projects are hard to test. Some code is hard to test too but sometimes that can be fixed and the developer learns a ton about refactoring, their language and design.

I had a project where I thought AWS services were hard to test and my app was breaking in weird ways everytime I pushed it out. Then I researched a bit and found some tricks and my app wasn’t so different between my laptop tests and public reality.

Complete Practical Testing

Some form of complete testing where units are tested and the system is tested (like UI testing or regression tests).

Engineers on the team have a culture of including tests with work estimates and expectation. The organization either supports this kind of work explicitly or implicitly. It doesn’t really matter. This is a purely practical decision and there is limited value in “abstract testing values”. Tests exist, that’s good enough.

This probably means the project probably run many different test suites where regression tests are run occasionally but some other group of tests are run more often as code happens.

Your repo:

src/...           # the source
test/...          # unit tests (fast)
integration/...   # end-to-end tests (slow)

This is trickier than it seems. That means that git commits, pushes, CI and other tools all have this culture baked in. You aren’t going to run integration tests all the time. Everyone has to know that to be the quickest they can be. Scripts to separate tests have to exist. You can’t just run all tests that match under test/*.

Where this ends is in philosophy and world view. There’s no perceived value (at least as far as schedule, job, work, too busy etc) in doing anything differently. As long as tests exist, that’s good enough. It stops production bugs.


  • Probably good enough for a lot of projects.
  • Production bugs are caught before-hand as long as the feature is correct.
  • Pragmatic and religion-free.


  • Requirements might be misunderstood or misremembered. Implementation comes first and the test comes later, how good are your notes?
  • Doesn’t really enable higher-level domain and customer modeling with things like BDD or user story, plain-english requirements gathering.

Test First Development

You simply write your tests before you start coding implementation. There’s a key difference to TDD that I will get to.

You can start top-down or bottom up. You can start a top-down test first:

# top-down starts with a story
When a user logs into the site
Then they see a logout button

Or you can start a bottom-up test first:

# bottom-up starts with low level details
describe "login controller returns a token on success"

It doesn’t matter. The thing you don’t do is write any code in src or lib. You don’t even start. You don’t even spike. You write a test. Hopefully your test is in a reaction to a conversation you just had with someone who is paying you. Hopefully your quickness to writing a test captures a conversation in an executable format that is checked in and lives and acts. Compare this with an email or a Slack message which sits and rots.

Pros/Cons: I don’t know many projects doing purely this. I guess the pro is not being religious about letting tests drive the design.

Test Driven Development

You let the tests lead design decisions. This is hands-off-the-wheel dogmatic stuff. You limit your thinking really. Requirements are captured early and tests are written first. But more so, you let go of the wheel. If something is hard to test, that means the design is wrong. If a test can’t be written then you don’t start the feature. If you can’t start the test, you ask someone a question. See how testing is the thing?

You don’t really need to do TDD to have confidence before deployment. But it’s trying to fight human behavior. Almost each step is trying to fight some historically problematic behavior (except for manual and no testing).

You probably need a tight feedback loop, tooling and automation to make this happen. It’s also not the best way to test just because it’s at the end of this list.


  • If you are completely lost, it doesn’t work
  • It can be more annoying at the start of the project if working bottom up


  • It can be a creative tool where a design emerges, the problem is solved and by definition the code is easy to test
  • It limits getting distracted while working on a feature - this is subtle and non-obvious from the outside

Enablers and Multipliers

testing multipliers


Let’s say your flow is something like this:

  1. Work on feature
  2. Write tests
  3. Run tests (old ones and new ones)
  4. Ship it!

While working, you might just run your tests or tests that are relevant to your feature/work. But before you ship it to production, you’re going to make sure you don’t break everything right? So, just have a tool that runs your tests. Have the tool tell you when it passes. Don’t deploy until it passes. Call this continuous integration (CI).

Now something else happens when you have continous testing. You can have continous deployment. So, tests passed and you have a build that those tests ran against right? Then that build is good to go! Why throw it away? Why deploy from your laptop? Deploy it! This is continous deployment (CD). Note that you don’t need to do CI or CD but testing is enabling you to do so.


After everything beyond manual testing, a non-obvious thing happens. Automation and tooling.

Let’s say your flow is something like this:

  1. An idea appears. It seems doable.
  2. Start work, maybe start tests.
  3. Finish work.
  4. Write tests maybe after.
  5. Tests seem to work.
  6. Maybe refactor your code and cleanup.
  7. Tests still work. Looks good? Maybe cleanup tests and try one more thing?
  8. Tests still work. Wow, this came out nice and I know I didn’t break anything.

Would you have refactored and tried one more thing if you didn’t have tests?

Boss: “Hey, right now our calculator only has numbers on it. Could we put a plus sign on there and call it addition?” You: “Sure!”

You go to the repo and start work. You add some code to handle the addition, in this contrived world life is simple.

  1. Add a function/method called add that takes two numbers
  2. Write a test for the happy path (numbers like 1 and 2)
  3. Write a few tests for failing paths (strings plus strings, not allowed)
  4. You don’t need to cover all permutations, in your head you probably have an idea of what bad data is, do that

Ready to deploy? Great! Oops. Someone noticed that you forgot to add a test in case an emoji character shows up. Ok, write a test for that. What is your confidence like now? You have a lot of edge cases covered. Is it going to work?

Executable Documentation

How does this code work? What is it supposed to do? Maybe you have code comments (but they bit-rot), maybe you have language features like annotations, typespecs or something. Maybe you don’t. But how do you use the calculator from the previous example? Can it handle numbers that are strings? It can’t! Did you write that down? Can you, yourself remember in a few months? Tests really aren’t docs but they are executable and they can stand-in as docs, especially as API usage. So until you write docs (and maybe you won’t), tests can act as capability documentation, ala “What was the developer thinking”.

Hooks and Addons

You have this testing habit now, why not add some libraries?

When you run your tests:

  1. Notify a channel in Slack that master has failed (broken build)
  2. Pass a random seed in to see if tests, modules or files interfere with each other
  3. See if race conditions happen
  4. See how much of your code executes (coverage)
  5. Load dummy data in to test against (fixtures and test factories)

Notice all these enablers and multipliers happening now that you have a test step.

Wrapping Up

Let me head back to the start of the spectrum. “We don’t have tests”

Compare the workflow where someone just types a program and copies it to a server and then closes their laptop. It seems insane from the very dogmatic land of TDD because their point of view is beyond just “having tests”. But that doesn’t mean it’s “best”. But there is a knowledge gap between each of the points on the spectrum. I’m ok with ignoring parts of the spectrum for a project if it’s understood. If you can explain this blog post to me then we’re cool. If you cannot then I feel like there is a blind spot and any pain points the project is feeling is fair game to improve. If you can explain this post then I’ll mark it up to semantics. If you cannot explain the spectrum to me, then the term TDD is being misused as “testing” and I’d like to explain and help because there is some wisdom to be shared.

There’s very little right or wrong in all this. I am trying to communicate that “no tests” does not mean “no testing”. You are probably doing manual testing. And for certain projects, who cares? Do you care? Do you see yourself hating the manual testing? Then automate it. Are you manually typing in a test user and password, clicking a login button and then clicking around to see if anything broke? Does that annoy you even to talk about it? Then automate it. Are you hand-crafting test data and sharing it with colleagues? Then automate the creating of your test data.

There’s no such thing as no testing and TDD is not completely required although I have enjoyed TDD or Test-First quite a bit when I’ve had the opportunity to use it as an approach. It’s not required to go all the way to TDD because testing is a gradient spectrum.

Golang gb project example

23 May 2017


Gb is a fantastic tool for Golang that let’s you define dependencies but more importantly (to me) is it lets you work out of a normal1 src directory wherever you want. You don’t have to mess with $GOPATH and you don’t have to put your own creations next to libraries. You could even code directly in Dropbox if you wanted to be super lazy about source control and sharing. Overall, I really like gb for projects. It’s more normal to other languages and I don’t have to have Go be the exception to my project backups / paths / scripts / everything.

But I think examples are lacking. The gb docs are great, I’m not saying that. I just wanted to walk through growing a project from small to medium to large and see how organization changes. First, we’ll start by building a fake calculator with no working pieces so it doesn’t need a lot of organization. Then as we add features, we’ll pretend that it needs lots of separation and structure for future expansion and work.

You’ll need to install gb with go get. You probably already have it installed and you know how to google so I’ll just skip that stuff.

I’m going to use the terms small / medium / large but please note that doesn’t mean stupid / insignificant / important. These size terms are just for labeling and explanation, don’t read anything else into it. If you make a small project, it’s not “stupid” just as a large project is not automatically “important” 2.

Minimum GB

First, a gb project is really just a directory with a src directory in it. Of course, nothing will work without some files for it to build. Below is the same error you’ll get even if you make gb_project/src (which gb looks under for source files).

$ mkdir gb_project && cd gb_project
$ gb build
FATAL: command "build" failed: no packages supplied

So, delete that directory and let’s do something more useful.

Gb wants a subdirectory for a package under src to tell it what to build. For our examples let’s make a pretend calculator. Our working directory is going to be pretend_calculator. This can be anywhere. Under your home, tmp or Desktop. Put it wherever you want. Just assume we’re in pretend_calculator as the project root after this point.

$ mkdir -p pretend_calculator/src/calculator

Let’s write minimal code for this to build.

// src/calculator/calculator.go
package main

func main() {
$ cd pretend_calculator
$ gb build
calculator  # showing us gb built the pkg, I'm going to omit this output from here on out

So our project tree looks like this:

├── src
    └── calculator
        └── calculator.go

When you gb build, it will create a binary ./bin/calculator that doesn’t print anything (not surprising, our main is empty). This project layout isn’t that great because the main is really a cmd. If we wanted to add more than executable, we’d have to change where the main() is and rename a few directories and files. So this isn’t great if we’re building an equivalent of Hello World, it’s hard to tell where func main() is if you just look at the filesystem.

$ tree -I pkg
├── bin
│   └── calculator
└── src
    └── calculator
        └── calculator.go

So let’s make this more obvious. Let’s create the start of a simple gb project with a command entry point.

Small Gb Project Example

In this case, we want some actual code that runs something. We’ll have everything in one file under cmd/. Later, we’ll move some code out to a package as the project examples grow in size. The cmd folder in gb projects tell gb to build binaries of that same name of the file or the package. It’s the executable we’re going to run from ./bin.

Now this is a bit tricky. If you name your source file src/cmd/calculator.go then you’ll get a binary called cmd. So what I’d do is name it something like src/cmd/calculator/main.go just show that this is where the main lives for this binary. You can name the file something other than main.go but it needs to be in a subdirectory. The gb docs are a bit vague in their example tree output describing this. Also, note that binaries will always show up in ./bin. So I’m skipping that output in the tree listings.

// src/cmd/calculator/main.go
package main

import "fmt"

func main() {
    fmt.Println("Calculator Fun Time™")
    fmt.Printf("2 + 2 = %d\n", 2+2)
└── src
    └── cmd
        └── calculator
            └── main.go  # <-- file can be named anything, needs .go extension
$ gb build && ./bin/calculator

Calculator Fun Time™
2 + 2 = 4

So this is a nice layout for a small CLI app with not too much logic that would be ok to put into a single file under cmd. If I wanted to break it apart more where the entry point (the main) and the app logic and functions were separated and kept organized, I’d use the medium project layout which we’re going to talk about next.

You could also just add functions to main.go to keep that file clean and then later move the functions around to different packages later.

Medium Gb Project Example

Let’s move some of the app logic to another file and package. This can be super confusing and yet it’s the most common thing to do (in my opinion) when working with Go projects. We’re going to make an add function in a new file and a new package called calculator. Note that this package is sort of arbitrary, it doesn’t need to be your project folder name or anything. Packages are subfolders under src. This will be more clear in the next gb project examples.

// src/cmd/calculator/main.go
package main

import (

    "calculator" // <- this is really our local package in src/calculator/*

func main() {
    fmt.Println("Calculator Fun Time™")

    result := calculator.Add(2, 2)
    fmt.Printf("2 + 2 = %d\n", result)
// src/calculator/calculator.go
package calculator

// Let's not name them num1 and num2 if we can :)
func Add(number int, addend int) int {
	return number + addend
└── src
    ├── calculator
    │   └── calculator.go
    └── cmd
        └── calculator
            └── main.go
$ gb build && ./bin/calculator

Calculator Fun Time™
2 + 2 = 4

Note that we would be planning on putting all functions into src/calculator/calculator.go here. If we wanted to only put the Add function into src/calculator/add.go, we could do that. In the context of a medium sized Go project, we might not want to do that.

Also note that the main.go needs to import calculator. This refers to the package we created. If we want sub-packages and more sub-division, we can do that but we’ll get to that in a bit.

Large-ish Gb Project Example

Just a reminder, my label of large is very arbitrary.

Ok, now what if we want to add more functions and packages. We can continue to do so across files and packages. Let’s add subtraction and the concept of memory storage (you know the MR button?).

Adding subtraction is the same as addition. We just add a Subtract function to src/calculator/calculator.go with a Capital letter to export it. It’s the same as Add. We could split this out to different files if we wanted. Maybe that’s more interesting. We’ll do that in the next example.

// src/calculator/calculator.go
package calculator

// Nothing changes here.
func Add(number int, addend int) int {
    return number + addend

// Subtract 1 from 4 is 3.
func Subtract(from int, number int) int {
    return number - from

Let’s add memory storage. We need to create a struct to store stuff in. So our memory.go code is going to have a struct initializer in it. The function naming is just Go convention, nothing here is specific to gb.

// src/calculator/memory.go
package calculator

type memory struct {
    register int

func NewMemory() memory {
    return memory{
        register: 0,

// MR means memory recall, it returns the contents of a number in memory
func (m *memory) MR() int {
    return m.register

// MS means memory store, it stores a number (normally would be the screen)
func (m *memory) MS(number int) {
    m.register = number

We only export the NewMemory function to keep people from creating structs themselves. Using this struct in main.go for the command goes like this:

package main

import (


func main() {
    fmt.Println("Calculator Fun Time™")

    add_result := calculator.Add(2, 2)
    fmt.Printf("2 + 2 = %d\n", add_result)

    subtract_result := calculator.Subtract(1, add_result)
    fmt.Printf("%d - 1 = %d\n", add_result, subtract_result)

    fmt.Println() // spacing

    // memory functions
    memory := calculator.NewMemory()
    fmt.Println("Storing the result in memory ...")
    memory.MS(subtract_result)                          // store
    fmt.Printf("Memory has <%d> in it.\n", memory.MR()) // recall

Running this now produces:

$ gb build && ./bin/calculator

Calculator Fun Time™
2 + 2 = 4
4 - 1 = 3

Storing the result in memory ...
Memory has <3> in it.

Our current tree structure looks like this. We are doing file organization at this point but we really still have one package (other than main).

└── src
    ├── calculator
    │   ├── calculator.go
    │   └── memory.go
    └── cmd
        └── calculator
            └── main.go

What does a more complicated project look like?

Larger Gb Example

Let’s split out every function into a file to make the project very easy to navigate. Intuition should drive file search. Add and subtract will go in their own files. We’ll add the concept of a tape to display information that will also have the opportunity to save state that will make the memory feature more realistic to how physical calculators work.

I said we’d break out functions into intuitive files. Let’s put Add() into add.go

// src/calculator/function/add.go
package function

func Add(number int, addend int) int {
    return number + addend

And the same for Subtract

// src/calculator/function/subtract.go                                               1 ↵
package function

func Subtract(from int, number int) int {
	return number - from

Next, let’s make a tape.

// src/calculator/tape/tape.go
package tape

import "fmt"

// represents an empty memory instead of using nil which does not communicate well
const emptyRegister = 0

// For simplicity's sake, the calculator tape is essentially the entire electronics 
// of this fake calculator.  A tape probably wouldn't care about current previous 
// values for undo functionality.
type tape struct {
    lastNumber    int
    CurrentNumber int

func NewTape() tape {
    return tape{
        lastNumber: emptyRegister,

func (s *tape) Clear() {
    s.CurrentNumber = emptyRegister

// Updates the internal state of the tape
func (s *tape) Update(number int) {
    s.lastNumber = s.CurrentNumber
    s.CurrentNumber = number

// Displays the current number
func (s *tape) Display(message string) {
    fmt.Printf("| %-22s|%7.1f|\n", message, float32(s.CurrentNumber))

// Just print a blank line like the calculator tape is advancing
func (s *tape) Advance() {
    fmt.Printf("|%31s|\n", "")

// Roll the tape back, behaves kind of like one-time undo
func (s *tape) Rollback() {
    s.CurrentNumber = s.lastNumber
    s.lastNumber = emptyRegister

func formatNumber(number int) string {
    if number == emptyRegister {
        return " "
    } else {
        return fmt.Sprintf("%1.0f", number)

It’s very similar to the last examples, just more code. We have some types and structs in this file but you can see that any Capitalized anything is expected to be used externally. The package has no hierarchy but later when we use it, we’ll need to alias it.

The changes to the last project are simpler that it seems. All we did was:

  • We made a directory called src/calculator/function. The package is now calculator/function.
  • We split Add and Subtract into files named add.go and subtract.go. We don’t explicitly need to care about this when importing.
  • Each of these new files have package function. You can’t declare package calculator/function at the top. Doing that won’t even pass go fmt, it will error.

Memory.go stays the same, it’s in the root calculator package just because.

// src/cmd/calculator/main.go
package main

import (

    fn "calculator/function"
    tape "calculator/tape"

func main() {
    fmt.Println("Calculator Fun Time™")
    fmt.Println(strings.Repeat("-", 32))

    tape := tape.NewTape()

    tape.Update(fn.Add(2, 2))
    tape.Display("2 + 2")

    tape.Update(fn.Subtract(1, tape.CurrentNumber))
    tape.Display("Subtract 1")


    // memory functions
    memory := calculator.NewMemory()
    memory.MS(tape.CurrentNumber) // store
    tape.Display("Hit Memory Store")

    tape.Display("Cleared screen")

    tape.Update(fn.Add(10, memory.MR()))
    tape.Display("Add 10 to memory")


    // rollback feature
    tape.Update(fn.Add(1, 1))
    tape.Display("1 + 1")
    tape.Update(fn.Add(1, tape.CurrentNumber))
    tape.Display("+ 1")
    tape.Display("Rolled the tape back")

Our main file has expanded dramatically as we try to exercise the new packages and files we’re making. We need to give calculator/function an alias (in this case fn) to use a hierarchical package. It’s very arbitrary. We still are using Memory out of the calculator package so we need to import that explicitly like we were before. If you wanted to break memory out, you’d follow what we did with add & subtract.

Our tree now looks like this:

└── src
    ├── calculator
    │   ├── function
    │   │   ├── add.go
    │   │   └── subtract.go
    │   ├── memory.go
    │   └── tape
    │       └── tape.go
    └── cmd
        └── calculator
            └── main.go

Running it shows how main.go works. It might be easier to skim the code and just read the output. It’s a very contrived example but more “real”.

$ gb build && ./bin/calculator

Calculator Fun Time™
| 2 + 2                 |    4.0|
| Subtract 1            |    3.0|
|                               |
| Hit Memory Store      |    3.0|
| Cleared screen        |    0.0|
| Add 10 to memory      |   13.0|
|                               |
| 1 + 1                 |    2.0|
| + 1                   |    3.0|
| Rolled the tape back  |    2.0|

Wrap Up

I hope this was interesting. I’ve been wanting an article like this to exist ever since I started using gb as a tool. I’ve found example gb projects on github that were useful examples but believe me when I’m blogging all this for myself as a future reference. Shoot me a note on twitter if you liked this or would like to see something else, it’s nice to know who’s reading.

[1] Nothing is normal

[2] I prefer better/worse over good/bad. In this case it'd be smaller and larger which is awkward in this case. The only rock we have to stand on in C.S. is metrics, everything else is opinion (like this very statement!).

Why A Dev Log is Great - Two Years Later

10 Apr 2017

pensieve from harry potter

I wrote about setting up a Dev Log about two years ago. At this point, I’ve been using this setup for two years so I thought I’d write a little bit about it as a follow up. After all, I hate uninvolved advice.

What Have I Learned

I’ve learned that the dev log works as a pensieve. It’s a dumping ground for code snippets and dreams. I found it a good outlet for frustrations too. But most importantly, it’s like an archeology site. Let me give you the best payoff of the dev log as a small story.

SSL Only Mystery

We use an external API for mobile stats tracking. It will track installs and other things from the app store. It’s wired up to our own API through a webhook. This webhook has a URL configuration. Originally, it was something like and it had a payload and other such details. We hadn’t received data from them in many, many months. I started looking at this but basically had no context other than this.

Of course, the first question in debugging is what has changed. So what did changed? I didn’t think anyone had messed with the config in months because essentially this service was a set and forget kind of thing. We also hadn’t received Android or iOS data on the same day. Too suspicious, too convenient. So, I knew the data it stopped working. Let’s go to the dev log!

What did we find on the day that it stopped working:

Switch to temporary SSL redirects by default

Later, there are some clues that we were working on making the temporary redirects into permanent redirects. There’s notes all around this timeframe that we were working on making the site SSL-only. Ah.

Change the webhook to https and bam, we start getting rows in the database of payloads. The URL didn’t jump out to me as wrong. It’s obvious now but the dev log helped trigger some clues around this. The clues were also in the git log but not the surrounding context that we were working on making the site SSL-only.

It especially didn’t seem wrong as redirects are supported. It turns out the service we are using doesn’t handle redirects (or at least seems to not). Just looking at the URL as http:// doesn’t seem wrong at all. But with the dev log context, it does. This is what changed.

The Surprise of the Double Me

Just as when you don’t have a pair and you need to be the “high level person” and the “low level person” all at once, I’ve found that my complaints and frustrations come off TO MYSELF as whining. This is amazing. Let me say this again.

Logging frustrations in my dev log comes off as “whining” to myself later.

I still think this is good if it’s a healthy outlet. It’s not good if it lets you polish your whining so it can be delivered as a pithy zinger to an unexpecting listener. The dev log is about capturing your thoughts. Be careful what your thoughts are, you might get what you want. I still like to capture task changes as this represents time lost or spent. Maybe this sounds like whining in the log but that’s ok.

Do Not

Don’t tag or organize your thoughts into an ontology or fancy structure. The idea is to get in and get out. One friend is good enough with org-mode that he was able to structure his log more than me. That’s fine. Make it your own. But don’t start making per-project logs, I think it would just self-destruct under ceremony burdens. The dev log is something I write in during context switches. Get in and get out. See my previous post for shortcut keys.

I however would leave clues for myself like LEARNED: or TIL which could be used for retros. Or PR: S3 refactor if I opened a pull request. The idea is to capture what have you been doing or what is your time being spent on. I capture interruptions or helping someone too. That’s a great thing to jot down when you first come back to your desk or switch from Slack.

Helped Dumbledore with Docker

Two Years Later

10435 lines of text and two years. My intention or goal was never length. It was always the pensieve. Reading back on it is a massive log of bugs, TILs, tech gotchas and a frustration heat-sink. There are face-palm mistakes, logs of miscommunications, “this library doesn’t do that” notes and details.

2017-02-28 - Tuesday

Trying to do a deploy, S3 goes down hard and breaks the Internet.

There are rabbit hole results with fantastic details right before you come back from the rabbit hole:

Envconsul won’t work for us because of our combination of unicorn zero downtime deployment configs, how we want to handle ENV restarts and a limitation of Go. Envconsul won’t work because it does in fact restart the app correctly if -once is passed and you -QUIT the worker. But since -once has been passed, you can’t reload the environment.

This is the detail I wanted to capture so I can chunk it later as “we can’t use envconsul” and then I can just text search for this later. This is how it actually worked many times.