Default DC Tech is Just Bad

dc_capitol

The opinions of this blog, but especially this post are mine and not my employers’.

I’m done with DC. I need to archive the reasons why for myself. I hope this serves as a free field trip to the DC area for anyone outside the beltway.

TL;DR

If you move to DC for the tech jobs, you are going to have to prune a lot of C-minus government work if you are good. All the while, you will be paying for local benefits you are not taking advantage of. This is the land of politics, military, intelligence, big government and lobbyists. I tried to influence from within but now it’s time for me to GTFO and move to Portland to try to find “actual reality” jobs.

I’ve Been Here Too Long

I’ve lived in the DC area for 30 years. I’ve been working for 14. You might as well say I’ve lived around DC my whole life. I went to School and College in Northern Virginia. Northern Virginia and Maryland are part of the DC metropolitan area or what we would call the “DC Metro”. This is a little bit confusing because if you say “The Metro” then you mean the subway. I’ve traveled around and lived other places but you could say I’ve been here for a long time. Recently, I’ve concluded that DC isn’t all bad but I’ve been here too long. Also I feel like a slow learner in that it has taken me so long to realize these things. I’m doing something about it and I’ll talk about that later.

I’ve done the DC government contracting circuit and I’ve concluded that I’ve seen enough. It doesn’t really matter what company you work for, it’s the same project with the same people and the same problems. However, a few years ago I found a small life raft. It’s a non-profit R&D shop and it’s the best big tech company I’ve found in DC. It’s not perfect but at the core of it is the non-profit bit. It’s not Beltway Bandit time.

Beltway bandit is a term for private companies located in or near Washington, D.C. whose major business is to provide consulting services to the US government.

At every company there’s been good and bad parts. I’ve been happier lately because of my move to something non-typical but I’m still done with the area. The problems are very complicated but at the core of it are a few common patterns which I’ll talk about later. It’s not the companies’ or the people’s fault, it’s the Beltway culture. It’s not micro, it’s macro. It’s not the individuals, it’s the whole. It’s This Town.

At the same time, there are some start-ups and smaller consulting shops that I’ve found very interesting. Viget Labs is outside DC a little bit and they do cool stuff like Rails consulting, UX, frontend stuff, iOS (iirc) and even have meetups in their corporate office space. But I think this is because they are bringing outside culture into DC not because they are native. Default DC tech doesn’t host meetups, have hack-a-thons or get valley culture right. It’s emulation. I look at the lame attempts to adopt agile in recent years and it’s just cargo culting.

Using Agile Effectively in a DoD Environment – Give me a break. It might as well say “doing fast things in a slow environment”. It’s a fight. You have to fight the default. If you listen to speakers at talks like this one, sometimes you will hear these exact words. “We had to fight … against the grain … it was a struggle to …”

No Good Advice Goes Unpunished

Of course, there’s no silver bullet. You can have a crappy job in a great company. You can have a terrible boss in a great town. You can have an awful commute in a great job with a great boss with awful pay. There are permutations on all these factors and more. Even more complicating is that one person’s experience, perspective and interaction with a position/job can be different from another person’s experience within the same situation. So, please treat all of the following as retrospective and a personal time capsule that, as a bonus, might be useful or cathartic for someone other there in The Ether.

It’s also particularly hard to give advice about such a large topic. I can’t tell you how your experience is going to be in your work life just like I can’t tell you how your experience is going to be in retirement or if you were going to college. College is like some things for all people and different things to each person.


Story Time

dc_old_mainframes

I have a lot of horror stories, some are mostly funny, some are mostly sad and some are a mix of funny/sad. I’ll try to focus on the funny/sad ones in that they are the most telling about the culture in DC.

The No Clue About Webapps Project

There was this project I sat near (was not in) and I heard all kinds of C.S. sins going on. At some point I got the impression that they didn’t know what Subversion was (this was a while ago) or even CVS. I said the phrase “check in” and the senior developer didn’t know what I meant. I mean, from a conceptual standpoint. Jesus Christ, even Sharepoint uses this terminology.

Anyway, at some point I got roped into installing local Oracle dev databases for everyone on Windows. They didn’t know how. (it’s just next, next, finish but whatever). After that, I was asked to come to a big team meeting. In the meeting the project manager starts off by apologizing for not being around for the last two years. He’s been “on-site” (btw this is a typical problem with closed networks, you are completely cut-off from everyone and everything). He says the customer is really mad because nothing has been done in the last four years. He starts going around the room and asking his team what programming language everyone knows (what). People seem to know Java so the manager said, “ok we’ll do it in Java”.

So I’m just sitting in there as the outsider with a box of popcorn enjoying the show because (little do they know) I’ve already given my two weeks notice because this place isn’t even close to being functional. In this meeting they are trying to plan out the project in panic mode, so they review what needs to be done. They break out the old project and demo it. Basically, they need to port it from some old language like Cold Fusion to Java (which they just decided after 2-4 years, wow).

The demo is an interactive map of the U.S. To me it looks like you can click around the map like an HTML imagemap. But they start planning out the project like this:

  • Ok, there’s 50 states.
  • It looks like you can drill down into about 4-5 counties per state.
  • That means we have 300-ish maps we need to port.
  • Kevin, you take 80 maps. Stacey, you take 120 maps …

I’m sitting there looking at the URL and what’s actually happening is this:
http://legacyapp/map.cfm?state=1&county=20

So I say, “Um. It looks to me like you guys have one thing to port. The map.cfm page is just swapping IDs out. There’s probably some geometry file somewhere that you could reuse or some database data somewhere that swaps out the images or imagemaps or something.”

Suddenly the room goes quiet. No one knows who I am. I’m not on the team. I had never seen the legacy app before. Some of them hadn’t seen the app they were supposed to be working on either. They just stared at me while the project manager tried to get me to join the project to help. “(to me) Wait, what project are you on right now?”

As much as I would have liked to just say “I’m quitting, GG” in front of everyone, I didn’t really want to bring the team down. So I just told him in private I had already given my two weeks notice after everyone left. He said, “darn”. Because he wanted me to fix and advise on this crazy train. But I was already leaving (phew).

This was a 4 year web porting project where people didn’t know about URL parameters. Wow, just wow.

The Porn Guy

I was tasked with inventory of memory or something of desktops (this was a while ago when I did sysadmin stuff). I walked around cubes in Kentucky (I forget where) and bugged people while they were working, “Hi. Can I look at your computer for a second? Thanks.” etc.

This one guy sat in a cube (his back facing a common hallway) and he was surfing porn. I cleared my throat and waited for him to be embarrassed or shut down the windows or apologize. I don’t know what the hell I expected, “this is awful” was what I was thinking. He kind of was startled and said, “oh!” in a surprised voice and closed an IE window. But there were more IE windows of porn underneath. So he said, “oh!” in a surprised voice and closed an IE window. But there were more IE windows of porn underneath. This happened over and over and it was hilarious at the time.

Looking back, I probably should have reported him and I feel awful about this story since I’ve been reading #YesAllWomen hashtags. Sigh.

Ok, maybe that was in Kentucky. But I remember there was this guy downtown that built a cardboard screen so he could watch porn at work. Sad. Granted, I’ve been involved in firing someone for this behavior (and harassment) at a commercial company. It’s more an ick story I guess. Why in the **** do people do this at work anyway?

The Sleeping Boss

In a similar desktop inventory task, I went downtown one time. I walked up and down a long cube row. It kind of looked like a Viking ship. The boss was at the head of the office and all the workers were lined up on either side of a main isle. I remember coming to this one guy who was playing a Playstation (ps1) on a TV that he brought in. He had a VCR too. His cube shelves were stacked with VHS tapes. I asked him what his job title was. He didn’t know. I asked him what his job was. He didn’t know. I asked him where he got the TV and Playstation from and he said he brought it in from home.

Wow! So here’s a guy who is telling a stranger with a clipboard that he doesn’t know what his job is. I went to the boss at the head of the ship because I needed to inventory his PC (although I should have started yelling about tax payer money) and here’s this guy snoring away in his chair. Literally sleeping in his little private office. I knocked really loud and woke him up. I received no apology. He just said “what huh” and then answered my questions.

Unapologetic isn’t alarming in that I deserve and apology. It’s alarming because no one cares. Nothing’s going to happen to them. They don’t feel shame or embarrassment because there’s no force of fitness like natural selection that would enable evolution pruning.


Great Stories

There’s no black and white. I’m probably anti-DC right now because I’m trying to move. As painful as it is to admit, I’ve spent a lot of time here and haven’t died from emo so it can’t be that bad.

  • I sat on Bush’s desk and set up Cheney’s Wife’s computer. That was nuts. My personal politics aside, it was cool to help people that were going to the White House.
  • I got to travel and see the country including Alaska which was awesome.
  • I helped some people catch some child molesters (may they burn in hell). There’s a lot of work in DC that isn’t about catching terrorists and other ethereal goals.
  • Once, someone asked me if I wanted to work on a quantum computer project. They were serious.
  • I’ve met some people that worked on The Internet. Like, the actual Internet. Everyday people have had historic roles. A lot of them are humble. I’m not sure the valley does that all the time. Who knows.
  • Was able to guide some (not all) projects away from rocks. If my goal was to influence from within, I did.
  • You can work on big systems. Clusters, huge storage, large concurrent queues, events, streams, billions of rows, zillions of things.
  • I learned Unix, Java and had time to learn Rails/Ruby and play with Python, Node, Scala, Golang and countless libraries and databases. There’s time to breathe.
  • #dctech is a good community. AgileTechDC is a good conference. But I think they are the outsiders to the default and that relationship should be flipped in a perfect world.
  • I met chinese, spanish, korean and other native language speakers just because DC is so diverse. I’ve learned a lot about languages just by osmosis.
  • I met the love of my life here.

The Good Stuff about DC

dc_arlington

  1. No burnout. If you work a normal DC job, you can go to work and not burn out. 9-5 is very, very normal. No one talks about Sustainability like they do in the valley, you don’t need to. Normal hours are the norm. This also means though that organizing hackathons are both unknown and foreign.
  2. There are lots of smart people in DC. DC has the highest literacy rate in the country. People want real answers for policy and practice that has real work to be done. If you are involved in practical problems, there’s a chance to work on national issues, crises and programs that make a difference without selling out. Sort of.
  3. DC has a lot of things to go see. I saw Stevie Wonder live with Michelle Obama in the audience at the Kennedy Center. Stuff like this is amazing.
  4. Seeing The Capitol when you go downtown never gets old.
  5. Schools are great. There’s this high school called Thomas Jefferson Science and Technology. I think they’re like #7 in the country. Amazing kids go there. In general almost everyone goes to college. I think my graduating class was like 98% went to college or something (maybe times have changed since the 90s, lol). Depends on your opinion of college and whether that matters. In-state universities (VA/MD) are as good as you need them to be, there are a ton to choose from.
  6. Crime is low in the ‘burbs. So many people have clearances around here and money is good so everyone’s a straight-edge (sort of).
  7. You can work on national problems. Sometimes a little flag waves in my head and I see a real democracy. Sometimes not so much.
  8. You can get a job anytime you want. A job. Maybe not a good job. Just a job.
  9. If you have a clearance, you can really, really, really get a job anytime you want.
  10. You will probably earn a lot of money. Two of the richest average income per household counties are in N.VA. I could care less at this point but this is true. Almost everyone is doing very well. Sometimes it’s sickening.
  11. Downtown is awesome. Arlington VA is full of life and young hip people. If you are in your early twenties (or feel like you are), live in Arlington (Clarendon/Ballston/Rosslyn). If you are rich, live in Georgetown or near Capitol Hill.
  12. Nats baseball, going to see a game. Fun times, nice park.
  13. The Washington Post is a good paper. Local Jazz radio is good.
  14. The formal arts scene (jazz, ballet, concerts, musicals, plays, displays, etc) is great. Big names come through because it’s a big town.

The Bad

dc_traffic

Traffic

I would swear very offensively here but I’m trying to cut back on my bad language. I can’t describe how bad the traffic is. Above is a picture of what you can expect on a Saturday (or maybe not, who knows!). My wife and I exclaim in horror at what we find when we go out. “Traffic on a Sunday. Wow.” The rush hour is insane. You will plan around the traffic. You will not take jobs because of the commute. You will find people who hate downtown. You will find recruiters who ask you where you live FIRST because they know you will never take a job where your commute is one hour if you are any good. And some people would kill for a 1 hour commute! I know people that commute from Leesburg downtown and it’s like 2 hours each way for them every day. What. How can people even.

Weird Cobweb Jobs

Infinite Sharepoint Administrator jobs. What. How can “Sharepoint Architect” even be a job title. Bad Government work is so hard to prune and avoid. There are really weird cobwebs out there. When you live here, you are competing with these salaries. They keep home prices up.

Sprawl

The Metro ride out to suburbia is pretty awful. You probably can’t afford to live downtown or close in. “Normal” houses cost $600-700k (there are places so much more) in the suburbs. It’s insanity.

You can spend one million dollars on a house anywhere. I think what is unique about DC is the average cost and wealth is very high compared to the rest of the country.

Un-Culture

Redskins won’t change their name, that’s status-quo in full effect. If status-quo aggravates you, don’t come here. Being so close to B.S. politics and you still can’t change it. Most people are here for the jobs so they really don’t care about you. East coast hardness. Most people don’t say hi.


Repeating Themes

I could probably go on with The Good vs The Bad. It’s just too much to list. I’m trying to keep this readable but I wanted to list out some recurring themes I’ve seen specifically within the tech industry in DC. This is not company specific. I’ve seen it over and over again. Worse yet, if you talk about this to someone who’s near the Beltway, they will know what you are talking about. So please don’t associate my employer or my past employers with this.

  1. Butts-in-seats contracting. People make money off other people working. You can make bank if you know the right people and get a fat contract. Hire bad people, doesn’t matter, a lot of government people won’t know the difference.
  2. Take a core sample, government folks aren’t technical enough. Fine –> Hire tech. Except they don’t know what they hired. Whoops. This is subtle but it causes sort of a chain reaction of problems on a project. They don’t know enough to ask the right questions. What is really needed is what Google already does: only promote ex-technical people to managers and have very few managers.
  3. Bureaucracies serve the needs of bureaucracies. The needs of an individual are not served. Case-in-point: The idea of a developer specifying and ordering his own workstation is comically alien. You might as well ask for your desk to be hung upside down.
  4. Security issues. Even if you aren’t in the cloak-and-dagger space, chances are you will be near it and that means a truck-load of issues that will decay your soul. Imagine no Internet at work.
  5. No designers. Designers and web developers live on their portfolio. No portfolio = no work history. Most government sponsors won’t let you take screenshots. So you have to design in your spare time? For the love of it? I’m not this type of person but I know a few people that have moved out of the area for this reason. What this means to me is that most UIs you work with are total pants. Total pants! UX? Ha. No one even knows what that means.
  6. Politics matter. Congressional BS will impact you. I have friends who have lost “stable” jobs because of some really weird contract expiring or the “wrong type” of money being in the “wrong bucket”. This space is insane. I’ve worked on one proposal before and I never want to do it again. Politics also means word-smithing. People are super careful about saying the right thing using the right words. A lot of the time this means corporate speak.
  7. Modified agile. Software craftsmanship people are moving on from the term agile because it’s been rendered meaningless. This decay is happening here.

So I’m Moving

mt_hood
I gave it 14 years but I’m moving to Portland. Hope springs eternal in the human breast. I understand that nothing is perfect. Things like the Calagator event calendar look absolutely amazing to me. Look at this schedule!

May 29 2014 PDX Weekly Hackathon
May 29 2014 Bruce Schneier: "Internet, Security, and Power"
May 30 2014 Ruby Programmers Informal Co-Working
May 30 2014 Python Web Development with Flask
...
May 31 2014 Scrum/Agile Workshop by Agile Coach/Trainer at Intel
...
Jun 2 2014 Triage-a-Thon at Puppet Labs
Jun 3 2014 Google Glass versus the COMPETITION
Jun 5 2014 PDXNode presentation night
....

Yes, there are similar events in DC (Bruce is probably talking here). There’s a lot of cool stuff happening in an Open Government mailing list called Sunlight Labs. I wrote a blog post attempting to enumerate awesome shops in DC (I picked Rails as a starting point). It wasn’t as long as I had hoped. Peter Corbett is an awesome guy. I’d seriously suggest networking with him if you get the chance.

I hope in a year I’m blogging about myself in a startup, lean company or more forward-leaning shop. Or maybe I’ll have created an art installation with Golang or something. We’ll see! :)

Passion

On a Ruby Rogues podcast about Passion, Avdi continued to enlighten and entertain me with his insights. I’ve really been enjoying his speaking style and voice lately through tapas and talks. If he reads this, I hope he understands I don’t disagree with what he is saying; I thought he would enjoy a related story.

Honestly, this topic is so massive I don’t think I can really offer too much more than the Rogues did on the podcast so I encourage you to listen to the episode yourself. It has almost nothing to do with programming or Ruby. I feel that philosophies and stories about passion are so close to the difficult and inevitable goal of “master yourself”, which is both complicated and personal, I can just barely approach the topic and then a rat’s nest of anecdotes and advice explodes all around us.

With that context laid out, here are a few stories.

On the podcast, the rogues talked about two PhDs that would scream at each other in the office on a daily basis. Avdi said that he’s had similar experiences personally and said:

I’ve gotten upset with people for their code because it was so stupid. I’ve gotten angry and I’ve said mean things. And you know what? All those instances, those were wastes of my passion. That was wasted emotional energy.

I know what he’s trying to say. There are moments when this is true. In fact, I would say in the majority of cases it’s better to just “get over it” (a challenge itself). Most of the time I try to run in this mode. Most of the time I fail. It’s especially hard when you feel like you need to “represent”.

Story Time

Here was my situation. A “lead architect” I sat with in a shared office room was named “Bradly”. Bradly was not his real name but it will help you remember that Bradly was Bad. He was a lead architect of our project but he couldn’t code and he couldn’t build servers. He had some very narrow skills in a certain problem domain but those skills weren’t general enough for him to be an “architect”. He got the title/position through a previous successful project. I’m just setting up the scenario here.

We had an application in the middle-tier, doesn’t really matter what it did except that it talks to a database. His bright idea was to install a database on every node to reduce network traffic. We were vendor-locked into Oracle. We had at least 10 servers that this design decision would impact. We really needed 1 database but we would be purchasing 10. I was the only one on the project that could or would argue against the decision.

It was very simple from my point of view:
- Three tier architecture is front, middle, back. Normally that’s web, app server, database.
- No one installs databases on their app servers to reduce network traffic.
- We didn’t know that network traffic is the (or a) bottleneck.
- Oracle database licenses sell for about $20k + $?? annual support.
- We were going to have a failover site so this single decision was on the order of $400k.
- Running 10 databases is hard. Replication is hard. Oracle RDBMS does not “want” this layout.

Bradly’s argument was:
- Networks (gigabit, brand new awesome switches) are slow.
- Local databases would avoid the network.


Just to be clear, this is a simplified version of what Bradly wanted.
tier_architecture_one


This is what I wanted.
tier_architecture_two


I did my best to remain calm but this is the represent problem. I felt compelled to bring the “outside world” to his mind. I knew that no one did it like this. And I imagined a stadium of my peers agreeing with me if they only knew what he was saying. “What would the Internet think? Oh my god! They would laugh! Imagine our embarrassment! What kind of project is this?!” etc.

I tried to approach it logically. I showed vendor diagrams and documentation of example architectures from the very vendor product we were using and no diagrams showed locally installed databases. Eventually it came down to his argument: “I own the architecture, I’m the architect.”

In this case, I was very passionate. I was furious. I had to represent the outside world. I couldn’t let it go. I had to teach him that no one does it this way. I had to represent.

Drama

We eventually got into the only screaming argument I’ve ever had in my 14-ish year career. It was bad. We didn’t talk anymore. We split the team. We pitched our designs to management separately. Management was trying to side with one of us since money was involved. I moved offices so I didn’t have to sit with him anymore. It was no fun, it was a bad situation.

In the end, he moved onto to other projects and I was able to influence the project back to normalcy. When new developers joined the project, some artifacts of this debate would surface (like an old diagram). Someone with three-tier experience would say “what the hell is this” and I’d have to explain the whole thing. In those rare cases I felt righteous but to this day I feel awful about the whole story. My working relationship was ruined with him and I hate the memory of that job just because of experiences like this one and others.

I’d rather teach than win. But he wasn’t there to be a teacher and I wasn’t given that role or power. When some people go to work they optimize for their career. They start off the day wanting to perform a job, move up and at the very least maintain the power they have. Because this is the best they’ve ever been in their career.

When I wake up, I optimize for experience. I start off the day ready to learn, improve and at the very least make myself or other people better at what we spend our time doing. Because this is the very worst I’ll ever be in my career.

So Avdi, in this and other things, I hear you.

DRY up Methods with Ruby Blocks

pixel-ribbon_redolution

Let’s do something terrible by hand. First, here’s our data. It comes from a database.

Now when working with these people, we probably could get away with doing something like this for a while:

Which is fine. Until you want to find out what people are on the Muffin Project:

But as you keep working, you might be getting a feeling of deja-vu. The two methods above are very similar. You might be inspired by other Ruby libraries which give you a tiny DSL or at least allow you to pass blocks into methods to be more expressive.

The Smell

Here’s the complete code smelly example.

We’re having a meeting between the admins and people who are on the Muffin Project. The only person not matching these rules in this case is Bob Barker (bbarker). He must be busy enjoying retirement eating pie, who knows.

Inspiration

Let’s take a look at Faraday. Faraday uses blocks to great effect to communicate intent just like most libraries in Ruby. In Faraday, this is how a HTTP POST is done using Faraday:

This is kind of nice! You can get more than one thing done at a time and it doesn’t require a lot of temporary variables. Let’s see if we can use blocks like this. We’ll get to blocks in a miniute. Let’s first refactor a little bit first.

The Fix

There’s a certain similarity between the two selects. We really want to get “admins” and “project people” all together, so let’s just do that. We’ll create two methods that essentially replace the instance methods but can be used in the future for other rules. We’ll call them .with_roles and .with_projects.

Next, we’ll create a method that takes a block.

The &block argument and yield block is optional. You could write this as:

But in that case, the block is optional, so you’ll want to check for block_given?. For this example, it’s easier for us to require a block to make this a shorter post … err, well I guess it’s longer now.

In any event, this method’s job is to filter results (users) with whatever code is passed in. Then it uniques the collected array because user IDs are assumed here to be unique. Finally, it returns just user_ids like it’s name implies.

The usage of this user_ids method that takes a block ends up reading very well.

Here’s the completed, less smelly example.

Wrap Up

This is pretty procedural. I’ll leave it to you to put it into a class, maybe add something better than a “plus” operator to combine the user list together. Maybe a UserList abstraction class could help get away from hashes too.

I like going down these paths because you end up with more expressive code that is flexible to change. At the same time, little hints of DSLs come out when using blocks to this effect. This is starting down the path of a Ruby DSL. I’ll be posting about that pretty soon.

Problems with “The Cloud”

pixel-ribbon_grass
I’ve been thinking about the problems with The Cloud outside it being a raging buzzword. It really comes down to Control and Connectivity. That’s the problem but allow me to elaborate.

Control

Google Wave is a great example of control loss. If you really put a lot of energy, stock and trust into Google Wave as a content store for your team, brain or idea then you might feel deflated by its cancellation. Even as an idea and a disruptive alternative to E-mail or SMTP crappiness, it’s a shame it had to die. So what now? Wait for an open source version? Host your own?

The idea was to “put it in the cloud” and forget about it. But when the cloud changes outside your control, you have to be aware of it again. Now you really have to think about the cloud itself. It’s not such a vague black box which is what the cloud diagram really means.

Another example of control is YouTube. I use YouTube favorites as a persistent list. I see a cool video, I favorite it and I feel like I sort of own it, or at least it’s in a list that I can refer to later. But take a look at this:

youtube_whoops

What were those things? Who knows! Now, I have to think about “the cloud” again. These are temporary videos that someone else ultimately controls. I’m just adding references to a list. I don’t own the clips. They are transient. They are ephemeral. I’m out of control again. I don’t even know what media I’ve lost. Do I mitigate again? Do I suck down a list periodically and do a diff?

Connectivity

I recently got a Roku box for my TV. It’s a great box. During registration it does a bunch of sign up and account creation. But it doesn’t work without uPNP enabled on the router. This isn’t even a connectivity outage thing, it’s a connectivity assumption that I have a certain kind of firewall that can’t have holes punched in it … or that I’m not capable of punching the holes myself. I don’t even really know why Roku does this uPNP thing. All I know is, it wouldn’t even finish the setup until I made this change. Now here’s a device that doesn’t work without connectivity or a clear path to connectivity.

Think about how picky that is for a second. If it’s not picky then think about how many technical barriers there are to pure or uniform Internet. Everyone brings their own quilted environment and it’s a mess.

IPv6 Spike

A spike is when you play around with something and then throw it away for the purposes of learning. So, let’s play around with IPv6. I had read a little bit about it but essentially my working experience with IPv6 was nothing except for disabling it. Let’s learn some stuff!

I’m going to skip over all the history of IPv6 and assume that you agree with me and think that this is important and relevant to the future of the Internet.

Setup

First, build 4 Ubuntu VMs. I’m using 13.04 but any current Linux distro should work, just the packages and paths will change. I found the best way is to build a simple VM and then clone it 3 more times (in Fusion this is copy/paste and resetting the MAC address). You’ll need four machines to simulate a local network. You won’t need any network hardware and VMware will be able to simulate everything we need. You can actually do this whole experiment on one real box (cool stuff)!

The goal of this spike is:

  • Build 4 VMs
  • Make a router, a web server, a dns server and a client
  • Hit a web page between two network boundaries over IPv6 only

Super practical IPv6 primer

Addressing is WEIRD. That’s really what I wanted to spike on. Getting comfortable with the addressing length, hexadecimal and understanding the addressing layout.

In IPv4, a network segment might look like this: 10.0.0.0/24
So a box with an IP on that network might be this: ip: 10.0.0.136 netmask:255.255.255.0

IPv6 is a lot different. Private addresses don’t start with 192.168., 172. or 10. Private addresses start with fc00 (from what I’ve read). So I made up two network segments called
fc00:deed:d34d:b33f
fc00:deee:deee:deee

But that’s only 4 sets of hex. IPv6 addresses have 8 sets of 4 hex bytes like this:

So let’s configure a box with an ip. Our boxes are named after onomatopoeias (boing, wap, rawr and piff). Boing’s address is “dot” 10.
boing: fc00:deed:d34d:b33f::10/64

So there’s a box that’s configured with an IP. Notice the double colons. That just means it fills in the zeros between. It’s shorthand. The /64 is the network segment. Like in ipv4 192.168.0.1/24 is a common private ip. The /24 is out of 32. So it means X.X.X.Y where Y is the host part and X.X.X is the network part. So 192.168.0.* is the network and .1 is the host. In IPv6 it’s /64 out of a total /128.

So my private address space is just like an IPv4 private range. I’m assigning this IPv6 space and I have 18 trillion private address for my ONE SUBNET. For a router to work, I need two subnets. So now I have 36 TRILLION free private addresses. O_o

Address Configuration

I’m using ipv4 just for remote admin and installing things through apt. So you’ll have to add another ip to your vm’s network card. Ubuntu does this in the file /etc/network/interfaces. Here’s an example.

If you type ifconfig or `ip addr` you will see that it has two IPs now. One IPv4 and one IPv6 address. We’re not quite done. I drew a picture of the network layout and you’ll have to configure all the VMs like this.

ipv6_lab

Router Configuration

This is really easy. You just need the Rawr box to forward IPv6 packets like a router but not like a firewall. So Linux can do that will a simple kernel switch. But first, you’ll need to add a second network card in VMWare. So:

  • Shutdown rawr
  • Add a second network card
  • Boot rawr
  • Edit /etc/sysctl.conf, change net.ipv6.conf.all.forwarding = 1
  • Run sysctl -p

Routing through rawr should work at this point. For example, from piff, you should be able to ping boing through ipv6 even though they aren’t on the same network segment. Use ping6 and traceroute6 to sanity check.

DNS Configuration

Boing is our DNS server so let’s make some changes. First, apt-get install bind9. Then edit these files below. I configured a temporary subdomain on squarism.com called ipv6.squarism.com but this can be anything you want.

Notice that the reverse zone (ip6.arpa) is super annoying to type out. It needs to be the reverse bytes (afaik).

Here we’re using the AAAA records for IPv6. In IPv4 these would be A records. Bind has supported AAAA records for a long time. CNAME records don’t change. Notice that you could easily run a DNS server that serves both stacks.

I used a reverse zone generator at rdns6.com for this last time. It’s amazingly annoying to type out. I’m not sure if there’s a more convenient form that could be used.

Restart bind: sudo services bind9 restart Check the logs in /var/log/syslog for any `named:` errors. I had a few typos I had to chase down. DNS can be tricky to set up so take your time.

See DNS working

Ok, let’s take a quick break from this infinite configuration and see how we are doing so far. At this point we should have routing and DNS working. So that means that Piff should be able to ping a DNS name and it should work.

But first, we need to tell Piff and all the other boxes to use our new dns server. Edit /etc/network/interfaces again and add


dns-search ipv6.squarism.com
dns-nameservers fc00:deed:d34d:b33f::10

For example, your eth0 block will look like this

Restart networking. Even if your VMs are running DHCP (they are by default), we’re just adding a DNS server to the static IPv6 address. In other Linux distros, this file will be different (sry).


piff:~$ ping6 www.ipv6.squarism.com
PING www.ipv6.squarism.com(fc00:deed:d34d:b33f::11) 56 data bytes
64 bytes from fc00:deed:d34d:b33f::11: icmp_seq=1 ttl=64 time=1.12 ms
64 bytes from fc00:deed:d34d:b33f::11: icmp_seq=2 ttl=64 time=0.553 ms

Ok great. DNS and routing are working. Now we are ready for the final part.

Web Server Configuration

Get dependencies installed on Wap (the web server).
aptitude install zlib1g-dev libssl-dev libpcre3-dev

We need ipv6 support built in and I’m not sure if the OS packages are going to come with it out of the box. Installing nginx is easy. So let’s download nginx, configure, compile.

# download latest stable and untar ...
./configure --with-ipv6 --prefix=/opt/nginx
make install
cd /opt/nginx
vi conf/nginx.conf
# change this line
listen [::]:80 default ipv6only=on;

Start nginx:
/opt/nginx/sbin/nginx -c /opt/nginx/conf/nginx.conf
Normally, I’d write a init.d script here but whatever. You can stop nginx like this:
sudo /opt/nginx/sbin/nginx -s stop

Sanity check:
netstat -nlp | grep nginx
tcp6 0 0 :::80 :::* LISTEN 24618/nginx.conf

Let’s create a dummy web page on Wap (the nginx box). We’ll test this page in the next section.

End to end test

Ok, everything is set up. So let’s see it work end to end. Our goal was to hit an IPv6 web server through DNS and a router.

Here you can see I’m pinging the webserver from the web client (piff -> wap).

IPv4 is routable directly because of vmware. But IPv6 traffic is split, the client isn’t on the same IPv6 segment as the web server. So I can ping directly with IPv4. So that’s our IPv4 sanity test but not why we did all this.

You can see when I try to hit that ipv6.html test page we created earlier it won’t work.

It actually says connection refused and this makes sense if you look at the netstat information from Wap. It’s not listening on 0.0.0.0:80, it’s listening on :::80. Crazy!

If I use ipv6 (curl needs some special settings for the URL)

And ipv6 DNS is working.

You can see it’s going through a router:

Wget works too

Ssh has no special flags, it just works.

Victory Lap

Even firefox works.
ipv6_firefox_dns

Just to prove that this isn’t IPv4, let’s use the weird numerical URL format for the IP.
ipv6_firefox_ip

Well this was a fun spike and I feel like I understand IPv6 a whole lot more and it doesn’t strike fear into my heart to think about big scary addressing. I think the key is to actually use DNS instead of fudging it with typing manual addresses or managing crazy hosts files. It should be interesting to see when ISPs and cloud providers start offering serious options for IPv6.

The Best Way to Read CSV in Ruby

pixel-ribbon_company_profile

CSV is awful. CSV isn’t well formed. It isn’t hard to use because it’s bloated and slow. CSV is hard to use because it’s just a dumb data format. However, sometimes all you have is stupid data and who cares, let’s do this thing and blot out the memories.

I assume you know how to use the CSV module that’s built into Ruby. It’s pretty easy. You just read a file in and you get some 2D array back. It usually comes out pretty horrible with long methods and little room for nice abstractions.

So what if you want to polish it up a little bit? Maybe you aren’t just going to kludge this thing again and hate yourself later? What if you aren’t just going to load this into a database? What if you want to do some quick CSV analysis but at the same time make it come out sort of readable?

Let’s take a look at an abstraction layer and see how we could write a CSV loader for a guest list. We’re going to have a dinner party and evite gave us a crappy CSV dump of who’s responded so far. Well, it’s what we have. But how many people are coming and how many groups aren’t allergic to peanuts? We want to know how many peanut M&Ms to buy.

Here’s our data:

Supermodel is pretty old and I like it a lot but it hasn’t been updated in a while and has some open pull requests. I took at look at some alternatives but it didn’t work out.
- ActiveModel from Rails 3 is hard to make generic
- ActiveModel::Model from Rails 4 is a great upgrade from 3.x. You can make anything look like a database object but it still doesn’t have the concept of a collection. So now I have to make an array variable called table? This is weird.
- Sequel has a nice interface to an in-memory sqlite3 database. It’s probably the most ‘real’ that I found but it requires you to do a CREATE TABLE statement even for your in-memory database.

None of these alternatives above are bad but let’s take a look and see how nice we can get it with Supermodel.

First, we are going to use a supermodel fork so that we automatically get rails 3.2.13 instead of 3.0.x. Create a project folder and a Gemfile file:

Run bundle.

You can see that Guest.all is much more intent revealing than manipulating a 2D array by hand.

Rails Dev Shops in Washington DC

pixel-ribbon_copper_leaf_hotel_lobby

What shops, companies, consultants, startups or other folks are using Ruby or Rails (on any level)? Contact me on twitter if you want to be added or you have corrections: @squarism or leave a comment below.

rails_dev_shops_radiant Radiant CMS Radiant is a no-fluff, open source content management system designed for small teams.
rails_dev_shops_triple_dog_dare Triple Dog Dare Has your Rails (or Ruby) project gone off of the tracks? Did you outsource your work on the cheap only to find that your application is bug-ridden and slow? I can help make it better. Wrangling chaos is one of my specialities.
rails_dev_shops_intridea Intridea We don’t just make web apps – we solve problems. At Intridea we write intelligently designed software to help businesses develop strategic solutions and launch new ideas.
rails_dev_shops_codesherpas Code Sherpas CodeSherpas is a full life-cycle software development and design firm based in Reston, VA.
rails_dev_shops_livingsocial Living Social LivingSocial is a deal-of-the-day website that features discounted gift certificates usable at local or national companies. Based in Washington, D.C., LivingSocial now has more than 70 million members around the world.
rails_dev_shops_bloomberg Bloomberg Bloomberg, the global business and financial information and news leader, gives influential decision makers a critical edge by connecting them to a dynamic network of information, people and ideas.
rails_dev_shops_monkeysee Monkeysee.com Monkey See captures the skill and knowledge of the world’s top experts and delivers it to inquisitive audiences everywhere.
Gannett ???
Comcast ???
Sprint ???

The Supermodel Ruby Gem Loses Data

pixel-ribbon_northface
Actually no.

I love Supermodel. It might be overcome by ActiveModel::Model in Rails 4 but until then Supermodel is a fantastic in-memory database for Ruby that has a lot of advantages over using just a plain hash or trying to roll your own.

However using it with a large amount of data, we noticed it loses data. Sometimes, a few records. Other times, a few more. It was really random. We were confused. Looking at the docs, this is the default class maccman has in his README.

That works no problem. We looked at the IDs that it uses and saw that it’s using the Ruby ObjectID which is about 14 digits long.

Ok, that ID of 70095779847820 seems good enough right? Let’s see!

Run it.

What.

Well. I’m no expert but I bet the object_ids in ruby aren’t very random. I would hope they wouldn’t be. Because you’re creating objects all the time right? Ruby is slow enough without some super accurate id field. Should we abandon all hope and scatter our dreams in despair? Nope.

Supermodel has a documented solution for this. Just add this mixin into your class.

This will make the IDs more random and you’ll find 1,000 pairs of fancy pants in your class. The odd ball thing for me was realizing that supermodel ‘loses data’. But it doesn’t. IMHO, this mixin should probably be the default. I find Supermodel an awesome quick and dirty database but a database shouldn’t lose records silently.

I still love Supermodel. I’ve played around with other in memory databases such as Rails3, Rails4, sqlite3 with datamapper and Supermodel works like I want it to.

Super Interesting Talks from RubyConf 2012

pixel-ribbon_chem

Trying to summarize someone’s 30-60minute talk is really hard. So apologies go out to anyone I’m trying to paraphrase here. I took it upon myself to watch every single video from RubyConf 2012 which started airing in November. It’s May now. There’s a lot of content there and you can’t just slurp it down and expect to process it all. So I thought I’d leave little breadcrumbs to myself noting which things were super interesting to me.

Real Time Salami – Aaron Patterson
Any presentation by @tenderlove is great and this one was fun and interesting as expected. Aaron talks about parallelism, streaming and making Salami (actual salami).

Tokaido: Making Ruby Better on OSX – Yehuda Katz
This was an exciting talk about Tokaido which is a work in progress to make a Rails.app one-click super-easy dev tool for Mac. He talks about other platforms too, don’t worry. This talk really makes you appreciate how hard this problem is. There are some super interesting low-level OSX details in there.

Why JRuby Works – Charles Nutter, Thomas Enebo
This was a great talk about JRuby and was very convincing presentation. Since watching it, I’ve been playing with Torquebox and JRuby. Unfortunately the audio and video are a bit weird. For me, I loved the part about garbage collection. It was a great summary about how good the JVM is at garbage collection.

Zero Downtime Deploys Made Easy – Matt Duncan
This talk was great. Matt walks through all the problems you will encounter when trying to reach a large number of nines. He covers a lot of gotchas, like “whoops that database migration locks the entire table and just took your site down”. He covers how Yammer does database changes, managing job queues and external services when you are trying to keep uptime at maximum. This was definitely an eye opening talk.

Y Not — Adventures in Functional Programming – Jim Weirich
OMG. This talk left my brain on the floor. I can’t really explain how awesome Jim (he wrote rake) is. If you want to see the best live coding I’ve ever seen and learn about the y-combinator, watch it. I didn’t follow along 100% but I was blown away.

The Celluloid Ecosystem – Tony Arcieri
This was a great intro into everything surrounding the celluloid gems. More importantly though, it was a _reference_ concurrency state of the union talk. If you want to learn why the actor model is the way to go (in Ruby or Scala actually) then watch it.

Ruby vs. the world – Matt Aimonetti
A great overview of languages other than Ruby. His starting point about the Sapir-Whorf Hypothesis – that language influences thought is a great opening to this talk. Matt chooses really interesting topics and does a good job. He covers Clojure, Scala and Go. This is a great talk if you don’t know what any of those are or want a quick ‘Rosetta Stone’.

Your app is not a black box – Josh Kalderimis
This talk is easy to watch. He does a great job of keeping it interesting. It’s basically a talk about DevOps but more importantly about tooling. I found this talk very interesting from an ops, polish and motivation. Please watch this.

How to build, use and grow internal tools – Keavy McMinn
One of my favorites. I forwarded to a bunch of people. Github is worth emulating and Keavy shares insight about tools, culture and teams.

Asynchronous Processing for Fun and Profit – Mike Perham
A great talk about sidekiq vs redis from the authority on sidekiq.

Change your tools, change your outcome – Dr. Nic Williams
Dr. Nic nailed this talk. Some NSFW language. Hilarious and interesting talk about getting over nice to haves (like fast MRI spin-up time) and making your app more awesome for Ops people. Super great talk.

Grow Your Unix Beard Using Ruby – Jesse Storimer
A reference talk all about Unix. I found this very educational even though I consider myself pretty unix savvy. Jesse is great, he has a book on pragprog.

Boundaries – Gary Bernhardt
Amazing talk by destroyallsoftware’s Gary. He talks about an imperative shell vs a functional core which is all the rage right now. Gary is brilliant.

Abstracting Features Into Custom Reverse Proxies – Nick Muerdter
Some great ideas about reverse proxies.

Service Oriented Architecture at Square – Chris Hunt
Chris walks you through creating a web service like they do at square as if you were working there. He introduced some amazing open source libraries from square that I need to check out (cane, fdoc, jetpack). For example, they use jetpack to auto-pack up and deploy a rails app with Jetty. So all you need is a JVM.

I’m pretty sure I picked more than half of the talks as ones that I found super interesting. There were many more but I can’t just pick everything. It takes a while to watch all these videos but they are worth your time.

Using a Redis as a Database

pixel-ribbon_cor

The Spike

I was spiking on Redis recently. I wanted to use the redis-objects gem to simulate a shopping cart app even though the README specifically says

Just use MySQL, k?

I wanted to see what would happen if I tried it anyway. So the README and examples for the redis-objects gem are great so I’m not going to rehash what’s there. However, I will say though that the example has you hardcode the id field to 1. That detail snuck up on me.

If you don’t set an ID then you can’t work with a redis-object instance. You get an exception: Redis::Objects::NilObjectId: Attempt to address redis-object :name on class User with nil id (unsaved record?)

It’s basically trying to tell you, “hey, save the record first or set an ID”. Well, honestly, I don’t want to set an id myself. This is where the meat of the README is. Redis-objects really fits organically in an existing ActiveRecord model. That means Rails. In this case though, I don’t want an entire Rails app. I can see the value though in a plain old Rails app. Just look at the examples if you want to see more.

Anyway, continuing on with the spiking, I tried to integrate the Supermodel gem with Redis-objects. That sort of worked. You just class User < Supermodel::Base and you can sort of get it to work. This is great because Supermodel gives you finders like User.find_by_email('bob@yahoo.com') to make it act like ActiveRecord but you can't use .create(email: 'bob@yahoo.com') to begin with because of the same errors as I mentioned above. Redis-objects really wants the record to have an ID already. Even using Supermodel's RandomID mixin didn't work. The initialize order and callback hooks don't really work (or at least I couldn't get them to work).

Finally, I tried combining just redis-objects and datamapper redis. That worked. And it's pretty nice. Check it out.

So using this is pretty easy.

When you look at Redis, the keys are already composited for you and magic has happened.

Yay!

The name field is from redis-objects and the create uses datamapper. This is a really odd pairing but I like the fact that I have no sql database in the mix but still have finders similar to an ORM. Something to keep in mind, datamapper's finders are a bit different than the Rails 3 ones (no .where method).

Benchmarking A Million Things

Ok fine. So maybe this works, maybe it doesn't. Maybe it's not the right idea. What about the good stuff? Like, how fast can we load a whole lot of names into MySQL versus Redis using the above code and techniques? Is it even relevant?

A gist of these test results is here.

A More Complete Example

If you know the ID and don't need something like an auto-incrementing column outside your code/control then you can greatly simplify the code above by getting rid of Datamapper. You can simply use redis-objects to fake an ORM. I had great success using it as long as you USE NATIVE REDIS TYPES. Listen to the redis-objects author, don't try to force the tool into the use case.