There seems to be a lack of understanding and experience in the same areas from project to project, company to company and sector to sector. It’s nothing specific to any one company. I’ve seen this in small .com companies to large contractors. It’s very odd to me. Even if everyone can’t know everything, it seems certain things are never taught or learned.

Some of these things are high-level skills and some are very specific. I think the specific ones are weirder. Osmosis, you’d think, would have some people learn certain tools or languages or whatever.

Troubleshooting
Basic troubleshooting skills. Like, change one thing at a time. Or, turn on debugging. Or what the likely culprit is versus chasing after every possibility. Or, feeling overwhelmed and not trying anything: it’s just broken. Broken how? What changed last? What is persisted? Is there a tmp file? Is there a cache? Can you repeat this every time? If not, what are the variables?

VNC
No one seems to know how VNC works. But everyone uses remote desktop just fine. With VNC, it’s an effort to get everyone connected and it’s a hand-holding operation. I understand it’s tricky. But VNC has been around for almost a decade and it hasn’t really changed. I figured osmosis would kick in.

X11
Cygwin, Xming, putty and getting X11 working. Or how X11 isn’t encrypted. Or security problems with xhost. Or running X as root. Or how X11 is different than the Windows display. Or how X11 is separate from the X11 shell. How to forward X11 over ssh. All these X concepts don’t come from the Windows world but are in every other OS: Solaris, OSX (X11 app), Linux.

Version control
Conflicts, merging, branching. Why SVN is different from SharePoint is different than git is different than just calling files .year.mon.day in a file share. Even ‘senior developers’ that don’t have any high-level knowledge of CVS, a program that was released in 1990. I’ve seen this everywhere.

Sudo and cron
Even if in a Windows environment, what the UAC momentary elevation of privileges is like (also see OSX ignorance). Or how `at` works on Windows. At is also a program in UNIX.

Myth of supportability
Everything has to be written in the language that’s delivered. If we’re delivering a COTS program that runs on C# then every script has to be C# because that’s what we know. If you offered to write a utility in a higher level language, then all the Java devs would cry foul because it’s too foreign. So everything is a hammer problem even though all tech changes in a few years. I wish someone would add up all the “this is going to save you so much money” promises of middleware.

Regarding COTS, Mike Taylor’s Whatever happened to programming? is a good read. I think of his libraries as my COTS/middleware.

Copy and paste
How to paste in X11 (middle click). How to paste into putty (right click). How to copy in X11 (select), how to copy in putty (select). How to copy and paste across VNC (enable clipboard support). How to strip out formatting (paste to notepad, start->Run, anything plain text).

Hierarchies in Software Architecture
How many nodes make up an X? How many X’s make up a Y? Can I have multiple Y’s per Z? What’s shared between Z and foo? Do I have to create foo first? Enough of the contrived examples. Let me give you some real examples:

I have a bunch of disks I’m going to share out for a database.

1. Combine one or more disks into raid or jbod array on SAN
2. Slice up SAN into one or more LUNs
3. Create volumes or filesystems with one or more partitions
4. Organize partisions into one or more diskgroups
5. Database is spread across one or more diskgroups
6. Database has one or more schemas
7. Schema has one or more tables
8. Tables can have one or more data partitions

And so on. It can keep going like this for quite a while. In terms of performance, if the partitions go to the same busy disks then there’s not a whole lot of point in sharding. In terms of design and planning, obviously I have to do step 1 before I can do step 5 and often undoing step 1 will cause the house of cards to tumble.

That’s not really a software thing like I said, ok. I can have multiple listeners in Oracle DB but I can’t have multiple listeners on the same port. I can have multiple reverse HTTP proxies on different boxes point back to the same web server on a single port though. I can have multiple IPs on a box but not multiple programs using a port on a single IP. I can have multiple virtualhosts against one IP in Apache but I can’t have name based virtualhosts against one IP with ssh. This hierarchy is especially important when load balancing webapps. Even the F5 has an internal hierarchy to make the configuration flexible.

Ok, ok. That’s still not software enough. I can have multiple sessions in a webapp but only one identity. The identity is in the persistence layer of the above SAN example and my webapp can scale out to the horizon with shared-nothing or whatever I want. I’m still really logging in only once even if I bounce between 500 boxes. The session could be just an instance of my identity.

Webapp vs webpage
Website is to webserver as webapp is to application server. HTML is a webpage/website. JSP/PHP/CF/ASP is a webapp. This is simplified but so many people don’t know the difference even when their job revolves around integrating or even developing ‘webpages’.

Plain-text
Many protocols, passwords and network traffic can be viewed in plaintext. There’s some protocols you can interact with just by typing text to an open telnet session. You can do IRC with telnet. You can do SMTP with telnet. You can do a HTTP GET.

Hands-On Security
Policy makers vs whitehats. Like actually checking Sans for actual 0-day remote root-level ultra-bad exploit on foo software. Security is always a huge blanket statement with very few actual experts. I’m certainly not one but I have met very few that I thought were teaching me something. I want the guy who actually knows what a XSS attack or a buffer overflow is. This might just be a reflection on the projects I’ve been on.

First off, this DroboFS is awesome. However some of the apps are a little thin on the docs and there were a few things I had to figure out.

So first a few assumptions, you didn’t change your drobo’s name. I actually did and that’s ok, just in this post, it’s called drobo-fs which affects mount paths and some of the commands. So just watch for “drobo-fs” below.

To get apps on the drobo, you have to install apache first, reboot and then install drobotools and reboot again. After that you can one-click install from the web admin. Just follow the official PDF docs.

Ok, if you install dropbear, we have the first password to reset: ssh root.

1. Change ssh root

ssh root@drobo-fs
(yes)
# passwd
(ignore error about /etc/shadow)

2. Change apache’s root

cd /mnt/Drobo-FS/Shares/DroboApps/droboadmin

Find a box with htpasswd on it (like a Linux box). There are ones online if you don’t like that. You’ll get a hash like:
root:$apr1$UNZ3DFk3$tspu/3z5Pkkn.h.TUytUl1

Backup your htpasswd:

1. cp htpasswd htpasswd.orig
2. vi htpasswd
3. Replace the root line: d,d,i,[paste hash]
4. Write/save: [colon], w, x, [enter]

Close your browser (clear the session) and try to go to:

http://drobo:8080/droboadmin/

The login should work now without restarting. If it doesn’t restore your htpasswd file and hit the web.

Looks like this thing has 128MB of RAM. It has some basic busybox commands but no bash, nano or even users. I haven’t figure out ctorrent yet. Also, big note: the root password resets on reboot. Run /mnt/DroboFS/Shares/DroboApps/dropbear/root_passwd to make the password change permanent.

I read this post about setting up autotest with growl. The little doomguy was a nice effect and got a literal lol out of me (llol?). However, the post was from 2007 and apparently autotest has some problems with 1.9. I’m not running 1.9 but on the eve of rails3, I probably will be. Ok, what’s cooler than autotest? Apparently not much, three hits on google.

Install a gem
I found Watchr. It’s not too hard to set up. sudo gem install watchr to get the gem. If you’re on *nix, grab the filesystem event gem: sudo gem install rev. Now we have watchr up but we need a config.

Create the watchr config
The docs tell you to edit a config/watchr.rb file. Mine came from the examples on their wiki. It worked out of the box. It’s posted here. It’s a basic rails testing config that I haven’t had to edit. It’s very readable if you need to futz with it.

Create a test case
Ok so watchr just runs `rake test` which you can do in your rails app root. But perhaps you don’t have any test cases created. The generators will create them for you. But assuming you don’t have one, create a file named [rails app root]/test/functional/hello_test.rb.

require 'test_helper'
 
class HelloTest < ActiveSupport::TestCase
  def test_hello
    s = "Hello"
    assert_equal("Hello", s)
  end
 
end

Obviously this is a crappy assertion.

Get Growlnotify
Obviously, you’ll need growl installed. But you also need the growlnotify binary. It’s good to have anyway (I use it in Automator). I didn’t have it by default on two of my machines so I assume that the normal Growl install doesn’t install it. It’s on the Growl 1.2 dmg under extras. You just have to copy it to /usr/local/bin. Download the growl dmg, open the dmg. You don’t need to reinstall growl.
sudo cp /Volumes/Growl-1.2/Extras/growlnotify/growlnotify /usr/local/bin

You can test it with:
growlnotify -t "Title" -m "Message"

Make doomguy happen
Create a directory for the growl icons: mkdir ~/.watchr_images.
Copy these two doomguy faces to ~/.watchr_images/. On mac, hit Shift+Cmd+G and go to the hidden folder.

Now we should have two pngs in our hidden folder.

The watchr.rb file references these images. If you want to put it somewhere else (like maybe under ~/Pictures), you can change the .rb.

Ok Go!
Launch with: watchr config/watchr.rb from your rails app root. Watchr should be in your $PATH because you installed the gem. You should have a blank screen now where watchr is waiting.

When you save a file, watchr will fire. Depending on the file you saved, a different test will run. If you save a controller, for example, the functional test for that controller will fire. If you save a model object, a unit test for that model object will fire. If you want to force a whole test suite to run hit Ctrl+\ in the watchr window.

When things pass:

When things fail:

The growl display setting is “Music Video”. I have it popping up on my 3rd monitor which is less distracting but still visible. However if you put it on your main monitor, the overlay won’t interfere with mouse clicks, which is nice. It makes for a killer setup where I don’t have to test my app with a browser or with contrived/non-automated tests. Now I just need to write more tests.

Railscast has a podcast on iTunes which lets you download all past episodes and thus this post is moot. But, I thought, what if it wasn’t? What if I found a cool site with a bunch of media files I want to download? I knew wget can do this but it’s been a while. So here’s what I did.

Get the links

First I downloaded the archive page. It’s full of links to other pages which have the .mov media files on it. But I really needed a list of those pages within the archive page. Ok, I wget’d the archive html source page in a file named archive.txt:
curl -o archive.txt http://railscasts.com/episodes/archive

However this page is full of html source and all I need is the list of episodes. So a simple:
grep href urls.txt|grep episodes > urls_href.txt gets me that. But it’s really full of text that looks like this: <a href="/episodes/195-my-favorite-web-apps-in-2009">My Favorite Web Apps in 2009</a>

Clean the links

Ok now I just need to trim all this crap out. First, there’s a bunch of whitespace out front in the links. Let’s use sed inline. This works on OSX and Linux but won’t work on Solaris (inline boo).
sed -ie 's/^[ ]*//' urls_href.txt. Now we’re missing a those spaces out front.

Ok, now we need to trim down to the relative link to the episode. We want a URL that looks like:
http://railscasts.com/episodes/30-pretty-page-title

So at this point sed was failing me because the regex syntax is different than I’m used to. So let’s switch to perl.
cat urls_href.txt | perl -e 'while(<>) { s/\<a\s*(.*)\>(.*)\<\/a\>/$1/; print}' > urls_href_clean.txt

We’re almost there. We have URLs that look like this:

"/episodes/17-habtm-checkboxes"
"/episodes/16-virtual-attributes"
"/15-fun-with-find-conditions"

We need a prefix of the domain and to get rid of those quotes:
cat urls_href_clean.txt | perl -e ‘while(<>) { s/href\=/http\:\/\/railscasts.com/; s/\”//; s/\”//; print}’ > urls_href_super_clean.txt

Despite our horribly unmaintainable “super” naming convention, we now have a text file full of URLs that looks like this:

http://railscasts.com/episodes/17-habtm-checkboxes
http://railscasts.com/episodes/16-virtual-attributes
http://railscasts.com/episodes/15-fun-with-find-conditions

Scrape

Fire wget using our text file as input (-i). Recurse (-r), go only two levels deep (-l), don’t download the file if we have one that’s newer and use timestamps to make this possible (-Nc), span hosts (-H), no directories (-nd), disrespect the robots.txt file (-erobots=off) and look for only .mov files (-A).
wget -r -erobots=off -l2 -H -Nc -nd -A.mov -i urls_href_super_clean.txt

You might want to create a working directory for this before you run it. And probably run it under “screen” if you have that command. After you do, you’ll eventually you’ll have a directory full of railscasts. Or, you could just subscribe to their feed on iTunes. Getting it through iTunes is a lot easier but that wasn’t the point.

I went to renew my membership on my iPhone developer account and although the membership is $99/year, my expiration date was only extended out for 3 months. I opened a ticket and it took about a month or two to get this thing resolved. I had to send screenshots, steps and evidence that I was doing the right thing (ok, fine, tier 1 support). Eventually I got a person that had a trick up their sleeve but didn’t quite get it right. I’m blogging this in case this is happening to someone else.

First, the symptoms. Let’s say your dev account is about to expire in January 2010. You want to renew for another year. You add a year subscription to your cart, pay for it. You’d expect your new expire date to be Jan-2011. But in your profile and in the “thanks for ordering” activation page it says March 2010 or something else wrong. Support send me activation links basically to the same page that wouldn’t activate any differently. Each time I checked out, it would say March 2010.

The trick was to click on renew membership (as if paying again), click the checkbox to select which membership you want to buy ($99 year) and hit continue. Now you’re on a “Review your purchase” page. Click continue here. Finally, it says “Proceed to your country’s Apple Online Store to purchase” and has an add to cart button. Stop here.

Click the following link: http://developer.apple.com/iphone/enroll/activate.action. It should bring you to a page with a single textbox for your activation code (the one you’ve been trying to use all along). Put this in and hit continue. Your expire date should be correct now. If not, try using a link formatted like this:
http://developer.apple.com/iphone/enroll/activate.action?activationCode=YOURCODEHERE

Seems like they have some bug in the dev store. Please comment if this helped you so it’s documented.

A question came up today about how to reverse a sentence. Word by word, preserving periods etc. I’ve done stuff like this before but I really saw an opportunity to solve this contrived quiz type question with TDD. It’s perfect really. Tedious string checking? Screw it. Let my test tell me when I’m done.

Ok so I could have probably done this more comfortably in Java but it would have been more lines. I banged this out in Ruby, including “learning” test cases (it’s really easy) in about an hour. At one point I realized how easy string manipulation is in Ruby for this and I literally said, “holy shit Ruby is amazing”. And then the edge cases started happening. I was only handling periods and my algorithm fell over flat on three sentences because I was trying to do a string[start, end] when it’s really string[start, length]. I fixed it with learning the String#slice syntax.

Ok enough fanboi service. There’s two files. One’s the test and one’s the class. Run the test and not the class (ie: ruby tc_reverser.rb).

tc_reverser.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

require 'test/unit'
require 'reverser'
 
  # Be wary of newlines in the test and expected trings.
  # Do not try to word wrap with escapes.  It's very picky.
class TestReverser < Test::Unit::TestCase
  # Optional
  #def setup
  #end
 
  def teardown
    if @test_passed then
      puts "\n #{@r.reverse}"  # more verbose for successful passes
      puts "#{@method_name.upcase} OK.\n\n"
    else
      puts "#{@method_name.upcase} FAIL.\n\n"
    end
  end
 
  # simple test
  def test_reverser
    test = "You space bastard.  You killed my pine."
    expected = "Bastard space you.  Pine my killed you."
    @r = Reverser.new(test)
    assert_equal(expected, @r.reverse, "Fail.")        
  end
 
  # Test multiple sentence delimiters  
  def test_multiple_sentence_delimiters
    test = "Shape up, man.  You're a slacker.  Do you want to be a slacker for the rest of your life?"
    expected = "Man, up shape.  Slacker a you're.  Life your of rest the for slacker a be to want you do?"
    @r = Reverser.new(test)
    assert_equal(expected, @r.reverse, "Fail.")
  end
 
  # Test no periods, question mark and commas
  def test_question_comma
    test = "Then tell me, future boy, huh, who's president of the United States in 1985?"
    expected = "1985 in States United the of president who's, huh, boy future, me tell then?"
    @r = Reverser.new(test)
    assert_equal(expected, @r.reverse, "Fail.")
  end
 
  # Big test  
  def test_everything
    test = "Our first television set. Dad just picked it up today. Do you have a television? Well, yeah, you know, we have two of them. Wow! You must be rich. Oh, honey, he's teasing you. Nobody has two television sets."
    expected = "Set television first our.  Today up it picked just dad.  Television a have you do?  Them of two have we, know you, yeah, well.  Wow!  Rich be must you.  You teasing he's, honey, oh.  Sets television two has nobody."
    @r = Reverser.new(test)
    assert_equal(expected, @r.reverse, "Fail.")
  end
 
end

I have to apologize for the scrollbars in the above posted code. I tried many different ways of escaping carriage returns for better formatting but it would have required a lot of changes to deal with the \n and so on in the tests. I had something nicely formatted working but the tabs and spaces for alignment then screwed the test and broke the pretty formatting a different way.

reverser.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

# A class that reverses sentences.
# "My dog has fleas." --> "Fleas has dog my."
 
class Reverser
 
  # tasty constructor, full of taste
  def initialize(text)
    @text = text
  end
 
  def reverse
    delim_location = 0  # char by char, remember !?. for sentence determination
    delim_last_location = 0  # allows us to move on to the next sentence
    delim_i = 0  # an iterator
 
    new_sentence = ""  # inits an empty string for string methods to work
 
    new_sentence_array = Array.new
    punc_array = Array.new  # list of punctuation marks in order
    @text.each_char do |char|
      delim_i += 1  # iterator
      if (char == "." || char == "?" || char == "!")
        # we hit a deliminter (?!.), remember it
        sentence = @text.slice(delim_last_location..delim_i) 
        sentence.strip!
        new_sentence_array.push(sentence)
        punc_array.push(char)             
        delim_last_location = delim_i  # remember our substring position
      end
    end
 
    i = 0
    new_sentence_array.each do |s|
        returned_sentence = do_reverse(s)  # reverse words sentence by sentence
        new_sentence += "#{returned_sentence}#{punc_array[i]}"  # append the delimiter back
 
        if (i+1 != new_sentence_array.length)  # look ahead for last sentence
          new_sentence += "  "  # only append spaces if this is not the last sentence
        end
        i += 1
 
    end
 
 
    new_sentence  # return new_sentence
 
  end
 
  def do_reverse(s)
    s.chop!  # trim delimiter
    text_array = s.split(" ")  # split on spaces to array
    text_array[0].downcase!  # downcase first word
    text_array.reverse!  # destructive reverse
 
    # need to move commas back
    text_array_i = 0
    text_array.each do |e|
       text_array_i += 1
        if e.include? ","
          e.chop!   # modify original array
          text_array[text_array_i - 2] += ","
        end
    end
 
    text_array[0].capitalize!
    text_array = text_array.join(" ")  # destructive join
    text_array.to_s  # return reversed array
  end
 
end

When you run the test case it looks like this:

Loaded suite ~/src/ruby/tc_reverser
Started

Set television first our. Today up it picked just dad. Television a have you do? Them of two have we, know you, yeah, well. Wow! Rich be must you. You teasing he’s, honey, oh. Sets television two has nobody.
TEST_EVERYTHING OK!

.
Man, up shape. Slacker a you’re. Life your of rest the for slacker a be to want you do?
TEST_MULTIPLE_SENTENCE_DELIMITERS OK!

.
1985 in States United the of president who’s, huh, boy future, me tell then?
TEST_QUESTION_COMMA OK!

.
Bastard space you. Pine my killed you.
TEST_REVERSER OK!

.
Finished in 0.003567 seconds.
4 tests, 4 assertions, 0 failures, 0 errors

Marty McFly would be proud. So some limitations. First, I don’t handle elipsis (…) characters at all. It splits on sentence delimiters which one is a period. The eplisis would cause major weirdness. I also don’t handle recapitalizing the word “Dad” as you can see in the “test_everything” test. This would require language parsing or a massive pick list. I also don’t handle slang or abbreviations. For example, “You told ‘em?” would turn into “‘em told you?” which might not be exactly right.

There are many other edge cases that this thing would fall flat on. My test case tests what functionality I wanted and nothing more.

Ruby’s String#slice was super handy on this as well as capitalize, reverse and even a regex split that I didn’t use (amazing!). You can see it here. It would split text into an array really easily:

sentence_array = @text.split(/[\.\!\?]/)  # split with regular expression
 
new_sentence_array = Array.new
sentence_array.each do |s|
    new_sentence_array.push(do_reverse(s))
end
 
puts sentence_array[2]
return

But then I’d lose the delimiter character (!?. etc) and would have to save it, search it or some other nonsense. Still, really neat that you can split on a regex.