SCP vs RSync vs SMB vs FTP

Blog — Dillon @ 11:55 pm

Update: as many have commented, my testing with /dev/zero might be skewing the results. I’ll be adding a random DVD iso test soon. I have a newborn so give me a sec. I’ll even add the NFS test. :)

@fearthepenguin made an rsync comment that made me curious. He said that rsync in cygwin is faster than native SMB in Windows. Ok I haven’t done this test in a while, let’s get a reminder about how fat SMB is.

Test Setup

SOHO gigabit switch. Ubuntu 9.10 and octo-core 2008 Mac Pro running 10.6. Both pretty fast boxes. Regular SATA drives in each, not very fast I/O. Whatever, network should be the bottleneck.

Get my test file generated.
$ dd if=/dev/zero of=1gb_file.zeros bs=1G count=1
1073741824 bytes (1.1 GB) copied, 133.577 s, 8.0 MB/s

Ignore that 8.0 MB/s. You can do a blocksize trick to make it output a file faster, I just didn’t feel like looking up the switches. Now you can see Mr. 1GB Zero File in all it’s empty and big glory.

$ ls -lh
total 1.2G
-rw-r--r-- 1 dude herd 1.0G 2010-02-11 12:29 1gb_file.zeros

The Tests

RSync
Copy from ubuntu box to Mac Pro:
$ time rsync -t /tmp/1gb_file.zeros dude@mac:~/tmp
real 0m17.694s
user 0m11.577s
sys 0m3.056s

SMB
Mount Mac share from Ubuntu box. Copy same file from ubuntu box to Mac Pro over SMB mount. I never do this. It’s stupid, slow, strips permissions and requires a mount. In the name of science!
# time cp /tmp/1gb_file.zeros /mnt/tmp
real 0m32.649s
user 0m0.008s
sys 0m0.568s

FTP
Ok let’s goddamn turn on OSX FTP and test that too. FTP is stupid.
The remote file "1gb_file.zeros" already exists.
Local: 1073741824 bytes, dated Thu 11 Feb 2010 12:29:06 PM EST.
(Files are identical, skipped)

Hey at least FTP is showing some smarts about it. Or maybe ncftp just rules. I dunno. Deleted it and got FTP time.

$ time ncftpput -u dude -p whoa mac /dest/tmp /tmp/1gb_file.zeros
/tmp/1gb_file.zeros: 1.00 GB 39.09 MB/s
real 0m26.507s
user 0m0.036s
sys 0m0.828s

SCP
Is SCP any different than rsync?
$ time scp /tmp/1gb_file.zeros dude@mac:~/tmp
1gb_file.zeros 100% 1024MB 42.7MB/s 00:24
real 0m24.303s
user 0m9.641s
sys 0m2.212s

Weird. It is. I wonder if rsync has compression flags whereas ssh does not without the -C magic switches. You can get SCP to be pretty quick with blowfish or arcfour:

$ time scp -c arcfour /tmp/1gb_file.zeros dude@mac:~/tmp
1gb_file.zeros 100% 1024MB 46.6MB/s 00:22
real 0m21.653s
user 0m5.452s
sys 0m2.032s

Conclusion

rsync SMB FTP SCP SCP arcfour
time 17.694 32.649 26.507 24.303 21.653
MB/sec 57.87 31.36 38.63 42.13 47.29

So RSync is pretty quick and SMB is pretty slow. @fearthepenguin was right.

Buying a cat with logic switches

Blog — Dillon @ 1:43 pm

Charles Petzold’s book Code is an awesome read. I’m reading it again actually. It’s so elegant and simple. Walks you through history and experiments in a style I find extremely invoking. I wanted to build a logic switch but I don’t want to blow anything up so I tried this java applet at falstad.com and it works pretty well. I tried qucs but it’s way too complicate for me and doesn’t include an LED.

An example from Code (this is not the exact example) is where he’s trying to buy a pet from a pet store and he creates a logic circuit that will light up a light bulb when the pet is correct. Let’s say I’m looking for a normal gray cat as a house pet (not a tiger! rawr!). When I flip all the switches correctly, the LED lights up but if I get a gray tiger (that’s technically a cat) then the LED (the red dot) says nope.

The salesman brings me a gray tiger. Nope.
logic_gray_tiger

The salesman brings me a gray cat that’s not a tiger. Yep.
logic_gray_cat

iPhoneDevCampDC 2009

Blog — Dillon @ 10:32 pm

iphonedevdc
Went to the first annual iPhone Dev Camp here in DC. It was a barcamp style event over two days. It was very enjoyable although there were quite a few heavy hitters there (I wasn’t one of them). There were a few different types of people there and I was one of the ones who doesn’t have anything on the app store and isn’t making any money off iPhone dev. A few people there had apps I knew from the press as well as an author (Dave Mark — who was great) that I’ve read quite a bit from.

What follows is some of the notes that I took at the conference. It’s by no means a transcription.

Peter Corbett – Apps for Democracy

@corbet3000
He met with Vivek Kundra (CTO/fed CIO DC) and started a dontest for DC, open data. Citizens created “fix my city” type apps. 43 webapps created, $2.3 EST value, $50k cost, 5000% roi. Huge success.

A few examples:

  • areyousafedc iphone app – a tachometer as you walk, green = safe, red = danger. Pulls data from open city sources, crime reports etc. Very simple interface, pretty cool
  • wethepeoplewiki.com – structured wiki, real-time crime data. I didn’t get this.
  • park it dc – very cool parking meter app. People report broken meters, displays broken meters on a google map mashup. City found their contractors were fixing their meters in avg of 7 days but their SLA was 24 hours. Broken meters not cities fault, app helped DC discover the real problem.
  • iPhone demo at DC311 – http://victorshillo.com/dc311/2/

Really good presentation. Peter is a cool guy. Nice, technical and well connected. Peter did another awesome presentation called No one cares about your crappy webapp at an Ignite Baltimore conference. I tried to tell him how awesome that was but it’s hard to put into words.

Jonathan Blocksom – OpenGL

@jblocksom

http://www.gollygee.com/weblogs/jblocksom

Jonathan works at google. He had an android shirt on. It was pretty funny. I seriously think he was there to convert some people. :P He was really nice and I enjoyed talking to him about gamedev (even if most of my stuff has been in Java). He did a really nice overview of OpenGL, computer graphics and his game Bubbles. His game has been on the app store since the beginning of the app store opening.

  • Z buffering is checking if a pixel is behind another, won’t render.
  • Use the iPhone boilerplate template to learn
  • Overview of the various buffers that the template creates.
  • You can use the bullet SDK, collada to import 3d models etc
  • You can use the Texture2D class to easily import textures from the apple Lunar lander sample code
  • You can use the touch fighter sample code. It shows how to overlay a high score list over opengl view
  • Don’t mix OpenGL and cocoa views together for performance reasons
  • He gave an overview of his sales history which was interesting.

A lot of people showed their sales tapering off after an initial burst of sales. Sometimes press coverage or even competing apps would create another bust of sales.

Leon Palm – Computer Vision

Leon also works at Google (not that anyone is judging people based on their day job. Hey, google is a cool company (currently). He was a really smart and nice guy. Easy to talk to. Had a cool Sudoku solver app to demo. I thought I had seen his app covered in the press but that turned out to be a competing one (oops!). His presentation walked through how the app works. Some parts are super confusing and hard but he did a good job in breaking it down.

How his app works:

  • Evolution algorithm
  • RANSAC to find the line
  • Walk the intersection of the lines
  • Find 8×8 inliers in checkers, 7×8 in connect four
  • Have lines, apply transform matrix to rectify image
  • Get pieces is easy
  • Sample region at expected center, create int array for piece config
  • Use open source solver etc

Drawing the results back.
We have: piece colors, positions, sizes and warp matrix. Derectify image and draw over solution.

Conclusion: easy to do if task is broken down. Use existing knowledge (whitepapers). Filtering/tweaking is the most important part. You have to tweak it to work with cameras lighting and make it accurate. He said tweaking and adjusting took the most time. I believe it.

I have done some test type stuff with OpenCV but Leon had really taken this all the way to the finish line. It was a really in-depth talk that was academically the most complicated of all the talks.

Kiril – Working w Designers from Imagini Studios

Kiril is the artist that worked on Harbor Master. It’s a “line drawing” game similar to Flight Control. Apparently it’s doing very well on the app store. These guys were super pro. They had a great presentation, super personalities and they had found success being an indie game dev shop. I was really green but then I hadn’t put in the hours etc.

Kiril talked about his mock ups, how he worked with the developers (2 of them) and showed his different iterative art pieces. He mentioned ffffound for art inspiration. And his most important advice to developers: don’t think that mockups are the final product. He said many people can’t make the jump from concept to final product.

Christopher Brown – App Store Data!

Christopher runs an analytics company called Tap Metrics. They had a super slick web app that scrapes data from the app store (I imagine only a few people can do this). He had run many reports and shared some interesting trends:

  • Most people that buy an app stay in that category and buy again
  • 94% of apps are in English, meaning German/English counts. Germany only counts wouldn’t count. All -> EN -> DE, FR, JP
  • 1% conversion freemium rate free->pro. Meaning 1M free downloads.
    1.99 better segment, .99 is saturated

If I was on the store, I’d talk to Chris about metrics. I can’t imagine anyone else having something similar in polish. I hope he gets some traction (if he hasn’t already) on his work. It was impressive.

Dave Smith – Audio on the iPhone

Dave had a presentation that I really enjoyed having worked with various audio APIs. I asked the most questions on this one. He walked through his audiobook app (which was really neat). He was friendly to talk to (for further notice … everyone was nice). He gave a good overview of real code and a real working audiobook app he works on. The app displays the text version of the audiobook while it plays and stays in sync. It’s very polished.

Some random notes (I wrote as fast as I could):

  • AVAudioPlayer level above openAL
  • mp3 format is hw decompression, good for batt/performance
  • To get started, add AVFoundation, AudioToolbox frameworks to project
  • Make a pointer: AVAudioPlayer* player
  • - (IBAction)play:(id)sender; // methods for button actions etc
  • In interface builder, mapping actions using touchUpInside is the best option to capture user button push
  • Useful command line utility in OSX: /usr/bin/afconvert -iaf4 (convert aiff to compressed formats, pre-compress best for iphone optimization)
  • UInt32 category = kAudioSessionCategory_MediaPlayback // kAudioSessionCategory_* has many diff options

He always released his memory correctly. :)
[player pause]
[player release]
player = nil // nice GC technique

His start method created the player and played at the same time. Pause destroyed it. This might seem odd but he said, “don’t keep player instances around for a long time, non deterministic things can start happening.”

Sze Wong – $1M app

Sze asked the question “what would a $1M iphone app look like?”. He also talked about enterprise development and asked if the iPhone could be a serious contender. He has a metric ton of experience doing enterprise and mobile development. He seems to like the iPhone (hey a lot of us are sick of doing J2EE) as a refreshing platform.

Sze had a really nice presentation that didn’t materialize for me until he showed his demo. I can only describe it as Oracle Forms for the iPhone. He has a slick web ui that can generate custom forms for many different uses. His forms could even include a signature box that the iPhone can use to create a UPS type delivery board. It was pretty compelling and he had a lot of nice backend stuff (like JSON, RSS, XLS exporters) created in the web ui.

Other topics

Things wound down and at the end they had a panel of the experts there give answers to various questions by Dave Mark. It was really neat to see an improvised conference.

  • A lot of people mentioned the importance of Touch Arcade.
  • Someone mentioned nsfetchrequest for nstableviews?
  • Ad hoc distribution for beta testers? I need to research that.
  • Imangi Studios mentioned getsatisfaction, a customer support portal to outsource support

Fun conference. I hope to see them next year or sooner. I think ruby dcamp is next for me.

OpenCV in Eclipse

Blog — Dillon @ 1:22 am

I was getting a weird runtime error about native library something in Eclipse while messing around with OpenCV (using Processing.org).
!!! library OpenCV not found
Exception in thread "Animation Thread" java.lang.UnsatisfiedLinkError: capture

I found the answer was to configure native libraries in my project. This is something I’ve never done in Eclipse so I thought I’d post it in case someone googles this error message. First, I set up my project like the following. Notice that I created a folder named “native”. I moved the shared library files (.dll, .so, .jndilib) into this folder. I am on a mac but I moved the DLL files anyway.
opencv_eclipse_1

Then I went into my project properties and set the Native Library Location as this folder. Viola. The processing applet launches.
opencv_eclipse_2

Macbill Intel

Blog — Dillon @ 10:40 pm

macbill intel
No credit go to me, everything goes to Macbill. I couldn’t get it to build a ubin for the Macbill source and I couldn’t find one anywhere.

So here’s Macbill for Intel.

Upgrade fest.

Blog,Unix — Dillon @ 5:48 pm

Following an upgrade guide on gentoo’s lovely doc site. GCC was majorly out of date (3.3 to 4.1.1) and hopefully you can still read this after all is said and done.

Right now, apache is in a weird state and I need to emerge a ton of crap:

# /etc/init.d/apache2 restart
* Apache2 has detected a syntax error in your configuration files:
Syntax error on line 6 of /etc/apache2/modules.d/70_mod_php.conf:
Cannot load /usr/lib/apache2-extramodules/libphp4.so into server: libXrender.so.1: cannot open shared object file: No such file or directory

Need X11 and a million other things put back on. Cobwebs from leaving it alone for so long.

SWAT

Blog — Dillon @ 4:26 pm

pagination
Swat, a nice PHP toolkit at silverorange.com has caught my attention a few times.

First, they (or someone in their group) designed the firefox and thunderbird vector icons. Next, they’ve released a few webapps (PHP). I ran into a demo they have online that showcases some widgets they’ve built.

It’s a fantastic little page full of good UI components that would really help out usability. Unfortunately, I’m not using PHP really anymore but I hope to find (or build) something similar to this in Ruby and Java. And not ugly components either, similar quality. I’m sure there are widgets and components that are not up to this quality.

« Previous Page
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
(c) 2012 SQUARISM | powered by WordPress with Barecity