I saw this animation from NVIDIA’s 2022 GTC presentation and I thought it was neat. So, I set out to re-create it.

You can see it here.

Source is here.

We are in the home-stretch now. Let’s show a few more details and then we’ll run the whole thing and show that it works.

This post is part 3 of a series. We are learning networking by building a network.

• Part 1 - covered networking basics and implemented MAC addressing
• Part 2 - implemented primitives like IP, ARP and an Interface

More Abstractions

In part 2, we created an interface and previewed how we can make “a box” at the end of the post. A box is just slang for a server, a node or a computer. There are many ways we could represent a box, but one way would be for it to own the things that it owns in the real world. Since we will not model an entire operating system here, this is just an approximation.

The Server

A box (or server) owns its hostname, has an interface and an ARP table.

pub struct Server {
  pub hostname: String,
  pub interface: Interface, // for now, one interface
  routes: Vec<Route>,       // routing table, not implemented
  pub arp_table: arp_cache::ArpCache,
}

It also owns its routing table as routes. But the routes part of a server is not implemented here because we never made a router. There is also a major gap in how an ICMP reply would be handled. In the real world, there would be a network stack listening that would generate a response. This is not implemented.

The Switch

pub struct Switch {
  ports: HashMap<u8, port::Port>,
  link_lights: HashMap<u8, bool>,
  cam_table: HashMap<MacAddress, u8>, // MAC to port number
}

The interesting part of the Switch is the cam_table. This functions the same as the arp_table that the Server has, except it tracks what ports have what MAC addresses. When a real switch is turned on, it doesn’t have any knowledge of who is physically in what port. This is going to be faked by a plug_in_interface function which knows this but on a real switch this information would be observed or be done by flooding all ports and watching who responds.

The switch contains ports. This would be layer 1. The cable and electrons are not modeled except for a message on the Port which prints out a "Sending frame out on port with MAC: {:02X?}" line. The simulation stops here. To continue the simulation, we could model the cable or connection to the server and send the packet along with whatever is connected to this port. But it doesn’t do that.

The Main

Time to run the simulation. Let’s walk through the main.

fn main() {
    // Let's say that a server `box1` pings `box2`.

    // We need to set up our network "by hand"
    let mut switch = switch::Switch::new();

    let box1_interface = server::interface::Interface::new(
        "11:12:13:14:15:16",
        "192.168.0.1".to_owned(),
        "255.255.0.0".to_owned(),
        None,
    );
    let mut box1 = Server::new("box1".to_owned(), box1_interface.clone());

    switch.plug_in_interface(1, &box1.interface);

    let box2_interface = server::interface::Interface::new(
        "21:22:23:24:25:26",
        "192.168.0.2".to_owned(),
        "255.255.0.0".to_owned(),
        None,
    );
    let box2 = Server::new("box2".to_owned(), box2_interface.clone());

    switch.plug_in_interface(2, &box2.interface);

This part above basically sets up our network. This is like going to the computer store and buying two computers, one switch and some cables. It also represents installing an operating system and assigning IPs. Notice that our IP addressing and MAC names are still the same from Part 2:

box1 will have the IP address of 192.168.0.1 and the MAC of 11:12:13:14:15:16
box2 will have the IP address of 192.168.0.2 and the MAC of 21:22:23:24:25:26

Notice switch.plug_in_interface(2, &box2.interface); takes the port number as the first argument. This is more readable in code with type hints on because the function hint would look like this:

plug_in_interface(&mut self, port_number: u8, interface: &Interface)

And then later on in plug_in_interface, there is a self.cam_table.insert(interface.mac, port_number); call. So, plugging in a port cheats by remembering the port number in the cam_table.

Then we have more “cheating” because we need to model a hosts file or DNS.

    // box1 calls getbyhostname(box2) which is simulated here
    let hosts_file = make_hosts_file();

And each box’s IP is saved as variables box1_host and box2_host (not shown for brevity). The function make_hosts_file is hardcoded as a helper function in main.

    // box1's IP stack figures out that the request is local, not needed to forward to the gateway
    let local_lan = crate::network::ip::same_subnet(
        box1_interface.ip.parse().expect("box1 ip is not an IP"),
        box2_interface.ip.parse().expect("box2 ip is not an IP"),
        box1_interface.subnet_mask.clone(),
    );

    if !local_lan {
        // TODO: routing and routers
        panic!("Routing not implemented.");
    }

This part is basically IP routing as discussed in Part 1. Box 1 and 2 are on the same subnet so local_lan is always true in this demo. This is here to exercise the IP routing feature but there is no other path at the moment. What would happen next is to implement a router and send the IP packet to the router instead of going to layer 2.

We continue instead and make an ICMP packet from the hosts file resolved IPs.

    // box1 crafts an ICMP echo request and IP packet
    let payload = "This is a ping, weee";
    let icmp_packet = icmp::packet(box1_host.ip.to_string(), box2_host.ip.to_string(), payload);

Then, we go to layer 2 and make an ethernet frame.

    // Once the IP address of box2 is known, box1 checks its ARP cache.
    // now, this is a one-shot simulation program so we will setup this scenario but later we
    // might turn this into a long running or concurrent or GUI program where this is situation
    // is not pre-determined, but in the meantime ... box1 looks up the MAC address for box2
    // using ARP The ARP response is not found so it broadcasts an ARP who-has and gets a
    // response box1 adds the response to its ARP cache
    let dest_mac = box1
        .arp_table
        .lookup(&box2_host.ip)
        .expect("The demo has gone south because box2 ARP resolution failed");

    let ethernet_frame =
        network::ethernet::build_ethernet(box1_interface.mac, *dest_mac, icmp_packet);

    // uncomment to see the packet and open in wireshark
    // write_pcap("ping.pcap", &ethernet_frame);

    // the packet is sent over ethernet to the switch which has its own MAC table etc
    switch.forward_frame(ethernet_frame);

    // the entire process is unwound on box2 which will not be covered here for now
}

Finally, the switch would handle layer 1 and the bits would arrive at box 2.

So, when we run this we get output like this.

Interface with MAC: "11:12:13:14:15:16" plugged into port: 1
Interface with MAC: "21:22:23:24:25:26" plugged into port: 2
Sending frame out on port with MAC: [21, 22, 23, 24, 25, 26]
Frame forwarded to MAC: "21:22:23:24:25:26"

But, there is something more interesting we can do. We can dump the Ethernet frame and see if we really are doing “networking”. In the main, there is a write_pcap() function that we can uncomment and it will dump a pcap file at the project root.

If you open this pcap file in Wireshark, it will parse correctly. These are the bits that would hit the modeled ethernet switch, in other words, the cable as bits because this is the frame that is from the message “Sending frame out …” above.

You’ll notice there are two warnings in yellow:

1. It doesn’t like the MAC address we made up because it probably breaks spec.
2. There isn’t an ICMP reply message seen.

Pretty funny. But otherwise, we see the payload “This is a ping, weee” in the right pane of Wireshark and our source and destination is all there and correct. You can even see the different layers. Ethernet is on top on the left pane, then Internet Protocol and finallly ICMP. These are the networking layers stacked, like envelopes containing envelopes.

The Network is Real

So, that’s a network simulation in Rust. What is fun to think about is that in Wireshark you can replay captured packets on a network. You could create a tun0 address and then send this packet out on a real network. If you had a host with that MAC address and IP then it would be delievered. But even if it didn’t deliver, something else would happen like the switch would ignore it or try to ARP broadcast. Or you could change the simulation to be a MAC and an IP that you really do have.

Seeing our simulated ping in Wireshark is nice visual validation of the simulation we made. We’ve gone from theoretical concepts to tangible output, bridging the gap between code and real-world networking. This experiment has hopefully taken a bit of the mystery out of networking. It did for me. I learned many things from doing all this.

1. I did not know how the CAM table worked.
2. I had not thought about which pieces went where on a server.
3. I had never used the Etherparse library before or generated a packet programmatically.

So, that was fun and I’m glad I got around to making three posts about it. Happy exploring and check out the repo and make it your own or implement something else to learn it. Learning by implementing is not my idea.

This post is part 2 of a series. We are learning networking by building it.

• Part 1 - covered networking basics and implemented MAC addressing
• Part 3 - finishes the abstractions and shows the whole thing working

More Primitives

Let’s model an IP address next. Luckily, Rust has one already as std::net::Ipv4Addr but it does not include all the functionality we need. It is just the data represenation of an IP address. We need what is routing.

I’m certain you’ve seen an IP address before. The interesting thing about an IP address is that it usually goes together with a subnet mask and a gateway address. These three things together tells the IP stack whether an address is local or not. If it’s not local, it sends it to a router. If it is local, it continues to send the message, usually to the local network.

In our simple example, we won’t implement a router but we still want to explore this network concept so we’ll implement it as a function even if it will inevitablly return true. Figuring out if a network address is local or not is not something Rust can do with stdlib. But the logic is pretty simple and can be expressed in 4 lines of Rust.

What we need to do to figure out if our IP message is local is look at the source IP, destination IP and subnet mask:

1. Take the source IP address and figure out the network address. The network address is a bitmask of the subnet mask and the IP address. This is why it’s called a subnet mask. You just AND the bits up. I explain this later.
2. Take the destination IP address and do the same thing.
3. Now you have two network addresses. Source network address and destiation network address.
4. Compare the two networks. If they are the same, send it locally (ie: Ethernet). If they aren’t, send the IP message to the router.

The Mask in Subnet Mask

Let’s say we have this IP address and subnet mask.

IP Address:  192.168.0.1
Subnet Mask: 255.255.255.0

We can explode this out into bits because each number is an 8-bit number in IP version 4.

IP:   11000000.10101000.00000000.00000001
Mask: 11111111.11111111.11111111.00000000

If we logically AND these together (only print a 1 if both are 1) then we get this:

11000000.10101000.00000000.00000000

Which if you turn into decimal again is 192.168.0.0. This tells us what the network address is. In other words, if we do the same for an IP address which is “next to” 192.168.0.1 like 192.168.0.42 then the network address is the same. This is what the mask does. It’s a bit mask by an bit-wise AND operation.

IP Address:  192.168.0.1
Subnet Mask: 255.255.255.0
Network:     192.168.0.0

IP Address:  192.168.0.42
Subnet Mask: 255.255.255.0
Network:     192.168.0.0

So, you can see that the Network address for ` 192.168.0.1 and 192.168.0.42 above are both the same 192.168.0.0`. That means, don’t route it. It’s on the same network, just send it locally (make an Ethernet message). So, let’s create a function to do all this.

use std::net::Ipv4Addr;

pub fn same_subnet(src: Ipv4Addr, dest: Ipv4Addr, subnet: String) -> bool {
    let subnet_parsed: Ipv4Addr = subnet.parse().unwrap();

    let src_network = ipv4_to_u32(src.octets()) & ipv4_to_u32(subnet_parsed.octets());
    let dest_network = ipv4_to_u32(dest.octets()) & ipv4_to_u32(subnet_parsed.octets());

    src_network == dest_network
}

// to_bits is nightly experimental on Ipv4Addr so we have to do it ourselves
fn ipv4_to_u32(octets: [u8; 4]) -> u32 {
    ((octets[0] as u32) << 24)
        | ((octets[1] as u32) << 16)
        | ((octets[2] as u32) << 8)
        | (octets[3] as u32)
}

So, part of IP routing is this network matching which the entire internet uses although there are many other details and specifics. But for now, know that the answer to

Are the IPs 192.168.0.1 and 192.168.0.42 on the same network?

Yes, they are both on the same network. And we can see this behavior in a test.

#[test]
fn test_subnet_routing() {
    let src = Ipv4Addr::new(192, 168, 0, 1);
    let dest = Ipv4Addr::new(192, 168, 0, 2);
    let subnet = "255.255.0.0".to_owned();

    let result = same_subnet(src, dest, subnet);
    assert!(result);
}

Because they are both on the same network, it means we don’t have to forward it to another IP device like a router. We can just send it locally (which means the next part).

ARP

IP is a protocol used for routing traffic over the internet and on local area networks. Similarly, there is another protocol which discovers hosts on a local network called ARP. It’s job is to discover what MAC addresses go with what IPs. When we send an IP message, that’s not enough. Remember in in Part 1, we talked about how messages go up and down abstraction layers. So ARP is sort of connecting Layer 3 and Layer 2 because it connects IP addresses and hardware addresses which (in wired networks) lead us to ports on a switch and eventually electrons on a wire.

Simulating a real ARP request would be complicated because it’s usually built into the operating system or network stack. For this simulation, we’re going to hardcode responses like this.

// src/network/arp.rs
impl Arp {
    // Simulating a broadcast, not accurate or realistic
    // In a real scenario, this would involve network communication
    pub fn broadcast_arp_request(ip_address: IpAddr) -> Option<MacAddress> {
        match ip_address.to_string().as_str() {
            "192.168.0.1" => Some([0x11, 0x12, 0x13, 0x14, 0x15, 0x16]),
            "192.168.0.2" => Some([0x21, 0x22, 0x23, 0x24, 0x25, 0x26]),
            _ => None, // do nothing and like it
        }
    }
}

For the remainder of this simulation, we’ll use a convention where box1 is all about the number 1. It has an IP with 1 and MAC hexadecimal that starts with 1 and eventually it will be plugged into port 1 on a network switch we will build. Eventually, we will observe success by seeing a real network message get passed and this convention will help with reading. This will be shown in Part 3.

box1 will have the IP address of 192.168.0.1 and the MAC of 11:12:13:14:15:16
box2 will have the IP address of 192.168.0.2 and the MAC of 21:22:23:24:25:26

In a real network, the ARP request would be a real-time event and simulating this would involve threads and concurrency which is not the point of this practice. So, ARP gives us a MAC address. What we do next, is make an Ethernet message that uses this MAC address.

Ethernet

The etherparse library has a packet builder which is very convenient but for this demonstration is unfortunately reversed in a particular way. What we are trying to demonstrate is going through the layers in order of the OSI model. But etherparse’s library does not allow us to do this.

Just as an example, this is how etherparse lets you create an Ethernet packet (frame) using their PacketBuilder.

pub fn packet(src_mac: MacAddress, dest_mac: MacAddress, payload: &str) -> Vec<u8> {
let builder = PacketBuilder::ethernet2(src_mac, dest_mac)
    .ipv4([192, 168, 6, 10], [192, 168, 6, 20], 42)
    .icmpv4_echo_request(123, 456);

We are starting with an IP packet and trying to make an ethernet packet. But etherparse doesn’t let us split up these method chains. So, unfortunately, we have to pass Vectors of bytes around. Fortunately, this is more accurate to the real world. Thinking of data flowing through the networking layers as binary is probably mostly accurate.

So, etherparse is reversed from what we want and coupled. Etherparse does ::ethernet2().ipv4() and we want ::ipv4().ethernet2() basically. And it’s coupled because you can’t break these apart. It’s just how Etherparse’s library works. In a real project, it wouldn’t matter. But for explaining the layers and learning, it’s just unfortunate.

// src/network/ethernet.rs

use etherparse::Ethernet2Header;

pub fn build_ethernet(source: [u8; 6], destination: [u8; 6], payload: Vec<u8>) -> Vec<u8> {
    let mut buffer = Vec::<u8>::new();

    let header = Ethernet2Header {
        source,
        destination,
        ether_type: etherparse::EtherType::IPV4,
    };

    header
        .write(&mut buffer)
        .expect("Failed to write Ethernet frame");
    buffer.extend(payload);

    buffer
}

There is our ethernet message. It’s a Vector of bytes and etherparse handles the bit offsets for us. Let’s finally create a network interface that can contain an IP address and a MAC address and send this Ethernet message.

An Interface

This is modeling a conceptual network interface. As we saw in Part 1, commands like ip addr and ifconfig on Linux can show us network interface information. But this is too much to implement.

// src/server/interface.rs
use crate::mac::{self, MacAddress};

#[derive(Clone)]
pub struct Interface {
    pub mac: MacAddress,
    pub ip: String,
    pub subnet_mask: String,
    pub gateway: Option<String>,
}

impl Interface {
    pub fn new(mac: &str, ip: String, subnet_mask: String, gateway: Option<String>) -> Interface {
        Interface {
            mac: mac::parse_mac_address(mac),
            ip,
            subnet_mask,
            gateway,
        }
    }
}

A Preview of putting this together

In the next part, we’ll finish up with a server, a switch and a main.rs that runs the entire thing.

So, for example when we are setting up box1 in our main.rs, we can do this.

let box1_interface = server::interface::Interface::new(
    "11:12:13:14:15:16",
    "192.168.0.1".to_owned(),
    "255.255.0.0".to_owned(),
    None,
);
let mut box1 = Server::new("box1".to_owned(), box1_interface.clone());

And we will “plug it in” to a switch on Port 1 that we haven’t created yet.

switch.plug_in_interface(1, &box1.interface);

See you in the next post.

Let’s take the magic out of networking, shall we?

First things first, let’s establish some ground rules:

1. Our aim is to provide an abstraction level akin to that experienced by a Linux server or networking device user, not delving into the intricacies of network engineering.
2. We’ll avoid nitty-gritty details of bits, electrons, and cables. After all, when people say they don’t grasp networking, they’re usually not referring to the physics behind it.
3. Our model won’t operate within an event loop. While scripting or CLI tools are fantastic, they’re inherently limited by their sequential nature. Real-world networking operates asynchronously, akin to a bustling city where events occur in parallel. To replicate this, we would need to veer into text-mode game territory, and I don’t want to do this. We can learn about networking without making a long-running program but some things will need to be faked.

Now that we’ve set some boundaries, let’s name some exciting topics ahead. We will explore bit-math, frames, packets, understand different abstraction layers, make a binary file that can actually be read by Wireshark and create a working local area network (LAN) using real specifications.

This is a series of posts exploring networking by building things:

• Part 2 - implements primitives like IP, ARP and an Interface
• Part 3 - finishes the abstractions and shows the whole thing working

For more in-depth background, here is a nice video series but I encourage you to replicate my work and build a network yourself in Rust or some other language. You’ll learn more by doing.

Approximately How Networking Works

Networking works in abstraction layers from the point of view of the OSI model. From the bottom up, it goes (1) Physical, (2) Data Link and then (3) Network layer. It continues to higher abstraction layers but our simulation and this blog post series will stop at layer 3.

Imagine a server named box1 pings box2.

user@box1 $ ping box2

The ICMP program ping is way up at layer 7 (not pictured below) and sends information down the left side of the picture below ICMP client -> IP -> Ethernet -> Switch Port 1. Eventually it turns into bits, electricity that flows across the switch through cables (not pictured) and then hits box2’s stack where it flows back up right side. This flow up the right side of the picture is shown below as Switch Port 4 -> Ethernet -> IP -> ICMP client. (The ICMP responder would really a part of the network stack on box2). So notice that the flow down the left side and the flow up the right side are opposite and reversed.

We will talk about all the details of all of this. But here are key takeaways at this point:

• Details are hidden by the abstraction layers.
• The layers are different. The physical layer could be copper wires, wifi, pigeons or glass. The other layers don’t know or care. Ping works the same over wifi or wires.
• Each layer has its own concerns (non-leaky abstraction).

Examples in the World

You’ve probably seen some of these concepts in terms of things you interact with like ip addr or ifconfig or a network cable. So, the point of this simulation is to try to model things around these common interactions and see if we can fake ping box2.

Take a look at the # ifconfig eno0 output above. This command has been replaced in many Linux distributtions with ip addr but the output is basically the same. The IP address is listed as inet 84.16.226.173. The netmask is 255.255.255.0 and the MAC address is under ether 00:9c:02:9b:fd:9c. The output for ip addr will be slightly different.

So in terms of modeling and the previous layer diagram, we will probably need at least the following nouns/objects/concepts/structs:

1. A server called box1 with an IP address. It could be 84.16.226.173 but we will use a more common private IP address of 192.168.0.1.
2. An IP address. Something that represents an IP. We won’t rebuild ip addr or ifconfig so we’ll need to make an IP address “thing” and a subnet mask.
3. An Ethernet device. Notice that in the ifconfig screenshot the interface is called eno0. The device name is arbitrary depending on your device driver and hardware model. We’re not going to model device driver details, all we need is an interface with a MAC address. The MAC address is burned onto a network card at manufacturing time. The MAC’s job is to locally identify an interface (sometimes called a NIC) on a local area network (LAN).
4. You’ve seen Ethernet cables. We won’t end up modeling cables but you’ve probably noticed the plug on a switch or a home router. You’ve also probably noticed that when you plug in a port a light turns on.

Notice that we can sort the things we have named so far into the same abstraction layers.

• Layer 1 / Physical - Cables, ports on a switch, pins, bits, electricity
• Layer 2 / Data Link - Ethernet, MAC addresses, a link light
• Layer 3 / Network - IP address, subnet mask, the ICMP protocol which ping uses
• Layer 7 / Application - A program named ping in Linux and Windows

Notice that we skipped nnumbers 4, 5 and 6. This is on purpose because these layers do not have concepts we have named. We can just ignore them for now for simplicity. We are going to focus on Layers 2 & 3 while faking Layer 1.

Starting our Model

So, with these concepts named and sorted, we can start thinking about how to implement this. There are many ways to implement all of this. First, I arbitrarily selected Rust because it’s my current low-level, CLI type of language. It’s also nice to have access to bytes and tooling that is lower down in abstractions. Specifically, we’ll be using a library called etherparse to avoid us having to define binary bit offsets from the specs and things like that.

First, let’s talk about the end-to-end flow of what happens with ping and then we’ll discover some things. When user@box1 $ ping box2 executes, this is approximately what happens:

1. Before anything happens, we had already created our network. IE: bought two servers, a switch and some cables.
2. We plugged in box1 into port1 and box2 into port2 on an ethernet switch.
3. The interfaces came with a MAC burned in at manufacturing time. box1 has 11:12:13:14:15:16. Each hex for box1 starts with 1 for ease of reading for the reader.
4. We assigned 192.168.0.1 to box1. It ends with .1 for ease of reading for the reader (and myself).
5. We did the same for box2 with 192.168.0.2 and the MAC 21:22:23:24:25:26.

When ping fires,

1. The program ping creates an ICMP packet and tells the network stack to make an ICMP packet with the destination of box2. This is layer 3.
2. The ICMP packet destiation of box2 isn’t good enough. It needs an IP address. So, before the IP packet is made, it looks up the IP address for box2. Normally, this would probably be done with DNS. We are going to fake it with a fake hosts file.
3. The IP layer now crafts a packet with the source of box1’s IP address and the destination IP of box2.
4. The IP layer now finds the interface that this network packet needs to go out on. Our simulation will only have one interface per server.
5. Now we hit layer 2. The IP is not enough, we need to put this network packet on the Ethernet network. So, we have to make an ethernet frame. The ethernet frame has a source and a destination just like IP does but it speaks in MAC addresses, hardware locations.
6. The server goes through a process of resolving the MAC address with a protocol called ARP which asks the network who owns the IP for box2 (192.168.0.2). ARP replies with 21:22:23:24:25:26. Now, box1 remembers this information in its ARP cache.
7. Now box1 sends the Ethernet packet out its interface to the switch on port 1.

This is where the simulation implementation ends. The switch in this example is a dumb layer 2 switch that hasn’t seen any traffic from anybody. All it has seen is an Ethernet packet coming from port 1 with a destination of 21:22:23:24:25:26. It doesn’t know where 21:22:23:24:25:26 is. So normally, it would flood all ports trying to find who that is. For our example, we stop here. If we implemented the switch some more, we would pass the ethernet frame into the interface attached to port 2.

There are many ways we could have modeled this but for now, this is enough. Notice that we have run into some concepts here like box, IP, switch, MAC, ARP and some others. These are some of the things that we will start modeling.

Our first model, the MAC address

A MAC address is what was in the ether field from the ifconfig screenshot we saw earlier. It’s a series of bytes in hex. An ethernet device has one but also dumb switches use them for passing along packets. We’ll talk about switches more in later posts but for now know that we have at least two things that need to know about MAC addresses. Without explaning each one now, these are the things we will find out that we need to have MAC addresses for:

1. An ARP broadcast asks the local network if anyone knows about the ownership of an IP address
2. A switch keeps a copy of MAC addresses it has seen in a table called a CAM table
3. An ethernet interface has its MAC address usually burned onto a ROM

So, this MAC address concept is coming up a lot. We should model a MacAddress type. We’ll also want to print this out in a friendly format for debugging so we will make a .to_string() method.

pub type MacAddress = [u8; 6];

pub fn to_string(mac: &[u8; 6]) -> String {
    // Convert each byte to a two-character hex string and join them with colons
    mac.iter()
        .map(|byte| format!("{:02x}", byte)) // Ensure two digits with padding if necessary, and lowercase hex
        .collect::<Vec<_>>()
        .join(":")
}

pub fn parse_mac_address(mac: &str) -> MacAddress {
    let parts: Vec<&str> = mac.split(':').collect();
    if parts.len() != 6 {
        panic!("Invalid MAC address format");
    }

    let mut mac_array = [0u8; 6];
    for (i, part) in parts.iter().enumerate() {
        mac_array[i] = u8::from_str_radix(part, 16).expect("Invalid hex value in MAC address");
    }
    mac_array
}


#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_parse_mac_address_valid() {
        let mac_str = "AA:BB:CC:DD:EE:FF";
        let expected = [0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF];
        assert_eq!(parse_mac_address(mac_str), expected);
    }
}

The tests section shows its usage. It will be more useful and clear what this type is doing when using it with an interface or an ARP function. But for now, that’s our first model. The others will be similar.

In the next post, we’ll continue modeling out concepts. The source code will not be completely explained and duplicated in following posts but is available on Github.

• The author calls it a microframework
• It has a small hello world example that looks like Sinatra in the README or the homepage
• It doesn’t pre-solve a common problem for you

So, Flask / Express / Sinatra are microframeworks as opposed to their sister projects Django / Next.js or others / Rails. If the language ecosystem offers a pair like Django vs Flask then comparatively, Flask is the microframework by comparison and not just because wikipedia also classifies it this way. It’s because it’s smaller than Django. This is not obvious if a language does not have a bigger framework.

The “pre-solve a common problem for you” is the big one. To make this more concrete, I will be using the use case of adding a database to your project as an inflection point. Larger frameworks usually have some kind of story around adding a database and the microframeworks do not.

It Started with Sinatra

There are many microframeworks in every language but it started with Sinatra. Sinatra had this very cute and small README snippet on their home page. “Put this in your pipe and smoke it” was the original tagline. The picture above is from The Wayback Machine. It was compelling and provocative at the time. You wire up a route and you get some plain text back. The whole idea fits in a code snippet and this was hard to ignore.

require 'sinatra'

get '/' do
  'Hello world!'
end

The trade-offs of Sinatra are not obvious in the above Hello World. To me, the simplicity trade-off is inticing to juniors when they don’t know what the trade-off is. The API surface is small which is easier to learn. But confusing things start to happen after that. If you start a Sinatra project, you will realize that when you change your code, you have to ctrl-c to stop the dev server and start your app again with ruby myapp.rb. Every code change, switch terminal, ctrl-c, up-arrow, enter. That was just changing code, manual testing. Yes, there was a plugin to do dev reloading but so many other questions and concerns would appear after this most basic flow. It doesn’t come with dev reloading and common use cases continue from there:

Where do I put passwords?
How can I add a database?
How do I build or deploy this thing?

So, as the project grew or if you simply kept working with it you would have to solve, research or enhance Sinatra yourself. Many times, myself and others would copy whole files out of Rails default projects. Like, we’d generate a Rails project off to the side and steal .rb files from it. Or, we’d end up stealing ideas from Rails. Ideas like development server reloading, where to put configs, test fixtures or the concept of dev/test/prod.

But I think these DX nit-picks are not where the test is. I think adding a database to a microframework is the inflection point where the lack of features makes things tedious and confusing.

The Database is the Inflection Point

I think configuring a database stresses the framework and most microframeworks fail here. This isn’t quite The Database Ruins All Good Ideas, it’s more like, The Database Makes the Framework Creak.

Some teams running Flask might say “we have a database in Flask already, it’s easy” but what they really have is a bunch of hidden context. Take this example of how some work with Flask:

1. There’s a production database which a Flask app connects to. This is the only database in the world for this application. There is no dev database, even on a laptop.
2. The SQL to create this empty database was copied to something like Dropbox, email or a file share. Or maybe it never was. “Why would you need an empty database?”
3. The database rarely changes but not by intent. When a change needs to happen, stress and confusion levels are high. It might even be deemed impossible.
4. The password for the database connection is in git and the repo is set to private. This password is never rotated when team members leave.
5. When you boot Flask to do development, Flask connects to the production database because that’s where the data is (“what else could we do?”).

In some cases, the fact that this setup works at all is sometimes related to the circumstance that the application is a read-only visualization or dashboard. If the application had become read-write, this entire idea would fall apart. Or, maybe there’s a “database person” that essentially is a mutex for the team. Or, maybe the app is so small in scope that this is fine.

So far, this hasn’t been my experience. My experience has been to take ideas as needed from full frameworks and bring them into Flask. These concepts are not obvious in Flask because Flask do not have these concepts in them. Take dev/test/prod. In order to add dev/test/prod, we need to:

• Get the passwords out of git and introduce the dotenv library
• Add some configuration files, maybe with Dynaconf
• Add database migrations and seeding
• Have every dev have a local database
• Have some example data or factories or something

If we get this far, a pull request could have code and schema changes proposed. Without it, we’re unlikely to mess around with the database structure.

You might be quick to blame the team. I’m quick to blame the tool but blame isn’t interesting here. I think Flask influences thinking and trades-off too far in the direction of simplicity. There is no dev/test/prod in Flask so I have to invent it by composing libraries together on top of Flask. Actually getting this to work with tests and conftest.py is non-trivial. I’m not saying there is no trade-off either. There is absolutely a trade-off. Adding a dev database might confuse juniors. We might have to introduce Docker or try to solve development environments. I might have just invented my own flask-unchained by selecting libraries.

For very simple applications, this simple setup might work fine. I don’t think it works for long. I think applications usually grow in complexity and features. I think it is common to have Flask fall apart in your hands I think teams in this situation have only been exposed to Flask. My experience in these cases has been to introduce ideas from other frameworks to Flask-only teams. My experience is that even small apps outgrow Flask because most apps have state in the database and the default experience (not just Flask) is awful.

Most of the Work is Explanation

For dev/test/prod and Flask, the change in thinking had to be the following. There isn’t just one database. There are many instances of this database in different environments for different purposes. This is a flawed “shared development server” style of thinking where the only database available is the production one. It’s not A database, it’s THE database.

┌────────────────┐
│ "THE" Database │
│                │
└────────────────┘
         ▲        
         │        
         │        
┌────────────────┐
│ Flask (laptop) │
│                │
└────────────────┘

Instead, thinking of it like as many instances in different contexts or environments this opens up many options.

┌───────────────────┐  ┌───────────────────┐  ┌───────────────────┐
│ A Database (dev)  │  │ A Database (test) │  │ A Database (prod) │
│                   │  │                   │  │                   │
└───────────────────┘  └───────────────────┘  └───────────────────┘
          ▲                      ▲                      ▲          
          │                      │                      │          
          │                      │                      │          
┌──────────────────┐   ┌──────────────────┐   ┌──────────────────┐ 
│   Flask (dev)    │   │   Flask (test)   │   │   Flask (prod)   │ 
│                  │   │                  │   │                  │ 
└──────────────────┘   └──────────────────┘   └──────────────────┘

There are many databases and not just one. There is no shared development server or shared state. There is no shared development anything. Test might be CI but test can also be your laptop. It is the dev -> test -> prod progression.

Thinking of it this way solves problems like:

• how can we rotate passwords or get them out of git?
• how can I change the database structure while it is running?
• how can we scale from 1 developer to 5?

But this is not what Flask comes with in the Hello World example nor does Flask teach you by example what this concept is. In order to get there you need to add plugins, libraries and configuration. Of course, full frameworks cannot solve all your problems (you have to code something). But when you need to diverge from the framework is usually much later in the project’s life and a good framework will be solving common problems for you (better than you could have done). It’s reuse.

Trade Offs

I think a lot of this is inevitable given an assumption of what class of application your application will turn out to be. Even with this viewpoint that I and others have, it was annoying when a greenfield would come along and we didn’t want to include all of Rails but we also knew that we’d regret picking Sinatra later because we’d have to at least handle things like passwords, ENVs, a database connection pool and development quality of life things. The trade-offs were known, which is why a microframework was being considered. But, do we need all of Rails, all the time? It’s annoying to have to have every concern included by default. Maybe we don’t have and won’t have an admin interface. Maybe we will never send email. But then we think through all the missing features and realize that we’d have to copy and steal ideas. This is also what happens in FastAPI to configure the database. Because FastAPI has “no database ideas” in it, you have to copy and paste configuration from documentation. Ideally, I’d want to opt-out of features in a full-framework and have it be configurable. Even with opt-out, the complexity is still there so I understand the draw to a small API surface area.

Flask was inspired by Sinatra. It has the same trade-offs as Sinatra. If you try to enhance Flask to have more concepts in it then you end up with your own framework. Look at flask-unchained. The features that flask-unchained are the features that Flask does not have. It comes with an opinion on databases, it has a structure for APIs etc. The issue with these projects is they are not authored by Flask Unchained. So the trade-off is a flexible plugin system counter open-source maintainance issues. Many Flask plugins are not maintained. The same issue happens in other languages. Full-featured frameworks usually control more of the stack. So, when they do a release, those things that they include or have written are bumped or fixed so they work in the new release. In theory, a plugin system sounds ideal because it has flexibility. What I’m arguing is that CORS, authentication and database state are extremely common and these things should be in the framework.

More than that, copying and pasting configs from FastAPI docs is one-way and subject to bit-rot. Controlling configuration even in a full framework is extremely challenging but usually there can be step-to-step upgrade guides but this only works when you can name the version you are on. If you are copying and pasting configuration and code, what version of FastAPI are you on?

A Tour of Small READMEs

There are many microframeworks in many languages. Most of them have a small hello world README just like Sinatra did.

Echo in Go

package main

import (
  "github.com/labstack/echo/v4"
  "github.com/labstack/echo/v4/middleware"
  "net/http"
)

func main() {
  // Echo instance
  e := echo.New()

  // Middleware
  e.Use(middleware.Logger())
  e.Use(middleware.Recover())

  // Routes
  e.GET("/", hello)

  // Start server
  e.Logger.Fatal(e.Start(":1323"))
}

// Handler
func hello(c echo.Context) error {
  return c.String(http.StatusOK, "Hello, World!")
}

Kemal in Crystal

require "kemal"

# Matches GET "http://host:port/"
get "/" do
  "Hello World!"
end

# Creates a WebSocket handler.
# Matches "ws://host:port/socket"
ws "/socket" do |socket|
  socket.send "Hello from Kemal!"
end

Kemal.run

Express in Node

const express = require('express')
const app = express()
const port = 3000

app.get('/', (req, res) => {
  res.send('Hello World!')
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

These examples are easily understood which is a pro in the complexity trade off. What is not shown is the other side of the trade off.

There are Few Fully-Featured Frameworks

There are very few fully-featured frameworks and there is usually only one or two in each language.

• Python has Django
• Javascript has Next or Remix or RedwoodJS
• Java has Spring Boot
• PHP has Laravel
• Ruby has Rails
• C# has .NET
• Elixir has Phoenix

But there are very few others. It seems that language communities tend to rally around one or two full frameworks and these frameworks last a long time. The microframeworks might churn a lot more because there is less to throw away or because they are easy to invent? Flask interest turns to FastAPI. Sinatra interest turns to Grape. Express interest turns to Fastify. But Django interest rarely turns into anything except a newer version of Django. Masonite is a rare exception (I have not tried it at scale). It’s much easier to create a hobby microframework project that is small in scope.

I think this is because full frameworks take 7 years to do right and this is assuming a productive or high-level language. The amount of work it takes to make a framework that is documented, tested and feature-rich is huge. Usually it seems to require a sponsored company or extraction out of a working business. Django was extracted out of a newspaper company. Rails was extracted out of a team management company. Next was extracted or built in parallel out of a hosting company. RedwoodJS is from an exited Github founder. I think it is interesting that this is the scale that is required but it also might help us predict what is possible. Can a hobbyist disrupt a full-framework? Can a single person invent a Django or Rails killer?

We can almost skip microframework attention because they will be the first to go. I am not trying to FUD people out of hobby projects. Do it, make your thing. But unless it has some very novel idea in it, then it’s unlikely to stick. We’ve had hundreds of microframeworks already. So if the real reason for invention is “I did it because I could” then this isn’t good enough. We need more “what we need” even if it takes years. This is my main point. Full-featured frameworks are harder to make and harder to learn. But they have ideas in them that you should probably know about. Critically think about the Hello World README example in Flask / Sintra / Express.

Ruby has known about the Python package manager Poetry for a long time as Bundler. This is not a smug victory lap. Bundler was not always the thing to use. If the sands of time are going to bury Ruby so that the language is deleted from all multiverse timelines, then we might as well listen to its dying words:

   “The Poetry transition is the Bundler transition, we already went through this. Ack.” 💀

Bundler is the de-facto standard on how to install a library in a Ruby project and has been for quite a while. But there was a time before Bundler existed, when we had library needs but we did not have Bundler. In that pre-Bundler time, some of us would use something like RVM (Ruby Version Manager) gemsets. There were other tools but I am going to talk about RVM, I will explain it later in this post in case you don’t know RVM.

Currently, as of writing, there is much debate about Python packaging. We are in the throws of AI hype. Python is being learned, in-use and popular. But those that are learning it are learning pip install. Or AI projects are not yet feeling operation concerns like repeatable builds.

I’m aware of prior-art posts:

• Why not tell people to “simply” use pyenv, poetry or anaconda
• Python dependency management difficulty is an unhelpful meme

These are good posts. I’m adding orthogonally to them. I want to talk about other prior-art. Bundler and RVM gemsets.

Translation

First, let’s translate a little bit. Ruby gems are like Python packages. Python has .whl and Ruby has .gem. Ruby has only had gems but Python has had many package formats over the years, so I will just say python package.

You can install packages yourself, sort of “raw” using pip install in Python and gem install in Ruby. Some people do this in Python but rarely do you do this in Ruby. There are exceptions like global installs or generators but basically you rarely would do the equivalent of pip install in Ruby.

Both sites have a package website. Python’s is pypi.org and Ruby’s is rubygems.org.

How RVM Gemsets Worked

Before Bundler was even invented, I was using RVM gemsets to keep my Ruby projects separated. RVM is a Ruby Version Manager (RVM) that would install and manage Ruby versions. But it also had this extra feature on it called gemsets. With gemsets, you could create a set of packages named after your project. You could even have RVM switch to the gemset when you cd‘d into your project directory. So this would be like if you venv activated automatically.

You would use RVM gemsets because if you didn’t, your Ruby install would fill up with packages, even the same one many times with different versions. Then your project would not know which to use, not to mention you wouldn’t know which packages you were using. So, if someone asked you this question:

Hey, are we using any GPLv2 stuff?

You’d have no idea. The same happens with raw pip.

When Bundler came out, I rejected it and kept using rvm gemsets. But there was a major difference. Bundler solved the dependency tree while doing isolation. This was something I didn’t understand. When someone told me that my gemset was just going become polluted, I said this was no big deal. I would just delete my gemset and then gem install all the libraries I needed. Maybe I could make a list of gems I needed in a text file in the project root.

I never tried Bundler and I didn’t know what problem it was solving.

When I Changed My Mind

So, I had my own devised system, mostly out of habit. The workflow was very similar to pip with requirements.txt. Except back then, we used README.md as a list of dependencies. It was pretty terrible for repeatability but also just developer experience in general.

This is how classic pip with requirements.txt would work too. pip install -r requirements.txt is append-only to your environment. There’s no cleanup function. pip list shows you all your downstream dependencies but requirements.txt only shows you what you want. If you remove a top-level dependency, it’s hard to cleanup the transitive dependencies. So, people delete their virtualenvs just like I did with gemsets.

Bundler came on the scene and I rejected it. “I have gemsets, why do I need Bundler?” But Bundler was doing something more than just project separation. Bundler was giving me higher level commands, semver and a lockfile I did not have to serialize to a README or a text file.

# usage of bundler
bundle init
bundle add some-library
git add Gemfile Gemfile.lock

# a user of my code
bundle install

This is compared to

rvm gemset create my-project
gem install some-library
# update README.md with some-library "hey, this project needs some-library"

# a user of my code
gem install some-library

The only caveat to bundler is that I need to prefix all commands with bundle exec because bundler did not use shell tricks to change ENVs or paths. So if I wanted to see all the gems I have listed: bundle exec gem list. I mitigate this by using an alias for be=bundle exec.

The Equivalent with Poetry

The amount of commands with Bundler and Poetry is about the same.

poetry init -n
poetry add some-library
git add pyproject.toml poetry.lock

# a user of my code
poetry install

The only caveat with poetry is that I need to prefix all commands with poetry run so poetry run pytest. I mitigate this by using an alias for pr=poetry run.

Ruby Has Known About Poetry

So, Ruby already went through the poetry transition. We learned a few things:

• Don’t raw install libraies with gem install.
• Deeply solve your dependencies as a tree (transitive deps).
• Prefixes on commands are annoying but sub-shell or shell tricks are worse.
• Lock files are good.
• Conventions are good.

This would translate to Python as something like this:

• Don’t raw install libraries with pip install.
• Deeply solve your tree with pip 23.1+. But, is it a good resolver? 🤷🏻‍♂️
• Get used to poetry run, alias it if you have to.
• Lock files are good, poetry comes with one. You don’t need to use pip freeze or piptools or addons.
• Conventions are good. The world will not use your homegrown system.

After I started using Bundler, I never went back to even another style. When I tried Go for 4 years, I used gb and other tools until go.mod was finalized. It was similar with Python. I searched for a Bundler-like tool and found Pipenv. Pipenv’s resolver failed me on a project and Poetry did not. I switched to Poetry. When I started with Rust, Cargo was very familiar because the people that worked on Cargo came from the Ruby community.

Lately, pip has been changing over to a pyproject.toml format which has a higher level of abstraction. I’m glad. It seems like pip is becoming more like Poetry. That’s fine, let the ideas be shared. I would use a Bundler-like tool, no matter the name. You could even say I would use a Cargo-like tool. The trickiest part of Poetry has been convincing people to try it and I went through the same thing with gemsets.

Our program depends on this function. How do we know our program works?

Mock the Randomness

Our program is just consuming the random function. If we print out the random number, we just need a number. We don’t care if the number is random or not. So the answer is simple: just avoid the randomness of the function.

It’s the XKCD comic 221.

int getRandomNumber()
{
  return 4;  // chosen by fair dice roll.
             // guaranteed to be random.
}

So, we could basically do the same by mocking out the random function to return 4. Problem avoided.

Leverage the Seed

If we need to functionally test what happens in different cases, maybe mocking the random function isn’t good enough. What if the number 9 crashes our program for some reason. We want 9 to be returned by the random number generator but also in a more complicated program (say a map generator), we might want to reproduce the state exactly for when 9 happens.

This is pretty easy to deal with, we just pass a seed in. Many things use this technique:

• A test suite’s order in pytest-random-order and rspec.
• The minecraft /seed command so your friends can play the map you have.
• An avatar generator might have a seed as well as other inputs.

Using the seed, you can replay or inspect the randomness as a way to make the behavior deterministic.

Not everything can be made deterministic in a sort of clean room setting. The next example is very different.

AI Models

Let’s take a sharp turn and assume that the context you are in is AI, Machine Learning (ML) or LLMs. Suddenly we are encountering a very different kind of randomness. These models are not deterministic and they don’t have a seed. We can test our program around the randomness with mocking, perhaps. But what about the model itself? What if we need to know that it works?

So, what are we talking about here? How do we know our model works? How do you test a model? How do you compare two models? How do you figure out if you have improved your model when you release a new version? How do we know if ChatGPT 4 is better than 3.5 beyond anecdotes? How does OpenAI determine that ChatGPT 4 is so good that it should be a paid upgrade over 3.5?

The answer is evaluation but I want to break down what an evaluation is a bit. I have to caveat that my experience with ML evaluations is mostly surface level from proximity to research, PHDs, etc. I have never designed an evaluation, so my numbers here might be very rough. I’ll also caveat that my specific examples (like Llama 2) are locked in time to the publish date of this post.

Let’s say that we wanted to see if Llama 2 is worse at generating Rust code than Llama 2 Code is. Seems reasonsable, right? Llama Code was made for coding and Llama was not. So, how do we really know? The models are not deterministic. We can ask it the same question and we will get different answers. We can’t mock anything because we don’t want to test the code around the model, we want to test the model. There is no seed or anything to make it deterministic. And, we can’t inspect the model because these models are not really explainable (or at least easily).

So, our previous approaches don’t work. We need a new approach. The new approach will be to throw experiments at it and measure our results using some methodology. There are many methodologies but for the sake of brevity, we’ll use the classic F1 score. I’m not going to go into how that works but how the evaluation would roughly be run.

The rough steps are going to be:

1. Generate or obtain ground truth questions and answers and call it the “Test Data”.
2. Keep the answers secret from the model.
3. Give the Llama 2 model each question from the Test Data and note its answer.
4. Do the same for the Llama 2 Code model.
5. Score the results.

These steps are easier said than done. Let’s try to break this down even more. What is involved in step 1?

Step 1 - Generate Rust Questions

One question will be this:

Generate hello world in Rust.

And the answer we expect is this:

fn main() {
    println!("Hello, World!");
}

We (as humans) all agree that this is the result that we expect given the question/prompt. No more context is allowed. This is one question and answer pair in our “Test Data” set. We might aim for 200 or 1,000 such questions. We might aim for a diverse set of questions, things way beyond hello world. As we select what we care about, we add bias to our evaluation. But bias is a different topic.

Step 2 is simply file organization, really. This is more important if we were fine-tuning Llama or making a model ourselves. But since Llama 2 models are already done, we can just move on.

As a side note, sometimes generating ground truth and test data can take a team of many people, many months to create.

Step 3 and 4 - Runs

For each question, run the model with the question. If we have Llama running somewhere, hit the API and record the result. We’ll get to the hard problem in Step 5. Repeat the calls for the other model. Organize the results. Maybe you have something like this:

results/llama_2/0001.json
results/llama_2/0002.json
...
results/llama_2_code/0001.json
results/llama_2_code/0002.json
...

The results files might be API JSON responses or a file format we’ve invented with metadata and the response as an attribute. The results file format is not important.

Step 5 - Scoring

Now the tricky part, scoring. What we need is a very exact number for our F1 calculation. But we have a text answer coming back. Consider this answer for our hello world question:

fn main() {
    println!("Hello, World!");
}

It’s an exact match. So, let’s talk about how this is scored. F1 score is calculated from 3 metrics around Precision and Recall:

True Positives (TP): 1 (since the generated code is correct)
False Positives (FP): 0 (since there are no incorrect results generated)
False Negatives (FN): 0 (since there are no correct results missed)

Precision = TP / (TP + FP)
Recall = TP / (TP + FN)

Precision is sort of like a function of quality and Recall is a function of quantity. They both have to be considered at the same time. In this case (one question, one answer) our F1 score is 1.0. It doesn’t get any better than that. But our scorer is way too simple. Consider this answer:

// this program prints hello world
fn main() {
    println!("Hello, World!");
}

We said that the generated code must match the answer. It doesn’t match the answer, even if we (as humans) know that code comments are ok. A tool like grep would fail on the complete string match of our Test Data answers

True Positives (TP): 0 (since the generated code is incorrect)
False Positives (FP): 1 (since there are no incorrect results generated)
False Negatives (FN): 0 (since there are no correct results missed)

What is our score now? It’s 0.0. But we (as humans) know that comments are ok, even encouraged in some cases. So, how could we get around this? We could pre-process our results to trim code comments to handle this specific case. What else could we do? Well, things get trickier after this point. Our scorer would have to improve or we’d have to use random sampling (as human domain experts) or other techniques to score our results. If we can make our scorer smarter then there is less and less manual work to do. Maybe we could use Levenshtein distance to fuzzy match bits of the program. Maybe we could break it apart with a parser. Maybe we could just run it and capture the results. Otherwise, lots of manual work.

In the end, we might be able to score an answer like this:

fn main() {
    let message = format!("Hello, World!");
    println!("{}", message);
}

Eh, that’s not really what we wanted but it’s ok. I give this a code quality score of 4/10 while marking it a True Positive. Then we assign weights and review feedback as a group. This process is very subjective and tricky. This is also why F1 scores cannot be used in a vacuum. Developers are used to something like this.

The scoring process might take even more months to do and many iterations. In the end, you would hopefully end up with a metric that you trust. This would also be something you would revisit like other performance benchmarks.

Wrap Up

Note how we had to have many questions and answer pairs. Notice how we exploited certain truths about the data that we knew as humans:

1. We know Hello World should be an exact match because it’s a simple program with a long history. In this case we are exploiting some string matching on Hello, World!" as the message to print.
2. We know that code comments are not important for our pre-processor. We can exploit the beginning of lines // are filtered out or something.
3. We can possibly exploit string nearness with Levenshtein distance although this is most likely a bad path to go down.
4. We know programs are executable so we could just run the thing and see what happens. This might involve changing our approach from string matching to having our ground-truth Test Data be actual Rust tests.

Conclusion

The main point I’m trying to make is that in some cases, AI/ML have had a different approach to solving problems than general software and I think machine learning evaluations need to be more understood especially by those who are integrating.

Testing randomness is solvable and recognizable in general software. I think I at least showed two basic approaches in about a paragraph while explaining evaluations even at a high level took a couple of pages. I have read a lot of people telling one-off stories about ChatGPT making mistakes. I think these are fine conversations to have as almost UX feedback but not as formal evaluations. Formal is just the term people use for using a methodology.

This gets even more complicated when people who are integrating LLMs for the first-time and wondering if their idea “works”. In general software, we usually avoid this problem with other tricks like mocking or seeding. I’m always curious how they are going to find out if they have not generated 100s of test data questions or know what an F1 score is. I barely understand evaluations and certainly have not designed one. I just have been around it a bit.

At the same time, I saw a determinism commonality between PRNGs and AI models and wanted to write a bit about its similarity. If we were testing the PRNG itself, we might follow a similar approach where we try many executions, collect the results and try to analyze it (maybe check its distribution). This is different than deterministic functional testing.

Example Bets in the GUI Domain

Someone somewhere:
“Electron is too slow and Qt is the future, $25 on Qt please.”

I agree on the impulse. I don’t agree on the bet. There’s a problem of feasibility, complete project context but also just historical trends. There are plenty of debates already online so there’s no need to rehash them here. I wanted to instead just focus on one aspect of this which is layout.

Layout has been implemented many times. Almost none of these technologies have gathered human effort like the web has but lets consider some past examples. In the early Web, if you wanted lots of functionality (or all you knew is one tech stack) you had to reach for Java applets, Flash or some other browser plugin.

So if you picked Java to do a form, you would pick a layout class to use. One is the GridBagLayout. An applet might have used this instead of form markup plus styling.

button = new JButton("5");
c.ipady = 0;       //reset to default
// snip ...
c.insets = new Insets(10,0,0,0);  //top padding
c.gridx = 1;       //aligned with button 2
c.gridwidth = 2;   //2 columns wide
c.gridy = 2;       //third row
pane.add(button, c);

Of course there might have been tools to help you generate these layouts but this is essentially the CSS of Java GUIs. If you squint hard enough, you can almost see a stylesheet in there. They call it insets whereas CSS would call it padding.

Qt does a similar thing. This is not me hating on Qt. I breathe a sigh of relief when I can use Qt. It’s quick, it’s light. It looks nice when the scope is small. I don’t want Qt apps to disappear.

The Bet Over Time

So given a goal of distributing a GUI form with project and team constraints, maybe you would select Java and its GridBag. But this technology is not compatible or related to the web version you didn’t write. Over time (with hindsight), this turned out to not be a good bet. Flex, Flash, Shockwave, Applets, Silverlight, ActiveX have come and gone and the pattern is still repeating today. The web tech version we have now has not been perfect and I understand the critics.

I would instead bet that distribution, updates, marketing, docs, interop and many other aspects of this hypothetical Java Applet project would eventually need something that is adjacent to web tech. Maybe the page itself that contains the “active form” or the rich email that you will later send.

Web tech doesn’t auto-win. I still like text. TUIs are great (but perhaps a concession). Native mobile is tricky and no one I talk to really likes generators or abstraction layers they are using (but this is 2nd hand).

Regardless of current abstractions, I feel that complete web tech avoidance is a liability and the implementations can be fixed. I have run into terrible web apis written late or poorly as a feature reluctantly bolted on. Misformed XML, weird JSON, wrong verbs. Whatever a bad API is, it’s usually not coming from a web-native team or culture. These are not specialized projects with an exempt domain. Is that the bet coming due?

If I had to bet I would bet that someone is going to solve Electron’s slowness before Qt displaces web tech or find a performance solution in general. The best thing would be to have a performance workaround or solution while keeping the Web APIs to enable the most interop and reuse. Then I’d rather adopt web tech related skills for the team.

The internal dichotomy I have is considering that I quite like Xcode from what I’ve used and I can’t imagine how you would have both. Trying to use web tech naturally leads to a web view which would have to have an entire engine in it. Now we don’t have web with applet/flash plugins; we have native code with a web plugin. It’s just swapping the framing. If browsers on the desktop can do lightweight native web tech then mobile will too, that’s my hope anyway.

Many Domains, One Tech Stack

The list of things web tech is solving is increasing. There’s very little left untouched. I’m almost speechless. I firmware flashed a teensy board using WebUSB. I changed a configuration on an audio interface using WebMIDI. Someone told me that their browser was opening native files and autosaving to their filesystem and I said “that’s impossible without a security vuln or something” and lo and behold, I was extremely incorrect.

In this vein, here is a list of technologies which were server-side, native or sacredly impossible to have a web alternative and now are in use or soon to be.

The list continues with similarities like what Web Workers would equivocate to in an operating system context. The list of things web tech is not solving is small regardless of what I think.

When I flashed a development board over WebUSB, there were two options: a binary or use the browser. I used the browser. Zero install and they can control distribution and the environment.

Look at the instructions at the bottom. Visit chrome://.../usbDevices? Amazing.

The Web is the Biggest Target

The web platform is the largest there is. The list of technologies is large. The exclusivity and importance of the operating system is ending and there is a focus and a force by all of us arriving and contributing to a single stack instead of reimplementing bespoke things over and over again. If WebWorkers give you something like threads, why not just use it “for free” with an extremely easy distribution model versus trying to package and maintain Windows/Mac/Linux once again?

It’s not all roses. I have a lot to say about nits and niggles in the web tech space but that will just have to be another post. This topic can extend easily to backend web frameworks with a javascript avoidance bias but I want to keep this focused. Consider these equivalent technologies and the problem with GUI technologies when betting against web tech. Without a major black swan event, I don’t see these technologies (and then naturally skills) going away soon.

$ docker images
REPOSITORY    TAG       IMAGE ID       CREATED         SIZE
<none>        <none>    77af4d6b9913   19 hours ago    1.089 GB
postgres      latest    746b819f315e   4 days ago      213.4 MB

Cache Invalidation is Hard

What’s the hardest problem with caching? Expiration! So knowing when to “expire” a docker image must be pretty tricky. Spoiler: it is. First, let’s talk about why we’d want to expire or even manage our docker images to begin with.

Docker has this image store but in the beginning they didn’t have a clear procedure for what you were supposed to do when you accumulate more and more images. You either fill your physical disk on Linux or fill a virtual disk on Mac/Windows and then docker stops working. There are countless threads about the qcow file but there was never any guidance as to what people were supposed to do to manage the images.

Spotify created a helper script called docker-gc to help manage this problem. It’s been archived in favor of docker system prune but this is not a complete solution. Removing images is like expiring the cache, the tricky part is knowing when to do so.

When you are building images, docker will print out intermediate steps as SHAs that you can enter and debug. When you tag a SHA hash, you are tagging a <none>. If you docker system prune then you could throw away data you care about and image prune is not much better. More so, you are blowing away your cache so builds will take longer. Spotify’s docker-gc let you specify images you want to keep which is useful for the previous examples when you are making images.

Sometimes, people suggest cron’ing the prune so the disk never fills up. If I cron docker system prune I’ll lose data (or at least cache hits) every day and might not know why. And it’s still not solving expiring the cache. Eventually your disk could fill up with tagged images and system prune will not have saved you. Your image store is a cache and you can’t tell if it’s out of date or full of unused things. What’s worse is, given no running containers these prune commands will basically empty your image store. So what’s the point of the image store?

You could devise a way to filter docker images using Go templates but this is far too advanced for the use case Docker is aiming for.

My Suggestions

I don’t have a list of quick fixes for the difficult problem of cache invalidation. However, in a perfect world:

1. Docker would run on all OS’s natively so the disk usage would be more obvious.
2. Keep using docker-gc and ignore their advice.
3. Write an updater that works like apt where it tells you that your images are old. This isn’t easy but this is event-driven and not time-driven. Event-driven caches are more precise.
4. Write an alternate utility that works like docker-gc using Go templates to tag and manage images.
5. Docker would provide tooling or some advice as to what to do when people use their product enough to fill their disk.
6. Run a local caching server to mitigate internet pulls? You’re still building all the time and losing partial images if you are cron’ing or running prune.
7. Do the typical cache shrug thing of using time as when to expire. docker system prune -af --filter "until=$((30*24))h" You’ll lose cache hits and lose data, just delayed. At least you’ll eventually get new images of latest I guess.

I hope this post explains a bit of what’s really going on and why this is so tricky. The docker image store is a cache.

Susan was hired to inject some life into the Nine Inch Nails (NIN) forum. NIN hadn’t really been making as much music as before and my take was that people were scared that their favorite band (one of my favs too) weren’t making music anymore and the glory days had passed. Hang onto this idea.

So she developed a series of augmented reality games and events that reinvigorated the fanbase. It was an extremely interesting series of hidden mp3 sticks and hidden puzzles, ending in a staged concert that was broken up by fake police and tear gas. The fans loved it. They created wikis and collected information. There were fake shutdowns and images “sent from the future”. It was an ARG.

You can see in the picture above a spectrogram of the found-in-a-bathroom mp3 file with a hidden image that was itself a pretend “leaked from the future” image of an alien. The fervor and excitement must have been crazy. But think about what I said about “Hang onto this idea”. This entire ARG campaign was started to inject new energy into a forum who were eating themselves alive. It didn’t go unnoticed.

I think about Slashdot, Perl forums, Usenet and yes even boards I associate myself with like Ruby. When Ruby came out it was in direct competition with Perl. Perl gained fear of Ruby. When Node came out, Ruby gained fear of Node. When Go came out, Node perhaps gained fear of Go. And so on to Rust and to Zig and to whatever else is next. Each generation causes fear from the old. But here I’m specifically talking about the lack of input and what the forums are like. Without input, things stew and ferment.

Elixir breathed new life into the Erlang community and Joe Armstrong was happy for it. To me, this is the most mature way to look at it. I can’t imagine average forums of fear having this kind of positive attitude with strangers over anonymous text.