Software Engineering

Bare metal clouds are hard

The problem, explains Eclypsium, is that a miscreant could rent a bare-metal server instance from a provider, then exploit a firmware-level vulnerability, such as one in UEFI or BMC code, to gain persistence on the machine, and the ability to covertly monitor every subsequent use of that server. In other words, injecting spyware into the server’s motherboard software, which runs below and out of sight of the host operating system and antivirus, so that future renters of the box will be secretly snooped on.

» about 500 words

There are no architects at Facebook

We get there through iteration. We don’t try to build an architecture that is failproof. Building an architecture and worrying about it for months and months at a time before you actually go deploy it tends to not get us the result we want because by the time we’ve actually deployed something the problem has moved or there are more technologies available to solve different problems.

We take it seriously enough to say “there are no architects on the team.”

We do a very “you build it you own it” process, where any team or any individual or any engineer that builds or designs something, they own it, and they do the on-call for it.

On call is where we learn, and that’s how we improve over time.

You build a system…you don’t have to be perfect. Deploy it, and as long as you have enough detection and mitigation capabilities, you will do OK. And you will learn, and you will iterate over it, and you will get better over time.

From the NANOG73 keynote: “Operations first, feature second” by Facebook VP of Network Engineering Najam Ahmad. It’s at about the 10:20 mark in the video:


This leads to the emerging pattern of “many clusters” rather than “one big shared” cluster. Its not uncommon to see customers of Google’s GKE Service have dozens of Kubernetes clusters deployed for multiple teams. Often each developer gets their own cluster. This kind of behavior leads to a shocking amount of Kubesprawl.

From Paul Czarkowski discussing the reasons and potential solutions for the growing number of Kubernetes clusters.

Hard solutions to container security

The vulnerability allows a malicious container to (with minimal user interaction) overwrite the host runc binary and thus gain root-level code execution on the host.

From Aleksa Sarai explaining the latest Linux container vulnerability.

To me, the underlying message here is: Containers are Linux.

From Scott McCarty washing his hands of it.

Kata Containers is an open source project and community working to build a standard implementation of lightweight Virtual Machines (VMs) that feel and perform like containers, but provide the workload isolation and security advantages of VMs.

From the Kata Containers website. The project is intended to be “compatible with the OCI specification for Docker containers and CRI for Kubernetes” while running those containers in a VM instead of a namespace.

The future of Kubernetes is Virtual Machines, not Containers.

From Paul Czarkowski, discussing multitennancy problems and solutions for Kubernetes.

Shuffle sharding in Dropbox's storage infrastructure

Volumes are spread somewhat-randomly throughout a cell, and each OSD holds several thousand volumes. This means that if we lose a single OSD we can reconstruct the full set of volumes from hundreds of other OSDs simultaneously. This allows us to amortize the reconstruction traffic across hundreds of network cards and thousands of disk spindles to minimize recovery time. » about 300 words

Parts of a network you should know about

If you’re running infrastructure and applications on AWS then you will encounter all of these things. They’re not the only parts of a network setup but they are, in my experience, the most important ones.

The start of Graham Lyons’ introduction to networking on AWS, which (though the terms may change) is a pretty good primer for networking in any cloud environment. Though cloud infrastructure providers have to deal with things at a different later, Graham’s post covers the basics—VPCs, subnets, availability zones, routing tables, gateways, and security groups—that customers need to manage when assembling their applications.

We're gonna need a bigger PRNG cycle length...

The general lesson here is that, even for a high quality PRNG, you can’t assume a random distribution unless the generator’s cycle length is much larger than the number of random values you’re generating. A good general heuristic is —

If you need to use n random values you need a PRNG with a cycle length of at least .

From a 2015 post by Mike Malone on PRNGs vs. random key collisions. The Chrome/V8 bug that caused Mike to write nearly 5000 words to explain has since been fixed, but you can check your browser’s PRNG here.

Interfaces, surface area, durability

A DOS program can be made to run unmodified on pretty much any computer made since the 80s. A JavaScript app might break with tomorrow’s Chrome update

From Joe Groff, who wonders if developers will choose old platforms running in emulators over more complex and volatile modern platforms. Also see Nikia Prokopov’s disenchantment.