a bunch of stuff I would have emailed you about


This leads to the emerging pattern of “many clusters” rather than “one big shared” cluster. Its not uncommon to see customers of Google’s GKE Service have dozens of Kubernetes clusters deployed for multiple teams. Often each developer gets their own cluster. This kind of behavior leads to a shocking amount of Kubesprawl.

From Paul Czarkowski discussing the reasons and potential solutions for the growing number of Kubernetes clusters.

Hard solutions to container security

The vulnerability allows a malicious container to (with minimal user interaction) overwrite the host runc binary and thus gain root-level code execution on the host.

From Aleksa Sarai explaining the latest Linux container vulnerability.

To me, the underlying message here is: Containers are Linux.

From Scott McCarty washing his hands of it.

Kata Containers is an open source project and community working to build a standard implementation of lightweight Virtual Machines (VMs) that feel and perform like containers, but provide the workload isolation and security advantages of VMs.

From the Kata Containers website. The project is intended to be “compatible with the OCI specification for Docker containers and CRI for Kubernetes” while running those containers in a VM instead of a namespace.

The future of Kubernetes is Virtual Machines, not Containers.

From Paul Czarkowski, discussing multitennancy problems and solutions for Kubernetes.

On asking the right questions

Instead of asking photographers what they might like, Fuji was said to have made up sets of comparison prints and slides: One set showed color as accurate as Fuji could make, the other sets had varying degrees of enhanced saturation—richer, warmer, deeper colors; healthier skin tones; bluer skies, greener grass, redder barns. Photographers, it seemed, consistently preferred the saturated versions. » about 400 words

Conflicting advice on time management

On the one hand:

As a leader, you want to encourage people to entertain “unreasonable ideas” and give them time to formulate their hypotheses. Demanding data to confirm or kill a hypothesis too quickly can squash the intellectual play that is necessary for creativity.

On the other hand:

[Force] teams to focus narrowly on the most critical technical uncertainties and [rapidly experiment for] faster feedback. The philosophy is to learn what you have gotten wrong early and then move quickly in more-promising directions.

From Gary P. Pisano writing on organizational culture for HBR. Paul E. McKenney emphasizes:

[S]tress-testing ideas early on avoids over-investing in the inevitable blind alleys.

But what kind of tests does Pisano suggest?

[do] not run experiments to validate initial ideas. Instead, […] design “killer experiments” that maximize the probability of exposing an idea’s flaws.

Government drinking game

The department of agriculture [had] an annual budget of $164bn and was charged with so many missions critical to the society that the people who worked there played a drinking game called Does the Department of Agriculture Do It? Someone would name a function of government, say, making sure that geese don’t gather at US airports, and fly into jet engines. Someone else would have to guess whether the agriculture department did it. (In this case, it did.) Guess wrong and you had to drink.

From Michael Lewis, colorfully illustrating the incredible work of presidential transition.

It just looks better that way

In Old English the past tense of “can” did not have an “l” in it, but “should” and “would” (as past tenses of “shall” and “will”) did. The “l” was stuck into “could” in the 15th century on analogy with the other two.

From Arika Okrent, in a MentalFloss piece about the weird history of some spellings. The piece has other examples of spelling changes to conform words to some aesthetic or another, even when those changes were inconsistent with the history and etymology of the word. And here’s a reminder to myself about the author’s book on invented languages.

On building a culture of candid debate

A good blueprint for [building a culture of candid debate] can be found in General Dwight D. Eisenhower’s battle-plan briefing to top officers of the Allied forces three weeks before the invasion of Normandy. As recounted in Eisenhower, a biography by Geoffrey Perret, the general started the meeting by saying, “I consider it the duty of anyone who sees a flaw in this plan not to hesitate to say so. I have no sympathy with anyone, whatever his station, who will not brook criticism. We are here to get the best possible results.”

Eisenhower was not just inviting criticism or asking for input. He was literally demanding it and invoking another sacred aspect of military culture: duty. How often do you demand criticism of your ideas from your direct reports?

From Gary P. Pisano in The Hard Truth About Innovative Cultures

Shuffle sharding in Dropbox's storage infrastructure

Volumes are spread somewhat-randomly throughout a cell, and each OSD holds several thousand volumes. This means that if we lose a single OSD we can reconstruct the full set of volumes from hundreds of other OSDs simultaneously. This allows us to amortize the reconstruction traffic across hundreds of network cards and thousands of disk spindles to minimize recovery time. » about 300 words

Parts of a network you should know about

If you’re running infrastructure and applications on AWS then you will encounter all of these things. They’re not the only parts of a network setup but they are, in my experience, the most important ones.

The start of Graham Lyons’ introduction to networking on AWS, which (though the terms may change) is a pretty good primer for networking in any cloud environment. Though cloud infrastructure providers have to deal with things at a different later, Graham’s post covers the basics—VPCs, subnets, availability zones, routing tables, gateways, and security groups—that customers need to manage when assembling their applications.