a bunch of stuff I would have emailed you about

Common root causes of intra data center network incidents at Facebook from 2011 to 2018

From A Large Scale Study of Data Center Network Reliability by Justin Meza, Tianyin Xu, Kaushik Veeraraghavan, and Onur Mutlu, the categorized root causes of intra data center incidents at Fabook from 2011 to 2018:

Maintenance17%Routine maintenance (for example, upgrading the software and firmware of network devices).
Hardware13%Failing devices (for example, faulty memory modules, processors, and ports).
Misconfiguration13%Incorrect or unintended configurations (for example, routing rules blocking production traffic).
Bug12%Logical errors in network device software or firmware.
Accidents11%Unintended actions (for example, disconnecting or power cycling the wrong network device).
Capacity planning5%High load due to insufficient capacity planning.
Undetermined29%Inconclusive root cause.

Two notes worth considering:

We use “failures” to refer to any network device misbehavior. The root cause of a failure includes not only hardware faults, but also misconfigurations, maintenance mistakes, firmware bugs, and other issues.


We use Govindan et al.’s definition of root cause: “A failure event’s root-cause is one that, if it had not occurred, the failure event would not have manifested.”

Pour one out for the Sears Catalog, the original market disrupter

Whet Moser pointed out this enlightening Twitter thread that explains an aspect of Sears I hadn’t considered before: by disrupting retail stores with mail-order, it was empowering a demographic that was often underserved in their communities:

The Sears catalog succeeded because it got the goods to people who couldn’t get to stores. One of those demographics? African-Americans. In a lengthy Twitter thread, Cornell historian Louis Hyman writes that it freed up black Southerners from going to general stores, which was often (at best) a humiliating experience. Store owners were so incensed that they “organized catalog bonfires in the street.”

It served as a similar resource for Appalachian coal miners, providing huge discounts over rip-off company stores. Sears also had “a policy that his company would fill any order it received, no matter what the medium or format”—a boon to customers who struggled with literacy.

On The Media’s Brook Gladstone spoke with Louis Hymen, the author of that twitter thread and professor at Cornell University’s School of Industrial and Labor Relations in a podcast extra last week. Listen or download.

How to date your foodstuffs

Whet Moser, suddenly making sell-by dates on food products relevant to me:

About a quarter of US methane emissions comes from food rotting in landfills.

The dates on our packaged food products look so authoritative, but the way Moser tells it, they were invented by marketing folks to increase sales at the cost of disposing of otherwise good products that have an expired sell-by date.

Fuji Instax back for Hasselblad

Isaac Blankensmith writing in PetaPixel about building an Instax instant film back for a Hasselblad 500:

Instant photos are magical. They develop before your eyes. You can share them, gift them, spill water on them, draw on them. The only problem is that most instant cameras are pretty cheap — that’s why I’ve always wanted to hack my medium format camera to take instant photos with shallow depth of field and sharpness.

Since Fuji ceased production of their Polaroid-compatible peel-apart films a few years ago, there has been significant interest in a Fuji Instax back. At least two Kickstarters have been announced, Hasselblad Square Instant Film Back and Rezivot Instant Film Processor, but neither of those was successful.

@instantmediumformat on Instagram converted an old Mamiya Press Camera, but has offered few details for those who wish to follow. Blankensmith’s post is a pretty good starting point for people who’ve been considering building one for themselves–possibly people like me.