I was pretty busy today, with most of my brain trying to figure out how to re-architect something that I didn't realize needed it until recently. So a few things piled up in my inbox:
And finally, Whisky Advocate has four recipes that balance whisky and Luxardo Maraschino cherries. I plan to try them all, but not in one sitting.
On this day in 1767, Charles Mason and Jeremiah Dixon completed their survey of the disputed Maryland-Pennsylvania border, which became even more contentious in 1780 when Pennsylvania aboolished slavery. A group of surveyors started re-surveying the border in 2019; I can't find out whether they finished.
Meanwhile, 255 years later, politics is still mostly local:
Finally, Chicago has perfectly clear skies for only the third time this month after yesterday and the 4th, getting only 39% of possible sunshine for almost the past three weeks.
Chicago Loop, Monday morning:
Apparently the rockets aren't the only colossal dicks at Blue Origin:
The company’s cultural issues came to light last month when Alexandra Abrams, the former head of Blue Origin’s employee communications, released an essay she said was written in conjunction with 20 other current and former Blue Origin employees. It said the company “turns a blind eye to sexism, is not sufficiently attuned to safety concerns and silences those who seek to correct wrongs.” The staffers were not identified in the essay, but three of them confirmed the allegations to The Post on the condition of anonymity for fear of reprisal.
[I]n 2017, [Amazon CEO Jeff] Bezos brought in [Bob] Smith to be the company’s first CEO, taking over from Rob Meyerson, the company’s president, who had been running its day-to-day operations.
Smith and the executives he brought in, many from legacy aerospace companies, sat in an executive suite in a new office building, isolated from the rest of the staff. While that is not unusual for many large corporations, it was off-putting for many employees at Blue Origin who had been used to their leaders sitting and mingling among them.
“That wasn’t appreciated,” one former executive said. “It was an I’m-above-you message.”
Concerned about the company’s leadership, the head of human resources brought in an outside management consultant, who interviewed Smith and the members of his team in 2019 and concluded that Smith’s micromanaging style was often ineffective, according to a former senior executive and confirmed by another person familiar with the matter.
Smith bristled at the report, which was first reported by CNBC, and refused to meet on the subject again.
Culture comes from many sources, but dickish, authoritarian behavior at the top gives people to act like authoritarian dicks below. That insight isn't exactly rocket science. But it does help explain how SpaceX keeps beating Blue Origin in every way that matters.
Remy Porter, owner of the hilarious blog The Daily WTF, responded to Facebook's catastrophic BGP update by pointing out how software actually gets made:
IT in general, and software in specific, is a rather bizarre field in terms of how skills work. If, for example, you wanted to get good at basketball, you might practice free-throws. As you practice, you'd expect the number of free-throws you make to gradually increase. It'll never be 100%, but the error rate will decline, the success rate will increase. Big-name players can expect a 90% success rate, and on average a professional player can expect about an 80% success rate, at least according to this article. I don't actually know anything about basketball.
But my ignorance aside, I want you to imagine writing a non-trivial block of code and having it compile, run, and pass its tests on the first try. Now, imagine doing that 80% of the time.
It's a joke in our industry, right? It's a joke that's so overplayed that perhaps it should join "It's hard to exit VIM" in the bin of jokes that needs a break. But why is this experience so universal? Why do we have a moment of panic when our code just works the first time, and we wonder what we screwed up?
It's because we already know the truth of software development: effing up is actually your job.
You absolutely don't get a choice. Effing up is your job. You're going to watch your program crash. You're going to make a simple change and watch all the tests go from green to red. That semicolon you forgot is going to break the build. And you will stare at one line of code for six hours, silently screaming, WHY DON'T YOU WORK?
Yep. And still, we do it every day.
Josh Marshall points out that the harm Facebook causes comes from its basic design, making a quick fix impossible:
First, set aside all morality. Let’s say we have a 16 year old girl who’s been doing searches about average weights, whether boys care if a girl is overweight and maybe some diets. She’s also spent some time on a site called AmIFat.com. Now I set you this task. You’re on the other side of the Facebook screen and I want you to get her to click on as many things as possible and spend as much time clicking or reading as possible. Are you going to show her movie reviews? Funny cat videos? Homework tips? Of course, not. If you’re really trying to grab her attention you’re going to show her content about really thin girls, how their thinness has gotten them the attention of boys who turn out to really love them, and more diets. If you’re clever you probably wouldn’t start with content that’s going to make this 16 year old feel super bad about herself because that might just get her to log off. You’ll inspire or provoke enough negative feelings to get clicks and engagement without going too far.
This is what artificial intelligence and machine learning are. Facebook is a series of algorithms and goals aimed at maximizing engagement with Facebook. That’s why it’s worth hundreds of billions of dollars. It has a vast army of computer scientists and programmers whose job it is to make that machine more efficient. The truth is we’re all teen girls and boys about some topic. Maybe the subject isn’t tied as much to depression or self-destructive behavior. Maybe you don’t have the same amount of social anxiety or depressive thoughts in the mix. But the Facebook engine is designed to scope you out, take a psychographic profile of who you are and then use its data compiled from literally billions of humans to serve you content designed to maximize your engagement with Facebook.
Put in those terms, you barely have a chance.
He goes on to draw a comparison between Facebook's executives and Big Tobacco's, circa 1975:
At a certain point you realize: our product is bad. If used as intended it causes lung cancer, heart disease and various other ailments in a high proportion of the people who use the product. And our business model is based on the fact that the product is chemically addictive. Our product is getting people addicted to tobacco so that they no longer really have a choice over whether to buy it. And then a high proportion of them will die because we’ve succeeded.
So what to do? The decision of all the companies, if not all individuals, was just to lie. What else are you going to do? Say we’re closing down our multi-billion dollar company because our product shouldn’t exist?
You can add filters and claim you’re not marketing to kids. But really you’re only ramping back the vast social harm marginally at best. That’s the product. It is what it is.
Yesterday's 6-hour reprieve from Facebook seems to have hurt almost no one. The jokes started right away, about how anti-vaxxers could no longer "do research" and how people have started reading again. I didn't even notice until I read that it had gone offline, because I had too much work to do. So maybe that's what regulators should do: limit the company to 8 hours a day or something. What a thought...
BGP stands for Border Gateway Protocol. It's a mechanism to exchange routing information between autonomous systems (AS) on the Internet. The big routers that make the Internet work have huge, constantly updated lists of the possible routes that can be used to deliver every network packet to their final destinations. Without BGP, the Internet routers wouldn't know what to do, and the Internet wouldn't work.
The Internet is literally a network of networks, and it’s bound together by BGP. BGP allows one network (say Facebook) to advertise its presence to other networks that form the Internet. As we write Facebook is not advertising its presence, ISPs and other networks can’t find Facebook’s network and so it is unavailable.
The individual networks each have an ASN: an Autonomous System Number. An Autonomous System (AS) is an individual network with a unified internal routing policy. An AS can originate prefixes (say that they control a group of IP addresses), as well as transit prefixes (say they know how to reach specific groups of IP addresses).
At 1658 UTC we noticed that Facebook had stopped announcing the routes to their DNS prefixes.
We keep track of all the BGP updates and announcements we see in our global network. At our scale, the data we collect gives us a view of how the Internet is connected and where the traffic is meant to flow from and to everywhere on the planet.
A BGP UPDATE message informs a router of any changes you’ve made to a prefix advertisement or entirely withdraws the prefix. We can clearly see this in the number of updates we received from Facebook when checking our time-series BGP database. Normally this chart is fairly quiet: Facebook doesn’t make a lot of changes to its network minute to minute.
But at around 15:40 UTC we saw a peak of routing changes from Facebook. That’s when the trouble began.
So, someone at Facebook may have applied a router update incorrectly. And as of now, they've corrected the problem.
The United States Supreme Court began their term earlier today, in person for the first time since March 2020. Justice Brett Kavanagh (R) did not attend owing to his positive Covid-19 test last week.
In other news:
So how did facebook.com disappear from root DNS, the day after 60 Minutes aired a segment on Haugen?
I've opened nearly every window in my house to let in the 15°C breeze and really experience the first real fall morning in a while. Chicago will get above-normal temperatures for the next 10 days or so, but in the beginning of October that means highs in the mid-20s and lows in the mid-teens. Even Cassie likes the change.
Since I plan to spend nearly every moment of daylight outside for the rest of this weekend, I want to note a few things to read this evening when I come back inside:
Finally, if you really want to dig into some cool stuff in C# 10, Scott Hanselman explains implicit namespace support.
So these things happened:
And finally, break out the Glühwein: Chicago's Christkindlmarket will return to Daley Plaza and Wrigleyville this winter.