So I have queued up stuff to read later:
About the Rogers outage: the CBC published a chart showing that network usage hit 100% of its capacity immediately before it started to fall steadily before collapsing entirely around 4am ET. I wonder if the sequence will turn out to resemble the 1965 northeast blackout?
Chicago's official temperature at O'Hare hit 35°C about two hours ago, tying the record high temperature set in 1994. Currently it's pushing 36°C with another hour of warming likely before it finally cools down overnight. After another 32°C day tomorrow, the forecast Friday looks perfect.
While we bake by the lake today, a lot has gone down elsewhere:
Finally, apparently John Scalzi and I have the same appreciation for Aimee Mann.
The security guru just posted a video he presented in November 2020:
I had to put out a new version of the Inner Drive Azure tools for my day job today, and I had more meetings than I wanted (i.e., a non-zero number), so these kind of piled up:
There were other things I'll read later, but it's past 6pm and someone is staring at me because she needs a walk.
Nathan Evans recorded his original 59-second TikTok on 27 December 2020. By January 18th...this had happened:
As I understand it, Evans has launched a recording career now. I hope a couple of other contributors to this mash-up get some recognition as well.
I will now take a break from my ongoing struggles to make Blazorise play nicely with Open ID authentication so I can read these:
And finally, WGN confirms we hit back-to-back record temperatures Wednesday and Thursday, both tied for 11th warmest December day in Chicago history.
I've had intermittent Internet all morning for no reason that Comcast/Xfinity can tell me. Everything I do professionally requires an Internet connection.
I am displeased.
My 8am meeting with colleagues in London had to wait until 9:30 because Comcast screwed the pooch this morning:
Reports indicate the system was down, or at least unsteady, in areas stretching from Chicago to Philadelphia, New Jersey, and South Carolina. Looking at DownDetector, issues had been reported earlier in the Bay Area, but it’s unclear if those are connected to the problems people saw this morning.
Comcast has released a statement regarding the outage. According to a spokesperson, “Earlier, some customers experienced intermittent service disruptions as a result of a network issue. We have addressed the issue and service is now restoring for impacted customers, as we continue to investigate the root cause. We apologize to those who were affected.” It appears that most of the people who reported problems have confirmed they’re back online. There’s still no word on exactly what caused the problem or how many people were impacted at its peak.
In Chicago, the outage affected thousands of people from about 7:30 to 9, by which time I'd already relocated to my company's Loop office.
Oh, and on the day before a trip, my bank called to let me know their fraud department killed my primary credit card. They hope the new one arrives before I leave for the airport.
BGP stands for Border Gateway Protocol. It's a mechanism to exchange routing information between autonomous systems (AS) on the Internet. The big routers that make the Internet work have huge, constantly updated lists of the possible routes that can be used to deliver every network packet to their final destinations. Without BGP, the Internet routers wouldn't know what to do, and the Internet wouldn't work.
The Internet is literally a network of networks, and it’s bound together by BGP. BGP allows one network (say Facebook) to advertise its presence to other networks that form the Internet. As we write Facebook is not advertising its presence, ISPs and other networks can’t find Facebook’s network and so it is unavailable.
The individual networks each have an ASN: an Autonomous System Number. An Autonomous System (AS) is an individual network with a unified internal routing policy. An AS can originate prefixes (say that they control a group of IP addresses), as well as transit prefixes (say they know how to reach specific groups of IP addresses).
At 1658 UTC we noticed that Facebook had stopped announcing the routes to their DNS prefixes.
We keep track of all the BGP updates and announcements we see in our global network. At our scale, the data we collect gives us a view of how the Internet is connected and where the traffic is meant to flow from and to everywhere on the planet.
A BGP UPDATE message informs a router of any changes you’ve made to a prefix advertisement or entirely withdraws the prefix. We can clearly see this in the number of updates we received from Facebook when checking our time-series BGP database. Normally this chart is fairly quiet: Facebook doesn’t make a lot of changes to its network minute to minute.
But at around 15:40 UTC we saw a peak of routing changes from Facebook. That’s when the trouble began.
So, someone at Facebook may have applied a router update incorrectly. And as of now, they've corrected the problem.
The United States Supreme Court began their term earlier today, in person for the first time since March 2020. Justice Brett Kavanagh (R) did not attend owing to his positive Covid-19 test last week.
In other news:
So how did facebook.com disappear from root DNS, the day after 60 Minutes aired a segment on Haugen?