With 58 days until the election, the noise keeps increasing. Here's some of it:
Finally, The Smithsonian describes how Greg Priore managed to steal priceless documents from the Carnegie Library of Pittsburgh, because he was in charge of security for those items.
This is my 55th post this month, and the fifth month in a row in which I've posted over 50 times. That brings my 12-month total to 581, the third record in a row and the fifth record this year. I guess Covid-19 has been good for something.
Here's what I'm reading today:
I'm excited to add a notch on the Brews and Choos project in a few hours. Check back tomorrow.
First: the difference between how Garmin handled a global outage that lasted 5 days, and how SendGrid managed one that lasted 5 hours. SendGrid handles billions of emails per day, including for Microsoft and other massive companies. So SendGrid going down didn't inconvenience a few athletes and pilots; it crippled Fortune-500 companies' marketing departments. (And it delayed a scheduled release on my own team.)
Within about an hour of their outage, SendGrid created an incident response page to which they posted updates every half-hour. They clearly stated what was going on and how they were trying to fix it, even as they were discovering for themselves what had happened:
Contrast that with Garmin, who still haven't really explained what happened or why it took so long to resolve the outage. (They finally declared their remediation "complete" yesterday, almost 6 days after the outage started.)
Second: the difference between how Germany (and other rich countries) have handled Covid-19 and how we have. Josh Marshall looked into the numbers:
The head of Germany’s equivalent of the CDC told reporters today that he’s “very concerned” about the rising case numbers in the country and accuses Germans of becoming “negligent” in their adherence to mitigation measures. He has good reason to be concerned.
Today Germany reported 638 new cases of COVID. That comes out to .76 cases per 100,000 residents. Let’s round that up to 1 new case per 100,000 for good measure. (The need for round numbers will become clear.)
Today New York State had 3 cases per 100,000. So Germany is concerned by 1/3 the number of cases as we have in New York state, a state which is probably controlling the disease as well as any other state in the country.
Florida today had 43 cases per 100,000. Florida’s outbreak is more than forty-three times the size relative to population.
To state the point baldly, Germany is very concerned about a rise in cases that would still be dramatically better than any other part of the United States. They’re ramping up border restrictions to get things back under control and chiding the population to redouble its collective efforts.
[S]eeing this as, "well, look, everyone’s having problems." Or "we’re not the only ones having new outbreaks" or "we can’t go back to shutdowns…" ... only captures how Americans are having a hard time grappling with just how many universes away we are from what is happening in other countries which are comparably affluent, industrialized and able to mount an effective response.
We’re failing that badly.
Anthony Fauci was on BBC just now, struggling not to call the president out on his criminal negligence. Only 98 days until we can vote the bastard out.
Even as Garmin picks up the pieces from what they now admit was a massive ransomware attack, bulk email provider SendGrid has gone down spectacularly.
I use SendGrid, as does my company, for status emails and such.
Here's my problem, though: I have a code update to put out that specifically targets a bug in SendGrid's .NET library that they claim to have fixed. My automated build pipelines won't release new code unless all the unit tests pass. Right now, the SendGrid tests fail sporadically, and at least one fails every time the tests run.
Ah, technology.
Today I finally solved a problem that has nagged me for months: Given a .NET Core 3.1 Web API, NLog, and a custom log target for NLog, how could I pass secrets into the log target using Azure Key Vault?
The last bit required me to add Azure Key Vault support to the InnerDrive.Logging NuGet package, which I did back in February. The KeyVaultSecretProvider class allows you to retrieve secrets from a specified Azure Key Vault. (I'm in the process of updating it to handle keys as well.) Before that, you'd have to put your secrets in configuration files, which anyone on your development team or who has access to your Git repository can see. For example, to use our SendGridTarget class, you would have to put this in your nLog.config file:
<extensions>
<add assembly="NLog.Web.AspNetCore"/>
<add assembly="InnerDrive.Logging"/>
</extensions>
<targets async="false">
<target xsi:type="SendGridTarget"
name="sendgrid"
apiKey="{supposedly secret key}"
applicationName="My Cool App"
from="service@contoso.org"
to="admin@contoso.org"
/>
</targets>
That is...suboptimal. Instead, you want to use a class that implements ISecretProvider and inject it into the SendGridTarget class through this constructor:
/// <summary>
/// Creates a new instance of <see cref="SendGridTarget"/> with
/// a specified <see cref="ISendGridSender"/> implementation.
/// </summary>
/// <param name="sender">The <see cref="ISendGridSender"/> to use</param>
/// <param name="secretProvider">The <see cref="IConfiguration"/> to use</param>
/// <param name="apiKey">The API key to use</param>
public SendGridTarget(
ISendGridSender sender,
ISecretProvider secretProvider = null,
string apiKey = null)
{
IncludeEventProperties = true;
if (null != secretProvider) Configuration = secretProvider;
if (!string.IsNullOrWhiteSpace(apiKey)) ApiKey = apiKey;
_sender = sender;
}
And now I'm going to save you about 30 hours of research and frustration. To get NLog to inject the secret provider into the target, I added this method to Startup.cs:
public void Configure(IApplicationBuilder app, IWebHostEnvironment env, IComponentContext container)
{
// Setup code elided
ConfigureLogging(container);
}
public static void ConfigureLogging(IComponentContext container)
{
var defaultConstructor = ConfigurationItemFactory.Default.CreateInstance;
var secretProvider = container.Resolve<ISecretProvider>();
ConfigurationItemFactory.Default.CreateInstance = type =>
{
if (type == typeof(SendGridTarget))
{
var sendGridSender = container.Resolve<ISendGridSender>();
return new SendGridTarget(sendGridSender, secretProvider);
}
return defaultConstructor(type);
};
LogManager.Configuration = LogManager.Configuration.Reload();
NLogBuilder.ConfigureNLog(LogManager.Configuration);
}
(Note that the ISecretProvider and ISendGridSender classes need to be registered with your dependency-injection framework.)
Today was a good day.
Four days after I switched from Fitbit to Garmin, all of Garmin's online services have gone offline:
The problems with those services also mean that a range of features can't be used on Garmin's own devices: it is not possible to create new routes to go running or cycling, for instance, or to share those activities on services like Strava once they are completed.
The devices themselves continue to work as normal with the data they do have, however, meaning that any data collected during the outage will be safe.
Garmin wrote on its official Twitter pages that the problems were also affecting its call centres, leaving users unable to get in touch through calls or online messages.
"We are currently experiencing an outage that affects Garmin Connect, and as a result, the Garmin Connect website and mobile app are down at this time," it wrote.
"This outage also affects our call centres, and we are currently unable to receive any calls, emails or online chats. We are working to resolve this issue as quickly as possible and apologise for this inconvenience."
This is not what Garmin customers want to see:
The error message on the Garmin Connect website suggests the problem is with their Cloudflare equipment:
Update: Based on Garmin employee's social-media posts, ZDNet now reports that the company experienced a catastrophic ransomware attack, most likely a new strain of WastedLocker. Fortunately, my Venu can hold 200 hours of data. So as long as they get it back up within a week or so, I shouldn't lose anything—unless the ransomware attack already destroyed my data from this past week.
I think today is Tuesday, the first day of my 10th week working from home. That would make today...March 80th? April 49th? Who knows.
It is, however, just past lunchtime, and today I had shawarma and mixed news:
Earlier, I mentioned that the state's unemployment office accidentally revealed thousands of records in an own goal. Turns out, Deloitte Consulting did the work, so I am no longer surprised. Note to anyone who needs software written: don't hire a big consulting firm. They don't attract the best developers because they use manager-driven development patterns that irritate the hell out of anyone with talent.
Charlie Pierce, noting that "[p]eople with firearms forced the civil government of the state of Michigan to shut itself down," wants to know in what sense this isn't terrorism. In other fun weekend stories:
And it's pouring, and will continue to do so for several more hours.
So believes NYU media professor Jay Rosen about how President Trump will try to win this fall:
The plan is to have no plan, to let daily deaths between one and three thousand become a normal thing, and then to create massive confusion about who is responsible— by telling the governors they’re in charge without doing what only the federal government can do, by fighting with the press when it shows up to be briefed, by fixing blame for the virus on China or some other foreign element, and by “flooding the zone with shit,” Steve Bannon’s phrase for overwhelming the system with disinformation, distraction, and denial, which boosts what economists call “search costs” for reliable intelligence.
Stated another way, the plan is to default on public problem solving, and then prevent the public from understanding the consequences of that default. ... The manufacture of confusion is just the ruins of Trump’s personality meeting the powers of the presidency. There is no genius there, only a damaged human being playing havoc with our lives.
In other fun stories:
Oh, and 151 years ago today, the Union Pacific and Central Pacific railroads completed the Transcontinental Railroad.
Remember slow news days? Me neither.
And finally, a cute diner in Toronto where I had breakfast last June has moved to delivery service during the lockdown. Too bad they can't deliver to Chicago.