The Daily Parker

It takes a while to transfer 162.4 million rows of data from a local SQL database to a remote Azure Tables collection. So far, after 4 hours and 20 minutes, I've transferred just over 4 million rows. That works out to about 260 rows per second, or 932,000 per hour. So, yes, the entire transfer will take 174 hours.

Good thing it can run in the background. Also, because it cycles through three distinct phases (disk-intensive data read, processor-intensive data transformation, and network-intensive data write), it doesn't really take a lot of effort for my computer to handle it. In fact, network writes take 75% of the cycle time for each batch of reports, because the Azure Tables API takes batches of 100 rows at a time.

Now, you might wonder why I don't just push the import code up to Azure and speed up the network writes a bit. Well, yes, I thought of that, but decided against the effort and cost. To do that, I would have to upload a SQL backup file to a SQL VM big enough to take a SQL Server instance. Any VM big enough to do that would cost about 67¢ per hour. So even if I cut the total time in half, it would still cost me $60 or so to do the transfer. That's an entire bottle of Bourbon's worth just to speed up something for a hobby project by a couple of days.

Speaking of cost, how much will all this data add to my Azure bill? Well, I estimate the entire archive of 2009-2022 data will come to about 50 gigabytes. The 2003-2009 data will probably add another 30. Azure Tables cost 6¢ per gigabyte per month for the geographically-redundant storage I use. I will leave the rest of the calculation as an exercise for the reader.

Update: I just made a minor change to the import code, and got a bit of a performance bump. We're now up to 381 rows per second, 46% faster than before, which means the upload should finish in only 114 hours or 4.7 days. All right, let's see if we're done early Monday morning after all! (It's already almost done with Canada, too!)

I mentioned yesterday that I purged a lot of utterly useless archival data from Weather Now. It sometimes takes a while for the Azure Portal to update its statistics, so I didn't know until this afternoon exactly how much I dumped: 325 GB.

Version 5 already has a bi-monthly function to move archival data to cool storage after 14 days and purge it after 3 months. I'm going to extend that to purging year-old logs as well. It may only reduce my Azure bill by $20 a month, but hey, that's a couple of beers.

As for Weather Now 5, if all goes well, I'll start downloading weather in the Production environment today, and possibly deploy the app over the weekend. Stay tuned...

I've spent today alternately upgrading my code base for my real job to .NET 6.0, and preparing for the Apollo Chorus performances of Händel's Messiah on December 11th and 12th.

Cassie, for her part, enjoys when I work from home, even if we haven't spent a lot of time outside today because (a) I've had a lot to do and (b) it rained from 11am to just about now.

So, as I wait for the .NET 6 update to build and deploy on our dev/test CI/CD instance (I think I set the new environments on our app services correctly), I have a few things to read:

• David Corn reports on how the previous administration's incompetence, if not outright malice, is responsible for 400,000 preventable Covid-19 deaths.
• James Fallows digs into the questions we should ask about China.
• Chicago's City Council looks likely to approve a narrower definition of "restaurant" that would let more bars allow dogs.
• Russian artist Lena Ruseva will make you some MAGA art for just a few thousand bucks. No scam there.

OK, the build has...well, crap. I didn't set the environment correctly after all.

Update: Fixed the build bit. And the rain stopped. But the test platform is the wrong version. FFS.

Update: Well, I have to pick something up from a store before 6, so I'll come back to this task later.

Update: Even though I've had 4 tiny commits of minor things that broke with the .NET 6 upgrade, this hasn't gone poorly. Kudos to Microsoft for providing a straightforward upgrade path.

I spent the morning unsuccessfully trying to get a .NET 5 Blazor WebAssembly app to behave with an Azure App Registration, and part of the afternoon doing a friend's taxes. Yes, I preferred doing the taxes, because I got my friend a pile of good news without having to read sixty contradictory pages of documentation.

I also became aware of the following:

• The FBI and Australian police completely pwned hundreds of criminals through a black-market "encrypted" cell phone app that they wrote and monitored, in one of the biggest sting operations in world history.
• Sue Halpern raises the alarm about Texas' and Arizona's unprecedented legislative attacks on election systems.
• El Salvador now accepts Bitcoin as legal tender, taking the question "what could possibly go wrong?" into new and frightening territory.
• "Ask the Pilot" Patrick Smith cannot figure out how United Airlines will make money with the Boom SSTs it just announced.
• As part of its $2.7 billion Red Line improvement project, the Chicago Transit Authority has agreed not to demolish a 127-year-old apartment building in Chicago's Lakeview neighborhood, and instead will move it about 10 meters to the southwest to make room for track improvements.
• Illinois has created a high-speed rail commission to study, among other things, a $20-billion, 350 km/h high speed line between Chicago and St Louis.
• As it legally must, the Treasury Dept. will investigate how ProPublica got its hands on the private tax information it reported on last week.

Tomorrow morning, I promise to make my WebAssembly app talk to our Azure Active Directory. Right now, I think someone needs a walk.

The deployment I concluded yesterday that involved recreating production assets in an entirely new Azure subscription turned out much more boring (read: successful) than anticipated. That still didn't stop me from working until 6pm, but by that point everything except some older demo data worked just fine.

That left a bit of a backup of stuff to read, which I may try to get through at lunch today:

• Duke University basketball coach Mike Krzyzewski (aka "Coach K"), the winningest basketball coach in NCAA history, plans to retire at the end of next season.
• United Airlines has made a deal with Boom Holdings to purchase 15 of the company's 65-to-88 seat supersonic airplanes for travel starting in 2029.
• Israeli opposition leader Yair Lapid formed a government last night, but outgoing prime minister Benjamin Netanyahu may try to forestall his ouster with parliamentary shenanigans Josh Marshall calls "Israel's January 6th."
• The Atlantic's Kate Julian worries that the pandemic has shifted Americans' drinking from social (good) to alone (bad).
• A skunk ate the three eggs endangered piping plovers Monty and Rose laid a few weeks ago, but the couple have reportedly started trying again.
• Geologists have determined that the 1861's 8.5-magnitude earthquake off Sumatra was actually just the last rumble of a continuous 32-year-long earthquake.
• On this day in 1800, President John Adams moved into the Union Tavern in Georgetown, D.C., to await the opening of the Executive Mansion on November 1st.

Finally, summer apparently arrives in full force tomorrow. We're looking forward to temperatures 5-10°C above normal through mid-June, which will continue northern Illinois' drought for at least a few more weeks.

I hope this helps someone else having this problem deploying a .NET function to Azure App Services.

At my day job, we created a new Azure directory and subscription for my group's product. As the product has gotten closer to release, we realized we needed a more complete separation from the company's Azure assets and our group's. So far, so good. I had some annoyances updating our deployment pipelines, but nothing I hadn't expected.

Then I tried to deploy our one function app. I followed the basic script in PowerShell:

Import-Module Az
Connect-AzAccount
CD c:\source\solution\project\bin\Debug\net5.0\
func azure functionapp publish function-name-dev

The script failed with: Can't find app with name "function-name-dev"

Undeterred, I modified the script:

Import-Module Az
Connect-AzAccount -Tenant 'guid' -SubscriptionId 'guid'
az account set --subscription 'guid'
CD c:\source\solution\project\bin\Debug\net5.0\
func azure functionapp publish function-name-dev

Same result. Googling and searching through Stack Overflow didn't help either. After a lot of experimentation, I finally got an error message that pointed me down the correct path, but only when I tried to create a new function app in the same subscription:

The following tenants require Multi-Factor Authentication (MFA). Use 'az login --tenant TENANT_ID' to explicitly login to a tenant.

And that was the solution. My new script, which worked fine, now looks like this:

Import-Module Az
Connect-AzAccount -Tenant 'guid' -SubscriptionId 'guid'
az login --tenant 'guid'
az account set --subscription 'guid'
CD c:\source\solution\project\bin\Debug\net5.0\
func azure functionapp publish function-name-dev

I may refine it further as I may have some redundancies in there. But I have now deployed the function app and tested it, much to my satisfaction.

I've just made a change to the side project I'm working on that will reduce my database costs about 94%. Maybe 96%. This is only in the dev/test environment, so it may make less of a difference in production, but still... Sometimes taking something out of your code can make an enormous difference.

I promise I'll write a lot about what I've been working on once it launches.

Bit of a frustrating day, today. I spent 2½ hours trying to deploy an Azure function using the Az package in PowerShell, before giving up and going back to the AzureCLI. All of this to confirm a massive performance issue that I suspected but needed to see in a setting that eliminated network throughput as a possible factor. Yep: running everything completely within Azure sped it up by 11%, meaning an architecture choice I made a long time ago is definitely the problem. I factored the code well enough that I can replace the offending structure with a faster one in a couple of hours, but it's a springtime Sunday, so I don't really feel totally motivated right now to do so.

Lest you worry I have neglected other responsibilities, Cassie already got over an hour of walks and dog park time today, bringing her up to 10½ hours for the week. I plan to take her on another 45-minute walk in an hour or so. Last week she got almost 14 hours of walks, however. I blame the mid-week rain we got.

I also have a 30-minute task that will involve 15 minutes of setup, 10 minutes of tear-down, and 5 minutes of video recording. I will be so relieved next fall when all of our chorus work happens in person again.

Before I do that, however, I'm going to go hug my dog.

A few articles caught my attention this week:

• Jennifer Rubin says the GOP's opposition to literally everything President Biden has proposed is killing their popularity.
• The New Republic, in collaboration with the Chicago Reader, tells the story of the last remaining men's hotel in Chicago.
• NPR host Steve Inskeep describes his difficulties getting his adoption records from the State of Indiana.
• Writing in The New Yorker, Daniel Alcarón mourns the loss of Puerto Rico's Arecibo Observatory last December.
• In Illinois, March had above-normal temperatures 23 of 31 days, making the month the 13th warmest March on record.

Also, I'm just making a note to myself of Yuriy Ivon's rundown on Microsoft Azure Cosmos DB, because I'm using it a lot more than I have in the past.

Microsoft Azure and Office 365 suffered an outage yesterday that affected just about everything in their cloud:

Microsoft Corp. was hit by a massive cloud outage today that took most of its internet services offline.

Microsoft’s Azure cloud services, as well as Teams, Office 365, OneDrive, Skype, Xbox Live and Bing were all inaccessible due to the outage. Even the Azure Status page was reportedly taken offline.

The first reports of the outage emerged from users on Twitter, and were confirmed by the website DownDetector which showed that reports began flooding in at around 5 p.m. ET. It says it received thousands of notices from Xbox Live, Teams and Office users.

Microsoft 365’s Twitter status account posted another update at 6.35 p.m. ET saying that traffic was being rerouted to resilient DNS capabilities and that it was already “seeing an improvement in service availability.”

Today, Microsoft reported as a preliminary root cause "We are continuing to investigate the underlying cause for the DNS outage but we have observed that Microsoft DNS servers saw a spike in DNS traffic." In other words, it looks like they suffered a distributed denial-of-service (DDOS) attack on their internal name servers. The final analysis will come out next Thursday.

This outage was like the familiar "collective amnesia" trope in sci-fi where suddenly none of the characters recognizes any of the others, though they retain their normal personalities and abilities. (See, e.g., Dollhouse and Buffy. Joss Whedon lurves this trope.) For example, The Daily Parker was still running, but no one could get to it because the mapping from www.thedailparker.com to the Microsoft App Service hosting it has to go through Microsoft's internal name servers.

I wonder if this was a DDOS attack from inside the house?