So, I finally figured out why Weather Now kept struggling: I deployed it to a too-small App Service Plan for the load it was getting. And since most of the load is coming from bots, I hope that the robots.txt file I finally deployed properly will get them to stop.
I fixed the mysterious exceptions being thrown, too. But it turned out, the problem was simply load. A bigger App Service Plan brought the effective CPU from 100% to 28% immediately. And it'll only cost another $66 per month...
I'll experiment with other ASP sizes later this week. At least the app works again!
I spent hours this weekend and a couple of nights this past week fixing Weather Now. The app has gotten a lot more traffic than usual lately, mostly because I didn't put the robots.txt file in the right location. Unfortunately, all the extra traffic made it really obvious that the app had some serious performance issues, which I traced to some bad asynchronous code design.
The miracle cure for these issues came from Microsoft, and the Microsoft.VisualStudio.Threading.Analyzers package. This easily found the places in the Inner Drive Extensible Architecture as well as Weather Now where I'd botched the async coding.
Along the way I also made a couple of small tweaks that should cut down on the number of error messages the thing sends me. After last weekend's deployment I started getting so many that I had to shut the app down for a while. (Good thing I don't have any paying users.)
Annnndddd...it's not better. I still haven't figured out why the production API keeps dying, even though it seems to work fine in the Dev/Test environment.
Crap.
I just finished a 75-minute open-level French test as part of a QA study that Duolingo invited me to participate in. What an eye-opener. And quelle épuisement!
The test started well enough but got a lot harder as it went on, for two principal reasons. First, the order of sections went precisely in the order of my abilities: reading, writing, listening, speaking. Turns out I read French a lot better than I write it, write it better than I understand it, and speak it like a reject from a Pink Panther film. Some poor evaluator will have to listen to me going on for nearly three minutes about how hard the job of cat-herder is. What's worse, I only just now learned the word berger. "Herder des chats" is, apparently, not a thing, but berger de chats potentially is. I hope whoever scores that response at least has a sense of humor.
The second reason it got harder is that "open level" bit I mentioned. Each section got progressively more difficult, such that by the end of the listening part I could barely pick out the topic let alone individual words. Senegalese fishermen, you may be surprised to learn, are harder to understand than recorded announcements at train stations.
Still, I'm glad I did it. I don't know if they'll share the results with me, because they only want the data to calibrate their language-learning product. I hope they do, particularly before I pop out of the Chunnel just over two weeks from now.
I'm dog-sitting again, so a nervous beagle wandered up to my office during the test to see why I hadn't fed her yet. I suppose they both could use an around-the-block and some kibble. I will try to speak French to them, if only for my own practice.
Oh, and if you haven't been able to get to Weather Now this afternoon, that's because I shut it down for a bit while I root out a connection-exhaustion problem. I believe there are too many bots hitting it the last few days, but it still shouldn't crash when they do. Until I can fix the problem, or get rid of the bots, I'm only going to have it up a little bit at a time. (Its data collection continues unaffected, however.)
I've added a new feature to Weather Now: user profiles. It's only the most basic implementation and, at the moment, doesn't actually do anything. But it will lead to a whole range of features that the application hasn't had since it was an old Active Server Pages app in 1999.
Unfortunately, the deployment required setting up additional features on the weather API, so that user IDs travel from the UI to the API securely. The deployment took two hours, and threw up several pipeline failures for a reason having nothing to do with the API changes.
Anyway, now that the base user profile feature works, I can now add:
- User preferences for measurement systems (metric or Imperial);
- User-selected home locations;
- User-selected home page weather lists;
- Multiple custom weather lists; and
- Lots of other personalization features.
At some point I'll also finish importing the whole (9-million-plus record) gazetteer, so users can search for more places.
Now, however, I'm going to make some lunch now and take Cassie on a very long walk in the amazing autumn weather we have today.
The forecast still predicts today will be the hottest day of the year. Last night at IDTWHQ the temperature got all the way down to 26.2°C right before sunrise. We have a heat advisory until 10pm, by which time the thunderstorms should have arrived. Good thing Cassie and I got a bit of extra time on our walk to day camp this morning.
Elsewhere in the world:
Finally, Garmin has released its latest fitness watch that doubles as a freaking Dick Tracy wrist phone. I mean, first, how cool is that? And second, how come it took 90 years after Dick Tracy got one?
The hot, humid weather we've had for the past couple of weeks has finally broken. I'm in the Loop today, and spent a good 20 minutes outside reading, and would have stayed longer, except I got a little chilly. I dressed today more for the 24°C at home and less for the cooler, breezier air this close to the lake.
Elsewhere in the world:
- I was waiting for Russia expert Julia Ioffe to weigh in on last week's hostage release.
- The Chicago White Sox failed to set the all-time record for most consecutive losses in the American League yesterday by winning their first game in the last 23.
- Of the $1.2 trillion Carbon Reduction Program funds allocated to reduce fossil-fuel emissions, $130 billion has been spent so far: but only $26 billion on rail, and $70 billion on highways.
- Even though Deutsche Bahn has faster, timelier, more convenient, more comfortable, and just more trains than the US, Germans say their national railroad is on the wrong track.
- Deadhorse, Alaska, which lies at 70° north latitude, set an all-time record yesterday with a high temperature just below 32°C.
- After CrowdStrike told Delta Airlines to go pound sand a couple days ago, Microsoft told the carrier off yesterday.
- Be careful taking dogs to fresh-water swimming holes: warmer weather has made blue-green algae blooms more common.
Finally, today is the 60th anniversary of the Gulf of Tonkin Resolution. If you don't know what that is, read up. It's probably the most direct cause of most of our military policy since then.
This shit amused me:
Finally, Thursday marks the 20th anniversary of the Dave Matthews Band tour bus dropping 350 liters of very literal, very stinky shit onto a boatload of sightseers in the Chicago River. "The culprit turned out to be the band’s tour bus driver, then-42-year-old Stefan Wohl, who pleaded guilty to charges of reckless conduct and discharging contaminates to cause water pollution. He got hit with 18 months on probation, 150 hours of community service and had to pay a $10,000 fine to Friends of the Chicago River."
I mean, what the shit?
Network security company CrowdStrike pushed a minor update to its Falcon Sensor product around 11pm Chicago time yesterday that managed to take down almost every virtual machine in Microsoft's Azure cloud:
Cascading technology errors stranded airline passengers around the world, halted hospital surgeries and crippled office workers’ computers on Friday in one of the most disruptive computer outages in years, highlighting how much of the world relies on potentially error-prone software from a handful of companies.
Technology experts said the meltdowns appeared to stem mostly from an error in a software update from CrowdStrike, whose technology is commonly used by businesses to defend against cyberattacks.
That defect affected computers that use Microsoft’s Windows, which powers hundreds of millions of personal computers and many back-end systems for airlines, digital payment, emergency services call centers and much more.
[B]ecause CrowdStrike’s digital protections are considered essential, its technology is given priority access on many computer systems. If something goes wrong with CrowdStrike software, that privileged access can grind computers to a halt.
CrowdStrike admitted that their software caused the problem:
- Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
- Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.
- Windows hosts which are brought online after 0527 UTC will also not be impacted
- Hosts running Windows 7/2008 R2 are not impacted
- This issue is not impacting Mac- or Linux-based hosts
- Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
- Channel file "C-00000291*.sys" with timestamp of 0409 UTC is the problematic version.
Don't worry, you probably don't have CrowdStrike software on your PC at home; but you probably do log into your Windows PC through Microsoft Active Directory, which runs on virtual machines in the Azure cloud that depend on Falcon Sensor.
This time, the random person in Nebraska turned out to be a multimillion-dollar corporation in Austin, Texas. Though, I suspect, several random people in Texas are now looking for new jobs.
If he were even a tiny bit better as a human being, I might have some empathy for the old man clearly suffering from some kind of dementia who spoke in Doral, Fla., yesterday. But he's not, so I don't. I mean...just read the highlights.
In other news:
Finally, I got two emails through the contact-us page from the "Brand Ambassador & Link Approval Specialist" at a little company in the Duchy of Grand Fenwick demanding that we remove a link from a post to their site. Each email was clearly the output of an automated process that must have scraped every post on The Daily Parker—all 9,479 of them—more than once, because each email had a different fully-qualified domain name and most of the links they included were for category or history pages. Clearly the BALAS hadn't actually read the post that contained the link.
The request read: "We kindly request the immediate removal of these links to SchengenVisaInfo.com from your page because SchengenVisaInfo maintains strict editorial control over the information it provides. As such, we do not endorse the linking of our website without our prior consent."
This is dumb for several reasons. First, the emails provide clear evidence that they ran a bot over The Daily Parker more than once, which is rude. Second, this particular link could only benefit the complaining firm as it appeared in context as a way of finding out more about exactly what the company offered. And finally, before you send an email like that, you should confirm that the site you're complaining to won't ridicule you and your firm in a subsequent post.
Of course I removed the link. There are many better sources of information on the topic out there.
(Note to self: remove the company's name before posting!)
But only for a moment. I've spent most of today trying to fix things, or at least trying to figure out what problems need fixing. One of the problems has generated a comment thread on a vendor website, now at 44 comments, and I think after all that work I found the problem in an interaction between my code and Microsoft Azure Functions. If I'm right, the confirmation will come around 3pm.
Naturally, I haven't had time to read any of these:
I wrote the intro to this post around 2:45 and had to pause for a while. It's now 3:25, and I appear to have solved the problem. I will now document the solution and apologize to the vendor. Fun times, fun times.