The Daily Parker

A longtime Daily Parker reader asked this about yesterday's post:

"The Daily Parker costs $4.87 per day" -- I'm really hoping that's a misprint, because that's almost $150 a month, which is ten times what I pay for my web hosting package which comes with unlimited domains, a full email service (IMAP+SMTP over TLS), click-to-install WordPress and MySQL database creation, SSH access to the back-end Linux machine, and excellent customer support.

Also -- and I *really* hate to say this to a fellow IT professional -- your web site often seems rather slow. So much so that I'd built a mental image of it running on an old PC in a corner of your apartment, and I'd put the slow response times down to the latency of a hard disk spinning up from idle.

So, he's not wrong: The Daily Parker right now is slow and buggy. And expensive*. (Ironically, when it was literally running on a PC in the corner of my apartment prior to 2013, it ran like Jesse Owens.)

Sherman, set the Wayback Machine to October 2015, when I deployed the current version of this blog. From the blog's separation from braverman.org in 2005 until 2015, it ran on DasBlog, a .NET 1.1 blog engine that worked most of the time and had a few features I liked. I dragged it kicking and screaming up to .NET 2.0 and later .NET 4.0, and there it stayed.

After 10 years and dozens of tweaks, I decided to modernize by moving to BlogEngine.NET, which I also forked and modified. This engine runs on .NET 4.8, which I had to shoehorn into an Azure App Service when Cloud Services went away a couple of years ago. BlogEngine.NET had modest performance problems when it had a nice virtual machine all to itself, as Cloud Services weren't too different from on-premises hardware. But Azure App Services don't quite work the same way, such that many of the performance optimizations in the BlogEngine.NET code actually cause performance headaches in App Services. For example, at app start, the engine loads the entire blog history into memory, because in 2007, when the project began, memory was fast and disks were slow. (NB: The Daily Parker has over 9,700 posts spanning 27 years.) Also, the code runs entirely synchronously, so under load it spins up more and more threads until it just collapses from exhaustion.

So here we are: running a very old blog engine on a nearing-end-of-life version of .NET that everyone is tired of.

But, aha! There is a solution, which I've been kicking around for almost as long as I've had a blog, and which I finally have the skills and time to work on. I'll simply build my own. It'll be idiosyncratic, sure, but it'll be fast and it'll be cool.

Or maybe I'll go back to DasBlog, now that someone has rebuilt it in .NET Core.

Nah. I'm going to write my own. Target date: October 15th, ten years after I released this version.

* It's actually now around $3.34 per day after a Microsoft Azure pricing change on February 12th which just showed up in the cost management tool today. The costs break down as follows: App Service type B2, $2.49; storage (media and event log), 52¢; database (serverless type B), 33¢. So, around $100 per month.

Because Microsoft has deprecated 2011-era database servers, my weather demo Weather Now needed a new database. And now it has one.

Migrating all 8 million records (7.2 million places included) took about 36 hours on an Azure VM. Since I migrated entirely within the U.S. East data center, there were no data transfer charges, but having a couple of VMs running for the weekend probably will cost me a few dollars more this month.

While I was at it, I upgraded the app to the latest Azure and Inner Drive packages, which mainly just fixed minor bugs.

The actual deployment of the updated code was boring, as it should be.

It's just past 9am on Monday and already I'm reduced to this kind of blog post. Tomorrow I may have some more time to read these things:

• Cranky Flier analyzes Malaysia Airlines' struggles.
• Microsoft is building subsea fibre cables between the U.S. and Europe and Asia.
• TPM explains exactly what Jade Helm 15 really is.
• Missed Microsoft Ignite this year? Here's the Channel 9 page.
• We're starting to set up JetBrains TeamCity to handle our continuous integration needs. Explain, however, why the user manual is all video? Guys. Seriously. I haven't got time for this.
• So now that Illinois actually has to pay the pensions we promised to pay, what now? (Hello, 9% income tax?)

Four-hour design review session is imminent. I may post later today...or I may lock myself in my office and stare at the wall.

...and also preparing for a fundraiser at which I'm performing tomorrow:

• Microsoft has moved Azure Premium Storage to general availability...
• ...and also improved SQL backup and export services.
• Coincidentally, my favorite performance analysis tool just added a feature I need this week. Color me happy.
• The United Center, where the Chicago Bulls and Chicago Blackhawks play, is getting beer robots.
• British photographer Marcus Lyon does not like sprawl. And his photos are kind of cool.
• There's also a poetry map of Toronto. One of my friends just discovered her street corner has a poem about it.
• The long-range climate forecast for Illinois predicts bog-standard weather. I can live with that.

And did I mention Apollo After Hours?

With meetings and a new developer on the team occupying almost all my time today, I've put these things aside for the half-hour I have at 6:30 to read them:

• If you look at data, you see Democrats have created more jobs than Republicans, no matter what people say about Saint Ronald.
• Microsoft has released a major update to Azure. There's even a slick video about it.
• Ted Cruz is going all-in with the white male vote. Mazel tov.
• Who doesn't like the Daily WTF?
• New York's 7th Avenue Subway extension is late but on-budget and nice to look at.

Now to jot down some policies on our new Microsoft Surface setups...

Interesting things to read:

• Climate change deniers, take note: even though 2014 was really cold in part of the U.S., it was still the warmest year ever worldwide.
• Two posts on the Microsoft Azure blog: how to add auto-complete suggestions using Azure Search, and how to tune Azure DocumentDB performance.
• Could airlines start giving landing preference to their own high-value flights?
• Chicagoist has their best brunches list up. Yum.
• We might start using JetBrains TeamCity for continuous integration. More on that if it happens.

Before reading all of those I need to get a production deployment ready for this weekend. It would help if I were completely certain what's in production right now...

Well, little time today. Since I'll be on an airplane for 8 hours on Sunday, I will probably have time to catch up on these:

• The Chicago Tribune's John Kass doesn't like the Episode 7 lightsaber.
• Chicago Public Media's Jim DeRogatis doesn't like Pomplamoose.
• Cranky Flier wonders why American Airlines doesn't like profit-sharing.
• I will probably really like Microsoft Azure DocumentDB partitioning (but it's a pretty esoteric topic for most Daily Parker readers).
• Your solid-state drive (SSD) probably likes defragmenting just fine, as long as Windows 8 is doing it. (My drives are all 0% fragmented right now, even the SSDs.)
• New Republic's Danny Vinik is indifferent about Amazon's slowing growth.
• James Fallows likes some things and doesn't like others about the politics of American reslience.
• I don't like that I had two consecutive days with fewer than 10,000 steps, but unless I want to walk up and down the aisle of an airplane a lot, it's hard to get them when traveling.
• Finally, I really, really like the look of this:

I mentioned over a month ago that, given some free time, I would fix the search feature on Weather Now. Well, I just deployed the fix, and it's kind of cool.

I used Lucene.NET as the search engine, incorporating it into the Inner Drive Gazetteer that underlies the geographic information for Weather Now. I won't go into too many details about it right now, except that I was surprised at how much the index writer was able to crunch and store (in Azure blobs). The entire index takes up 815 MB of blob space. That's so small a fraction of a cent per month I can't even calculate it right now.

The indexing process took about 6 minutes per 500,000 rows. (The entire database has 7.25 million rows.) It helped that I ran the indexing process on an Azure virtual machine, because at one point during index optimization I clocked the data throughput at 200 Mbps. Yes, two hundred megabits per second. The entire index ran in a little less than two hours on a VM while I was doing other things. And once the index initializes in the Weather Now app, searches only take a second or so.

Go ahead. Try a search. Put in your ZIP code or the name of a prominent building near you.

I still have a lot I want to do with the application, including updating it to a responsive theme and MVC, but this is a pretty big leap.

I have to dash off to a meeting in a few minutes, then to Wrigley. So this is more of a note to myself.

Lucene.NET will be coming to Weather Now, I hope in a few weeks. This will massively improve its piss-poor searching, and allow me to do a few other things as well given Lucene's amazing search capabilities.

Unfortunately, Weather Now ranks third in development priorities behind my employer and my long-suffering freelance client. At least it kind of runs itself these days.

For the last couple of days, I've had trouble getting to Microsoft's Azure blog. From my office in downtown Chicago, clicking the link gives me an error message:

The resource you are looking for has been removed, had its name changed, or is temporarily unavailable.

However, going to the same URL from a virtual machine on Azure takes me to the blog. So what's going on here? It took a little detective work, but I think Microsoft has a configuration error one of a set of geographically-distributed Azure web sites, they don't know about it, and there's no way to tell them.

The first step in diagnosing a problem like this is to see if it's local. Is there something about the network I'm on that prevents me from seeing the website? This is unlikely for a few big reasons: first, when a local network blocks or fails to connect to an outside site, usually nothing at all happens. This is how the Great Firewall of China works, because someone trying to get to a "forbidden" address may get there slowly, normally, or not at all—and it just looks like a glitch. Second, though, the root Azure site is completely accessible. Only the Blog directory has an error message. Finally, the error message is coming from the foreign system. Chrome confirms this; there's a HTTP 200 (OK) response with the content I see.

All right, so the Azure Blog is down. But that doesn't make a lot of sense. Thousands of people read the Azure blog every day; if it were down, surely Microsoft would have noticed, right?

So for my next test, I spun up an Azure Virtual Machine (VM) and tried to connect from there. Bing! No problem. There's the blog.

Now we're onto something. So let's take a look at where my local computer thinks it's going, and where the VM thinks it's going. Here's the nslookup result for my local machine, both from my company's DNS server and from Google's 8.8.8.8 server:

Now here's what the VM sees:

Well, now, that is interesting.

From my local computer, sitting in downtown Chicago, both Google and my company's DNS servers point "azure.microsoft.com" to an Azure web site sitting in the North Central U.S. data center, right here in Chicago. But for the VM, which itself is running in the East U.S. data center in southern Virginia, both Microsoft's and Google's DNS servers point the same domain to an Azure web site also within the East U.S. data center.

It looks like both Microsoft and Google are using geographic load-balancing and some clever routing to return DNS addresses based on where the DNS request comes from. I'd bet if I spun up an Azure VM in the U.S. West data center, both would send me to the Azure blog running out there.

This is what massive load balancing looks like from the outside, by the way. If you've put your systems together correctly, users will go to the nearest servers for your content, and they'll never realize it.

Unfortunately, the North Central U.S. instance of the Microsoft Azure blog is down, has been down for several days, and won't come up again until someone at Microsoft realizes it's down. Also, Microsoft makes it practically impossible to notify them that something is broken. So those of us in Chicago will just have to read about Azure on our Azure VMs until someone in Redmond fixes their broken server. I hope they read my blog.