The Daily Parker

I have to dash off to a meeting in a few minutes, then to Wrigley. So this is more of a note to myself.

Lucene.NET will be coming to Weather Now, I hope in a few weeks. This will massively improve its piss-poor searching, and allow me to do a few other things as well given Lucene's amazing search capabilities.

Unfortunately, Weather Now ranks third in development priorities behind my employer and my long-suffering freelance client. At least it kind of runs itself these days.

For the last couple of days, I've had trouble getting to Microsoft's Azure blog. From my office in downtown Chicago, clicking the link gives me an error message:

The resource you are looking for has been removed, had its name changed, or is temporarily unavailable.

However, going to the same URL from a virtual machine on Azure takes me to the blog. So what's going on here? It took a little detective work, but I think Microsoft has a configuration error one of a set of geographically-distributed Azure web sites, they don't know about it, and there's no way to tell them.

The first step in diagnosing a problem like this is to see if it's local. Is there something about the network I'm on that prevents me from seeing the website? This is unlikely for a few big reasons: first, when a local network blocks or fails to connect to an outside site, usually nothing at all happens. This is how the Great Firewall of China works, because someone trying to get to a "forbidden" address may get there slowly, normally, or not at all—and it just looks like a glitch. Second, though, the root Azure site is completely accessible. Only the Blog directory has an error message. Finally, the error message is coming from the foreign system. Chrome confirms this; there's a HTTP 200 (OK) response with the content I see.

All right, so the Azure Blog is down. But that doesn't make a lot of sense. Thousands of people read the Azure blog every day; if it were down, surely Microsoft would have noticed, right?

So for my next test, I spun up an Azure Virtual Machine (VM) and tried to connect from there. Bing! No problem. There's the blog.

Now we're onto something. So let's take a look at where my local computer thinks it's going, and where the VM thinks it's going. Here's the nslookup result for my local machine, both from my company's DNS server and from Google's 8.8.8.8 server:

Now here's what the VM sees:

Well, now, that is interesting.

From my local computer, sitting in downtown Chicago, both Google and my company's DNS servers point "azure.microsoft.com" to an Azure web site sitting in the North Central U.S. data center, right here in Chicago. But for the VM, which itself is running in the East U.S. data center in southern Virginia, both Microsoft's and Google's DNS servers point the same domain to an Azure web site also within the East U.S. data center.

It looks like both Microsoft and Google are using geographic load-balancing and some clever routing to return DNS addresses based on where the DNS request comes from. I'd bet if I spun up an Azure VM in the U.S. West data center, both would send me to the Azure blog running out there.

This is what massive load balancing looks like from the outside, by the way. If you've put your systems together correctly, users will go to the nearest servers for your content, and they'll never realize it.

Unfortunately, the North Central U.S. instance of the Microsoft Azure blog is down, has been down for several days, and won't come up again until someone at Microsoft realizes it's down. Also, Microsoft makes it practically impossible to notify them that something is broken. So those of us in Chicago will just have to read about Azure on our Azure VMs until someone in Redmond fixes their broken server. I hope they read my blog.

A Comcast installer showed up this morning within the appointed time frame, and in about an hour had taken my apartment the Inner Drive Technology World Headquarters from this:

To this:

I almost want to dance around singing "A Whole New World" but that would be very disturbing to my self image.

Instead I'll head into the office, getting in a little earlier than I expected, and come home to real Internet speeds. In fact, I think right now I'll watch something on YouTube just because I can.

Goodbye, AT&T. Hello Comcast, you gorgeous thing.

No, not aviation routing; IP routing.

From the Terminal 2 American Airlines club, I am unable to hit most *.cloudapp.net IP addresses. This is significant because it's basically all of Microsoft Azure, including logon.microsoft.net, Weather Now, and a bunch of other sites I use or have some responsibility for.

I've just spent a few minutes testing DNS (everything is fine there) and then using tracert and pathping, and it looks like the entire 168.62.0.0/16 and 168.61.0.0/16 ranges are just not visible from here. (The Daily Parker is also in Azure, but its IP is in the 191.238.0.0/16 subnet, which seems to be visible just fine.)

I wonder if Microsoft knows that its U.S. East data center is being blocked by some French ISP? Or why?

We've been using Microsoft Azure virtual machines for development for a while. This means we run our Visual Studio instances in the cloud up on special virtual machines that have nothing on them except the bare minimum required for writing software. This keeps different projects separate from each other, and also speeds up network access, which is useful for network-intensive applications.

We started noticing, however, that going to MSDN or Google or other big sites became...challenging. All of these sites started acting as if our VMs were located in Brazil, when we knew perfectly well that they were in Virginia. Microsoft has finally explained the problem:

IPv4 address space has been fully assigned in the United States, meaning there is no additional IPv4 address space available. This requires Microsoft to use the IPv4 address space available to us globally for the addressing of new services. The result is that we will have to use IPv4 address space assigned to a non-US region to address services which may be in a US region. It is not possible to transfer registration because the IP space is allocated to the registration authorities by Internet Assigned Numbers Authority.

At times your service may appear to be hosted in a non-US location.

It is important to note that the IP address registration authority does not equate to IP address physical location (i.e., you can have an IP address registered in Brazil but allocated to a device or service physically located in Virginia). Thus when you deploy to a U.S. region, your service is still hosted in U.S. and your customer data will remain in the U.S.

In other words, Microsoft's cloud service is so popular that they have run out of addresses to assign to it. Microsoft, it should be noted, has tens of millions of IPv4 addresses available. (Of course, IPv4 has only 4.2 trillion possible addresses, though fully 43 billion are in private IP ranges.)

The deployment, I mean. Everything works, at least on the browsers I've used to test it. I ran the deployment three times in Test first, starting from a copy of the Production database each time, so I was as confident as I could be when I finally ran it against the Production database itself. And, I made sure I can swap everything back to the old version in about 15 minutes.

Also, I snuck away to shoot publicity photos for Spectralia again, same as last year. I'll have some up by the end of the week, after the director has seen them.

Jez Humble, who wrote the book on continuous delivery, believes deployments should be boring. I totally agree; it's one of the biggest reasons I like working with Microsoft Azure.

Occasionally, however, deploying software is not at all boring. Today, for example.

Because Microsoft has ended support for Windows Server 2008 as of next week, I've upgraded an old application that I first released to Azure in August 2012. Well, actually, I updated it back in March, so I could get ahead of the game, and the boring deployment turned horrifying when half of my client's customers couldn't use the application because the OS upgrade broke their Windows XP/IE8 user experience. Seriously.

All of my client's customers have now upgraded to Chrome, IE11, or Firefox, and I've tested the app on all three browsers. Everything works. But now I have to redeploy the upgrade, and I've got a real feeling of being once-bitten.

The hard part, the part that makes this a one-way upgrade, is a significant change to the database schema. All the application's lookup lists, event logging, auditing, and a few other data structures, are incompatible with the current Production version. Even if there weren't an OS upgrade involved, the database changes are overdue, so there is no going back after this.

Here are the steps that I will take for this deployment:

1. Copy current Production database to new MigrationTest database
2. Upgrade MigrationTest database
3. Verify Test settings, connection strings, and storage keys
4. Deploy Web project to Test instance (production slot)
5. Validate Test instance
6. Deploy Worker project to Test instance (production slot)
7. Validate Worker instance
8. Shut down Production instance
9. Back up Production database to bacpac
10. Copy Production database within SQL instance
11. Upgrade Production database
12. Verify Production settings, connection strings, and storage keys
13. Deploy solution to Production instance (staging slot)
14. Validate Production Web instance
15. Validate Production Worker instance
16. VIP swap to Production

Step 1 is already complete. Step 2 will be delayed for a moment while I apply a patch to Visual Studio over my painfully-slow internet connection (thanks, AT&T!). And I expect to be done with all of this in time for Game of Thrones.

Short answer: You can't. So don't try.

Back in 2007, when I wrote a scheduling application for a (still ongoing!) client, Azure was a frustrating research project at Microsoft. Every bit of data the application stored went into SQL Server tables including field-level auditing and event logs.

The application migrated to Azure in August 2012, still logging every audit record and event to SQL tables, which are something like 10x more expensive per byte than Azure Table Storage. Recently, I completed an upgrade to the Inner Drive Extensible Architecture™ so it can now use Azure table storage for both.

The old application knew nothing about this. So upgrading the application with the new IDEA bits worked fine for writing to the audit and event logs, but completely broke reading from them.

Here's the code the app uses displaying a specific audit record so an administrator can see the field-level details:

// Get the repository from Castle IoC:
var repo = ActiveContainer.Instance.Container.Resolve<IAuditRepository>();

// Get the individual audit record by ID:
var audit = repo.Find(id);

That's great if the audit record uses a database identity key. Unfortunately it does not; it uses an Azure partition and row key combination.

I agonized for a couple of days how to fake database identities in Azure, or how to write a mapping table, or do some other code- or data-intensive thing or another. Then this afternoon, epiphany!

The user viewing an audit record is doing it in the context of reviewing a short list of audit headers. They click on one of these headers to pop up the detail box. The detail box uses the audit ID like this: https://scheduler.myclient.com/Auditing/AuditDetailViewer/12345.

It turns out, the only time a user cares about audit details is when she has the audit list right in front of her. So the audit ID is irrelevant. It only has to be unique within the context of the user's experience.

Here's the solution. In the Auditing class, which generates the audit list, I do this:

foreach (var item in orderedAudits)
{
	// Temporal cohesion: Add identity to Audit before using
	AddIdentity(item);
	Cache(item);
	CreateAuditRow(placeHolder, isObjectSpecified, item, timeZone);
	// End temporal cohesion
}

The cache uses least-frequently-used scavenging with a capacity of 512 items. (If the one user who cares about auditing ever needs to see more than 512 audit items in one list, I'll coach him not to do that.) Items live in the cache until it gets full, at which time the least-used ones are removed. This lets me do this in the audit detail view's control code:

var audit = Auditing.Find(id);

The Auditing.Find method simply plucks the item from the cache. If it returns null, oops, the audit details are missing or have expired from the cache, sorry. Just rerun the list of audits that you just clicked on.

I'm going to use a similar approach to the event log.

I want to try this:

In less than an hour [my website] went from a small prototype in a data center in Chicago and then scaled it out to datacenters globally and added SSL.

The step-by-step explanation is worth a read if you do anything in .NET.

Yesterday I migrated this blog and four other ASP.NET websites from a Windows 2008 Microsoft Azure virtual machine (VM) to a brand-new Windows 2012 R2 VM. I did this because Microsoft has announced the end-of-life for Windows 2008 VMs on June 1st, so I thought I'd get a jump on it.

VMs usually mean never having to say "reinstall." Unfortunately, since this involved upgrading three steps at once, I decided it would be simpler just to launch a new VM and migrate the applications using FTP.

Seven hours and 25 minutes later, everything works, and I've archived the old VM's virtual hard disk (VHD). Why did it take 7:25 to complete?

Forget it. I'm not reliving those hours. I will say only that at least 90 minutes of that time was completely wasted because my AT&T Uverse FiOS line doesn't...quite...make it to my building, limiting it to 1.5 Mbps. Yes, I have a 1.5 Mbps Internet line. While waiting for things to download and upload yesterday, I spoke with them, and they assured me that I have the fastest Uverse service available to me.

Which brings up the other problem with doing so much in Microsoft Azure: you need good Internet connectivity. Which I don't have. Which meant I spent a lot of time yesterday rubbing Parker's belly and cursing AT&T.