The Daily Parker

Politics, Weather, Photography, and the Dog

Yes, that's an improvement (Comcast win!)

A Comcast installer showed up this morning within the appointed time frame, and in about an hour had taken my apartment the Inner Drive Technology World Headquarters from this:

To this:

I almost want to dance around singing "A Whole New World" but that would be very disturbing to my self image.

Instead I'll head into the office, getting in a little earlier than I expected, and come home to real Internet speeds. In fact, I think right now I'll watch something on YouTube just because I can.

Goodbye, AT&T. Hello Comcast, you gorgeous thing.

Scary software deployment

Jez Humble, who wrote the book on continuous delivery, believes deployments should be boring. I totally agree; it's one of the biggest reasons I like working with Microsoft Azure.

Occasionally, however, deploying software is not at all boring. Today, for example.

Because Microsoft has ended support for Windows Server 2008 as of next week, I've upgraded an old application that I first released to Azure in August 2012. Well, actually, I updated it back in March, so I could get ahead of the game, and the boring deployment turned horrifying when half of my client's customers couldn't use the application because the OS upgrade broke their Windows XP/IE8 user experience. Seriously.

All of my client's customers have now upgraded to Chrome, IE11, or Firefox, and I've tested the app on all three browsers. Everything works. But now I have to redeploy the upgrade, and I've got a real feeling of being once-bitten.

The hard part, the part that makes this a one-way upgrade, is a significant change to the database schema. All the application's lookup lists, event logging, auditing, and a few other data structures, are incompatible with the current Production version. Even if there weren't an OS upgrade involved, the database changes are overdue, so there is no going back after this.

Here are the steps that I will take for this deployment:

  1. Copy current Production database to new MigrationTest database
  2. Upgrade MigrationTest database
  3. Verify Test settings, connection strings, and storage keys
  4. Deploy Web project to Test instance (production slot)
  5. Validate Test instance
  6. Deploy Worker project to Test instance (production slot)
  7. Validate Worker instance
  8. Shut down Production instance
  9. Back up Production database to bacpac
  10. Copy Production database within SQL instance
  11. Upgrade Production database
  12. Verify Production settings, connection strings, and storage keys
  13. Deploy solution to Production instance (staging slot)
  14. Validate Production Web instance
  15. Validate Production Worker instance
  16. VIP swap to Production

Step 1 is already complete. Step 2 will be delayed for a moment while I apply a patch to Visual Studio over my painfully-slow internet connection (thanks, AT&T!). And I expect to be done with all of this in time for Game of Thrones.

How to look up Azure table data by ID

Short answer: You can't. So don't try.

Back in 2007, when I wrote a scheduling application for a (still ongoing!) client, Azure was a frustrating research project at Microsoft. Every bit of data the application stored went into SQL Server tables including field-level auditing and event logs.

The application migrated to Azure in August 2012, still logging every audit record and event to SQL tables, which are something like 10x more expensive per byte than Azure Table Storage. Recently, I completed an upgrade to the Inner Drive Extensible Architecture™ so it can now use Azure table storage for both.

The old application knew nothing about this. So upgrading the application with the new IDEA bits worked fine for writing to the audit and event logs, but completely broke reading from them.

Here's the code the app uses displaying a specific audit record so an administrator can see the field-level details:

// Get the repository from Castle IoC:
var repo = ActiveContainer.Instance.Container.Resolve<IAuditRepository>();

// Get the individual audit record by ID:
var audit = repo.Find(id);

That's great if the audit record uses a database identity key. Unfortunately it does not; it uses an Azure partition and row key combination.

I agonized for a couple of days how to fake database identities in Azure, or how to write a mapping table, or do some other code- or data-intensive thing or another. Then this afternoon, epiphany!

The user viewing an audit record is doing it in the context of reviewing a short list of audit headers. They click on one of these headers to pop up the detail box. The detail box uses the audit ID like this: https://scheduler.myclient.com/Auditing/AuditDetailViewer/12345.

It turns out, the only time a user cares about audit details is when she has the audit list right in front of her. So the audit ID is irrelevant. It only has to be unique within the context of the user's experience.

Here's the solution. In the Auditing class, which generates the audit list, I do this:

foreach (var item in orderedAudits)
{
	// Temporal cohesion: Add identity to Audit before using
	AddIdentity(item);
	Cache(item);
	CreateAuditRow(placeHolder, isObjectSpecified, item, timeZone);
	// End temporal cohesion
}

The cache uses least-frequently-used scavenging with a capacity of 512 items. (If the one user who cares about auditing ever needs to see more than 512 audit items in one list, I'll coach him not to do that.) Items live in the cache until it gets full, at which time the least-used ones are removed. This lets me do this in the audit detail view's control code:

var audit = Auditing.Find(id);

The Auditing.Find method simply plucks the item from the cache. If it returns null, oops, the audit details are missing or have expired from the cache, sorry. Just rerun the list of audits that you just clicked on.

I'm going to use a similar approach to the event log.

Waiting for software to deploy...

I'm uploading a couple of fixes to Inner-Drive.com right now, so I have a few minutes to read things people have sent me. It takes a while to deploy the site fully, because the Inner Drive Extensible Architecture™ documentation (reg.req.) is quite large—about 3,000 HTML pages. I'd like to web-deploy the changes, but the way Azure cloud services work, any changes deployed that way get overwritten as soon as the instance reboots.

All of the changes to Inner-Drive.com are under the hood. In fact, I didn't change anything at all in the website. But I made a bunch of changes to the Azure support classes, including a much better approach to logging inspired by a conversation I had with my colleague Igor Popirov a couple of weeks ago. I'll go into more details later, but suffice it to say, there are some people who can give you more ideas in one sentence than you can get in a year of reading blogs, and he's one of them.

So, while sitting here at my remote office waiting for bits to upload, I encountered these things:

  • The bartender's iPod played "Bette Davis Eyes" which immediately sent me back to this.
  • Andrew Sullivan pointed me (and everyone else who reads his blog) towards the ultimate Boomer fantasy, the live-foreverists. (At some point in the near future I'm going to write about how much X-ers hate picking up after both Boomers and Millennials, and how this fits right in. Just, not right now.)
  • Slate's Jamelle Bouie belives Wisconsin's voter rights decision is a win for our cause. ("Our" in this case includes those who believe retail voter fraud is so rare as to be a laughable excuse for denying a sizable portion of the population their voting rights, especially when the people denied voting rights tend to be the exact people who Republicans would prefer not to vote.)

OK, the software is deployed, and I need to walk Parker now. Maybe I'll read all these things after Game of Thrones.

Clbuttic software mistake

Apparently OCR software sometimes still has trouble interpreting older books:

[A]s Sarah Wendell, editor of the Romance blog  Smart Bitches, Trashy Books noticed recently, something has gone awry. Because, in many old texts the scanner is reading the word ‘arms’ as ‘anus’ and replacing it as such in the digital edition. As you can imagine, you don’t want to be getting those two things mixed up.

The resulting sentences are hilarious, turning tender scenes of passionate embrace into something much darker, and in some cases, nearly physically impossible. The Guardian’s Alison Flood quotes some of the best:

From the title Matisse on the Loose: “When she spotted me, she flung her anus high in the air and kept them up until she reached me. ‘Matisse. Oh boy!’ she said. She grabbed my anus and positioned my body in the direction of the east gallery and we started walking.”

And ‘”Bertie, dear Bertie, will you not say good night to me” pleaded the sweet, voice of Minnie Hamilton, as she wound her anus affectionately around her brother’s neck. “No,” he replied angrily, pushing her away from him.”‘ Well, wouldn’t you?

As Flood notes, a quick search in Google Books reveals that the problem is widespread. Parents should keep their children away from the ebook edition of the 1882 children’s book Sunday Reading From the Young. It all seems perfectly innocent until… “Little Milly wound her anus lovingly around Mrs Green’s neck and begged her to make her home with them. At first Mrs Green hesitated.” And who can really blame her?

Clbuttic.

Not embracing open source so much as shaking hands

The Inner Drive Extensible Architecture™ is about to get wider distribution.

After 11 years of development, I think it's finally ready for wider distribution. And, who knows, maybe I'll make a couple of bucks.

I've updated the pricing structure and the license agreement, and in the next week or so (after some additional testing), I'm going to release it to NuGet.

That doesn't make it free; that makes it available. (Actually, I am making it free for development and testing, but I'm charging for commercial production use.)

I'll have more to say on this once it's released.

Head-scratching software bug

I don't know how this little snippet of code got into a project at work (despite the temptation to look at the file's history):

Search = Search.Replace("Search…", "");

// Perform a search that depends on the Search member being an empty string
// Display the list of things you find

First, I can't fathom why the original developer made the search method dependent on a hard-coded string.

Second, as soon as the first user hit this code after switching her user profile to display Japanese, the search failed.

The fix was crushingly simple:

var localizedSearchPrompt = GetLocalizedString("SearchPrompt");
Search = Search.Replace(localizedSearchPrompt, string.Empty);

(The GetLocalizedString method in this sample is actually a stub for the code that gets the string in the current user's language.)

The moral of the story is: avoid hard-coded strings, just like you've been taught since you started programming.

Weakest link in the chain

I had planned to post some photos tonight showing the evolution of digital cameras, using a local landmark, but there's a snag. The CF card reader I brought along isn't showing up on my computer, even though the computer acknowledges that something is attached through a USB port.

As I'm visiting one of the most sophisticated and technological cities in the world, I have no doubt I can fix this tomorrow. Still, it's always irritating when technology that worked a few days ago simply stops working.

For those doubting my troubleshooting skills, I have confirmed that the CF card has all the photos I shot today; that the computer can see the CF card reader; and that the computer can connect effectively to other USB attachments. The problem is therefore either in the OS or in the card reader, and I'm inclined to suspect the card reader.

It's not the good times they care about, it's the bad

The repercussions from Monday's data-recovery debacle continued through yesterday.

By the time business started Tuesday morning, I had restored the client's application and database to the state it had at the moment of the upgrade, and I'd entered most of their appointments, including all of them through tomorrow (Thursday). When the client started their day, everything seemed to be all right, except for one thing I also didn't know about their business: some of their customers pay them based on the appointment ID, which is nothing more than a SQL IDENTITY column in the database.

If you know how databases work, you know that IDENTITY columns are officially non-deterministic. In this specific case, the column increments by one every time it adds a row, but also in this specific case, I didn't re-enter the data in the same order it was originally entered, since I prioritized the earlier appointments.

We've gotten through the problem now, and the client no longer want to put my head on a spike, so I will now take a moment for an after-action review that might help other software developers in the future.

First, the things I did right:

  • When I deployed the upgrade Saturday, I preserved the state of the database and application at exactly that moment.
  • All of the data in the system, every field of it, was audited. It was trivially easy to produce a report of every change made to the system from roll-out Saturday afternoon through roll-back Monday night.
  • When I rolled back the upgrade Monday night, I preserved the state of the upgraded database and application at exactly that moment.
  • When the client first noticed the problem, I dropped everything else and worked out a plan with them. The plan centered around getting their business back up first, and then dealing with the technology.
  • Their customers were completely back to normal at the start of business Tuesday.
  • The application runs on Windows Azure, which made preserving the old application state not only easy, but possible.

So what should I have done better?

  • My biggest error was overconfidence in my ability to roll back the upgrade. No matter what other errors I made, this was the root of all of them.
  • The second major error was not testing the UI on Internet Explorer 8. Mitigating this was the fact that neither I nor my client was aware that the bulk of their customers used IE8. However, given that people using IE8 were totally unable to use the application, even if the numbers of customers using IE8 was very small, the large impact should have put IE8 near the top of my regression test checklist.
  • Instead of spending a couple of hours re-entering data, I should have written a script to do it.
  • I have always regretted (though never more than today) publicizing the appointments IDENTITY column to the end user, because it's normal they'd use this ID for business purposes. This illustrates the danger—not just the sloppy design—of using a single database field for two purposes. Any future version of the application will have an OrderID field that is not a database plumbing field.

All in all, the good things outweighed the bad, and I may get back in my client's good graces when I roll out the next update. You know, the one that works on IE8, but still solves the looming problem of the platform's age.