The Daily Parker

Politics, Weather, Photography, and the Dog

Hacking the Vatican

Security guru Bruce Schneier examines Papal election security:

Probably the biggest risk is complacency. What might seem beautiful in its tradition and ritual during the first ballot could easily become cumbersome and annoying after the twentieth ballot, and there will be a temptation to cut corners to save time. If the Cardinals do that, the election process becomes more vulnerable.

A 1996 change in the process lets the cardinals go back and forth from the chapel to their dorm rooms, instead of being locked in the chapel the whole time, as was done previously. This makes the process slightly less secure but a lot more comfortable.

There are also enormous social -- religious, actually -- disincentives to hacking the vote. The election takes place in a chapel and at an altar. The cardinals swear an oath as they are casting their ballot -- further discouragement. The chalice and paten are the implements used to celebrate the Eucharist, the holiest act of the Catholic Church. And the scrutineers are explicitly exhorted not to form any sort of cabal or make any plans to sway the election, under pain of excommunication.

Of course, no amount of security in the world will prevent the electors from replacing Joseph Ratzinger with someone at least as out-of-touch and reactionary as he is, given the constitution of the cardinality these days.

Two Microsoft Azure outages in 24 hours

Over the past two days, Microsoft Azure had two outages they're still investigating. The first, from 18:26 CST through 20:00 CST Monday (0026 to 0200 UTC Tuesday), and the second, from 13:50 to 15:27 CST (1950-2127 UTC) yesterday, affected SQL Database and related services in the Azure datacenter outside Washington, D.C.

I noticed the Monday evening outage as it happened, because when a database goes down, a number of applications start sending me emails. A couple of people had minor inconveniences, but as it happened on a holiday evening, the damage wasn't too severe.

I did not notice the Tuesday afternoon outage, which did affect a lot of people and made some of my clients very angry, because I was on an airplane. When I landed and turned on my phone, I had 300 emails from various applications and mercifully only 4 from angry clients. (Welcome home!)

Microsoft hasn't determined reported the cause yet, but given the maintenance they had planned, started, and then backed out on Sunday night, they may have a clue. They have a second round of maintenance planned for tonight at midnight CST (0600 UTC). I'll be watching carefully tomorrow morning.

21st Century Rituals

Via Sullivan, a catalog of strange things we do with gadgets:

You’re on your cell phone, talking to a friend, pacing in circles, fidgeting with your hands, checking your cuticles–whatever it is you do while you’re on the phone. They’re odd, pointless behaviors, but we do them nonetheless, and a group of designers from the Art Center College of Design has taken it upon themselves to illustrate and document all of them (sort of like that Illustrated Dictionary of Cyborg Anthropology).

There’s the “Security Blanket” (checking your smartphone for no particular reason when faced with the slightest discomfort in a social situation), the “Halfway Courtesy” (taking one earbud out in order to show a person you’re listening to them), the “Haunted Interface” (performing actions an interface can’t react to, like shaking a video game controller), and many others. All of the actions are collected in a free ebook called Curious Rituals. Researcher Nicolas Nova explains in the book’s introduction.

Meanwhile, I'm doing my strange ritual of camping at Peet's Coffee before dawn to make sure I stay reasonably close to Chicago time for the weekend. Otherwise, Wednesday will be hell.

When the Azure emulator is more forgiving than real life

Last night I made the mistake of testing a deployment to Azure right before going to bed. Everything had worked beautifully in development, I'd fixed all the bugs, and I had a virgin Windows Azure affinity group complete with a pre-populated test database ready for the Weather Now worker role's first trip up to the Big Time.

The first complete and total failure of the worker role I should have predicted. Just as I do in the brick-and-mortar development world, I create low-privilege SQL accounts for applications to use. So immediately I had a bunch of SQL exceptions that I resolved with a few GRANT EXEC commands. No big deal.

Once I restarted the worker role, it connected to the database, loaded its settings, downloaded a file from NOAA and...crashed:

Inner Drive Weather threw System.Data.Services.Client.DataServiceRequestException
...
OutOfRangeInput

One of the request inputs is out of range.
RequestId:572bcfee-9e0b-4a02-9163-1c6163798d60
Time:2013-02-10T06:05:41.5664525Z

at System.Data.Services.Client.DataServiceContext.SaveResult.d__1e.MoveNext()

Oh no. The dreaded Azure Storage exception that tells you absolutely nothing.

Flash forward fifteen minutes (now past midnight; and for context, I'm writing this on the 9am flight to Los Angeles), with Fiddler running on a local instance connecting to production Azure storage, and I found the XML block on which real Azure Storage barfed but the Azure storage emulator passed without a second thought. The offending table entity is metadata that the NOAA downloader worker task stores to let the weather parsing worker task know it has work to do:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
   <entry xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" 
   xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" 
   xmlns="http://www.w3.org/2005/Atom">
  <title />
  <author>
    <name />
  </author>
  <updated>2013-02-10T05:55:49.3316301Z</updated>
  <id />
  <content type="application/xml">
    <m:properties>
      <d:BlobName>20130209-0535-sn.0034.txt</d:BlobName>
      <d:FileName>sn.0034.txt</d:FileName>
      <d:FileTime m:type="Edm.DateTime">2013-02-09T05:35:00Z</d:FileTime>
      <d:IsParsed m:type="Edm.Boolean">false</d:IsParsed>
      <d:ParseTime m:type="Edm.DateTime">0001-01-01T00:00:00</d:ParseTime>
      <d:PartitionKey>201302</d:PartitionKey>
      <d:RetrieveTime m:type="Edm.DateTime">2013-02-10T05:55:29.1084794Z</d:RetrieveTime>
      <d:RowKey>20130209-0535-41d536ff-2e70-4564-84bd-7559a0a71d4d</d:RowKey>
      <d:Size m:type="Edm.Int32">68202</d:Size>
      <d:Timestamp m:type="Edm.DateTime">0001-01-01T00:00:00</d:Timestamp>
    </m:properties>
  </content>
</entry>

Notice that the ParseTime and Timestamp values are equal to System.DateTimeOffset.MinValue, which, it turns out, is not a legal Azure table value. Wow, would it have helped me if the emulator horked on those values during development.

The fix was simply to make sure that neither System.DateTimeOffset.MinValue nor System.DateTime.MinValue ever got into an outbound table entity, which took me about five minutes to implement. Also, it turned out that even though my table entity inherited from TableServiceEntity, I still had to set the Timestamp property when using real Azure storage. (The emulator sets it for you.)

By this point it was 12:30 and I needed to get some sleep, however. So my plan to run an overnight test will have to wait until this evening at my hotel. Then I'll find the other bits of code that work fine against the emulator but, for reasons that pass understanding, the emulator gets completely wrong.

Thinking about vacation...where to go?

Unfortunately, that's not going to happen for a while. I'm going to spend a lot of time in airplanes over the next 11 days, including a long weekend with the folks. Good thing wifi is ubiquitous, even on airplanes, because it also looks like I'm going to burn at over 120% of utilization again this month. (Last month I was 118% billable, but if you add non-billable time I actually worked 134% of full time.)

The madness ends soon. We're hiring, projects are gelling, other projects are winding down, and at some point I'll just get on a plane for four days without taking my laptop.

I did take three hours yesterday to play pub trivia with my droogs, owing to the start of a four-week trivia tournament. We're in second place—by one point. I sincerely hope to make the next three Thursdays.

Quick link round-up

I'll be a lot less busy in March, they tell me. Meanwhile, here are some things I want to read:

I will get to them...soon...

Under the hood of Weather Now

My my most recent post mentioned finishing the GetWeather component of Weather Now, my demo project that provides near-real-time aviation weather for most of the world. I thought some readers might be interested to know how it works.

The GetWeather component has three principal tasks:

In the Inner Drive Technology world, an Azure worker process uses an arbitrary collection of objects that implement the IWorkerTask interface. The interface defines Interval and LastRun properties and an Execute method, which is all the worker process needs to know. The tasks are responsible for their own lifespans, reentry prevention, etc. (That's another discussion.)

In order to decouple the data source (NOAA now, other sources in the future) from the application, I split the three tasks into two IWorkerTask classes:

  • The NoaaFileDownloadingWorkerTask opens an FTP connection to the NOAA public weather servers, retrieves the files it hasn't already retrieved, and stores the contents in Azure Blob Storage; and
  • The NoaaFileParsingWorkerTask pulls the files out of Azure Storage, parses them, and stores the results in an Azure SQL Database and Azure table storage.

I'm using Azure storage as an intermediary between the two sides of the process because my analysis led me to the conclusion that they're really independent of each other. Coupling of the two tasks in the current (2002) version of GetWeather causes all kinds of problems, not least that a failure in one task can stop the whole thing. If, as happens given the nature of the Internet, the FTP side has an unrecoverable problem, the application has to restart. In actual practice it simply kills itself and waits for the next time it runs, which can be a while because it's running on a Windows Server 2008 Scheduler job every 30 minutes.

The new architecture will allow the parser to run every minute or two, see if it has anything to do by looking at some metadata, and do its job if needed. I can change a system setting to stop it from running (for example, because I need to do some database maintenance), while letting the downloader continue to work separately.

On the other side, the downloader can run every 5 minutes, snatch the one or two files it needs from NOAA, and shut down cleanly without waiting for the parser. NOAA likes this because the connection is only open for a few seconds, instead of the 27 minutes it stays open right now. And if the NOAA server isn't available, so what? It's a clean shutdown and a clean start a few minutes later.

This design also allows me to do something else: manually upload files for parsing and storage. This helps with testing, migration, service interruptions—all things that the current architecture has made nearly impossible.

I'm not entirely done with the application (and while writing this I just thought of an improvement I'll need to make to prevent infinite retries), but it's close. And I'm really pleased with the application so far. Stay tuned; I can now set a tentative public launch date of March 31st.

Resolving the oldest case

Five years ago, on 6 January 2008, I opened a FogBugz case (#528) to "Create NOAA Downloader". The NOAA downloader goes out to the National Weather Service, retrieves raw weather data files, and stores the files and some metadata in Windows Azure storage. Marking this work item "resolved"

Well, I just finished it, and therefore I have finished all of the pieces of the GetWeather application. And with that, I've finished the last significant piece of the Weather Now 4.0 rewrite. Total time to rewrite GetWeather: 42 hours. Total time for the rewrite so far: 66 hours.

Now all I have to do is...let's see...create worker role tasks to run the various pieces of the application (getting the weather, parsing the weather, storing the weather, and cleaning up the database), upgrading the Web site to a full Cloud Services application, deploy it to Azure, and deploy its gazetteer. That should be about 5 hours more work. Then, after a couple of weeks of mostly-passive testing, I can finally turn off the Inner Drive Technology Worldwide Data Center.

How to build software

Via Fallows, a software designer explains how a simple feature isn't:

This isn’t off the shelf, but that’s OK — we’ll just build it, it’s not rocket science. And it’s a feature that’s nice, not one that’s essential. Lot’s of people won’t use these tabs.

So, what do we need to think about when adding a bar of tabs like this?

  • The whole point is to have a view state that summarizes what you’re looking at and how it’s presented. You want to switch between view states. So we need a new object that encapsulates the View State, methods for updating the view state when the view changes or you switch tabs, methods for allocating memory for the view state and cleaning up afterward.
  • You need a bar in which the tabs live. That bar needs to have something drawn on it, which means choosing a suitable gradient or texture.
  • The tab needs a suitable shape. That shape is tricky enough to draw that we define an auxiliary object to frame and draw it.
  • Whoops! It gets drawn upside down! Slap head, fix that.

...and on for another 16 steps. He concludes, among other things:

This is a hell of a lot of design and implementation for $0.99. But that’s increasingly what people expect to pay for software. OK: maybe $19.95 for something really terrific. But can you sell an extra 100 copies of the program because it’s got draggable tabs? If you can’t, don’t you have better things to do with your time?

He's developing for a commercial application that he sells, so he may not be figuring the costs of development the same way I do. Since clients pay us for software development, it's a reflex for me to figure development costs over time. I don't know how much the tab feature cost him to develop, but I do know that to date, migrating Weather Now to Azure (discussed often enough on this blog) would have cost a commercial client about $9,000 so far, with another $3,000 or so to go. And the Inner Drive Extensible Architecture? That's close to $150,000 of development time—if someone else were paying for it.

And all you wanted was a little tab on your word processor...