The Daily Parker

Taking a brief rest from my temporary insanity, I read Sullivan:

Someone in the GOP needs to take Bush-Cheney apart, to show how they created the debt crisis we are in, by throwing away a surplus on unaffordable tax cuts, launching two unfunded wars, and one new unfunded entitlement. They need to take on the war crimes that has deeply undermined the soul of the United States. They need to note the catastrophic negligence that gave us the worst national security lapse since Pearl Harbor (9/11) despite being warned explicitly in advance, accept weak and false intelligence to launch a war they were too incompetent to fight or win, sat back as one of the worst hurricanes all but took out a major city, and was so negligent in bank regulation that we ended up with Lehman and all that subsequently took place.

These were not minor errors. They were catastrophic misjudgments which took an era of peace, surplus and prosperity and replaced it with a dystopia of massive debt, a lawless executive branch, two unwinnable wars, and a record of war crimes that had their source in the very Oval Office.

That seems about right to me.

Unfortunately, that's not going to happen for a while. I'm going to spend a lot of time in airplanes over the next 11 days, including a long weekend with the folks. Good thing wifi is ubiquitous, even on airplanes, because it also looks like I'm going to burn at over 120% of utilization again this month. (Last month I was 118% billable, but if you add non-billable time I actually worked 134% of full time.)

The madness ends soon. We're hiring, projects are gelling, other projects are winding down, and at some point I'll just get on a plane for four days without taking my laptop.

I did take three hours yesterday to play pub trivia with my droogs, owing to the start of a four-week trivia tournament. We're in second place—by one point. I sincerely hope to make the next three Thursdays.

This is alarming:

A new and worrisome benchmark has been reached with the announcement Tuesday by the U.S. Army Corps of Engineers that Lakes Michigan and Huron have dipped to new record lows. It’s been a 14 year journey. That’s how long water levels have been below historic averages--the most extended run of below normal water levels in the 95 year record of Great Lakes dating back to 1918.

The numbers are as stunning as they are disturbing with serious implications to shipping interests, all manner of creatures which populate the lakes, plus the millions who enjoy these natural treasures recreationally and depend on them as a source of water.

Water levels have fallen 1.9 m from the record highs established in October 1986 and currently sit at levels 735 mm below the long term average. Lake Michigan's water level is 430 mm lower than a year ago

We're getting more precipitation than we have in a while, but it hasn't been enough to end the drought. And because of dredging near Detroit, the lakes are emptying faster than ever right now.

More things I gotta read later:

• The New York Times has a thorough look at their recent security problems, and security writer Nick Selby takes Symantec to task for its response.
• The Atlantic posted photos of Grand Central Station commemorating its 100th birthday this month.
• Sarah Goodyear imagines drone-proof cities in the Middle East.

Now, back to rewriting an authentication provider...

The UK's Conservative government has passed a marriage equality bill by 400 to 175:

[UK Prime Minister David] Cameron, who described gay marriage as “an important step forward for our country”, smiled broadly as the result was revealed. [Deputy PM] Nick Clegg called the vote “a landmark for equality in Britain”. [Opposition leader] Ed Miliband said it was “a proud day”.

However, the details of the vote quickly showed that Mr Cameron’s decision to push through the legislation has left him in a minority within his own party over the issue.

The result followed a debate in which several gay MPs made impassioned arguments for the change in the law. Mike Freer, the Conservative MP for Finchley and Golders Green, appealed directly to his party colleagues over the vote, declaring: “I am not asking for special treatment. I am simply asking for equal treatment.”

The United Kingdom has a state church and a right-leaning government. The U.S. has a constitutional prohibition against a state church and has a left-leaning government. Yet the UK passed marriage equality before most legislatures in the U.S.

I remember fondly when the U.S. was a beacon of freedom and tolerance in the world. Maybe it will be again, someday.

I'll be a lot less busy in March, they tell me. Meanwhile, here are some things I want to read:

• The Atlantic Cities blog has an analysis of class in Chicago by census tract.
• Seth Godin doesn't like airports, because they're organizationally horrible. Best bit: "There are plenty of potential bad surprises, but no good ones."
• Liz Keogh advocates Behavior-Driven Development as a new way of looking at Test-Driven Development.

I will get to them...soon...

Ho did the accounting firm CliftonLarsonAllen LLP miss that Dixon, Ill., comptroller Rita Crundwell embezzled $53 million?

CliftonLarson in 2005 resigned as auditor for Dixon in order to keep other city assignments such as ledger-keeping after an influx of federal funds required the town to hire an independent auditor.

In its lawsuit, however, Dixon contends that CliftonLarson continued to do the annual audit and get paid for it, while hiring a sole-practitioner CPA from nearby Sterling to sign off on the work, thereby preventing competitors from grabbing the business. CliftonLarson says it prepared only a bare-bones “compilation” of financials after 2005 to aid the new auditor, Samuel Card, 56, who also is a defendant.

In depositions in late 2012, Power Rogers attorney Devon Bruce produced CliftonLarson emails after 2005 that referred to the firm's “audit” of Dixon. Also entered into the record were invoices submitted by Ms. Crundwell supposedly from the Illinois Department of Transportation that lacked an IDOT heading or logo and, in one instance, carried a nonexistent date—Nov. 31, 2004.

“Had a two-minute phone call been made by a Clifton employee to the Illinois Department of Transportation regarding any of these false invoices, Rita Crundwell's theft would have been identified at that time,” the lawsuit argues.

Oops.

My my most recent post mentioned finishing the GetWeather component of Weather Now, my demo project that provides near-real-time aviation weather for most of the world. I thought some readers might be interested to know how it works.

The GetWeather component has three principal tasks:

• Get the raw data from the National Aeronautics and Atmospheric Administration (or, in future, any other source);
• Parse the data; and
• Store the data for the web application to use.

In the Inner Drive Technology world, an Azure worker process uses an arbitrary collection of objects that implement the IWorkerTask interface. The interface defines Interval and LastRun properties and an Execute method, which is all the worker process needs to know. The tasks are responsible for their own lifespans, reentry prevention, etc. (That's another discussion.)

In order to decouple the data source (NOAA now, other sources in the future) from the application, I split the three tasks into two IWorkerTask classes:

• The NoaaFileDownloadingWorkerTask opens an FTP connection to the NOAA public weather servers, retrieves the files it hasn't already retrieved, and stores the contents in Azure Blob Storage; and
• The NoaaFileParsingWorkerTask pulls the files out of Azure Storage, parses them, and stores the results in an Azure SQL Database and Azure table storage.

I'm using Azure storage as an intermediary between the two sides of the process because my analysis led me to the conclusion that they're really independent of each other. Coupling of the two tasks in the current (2002) version of GetWeather causes all kinds of problems, not least that a failure in one task can stop the whole thing. If, as happens given the nature of the Internet, the FTP side has an unrecoverable problem, the application has to restart. In actual practice it simply kills itself and waits for the next time it runs, which can be a while because it's running on a Windows Server 2008 Scheduler job every 30 minutes.

The new architecture will allow the parser to run every minute or two, see if it has anything to do by looking at some metadata, and do its job if needed. I can change a system setting to stop it from running (for example, because I need to do some database maintenance), while letting the downloader continue to work separately.

On the other side, the downloader can run every 5 minutes, snatch the one or two files it needs from NOAA, and shut down cleanly without waiting for the parser. NOAA likes this because the connection is only open for a few seconds, instead of the 27 minutes it stays open right now. And if the NOAA server isn't available, so what? It's a clean shutdown and a clean start a few minutes later.

This design also allows me to do something else: manually upload files for parsing and storage. This helps with testing, migration, service interruptions—all things that the current architecture has made nearly impossible.

I'm not entirely done with the application (and while writing this I just thought of an improvement I'll need to make to prevent infinite retries), but it's close. And I'm really pleased with the application so far. Stay tuned; I can now set a tentative public launch date of March 31st.

Five years ago, on 6 January 2008, I opened a FogBugz case (#528) to "Create NOAA Downloader". The NOAA downloader goes out to the National Weather Service, retrieves raw weather data files, and stores the files and some metadata in Windows Azure storage. Marking this work item "resolved"

Well, I just finished it, and therefore I have finished all of the pieces of the GetWeather application. And with that, I've finished the last significant piece of the Weather Now 4.0 rewrite. Total time to rewrite GetWeather: 42 hours. Total time for the rewrite so far: 66 hours.

Now all I have to do is...let's see...create worker role tasks to run the various pieces of the application (getting the weather, parsing the weather, storing the weather, and cleaning up the database), upgrading the Web site to a full Cloud Services application, deploy it to Azure, and deploy its gazetteer. That should be about 5 hours more work. Then, after a couple of weeks of mostly-passive testing, I can finally turn off the Inner Drive Technology Worldwide Data Center.

Via Fallows, a software designer explains how a simple feature isn't:

This isn’t off the shelf, but that’s OK — we’ll just build it, it’s not rocket science. And it’s a feature that’s nice, not one that’s essential. Lot’s of people won’t use these tabs.

So, what do we need to think about when adding a bar of tabs like this?

• The whole point is to have a view state that summarizes what you’re looking at and how it’s presented. You want to switch between view states. So we need a new object that encapsulates the View State, methods for updating the view state when the view changes or you switch tabs, methods for allocating memory for the view state and cleaning up afterward.
• You need a bar in which the tabs live. That bar needs to have something drawn on it, which means choosing a suitable gradient or texture.
• The tab needs a suitable shape. That shape is tricky enough to draw that we define an auxiliary object to frame and draw it.
• Whoops! It gets drawn upside down! Slap head, fix that.

...and on for another 16 steps. He concludes, among other things:

This is a hell of a lot of design and implementation for $0.99. But that’s increasingly what people expect to pay for software. OK: maybe $19.95 for something really terrific. But can you sell an extra 100 copies of the program because it’s got draggable tabs? If you can’t, don’t you have better things to do with your time?

He's developing for a commercial application that he sells, so he may not be figuring the costs of development the same way I do. Since clients pay us for software development, it's a reflex for me to figure development costs over time. I don't know how much the tab feature cost him to develop, but I do know that to date, migrating Weather Now to Azure (discussed often enough on this blog) would have cost a commercial client about $9,000 so far, with another $3,000 or so to go. And the Inner Drive Extensible Architecture? That's close to $150,000 of development time—if someone else were paying for it.

And all you wanted was a little tab on your word processor...