The Daily Parker

Weather Now is fully deployed to the Cloud. As soon as the Worker Role finishes parsing the last few hours of weather, I'll cut over the DNS change, and it will be live.

Actually, that's not entirely true; I'm going to cut over the DNS in the morning, after I know I fixed the bugs I found during this past week's shake-down cruise.* So if you want to see what a weather site looks like while it's back-filling its database, you can go to its alias, http://wx-now.cloudapp.net. (Because of how Azure works, this will remain its alias forever.)

Time to meet my friends, who are wondering where I am, no doubt.

* Bugs fixed: 13. Total time: 6.9 hours (including 2.4 to import and migrate the Gazetteer).

After a couple of days in which I'm glad we keep bourbon in the 10th Magnitude office, Scott Hanselman's examination of working remotely seems timely:

I see this ban on Remote Work at Yahoo as one (or all) of these three things:

• A veiled attempt to trim the workforce through effectively forced attrition by giving a Sophie's Choice to remote workers that management perceives as possibly not optimally contributing. It's easy to avoid calling it a layoff when you've just changed the remote work policy, right?
• A complete and total misstep and misunderstanding of how remote workers see themselves and how they provide value.
• Pretty clear evidence that Yahoo really has no decent way to measure of productivity and output of a worker.

All this said, it's REALLY hard to be remote. I propose that most remote workers work at least as hard, if not more so, than their local counterparts. This is fueled in no small part by guilt and fear. We DO feel guilty working at home. We assume you all think we're just hanging out without pants on. We assume you think we're just at the mall tweeting. We fear that you think we aren't putting in a solid 40 hours (or 50, or 60).

Because of this, we tend to work late, we work after the kids are down, and we work weekends. We may take an afternoon off to see a kid's play, but then the guilt will send us right back in to make up the time. In my anecdotal experience, remote workers are more likely to feel they are "taking time from the company" and pay it back more than others.

I like working from home when I have a lot of creative or intense work to do, but generally I prefer working in the office. I've also been thinking about the compromise solution of moving to within, say, 500 meters of the office, so I can get home in 5 minutes if I need to.

Meanwhile...back to work.

I don't have time to read these must-read articles:

• Noam Schreiber on Aaron Swartz
• Marlene Zuk on our fascination with the paleolithic
• Sean Flynn on Pope Benedict's butler
• Microsoft on the Azure outage a year ago (interesting because it may be related to last week's outages
• Jennifer Lawrence rocks

Back to the mines...

Unfortunately, that's not going to happen for a while. I'm going to spend a lot of time in airplanes over the next 11 days, including a long weekend with the folks. Good thing wifi is ubiquitous, even on airplanes, because it also looks like I'm going to burn at over 120% of utilization again this month. (Last month I was 118% billable, but if you add non-billable time I actually worked 134% of full time.)

The madness ends soon. We're hiring, projects are gelling, other projects are winding down, and at some point I'll just get on a plane for four days without taking my laptop.

I did take three hours yesterday to play pub trivia with my droogs, owing to the start of a four-week trivia tournament. We're in second place—by one point. I sincerely hope to make the next three Thursdays.

Five years ago, on 6 January 2008, I opened a FogBugz case (#528) to "Create NOAA Downloader". The NOAA downloader goes out to the National Weather Service, retrieves raw weather data files, and stores the files and some metadata in Windows Azure storage. Marking this work item "resolved"

Well, I just finished it, and therefore I have finished all of the pieces of the GetWeather application. And with that, I've finished the last significant piece of the Weather Now 4.0 rewrite. Total time to rewrite GetWeather: 42 hours. Total time for the rewrite so far: 66 hours.

Now all I have to do is...let's see...create worker role tasks to run the various pieces of the application (getting the weather, parsing the weather, storing the weather, and cleaning up the database), upgrading the Web site to a full Cloud Services application, deploy it to Azure, and deploy its gazetteer. That should be about 5 hours more work. Then, after a couple of weeks of mostly-passive testing, I can finally turn off the Inner Drive Technology Worldwide Data Center.

Via Fallows, a software designer explains how a simple feature isn't:

This isn’t off the shelf, but that’s OK — we’ll just build it, it’s not rocket science. And it’s a feature that’s nice, not one that’s essential. Lot’s of people won’t use these tabs.

So, what do we need to think about when adding a bar of tabs like this?

• The whole point is to have a view state that summarizes what you’re looking at and how it’s presented. You want to switch between view states. So we need a new object that encapsulates the View State, methods for updating the view state when the view changes or you switch tabs, methods for allocating memory for the view state and cleaning up afterward.
• You need a bar in which the tabs live. That bar needs to have something drawn on it, which means choosing a suitable gradient or texture.
• The tab needs a suitable shape. That shape is tricky enough to draw that we define an auxiliary object to frame and draw it.
• Whoops! It gets drawn upside down! Slap head, fix that.

...and on for another 16 steps. He concludes, among other things:

This is a hell of a lot of design and implementation for $0.99. But that’s increasingly what people expect to pay for software. OK: maybe $19.95 for something really terrific. But can you sell an extra 100 copies of the program because it’s got draggable tabs? If you can’t, don’t you have better things to do with your time?

He's developing for a commercial application that he sells, so he may not be figuring the costs of development the same way I do. Since clients pay us for software development, it's a reflex for me to figure development costs over time. I don't know how much the tab feature cost him to develop, but I do know that to date, migrating Weather Now to Azure (discussed often enough on this blog) would have cost a commercial client about $9,000 so far, with another $3,000 or so to go. And the Inner Drive Extensible Architecture? That's close to $150,000 of development time—if someone else were paying for it.

And all you wanted was a little tab on your word processor...

I have a new post up at my employer's developers blog.

Hard-core Daily Parker enthusiasts may have seen it already. Still, click through to XM. We like blog visitors!

The Inner Drive Technology International Data Center continues to whir away (and use electricity), despite my best efforts to shut it down by moving everything to Microsoft Windows Azure.

Most of the delay finishing the move has nothing to do with its technology. Simply, my real job has taken a lot of time this month as we've worked toward launching a new application tomorrow. Against the 145 hours spent on that project this month, not counting the 38 hours spent helping with other projects, squeezing out the 22 hours I've managed to find for Weather Now has left me falling behind on the Oscar nominees.

For those just joining our story, Weather Now remains the last living application in the IDTIDC. This application shows real-time aviation weather for almost every airport in the world. I wrote the first version in 1998, moved it to its own domain in 2000, and published the last significant update in 2010.

The application benefited for most of its life by having practically unlimited hardware and system software to run on. As a Microsoft partner, I've gotten access to Windows Server, SQL Server, and other goodies for my entire professional life. Moving to Azure changes the calculus radically.

Weather Now runs on Microsoft SQL Server 2012 Enterprise with essentially limitless disk space. In the past 14 years, the application has quietly gathered 50 GB of data, merrily occupying a physical partition scheme that takes up a good bit of a RAID-5 volume. Creating a similar architecture in Azure exceeds my budget a bit: a single medium VM to run the application and its GetWeather component plus a 50 GB SQL Database would cost about $250 per month.

Fortunately, I don't have to do that. Most of the data, you see, hardly ever gets used.

Weather Now usually has around 4,500 current weather observations and 165,000 observations from the last 24 hours. Since each row is small, and the index is positively tiny (only the station ID and observation times are indexed), the current table uses about 1.5 MB of space and the last-24 table uses 43 MB. That doesn't even break a sweat on Azure SQL Database.

No, it's the Archive table that grows like the Beast from Below. That one has all of the past observations since the site started. In some cases I've pruned the table, but basically, it has one row per observation per station. For an average station like O'Hare, that means about 10,500 per year. For a chatty, automated station that spits out a report every 20 minutes, it stores about 27,000 per year. For 2012 alone, that works out to about 47 million* rows—growing at 4 million per month.

What to do? Well, stop storing it was my first thought. It hardly ever gets used, partially because the UI doesn't have a way to pull out historical data.

On the other hand, I've frequently wanted to illustrate blog entries with specific weather reports that have permanent links. And this problem, such as it is, does not have a difficult solution.

So, among its other features, Weather Now 4.0 will store archival weather reports in Azure table storage. It won't have the full 50 GB of material initially, possibly ever; but even if it did, it would only cost about $5 per month to store it. And I've hit on a partitioning scheme that will, eventually, make finding archival data really quick and easy, no matter how much of it there is.

The conclusion should be obvious: If you start looking at things the Azure way, using Azure can save you tons of money. My current estimate of the monthly cost to run Weather Now, assuming current visitor levels and acceptable performance on "very small" Cloud Services instances, is $40 per month. If it eventually amasses 50 GB of archives, it will cost...$42 per month. And if I get thousands of visitors that require upgrading to a "small" instance, I'll start selling subscriptions, but I won't have to buy new equipment because it's Azure.

More on this later. Right now, I've got to get back to work.

* Actually, 47,704,735 rows for 2012, an average of 130,341 new rows per day.

Sigh. Zipcar, short-term rental car service that has occasionally made my life a lot easier, just got swallowed by Avis:

Zipcar Inc. has been growing as more people in urban areas forgo owning a car and instead tap car-sharing and hourly rental services when they need a vehicle. The company’s third-quarter sales grew 15% to $78.2 million while its membership (renters) grew 18% to more than 767,000. Zipcar earned $4.3 million in the three-month period and has said it expected 2012 to be the first full year for which it posts a profit.

Avis Budget, the nation’s third-largest car rental company -- after Enterprise Holdings and Hertz -- will pay $12.25 a share in cash for Zipcar, a 49% premium over the stock's closing price Monday.

Avis believes it can whittle $50 million to $70 million of expenses out of the combined operations of the companies by eliminating duplication of functions such as the cost of maintaining Zipcar as a publicly traded company.

Just once I'd like to see a cool, niche company grow to a sustainable size without being acquired by a huge corporation.

I don't know how Amazon figures out what to recommend, nor do I know who's buying what from them. Sometimes I wonder, though, like when it gives me these helpful suggestions:

But of course it's important to have cookies when the revolution comes...