The Daily Parker

The Inner Drive Technology International Data Center continues to whir away (and use electricity), despite my best efforts to shut it down by moving everything to Microsoft Windows Azure.

Most of the delay finishing the move has nothing to do with its technology. Simply, my real job has taken a lot of time this month as we've worked toward launching a new application tomorrow. Against the 145 hours spent on that project this month, not counting the 38 hours spent helping with other projects, squeezing out the 22 hours I've managed to find for Weather Now has left me falling behind on the Oscar nominees.

For those just joining our story, Weather Now remains the last living application in the IDTIDC. This application shows real-time aviation weather for almost every airport in the world. I wrote the first version in 1998, moved it to its own domain in 2000, and published the last significant update in 2010.

The application benefited for most of its life by having practically unlimited hardware and system software to run on. As a Microsoft partner, I've gotten access to Windows Server, SQL Server, and other goodies for my entire professional life. Moving to Azure changes the calculus radically.

Weather Now runs on Microsoft SQL Server 2012 Enterprise with essentially limitless disk space. In the past 14 years, the application has quietly gathered 50 GB of data, merrily occupying a physical partition scheme that takes up a good bit of a RAID-5 volume. Creating a similar architecture in Azure exceeds my budget a bit: a single medium VM to run the application and its GetWeather component plus a 50 GB SQL Database would cost about $250 per month.

Fortunately, I don't have to do that. Most of the data, you see, hardly ever gets used.

Weather Now usually has around 4,500 current weather observations and 165,000 observations from the last 24 hours. Since each row is small, and the index is positively tiny (only the station ID and observation times are indexed), the current table uses about 1.5 MB of space and the last-24 table uses 43 MB. That doesn't even break a sweat on Azure SQL Database.

No, it's the Archive table that grows like the Beast from Below. That one has all of the past observations since the site started. In some cases I've pruned the table, but basically, it has one row per observation per station. For an average station like O'Hare, that means about 10,500 per year. For a chatty, automated station that spits out a report every 20 minutes, it stores about 27,000 per year. For 2012 alone, that works out to about 47 million* rows—growing at 4 million per month.

What to do? Well, stop storing it was my first thought. It hardly ever gets used, partially because the UI doesn't have a way to pull out historical data.

On the other hand, I've frequently wanted to illustrate blog entries with specific weather reports that have permanent links. And this problem, such as it is, does not have a difficult solution.

So, among its other features, Weather Now 4.0 will store archival weather reports in Azure table storage. It won't have the full 50 GB of material initially, possibly ever; but even if it did, it would only cost about $5 per month to store it. And I've hit on a partitioning scheme that will, eventually, make finding archival data really quick and easy, no matter how much of it there is.

The conclusion should be obvious: If you start looking at things the Azure way, using Azure can save you tons of money. My current estimate of the monthly cost to run Weather Now, assuming current visitor levels and acceptable performance on "very small" Cloud Services instances, is $40 per month. If it eventually amasses 50 GB of archives, it will cost...$42 per month. And if I get thousands of visitors that require upgrading to a "small" instance, I'll start selling subscriptions, but I won't have to buy new equipment because it's Azure.

More on this later. Right now, I've got to get back to work.

* Actually, 47,704,735 rows for 2012, an average of 130,341 new rows per day.

Well, Chicago finally found out how long was the longest stretch in recorded history without a 25 mm snowfall: 335 days. The official tally through 6 am was 28 mm, which looked like this in Lincoln Park:

It really won't last. The forecast calls for 11°C by Tuesday.

For the first time in almost two years, Chicago woke up to below--18°C temperatures. We last had a day this cold on 11 February 2011, when it got down to -19°C. And we still haven't got any snow:

Lake snowfall across Michigan, despite the relatively low westerly wind-fetch (the "fetch" is the distance over which winds travel across Lake Michigan's comparatively "warm" waters) which is generating it had produced as much as 100-150 mm accumulation late Monday—and more snow is to fall there Tuesday.

Despite snowfall there, all but a comparatively small swath of downstate Illinois, Indiana and Ohio, is reporting sub-par snowfall this season. Chicago, with just 33 mm of snow to its credit, leads the pack of snow-deprived Midwest sites with just 8% of its typical seasonal snow to date--an amount 394 mm below normal.

And we're still pushing out three snow records: the longest period ever without a 25 mm snowfall (333 days, still going); the longest period ever with less than 25 mm of snow on the ground (331 days, still going); and the latest-ever 25 mm-or-greater snowfall (last broken on 17 January 1899—so we're now 5 days past the record).

Weirdest winter in memory, I tell you.

We've had a more-or-less normal 24 hours in January, with temperatures between -1°C and -11°C—bog standard.

That said, we've also had the latest sub-freezing high temperature ever (January 1st), which ended the longest-ever stretch without sub-freezing high temperatures (310 days); the second-most days in a calendar year without a sub-freezing high temperature (354); and the fourth-longest stretch without 25 mm of cumulative snow (through January 5th). More records: the longest period ever without a 25 mm snowfall (325 days, still going); the longest period ever with less than 25 mm of snow on the ground (323 days, still going); and by Thursday, given the forecast, the latest-ever 25 mm-or-greater snowfall (last broken on 17 January 1899).

Meanwhile, it snowed in Jerusalem last week, an event as common as...well, snow in Los Angeles.

Now, with more extreme weather in more places, the *New!* *Improved!* Anthropogenic Climate Change! Yay!

It's 12°C in Chicago right now, a temperature perfectly normal for April 1st, and 12.2°C above normal for January 13th.

This week's spring-like temperatures and the medium-term climate forecast make it likely that January 2013 will be the 335th consecutive month of above-normal temperatures. That means, if you're under 28 years old and lived in the U.S., you have never experienced a normal month.

Sadly, tonight things get back to normal:

Here's the forecast graph:

Boy, I can't wait for the freezing rain and sleet coming in 15 hours.

As Chicago today goes through its record 321st day without a 25 mm snowfall on its way to 13°C temperatures this weekend, we can take some comfort knowing that this summer's weather in Australia has been unimaginably worse:

Heat is part of the national mythology. It killed some of the country’s first white explorers, and has sparked many devastating fires. The worst, “Black Saturday” in Victoria, killed 173 people four years ago. Thanks to better preparation, firefighting skills and a good dose of luck, fires raging in four states in the latest heatwave have spared humans. Yet Australia is getting ever hotter. The 2013 heatwave has set a new record, 40.3°C, for the highest national average temperature. So far, Leonora, a town in Western Australia, has been the hottest place of all, at 49°C on January 9th. That is still below the highest temperature ever recorded in Australia, 50.7°C at Oodnadatta 53 years ago.

The punchline: the Australian weather service has added colors to its map to account for the heat:

The scorching conditions are set to continue into the coming week and prompted the national Bureau of Meteorology to take the extraordinary measure of revamping its weather charts. New colours have been added to forecasting maps — deep purple and pink — to mark out areas experiencing peaks above 50°C.

The colours have come in for immediate use, with large purple blotches appearing on the weather map for next Sunday and Monday. Temperatures in parts of the state of South Australia are tipped to exceed 50°C.

But hey, New Zealand looks perfectly comfortable on that map...

Today is the 319th day since Chicago has had a 25 mm snowfall, tying the record set in 1940. As our forecast calls for 10°C-plus temperatures this coming weekend, the record will just get harder to beat.

Unfortunately, the lack of snowfall is also a general lack of precipitation, so water levels in Lake Michigan has hit an all-time low:

he U.S. Army Corps of Engineers reported water levels in Lakes Michigan and Huron hit record lows in December, at nearly two and a half feet below average. Army Corps projections for lake levels have been dire since September, when it became clear that a relatively warm, dry fall and winter would not provide relief from a long drought and one of the the hottest summers ever.

Now the water is an inch below its record low for this time of year in 1964, and continues to drop. Shippers, fishermen, and small-town tourist harbors say federal help with digging out channels and repairing infrastructure could keep the low water problem from becoming a crisis.

The water will likely go back up in spring and summer, as it does every year; late winter is generally the lowest time in the lakes’ yearly cycle. But another summer of extreme heat or drought, and this winter’s woes will seem like kid stuff.

Also announced today, 2012 was the hottest year ever in the United States: "The average temperature for 2012 was 12.9°C, 1.8°C above normal and a full degree [Fahrenheit] higher than the previous warmest year recorded -- 1998 -- NOAA said in its report Tuesday."

I'm in the home stretch moving Weather Now to Azure. I've finished the data model, data retrieval code, integration with the existing UI, and the code that parses incoming weather data from NOAA, so now I'm working on inserting that data into the database.

To speed up development, improve the design, and generally make my life easier, I'm using Entity Framework 5.0 with database-first modeling. The problem that consumed me yesterday afternoon and on into this morning has been how to ramp up to realistic volumes of data.

The Worker Role that will go out to NOAA and put weather data where Weather Now can use it will receive somewhere around 60,000 weather reports every hour. Often, NOAA repeats reports; sometimes, NOAA sends truncated copies of reports; sometimes, NOAA sends garbled reports. The GetWeather application (soon to be Azure worker task) has to handle all of that and still function in bursts of up to 10,000 weather reports at once.

The WeatherStore class takes parsed METARs and stores them in the CurrentObservations, PastObservations, and ClimateObservations tables, as appropriate. As I've developed the class, I've written unit tests for each kind of thing it has to do: "Store single report," "Store many reports" (which tests batching them up and inserting them in smaller chunks), "Store duplicate reports," etc. Then yesterday afternoon I wrote an integration test called "Store real-life NOAA file" that took the 600 KB, 25,000-line, 6,077-METAR update NOAA published at 2013-01-01 00:00 UTC, and stuffed it in the database.

Sucker took 900 seconds—15 minutes. In real life, that would mean a complete collapse of the application, because new files come in about every 4 minutes and contain similarly thousands of lines to parse.

This morning, I attached JetBrains dotTrace to the unit test (easy to do since JetBrains ReSharper was running the test), and discovered that 90% of the method's time was spent in—wait for it—DbContext.SaveChanges(). As I dug through the line-by-line tracing, it was obvious Entity Framework was the problem.

I'll save you the steps to figure it out, except to say Stack Overflow is the best thing to happen to software development since the keyboard.

Here's the solution:

using (var db = new AppDataContext())
{
	db.Configuration.AutoDetectChangesEnabled = false;

// do interesting work

	db.SaveChanges();
}

The result: The unit test duration went from 900 seconds to...15. And that is completely acceptable. Total time spent on this performance improvement: 1.25 hours.

Here's the semi-annual Chicago sunrise chart. (You can get one for your own location at http://www.wx-now.com/Sunrise/SunriseChart.aspx.)

You can get sunrise information for your location at wx-now.com.

Nearly 65% of the lower 48 United States has snow cover, including parts of every state except for the seven between Louisiana and South Carolina.

Chicago, for reasons not well understood, has just a trace on the ground and has gone 314 days without 25 mm of snow, 4 days short of the record set in 1940. Since we have no significant snow in the forecast, it looks like we'll break that record too.

Other records threatened: number of days without 25 mm total snowfall accumulation, 312 (record is 313); latest date without a 25 mm snowfall, ranked 10th (record is 17 January 1899; 9th is 5 January 1994); and latest day for total 25 mm accumulation for the season, ranked 4th (record is 8 January 1944; 3rd is 6 January 1913).

And the forecast for March-like weather next week has gotten clearer, with the National Weather Service now predicting 8°C on Wednesday.

Weird weather continues.