The Daily Parker

Politics, Weather, Photography, and the Dog

Fun Gazetteer facts

I meant to add this earlier today, but I had to do some work for my real job.

Uploading 15.4 million place records into Weather Now revealed some unexpected statistics. As you might expect from a military website, the National Geospatial-Intelligence Agency got a lot of its data from military sources. And the military tends to map things they care about in great detail. So the top 10 countries by place counts turn out to be:

  1. China, 2.1 million
  2. United States, 1.1 million
  3. Russia, 857,000
  4. Iran, 686,000
  5. Indonesia, 543,000
  6. Thailand, 530,000
  7. Finland, 495,000
  8. Mexico, 490,000
  9. Republic of Korea, 346,000
  10. Afghanistan, 343,000

The top 5 states by USGS place count might surprise you, too:

  1. California, 52,700
  2. Texas, 43,300
  3. Oregon, 36,800
  4. Tennessee, 33,400
  5. Arkansas, 29,800

Florida, New York, Pennsylvania, and Illinois, the 3rd through 6th most populous states, come in at 19th, 11th, 8th, and 33rd, respectively. I have no idea why.

And the smallest places? All of these NGIA files had fewer than 8 locations—probably because they contain places that are really part of some other country:

  1. Dramana and Shakhatoe, 1
  2. Kalapani, 1
  3. Minerva Reefs, 1
  4. Conejo Island, 2
  5. Geyser Reef, 2
  6. Hans Island, 2
  7. Siachen, 2

I honestly don't know why NGIA puts those places in their own files. I'll correct them eventually.

If you want to have a bit of fun, try searching for places you know and see what weather is closest.

Almost everywhere in the world

Late Tuesday night, Weather Now finished importing and indexing 15,430,045 places from around the world, ending with Mutirikwi Dam, Masvingo, Zimbabwe at 9:29 pm CST. (I need to re-import about 11,000 records for places that don't belong to any particular country, but that's low-priority.)

When I first built the Weather Now Gazetteer in July 2002, I only imported populated places, because database space was a lot more expensive then. So from 2002 until the v5 upgrade launched 3 years ago, the Gazetteer had about 7.5 million records and lived in a relational SQL database that migrated to the Cloud in 2013.

This time, I imported every scrap of geographic data I could find in the US, which added 8 million more. And starting later this year, I'll automate updates, particularly from sources that change frequently like the National Geospatial-Intelligence Agency and the Federal Aviation Administration. I may import other databases later on, but for now, I think we're good.

Pricing for Cosmos DB works differently than it used to for SQL Server. Weather Now v4 (2013-2022) used an Azure SQL DB size that cost $10 per month. That gave me up to 10 GB of relational storage for everything, which is why all the weather data moved to really inexpensive flat storage (Azure Tables) that cost about 4.5c per GB per month. When Weather Now lived on-premises through 2013, the Gazetteer and weather data took up about 50 GB (25 GB for the Gazetteer and 25 GB for the first 11 years of weather data), but it cost nothing except electricity—$100 a month of electricity, in fact.

With Cosmos DB, I pay for database transactions, called Request Units, plus a small fee of 25c per GB per month for storage. It gets a little complicated, but basically, the biggest expense for the database is the import, which cost about $75. Going forward, the biggest database expense will be the search service, which costs $2.42 per day. Storing 20 years of weather data costs $1.65 per day. Of course, the application service hosting Weather Now runs $3.60 a day, so it does all add up.

Because the daily cost summary takes a full UTC day to update, I don't have the new run rate for the application yet. When that comes out tonight, I may have to look into ways to defray the cost of the app, whether by voluntary donations or *gulp* advertising.

Incidentally, The Daily Parker costs $4.87 per day ($3.60 for the app service, 79c for storage, and 48c for the database. Now that Weather Now is pretty much where I want it, my next project will be to write a new blog engine and port this app to something that will cost about half that. So be on the lookout for a tip jar on this app as well.

Still chugging along

The Weather Now gazetteer import has gotten to the Ps (Pakistan) with 11,445,567 places imported and 10,890,186 indexed. (The indexer runs every three hours.) I'll have a bunch of statistics about the database when the import finishes, probably later tonight or tomorrow morning at the latest. I'm especially pleased with the import software I wrote, and with Azure Cosmos DB. They're churning through batches of about 30 files at a time and importing places at around 10,000 per minute.

Meanwhile, in the rest of the world:

Finally, in February 1852, a man calling himself David Kennison died in Chicago. He had clamed to be 115 years old, participated in the Boston Tea Party, and hobnobbed with the great and good in the early days of the Republic. And in the proud tradition of people giving undue acclaim to total charlatans, the entire city turned out for his funeral—173 years ago yesterday.

Some good news

First, the temperature at Inner Drive Technology WHQ has gotten up to 10.5°C for the first time since 3:33 pm on Monday December 9th. If it goes up just 0.1°C more, that will make today the warmest day since Monday November 25th. Fingers crossed.

Second, the National Geospatial-Intelligence Agency returned to DNS this morning. Someone at the Department of Defense must have noticed that the government's maps had vanished and was able to get the DNS entries restored. In consequence, I have downloaded everything through Romania, imported everything through Comoros (except China; that's importing right now), and the automatic indexer has captured 274,194 new places for a total gazetteer size of 2,668,565 places. That will rise dramatically later today; there are 2.05 million records for China, of which the import tool has already saved 1 million.

Updates as conditions warrant.

Update: As of 2:29 pm the temperature hit 10.6°C for the first time since 1:49 pm on Monday November 25th. Today is officially the warmest day of the 2024-25 winter season!

National Geospatial-Intelligence Agency removed from the Internet

By yesterday evening I managed to import all the National Geospatial-Intelligence Agency country place data through the Bs. This morning, I couldn't get to the NGIA website.

All right, sometimes these things happen. No biggie.

Yet, knowing a little about how the OAFPOTUS and Clown Prince Elon have operated the last 30 days, I did some digging. And I discovered yet another example of how imbecilic these infants are.

Simply: someone has removed the agency from the Internet. All DNS records for the agency are gone. Someone just deleted the entire domain.

NGIA has 14,500 employees, mainly in its Virginia headquarters. Its products include the NGIA name files that cartographers all over the world use to maintain their maps. Google, Microsoft, and Apple, for example, are big NGIA consumers, as you might want to know if you've ever used their maps.

I knew that the OAFPOTUS could seriously degrade Weather Now, but I thought I had a bit more time. I guess not. So if you use Weather Now and want to search for anyplace in a country whose name begins with C to Z (except Canada, Jamaica, Macau, and the UK, for reasons), I guess we'll both be disappointed.

This could just be a temporary error, like how the Clown Prince accidentally shut down vital food services here and abroad, or mistakenly halted cancer research. At least I know the data are still there, even if the computers the data live on are no longer accessible from outside Fort Belvoir. And I expect Google, Microsoft, and Apple might have something to say about this when they notice it's happened.

At least the National Weather Service still has its DNS entries. For now.

Weather Now update

Last night I released Weather Now v5.0.9183, with a few bug fixes including a patch to the Gazetteer that recognizes the UK's four constituent parts (example). I've spent a few evenings the past week and a half fixing everything I could think of in the Gazetteer code, plus integrating with Azure Maps to allow me to correct time zones and parent places.

Then, starting around 5pm yesterday, I re-imported the existing data from fresh sources, including the NCDC update Monday and the FAA update yesterday. And just now I've completed importing all 970,000 USGS names records for the 50 United States plus DC and Puerto Rico. The Gazetteer now has 1.6 million records, and there are still a lot of places to import from the National Geospatial-Intelligence Agency.

If you're curious, you can go to the About the Gazetteer page to see what all I've imported. I plan to start importing the NGIA country files this afternoon, once I've downloaded them.

The Azure Search indexer that populates the full-text search for the application runs every three hours, so most of the USGS records won't show up in the search feature until just after noon US Central time (a bit more than an hour for now).

Also, after lunch, I'll have a Butters update. Spoiler: she sleeps 21 hours a day.

New Weather Now release

I've added a couple of new features to Weather Now, including a view of the best weather in the world. More usefully, I've added a release notes page.

This weekend, I plan to to rebuild the Gazetteer, and import all of the USGS records—about 900,000 places. This will also mean rebuilding the search index in the Basic tier. The Free tier is great for testing, but it only lets me index 10,000 items at a time. Even if the Free tier could support 900,000 records, I don't want to hit "Rebuild" 90 times.

I'll also be setting up a Donate button, for obvious reasons.

Enjoy.

Open season for scammers

The OAFPOTUS's principal motivation has always been self-enrichment. He has scammed and grifted his whole life, though he sucks at it so hard he managed to burn through so much of his inheritance that he'd have been better off stuffing it in a savings account.

So it should come as no surprise that the first few weeks of his second term have seen remarkable gifts to other grifters and scammers worldwide, not to mention our adversaries:

In other stupidity directly attributable to the OAFPOTUS:

But hey, he's really intelligent, isn't he? He's the Weave-Meister! The Weave-o-Matic! Weavy Wonder! And he really does inspire all the people with sub-100 IQs out there that, someday, they too could be president.

The damage he's done in the last three weeks to our standing in the world is a gift to Russia, China, and everyone else who wants to end Pax Americana. As Krugman said yesterday, "what we’re seeing is what you’d expect if China and Russia had somehow managed to install people who wanted to sabotage America’s international position at the highest levels of the U.S. government."

It's also a testament to the Republican Party's 50-year endeavor to destroy public education. Makes me proud to be an American.

Slippery walk to the train

Chicago got a few millimeters of ice last night, which made my 15-minute walk from my house to Cassie's day camp into a 24-minute walk. The poor girl could not understand my difficulty, but she also can't count all four of her paws, so we work with what we have. Fortunately the temperature has gotten above freezing and promises to stay there at least until late tonight.

Elsewhere in the world:

In other news, I deployed an update to Weather Now last night that corrected a couple of bugs, and I also imported a few thousand places from the US Census Bureau and the US Geological Survey. By the end of February I should have the entire USGS gazetteer imported, plus a vastly expanded search service that will speed up finding places in the world to see their weather. I also hope to (finally!) allow registered users to choose measurement systems and to see where the best and the worst weather in the world is currently reported. Fun!

The good, the bad, and the stupid

First: the good. My friend Kat Kruse has a new book of her short stories coming out. She let me read a couple of them, and I couldn't wait to pre-order the entire collection. I should get it on February 17th.

Still on the good things—or at least the things that don't seem so bad, considering:

Now for the bad:

And, of course, the stupid:

I might as well finish with a good thing. The temperature has gotten all the way up to 6.2°C at Inner Drive Technology WHQ and 7.8°C at O'Hare. It was last this warm at WHQ on December 29th. If O'Hare can get up to 11.1°C, it will eke past December 27th.