Strange Azure Tables behavior

I had a maddening experience last week in this app's dev/test environment: Azure Table Storage stopped working. But it only stopped for table storage, and only in one storage account. I tried everything I could think of to reach it, but it stubbornly refused all read and write requests, even though it allowed me to list the tables. So I could see the tables but not what was in them. This seemed a lot like an authorization problem both to Google's and Anthropic's AIs, and I could find nothing by searching sixteen different ways from Tuesday if anyone else had seen a similar problem.

Without table storage, the application has no indices, so it can't find any data by date range or tag. This means, without table storage, the app appears to have no data. Oh, if you know the ID or permalink of an item, you can still navigate directly to the item (though it won't have any tags). Except that because the app couldn't write to table storage either, it crashed trying to write event and security logs.

Also, because it was a dev/test envirnonment, I hadn't turned on any kind of redundancy. So it appeared I had lost all the dev/test data.

This prompted me to do a couple of things. First, I created a new storage account and confirmed that everything worked normally. Next, I created a utility that can re-scan a project and rebuild its indices, though tags and comments would still be inaccessible. Once I switched over to the new storage account and re-indexed, the app worked fine again.

Last, I did one thing that turns out probably fixed the problem. I turned on zone-redundant storage, the least robust redundancy setting that actually has redundancy. This forced Azure to copy everything in the storage account to another set of blades in the Central US data center. And that, apparently, restored all the data.

So, the dev/test environment is back on its original storage account, and everything appears to work fine.

I still have no idea what caused the problem, and I didn't want to spend $150 submitting a support request to find out. It was probably local corruption on the blade, which is why I thought turning on ZRS would actually solve the problem. I may have been correct in my diagnosis or I may have been completely wrong, but the fix worked. If you have an infection and it goes away when you take penicillin, it was probably bacterial, but you'll never know for sure.

Also, the production environment has always had geo-redundant storage, meaning a copy of the data is maintained in a different data center entirely. For Central US (West Des Moines, Iowa), I believe the backup location is South Central US (San Antonio, Texas). If the production storage account goes down, we have much bigger problems in the world than just the Daily Parker not working.