The Daily Parker

This morning I posted about some frustrations in getting our CRM system to import donations from our fundraising events so that we can then match donations with addresses to send out end-of-year tax letters. The frustrations have grown to the point where naming names seems appropriate, if only because Neon One, the CRM company, has a web-based ticketing system that doesn't really handle the level of detail their developers will need to (a) understand the problem, (b) understand the frustration, and (c) understand the features needed to solve (a) and (b).

As you read this, keep in the back of your head that I'm a software developer with 25 years of professional experience and another 15 of hobby experience before that. In other words, I've been writing software longer than almost everyone at the CRM developer has been alive.

Neon will probably consider this a feature request, though on any of the product teams I've run in the past 15 years, this would be a usability bug report.

tl;dr: Neon's import feature is software-centric, not user-centric. Instead of the feature helping the user, it expects the user to help the software. This causes grief for any user who is not a piece of software.

The simple problem

Because Neon doesn't have adequate (or, it seems, any) support for silent auctions and other day-of-event realities, we use a different system for our fundraising events. The other system spits out perfectly reasonable Excel documents with all the information we need to track donations along with the fair-market valuations of silent auction winnings. We want to import that information into Neon so that we (a) can print out end-of-year tax letters and (b) accurately track giving in the long term.

The obvious path, which doesn't work

People have studied usability for almost as long as I've worked in software professionally. Jakob Nielsen has written about since 1998. The first principle of usability has never changed: make the obvious path work in an obvious way.

Neon has an import function that appears, at first glance, to import exactly the kind of data our event system produces. It seems like one should be able to import a flat file containing the donor name, address, email, and phone number; the date and amount of the donation; maybe a note or other optional information. You would think you could map the columns on the file to fields in Neon, and the software would read the data file and import the data. Maybe you'll get one or two spurious donor records when the information in the file doesn't exactly match an existing record's data, so maybe you'll have a few minutes of clean-up that you can do right from the import report after it's done.

Anyway, that's how I'd design it. That's how Jakob Nielsen would design it. That's how the 22-year-old newbie with a still-wet bachelor's degree in design would do it.

That's not how Neon designed it.

OK, so there may be an extra step

The first thing the Neon import feature asks is: does your data have Neon account ID numbers? If not, it warns you it won't be able to match your data with existing donor records.

Wait, what? The CRM already has a decent "find duplicate records" feature, so why can't this run automatically on new or imported entries? (Seriously, Neon: why do we spend time after every concert de-duping our data because your software doesn't think to match existing records with new ticket orders even when all of the data are the same in both records? More on this in a moment.)

All right. I'll spend 15 minutes adding Neon IDs to all the donation records in the exported file that correspond to existing donor records. It's an extra step that the software should be able to accomplish on its own, but whatever, no one likes writing record-matching code.

Now what I expect is that Neon will add the new donations to the existing records, and add new account records if I don't supply an ID.

Nope. My first pass through this process looks only at the exported records whose IDs I've provided and updates those donor records, while completely ignoring the new records. And it then takes me to a screen that looks suspiciously like it will make a total hash of the donation records in the same export file if I click "next." So I abort.

Maybe a few extra steps?

I think, perhaps I should give Neon a little extra help now. Let me scrub the data going into the import so that we have the best chance of good data getting into the CRM. So I separate the export into two files, one containing Neon IDs and one with all the new donors who came to the event. Then I go through both to make sure mailing addresses conform to USPS standards, phone numbers are uniformly 10 digits without separators, and email addresses are validly formed.

Next, I import the file that does not have Neon IDs, hoping it will create new records for me. It does! And it even exports a report containing the new IDs it created, albeit with only the donor's full name and not the donor's first and last names, which creates a bit of extra work as now I have to manually map them to my donor export.

So after I add the new IDs where needed in my donation export file, I'm ready to import the donations. I start the donation wizard, it accepts that my records have valid Neon IDs, and I map the donation date and amount columns to Neon fields. And then it tells me that I don't have a "donation type" mapped.

I'm not going to go through the steps required to figure out what a valid "donation type" is. Suffice to say, I add a column to the export called "Donation Type" and fill every row with the value "Donation". (Why can't we just specify constant values for required fields at import?)

I go through the import wizard again, map everything required, and hit import. Oof! Error! Apparently, dates are hard. The export file has donation dates in the format "Jul 17, 2020 7:27:19 PM", but Neon says "Time field must be a date;now it supports 'MM/DD/YYYY','MM-DD-YYYY' and 'MMDDYYYY' format."

Now, without going into the rabbit hole too deeply, a few things immediately occur to me as someone who has successfully parsed dates for a quarter century in a half-dozen programming languages. First, why the fuck does Neon only accept those three formats when the format presented is unambiguous and can be parsed by nearly every programming language out there? Second, none of the formats presented or accepted in this case conform to ISO-8601, the international standard for date and time representation, so everyone is on shaky ground here. Third, I got all the way to this point and now it tells me I have to go all the way back and change the date format by hand? Because it turns out, Excel can't parse the dates either. Good work, 3rd-party event package. Nicely done.

Once more unto the breach, dear friends, once more;
Or jam the import with our ISO dates.

After entering all the dates by hand (made easier by 90% of them being the date of the event), I re-exported the file to .csv and tried the import again. (Oh, yes, Neon can't read XLSX files, so I have to export to CSV every. Single. Time.)

Success! Finally!

And now I get to do it again with the silent-auction winners list. All of it. Again. This time I just entered the 9 new accounts by hand, and discovered 7 duplicate accounts that had to be merged, and thanked the universe we only had 60 silent-auction items.

So I go to import and...Goddammit I forgot the "donation type" field.

So I go to import again. And it mostly worked. Except even though I specified which field contained the fair-market value to map against the donation, Neon ignored that. It appears nowhere in the individual donation records. Why even present the field as an option if...I mean...what the hell, Neon? Yet more shit I have to map by hand later on.

But wait, it still doesn't work

After all that effort, I did some spot-checks on various accounts and found that even though none of the donation records shows fair-market value for auction items, at least all of the donations that I expect to see in each record appear in each record. So now it's time for the 2020 donor report, and...

Um...you're fucking kidding me.

None of the imported donations shows up in the donor report's "2020 Donation Amount" field. After checking a few donor records, I promote my hypothesis (that Neon groups by the record-creation date for the annual sum instead of the actual donation date) to a theory. Neon support case #00316491 is born.

How Neon should have coded this

Many, many developers have solved this problem before. Importing donations should require only one pass through the Import feature, and follow this heuristic:

1. As soon as the user selects "import donations" from whatever UI control offers the choice, Neon presents a page of simple documentation explaining the process, listing the required fields, and offering some insight into how it will work.
2. The user selects the import file, which can be CSV, Excel, or any other common delimited format.
3. The user maps all of the relevant columns from the import file to Neon fields. Neon does not present the user with fields that it will ignore for no stated reason later on.
4. The user provides constant values (from drop-downs if necessary) for required fields that do not appear in the import file.
5. The user clicks "upload."
6. For each record:
   1. If the imported donor account has an ID that matches an existing account by ID, use that donor account.
   2. If the imported donor account matches the name and email of an existing account (or some other criteria), use that donor account but flag the row for review.
   3. If the imported donor account does not match an existing account on either criteria, create a new donor account.
7. Stage the donation record with the specified, found, or new account ID, including all of the fields that the user mapped in step 3.
8. Present the list of "for review" items to the user before committing the import, allowing the user to make edits to the imported data, or move back a step in the process.
9. Once the user is satisfied, the user clicks "commit" to write the data to Neon.

I get it: Neon's dev group have process problems

I mean, guys, this really isn't that hard. You want hard? Write an IBM360 to MS-DOS import, complete with different endian values, and where mapping has to be hard-coded because configuration files haven't been invented yet.

I know how software development works. I expect any Neon folks reading this may think this is unfair, that the devs told management the features weren't finished, that management told leadership the devs weren't 100% finished but the stuff works well enough, and that leadership looked at the growing list of must-have features for the brochure and forgot to finish the must-have features for the users, that "it's not my fault." I'd also bet you a dollar that any dev reading this will think "I told you so" (unless they think "it works on my machine, you DFU," in which case you have other problems.)

In other words, you guys have a process problem. Somewhere the definition of "done" that passed QA for the import features didn't match the definition of "done" that users need.

For this we're paying $3600 a year. NB: we're willing to pay a lot more for software that works the way we need it to.

Meanwhile, though, I'll have to hand-correct what all this automation should have given me already, and get the damn tax letters out.

And Neon CRM support incident #00316497 is born.

Facebook reminded me this morning that 10 years ago today I got the first digital camera I've ever owned whose photo quality approached that of the film cameras I had growing up. My new Canon 7D replaced my 5-year-old Canon 20D, and between the two I took over 32,700 photos in just over nine years. In May 2015 I upgraded to the Canon 7D mark II, the first digital camera I've owned whose capabilities exceeded my 1980s and 1990s film cameras.

I've updated the chart showing all the photo-capable devices I've owned since I got my first SLR in June 1983, along with other data showing, to some extent, how technology marches on:

Here are four example photos. (To see all the details, right-click the photos and open them in separate windows.) First, one of the earliest photos I took with my AE-1 Program, in Raton Pass, N.M., mid-August 1983:

Keep in mind, this is a Kodachrome 64 photo scanned some 38 years later, with a bit of help from Adobe Lightroom. Printing directly from the slide would make a better-looking photo...maybe. In any event, the resolution of the slide exceeds the resolution of the scan by an order of magnitude at least, so there really is no way without specialized equipment to produce a JPEG image that looks as good as the slide itself.

Jump ahead a few decades. Here's an early photo I took with the 20D on 20 May 2006 in Portsmouth, N.H.:

The original photo and this edit have the same resolution (2544 x 1696) and the same format (JPEG). Other than a few minor burns and dodges, this is what the camera recorded. It almost approaches film quality, but had I shot this image with Kodachrome 64, it would have much more vibrant color and a depth of texture that the 20D just couldn't achieve.

Now from the 7D that I got 10 years ago today, near Saganonomiyacho, in Kyoto, Japan, in November 2011:

With the first 7D, I gave it a 32 GB memory card and switched to the lossless CR2 format. The JPEG above has as much depth and range as a JPEG can have, but the CR2 file it came from finally has as much detail and photographic information as consumer-quality negative film from the 1980s or 1990s. Its 5184 x 3456 resolution comes awfully close to the density of, say, 100-speed Kodacolor VR-G from the mid-1980s, but the 7D's CMOS chip has literally 32 times the sensitivity of the fastest consumer film then available (ISO 12,800 vs. ISO 1600). I shot the photo above with a focal length of 250 mm from a bridge 250 meters from the subject, at ISO 1600, 1/500 second at f/5.6. The same shot on Kodacolor VR-G 1600 would have massive grain, and the same shot on Kodachrome 64 would have required an exposure of 1/15 second—guaranteeing camera shake. (I've re-edited this photo slightly from the quick-and-dirty treatment I gave it in my Tokyo hotel room after getting back from Kyoto.)

Finally, take a look at this photo from my current camera, the 7D Mark II, of the Chiesa de San Giorgio Maggiore, Venice:

I mean...wow. Even cropped slightly from the raw photo's 5472 x 3648 resolution, the detail is just as fine as Kodachrome 64 ever gave me. I shot this at 1/1000 sec., f/5.6, at 105 mm using a borrowed EF24-105 f/4L lens. (I posted a similar shot in June 2015 when I got back from the trip. I think this one has better composition and editing.)

One more thing, which I won't illustrate with a comparison photo but which I do think bears mentioning: these days, I only pull out the 7Dii for serious work. For day-to-day photos and snapshots, my smartphone's camera works better than digital SLRs from 15 years ago. We do live in the future.

Yesterday I predicted that I would not get 10,000 steps for the first time in 2021. I was right: I got 7,092. Respectable, but not a goal.

Right now I'm at 4,753, so I could get to 10,000 just by going for a 30-minute walk and then doing normal things the rest of the day. Of course, it's -15°C outside, an improvement over this morning's -22°C but still so cold that only obscenities suffice to describe how cold.

OK, I can do this...I just don't want to.

At least, I don't think so. I'm about to go to a very small wedding where somehow we'll all stay two meters apart, meaning I'll be in my car or indoors all day. Outdoors, meanwhile, it's -13°C. It got down to -16°C overnight, so this qualifies as an improvement.

I was hoping to make 10k steps every day this year, but living in Chicago and having some ability to balance "would like to" against "have to" goals, I say no to today.

I will get 5,000 though. I haven't missed that number in six years.

These are just some of the things I read at lunch today:

• Ezra Klein looks at how a $1.9 trillion proposal got through the US Senate and concludes the body has become "a Dadaist nightmare."
• Several groups of ice fishermen, 66 in total, found themselves drifting into Green Bay (the bay, not the city) yesterday, when the ice floe they were fishing on broke away from the shore ice. Given that Lake Michigan has one of the smallest ice covers in years right now, this seems predictable and tragic.
• Writing in the Washington Post, Bruce Schneier laments that government security agencies have to customize President Biden's Peloton stationary bicycle to make it safe to use in the White House—not because of the effort involved to keep the president safe, but because very few people will have a Peloton with that level of security.
• The resident Orca population in the Salish Sea between British Columbia and Washington has immigration issues and declining standards of living. (So far, none of them has joined the Proud Whales.)

Finally, McSweeney's translates US Representative Marjorie Green's (R-GA) non-apology for being a racist whacko into simpler terms.

Writing for Just Security, Yale philosophy professor Jason Stanley decodes the structural and content similarities between the propaganda film that introduced the XPOTUS on the Ellipse on January 6th and the propaganda films a certain central-European country produced in the first half of the previous century:

Fascism is a patriarchal cult of the leader, who promises national restoration in the face of supposed humiliation by a treacherous and power-hungry global elite, who have encouraged minorities to destabilize the social order as part of their plan to dominate the “true nation,” and fold them into a global world government. The fascist leader is the father of his nation, in a very real sense like the father in a traditional patriarchal family. He mobilizes the masses by reminding them of what they supposedly have lost, and who it is that is responsible for that loss – the figures who control democracy itself, the elite; Nazi ideology is a species of fascism in which this global elite are Jews.

The future promised by the fascist leader is one in which there are plentiful blue collar jobs, reflecting the manly ideals of hard work and strength.

Fascism uses propaganda as a way of mobilizing a population behind the leader. Fascist propaganda creates an awesome sense of loss, and a desire for revenge against those who are responsible. The goal of fascist propaganda is to mobilize a population to violently overthrew multi-party democracy and replace it with the leader.

This history, both European and American, illuminates the dangers we face today, laid bare in the video. In it, Trump is repeatedly represented as the nation’s father figure. It is laced through with images of masculinity, and mournful loss at the hands of traitors, clearly justifying a violent restoration of recent glory.

The message of the video is clear. America’s glory has been betrayed by treachery and division sown by politicians seeking to undermine and destroy the nation. To save the nation, one must restore Trump’s rule.

We didn't dodge a bullet on January 6th; the shooter just missed, and not by much.

It's exactly 0°C in Chicago this afternoon, which is a bog-standard temperature for February 3rd. And it's sunny, which isn't typical. So, with the forecast for a week of bitter cold starting Friday evening, I'm about to take a 30-minute walk to take advantage of today's weather. First, though:

• Trump political appointees who knew or should have known they would lose their jobs on January 20th are throwing tantrums because they lost their parental leave benefits at noon that day, despite other Trump appointees telling them previously that they wouldn't.
• David Leonhardt investigates why Democratic presidents typically oversee better economies than Republican ones. (Hint: which party repeatedly tanks the government, and which party typically puts it back together?
• An aluminum can shortage has given local breweries headaches and local beer distributors backaches, with no respite in sight.
• The second Covid-19 shot "is a rude awakening for immune cells," which means it won't feel great but it will confer live-saving benefits.

Early February is also the time of year when we start imagining spring. Tomorrow's sunrise is at 7am for the first time since December 1st, and we had 10 hours of daylight last week for the first time since mid-November. Yes, Chicago typically has an Arctic blast sometime during February. But Spring begins in 25 days. We can make it.

Senate Democrats gave the opposition three whole days to stop dicking around with the latest Covid-19 relief package. Then today, with no more than a shrug, they told the Republicans they're tired of the crap:

Senate Democrats took the first step Tuesday toward passing a $1.9 trillion stimulus bill without Republican support, advancing their efforts to avoid a GOP filibuster.

The vote to kickstart the budget reconciliation process, which passed 50-49, is a sign that leadership expects to have the full Democratic caucus on board for the final package.

The vote comes a day after President Joe Biden met with a group of Senate Republicans, who are offering a $618 billion counterproposal. Although Biden told Senate Democrats Tuesday on a private caucus call that the meeting went well, he also said the Republican proposal is not sufficient, according to sources on the call.

Economist Paul Krugman has already explained the ways the GOP's $618 billion "offer" wasn't serious:

It’s not just that the G.O.P. proposal is grotesquely inadequate for a nation still ravaged by the coronavirus pandemic. Beyond that, by their behavior — not just over the past few months but going back a dozen years — Republicans have forfeited any right to play the bipartisanship card, or even to be afforded any presumption of good faith.

But what about bipartisanship? As Biden might say, “C’mon, man.”

First of all, a party doesn’t get to demand bipartisanship when many of its representatives still won’t acknowledge that Biden won legitimately, and even those who eventually acknowledged the Biden victory spent weeks humoring baseless claims of a stolen election.

Complaints that it would be “divisive” for Democrats to pass a relief bill on a party-line vote, using reconciliation to bypass the filibuster, are also pretty rich coming from a party that did exactly that in 2017, when it enacted a large tax cut — legislation that, unlike pandemic relief, wasn’t a response to any obvious crisis, but was simply part of a conservative wish list.

Yes. It only took, what, 12 years? But our party's leadership have finally figured out not to play this game. We're not giving Lucy the football on this one.

I've often compared this era in American politics with previous eras, most notably the Antebellum period from Jackson on. Yale historian Joanne Freeman zeroes in on 1850—at least as far as our representatives' behavior in Congress

Although couched in calls for unity, [Republican] warnings are remarkably one-sided. There is no talk of reconciliation or compromise. No acceptance of responsibility. Lots of blame casting. And little willingness to calm and inform their base. Even now, some Republicans refuse to admit that Joe Biden won the election, and the Senate vote on an impeachment trial on Jan. 26 suggests that most Republicans want no investigation and will place no blame. They want reconciliation without apologies, concessions without sacrifice, power without accountability.

This is bullying as politics, the modus operandi of our departed chief. Hardly a Trumpian innovation, its heyday was in the decades before the Civil War. During the 1840s and 1850s, America was divided over the fate of slavery. Political parties were splintering under the strain. National institutions were struggling — and failing — to contain it. The press sensationalized the struggle to serve a cause and sell papers.

Southerners had long dominated the national government, and they felt entitled to that power. They were also worried about losing it, a product of the rise of abolitionism and potentially free states forming in the West. Viewing their violence as defensive and justifiable, they fended off attacks on slavery with force when necessary, demanding compliance or silence from their foes, and sometimes getting it.

For some Southerners, this brand of politics wasn’t much of a stretch; their slave regime was grounded in violence, and mastery was their way of life. Between 1830 and 1860, there were at least 70 violent incidents on the House and Senate floors, most of them prompted by Southerners.

With the centuries-long history of Southern white politicians advocating for Southern white minority power, does their behavior today surprise anyone? Freeman concludes:

It needs to assert the power of democratic institutions and the rule of law with a comprehensive investigation into the attack and punishment of its enablers, not in a spirit of revenge, but justly, fairly and consistently. It’s stunning that it needs to be said, but it must be: People who stage, support, or incite violent attacks on the federal government — whether they’re American citizens, members of Congress or the president himself — should be held accountable, not only to restore order in the present, but to fend off similar attacks in the future. Uniting the nation requires no less.

Yes. Lest we find ourselves lurching through the 1850s again. We all know how that ended.

Annalee Newitz, author of Four Lost Cities, explains that urban collapse doesn't look anything like dystopian fiction would have it:

It’s always lurking just around the corner, seductive and terrifying, but it never quite happens. Lost-city anxieties, like the ones aroused by the pandemic, result from a misunderstanding of what causes cities to decline. Pandemics, invasions, and other major calamities are not the usual culprits in urban abandonment. Instead, what kills cities is a long period in which their leaders fail to reckon honestly with ongoing, everyday problems—how workers are treated, whether infrastructure is repaired. Unsustainable, unresponsive governance in the face of long-term challenges may not look like a world-historical problem, but it’s the real threat that cities face.

This slow-motion catastrophe—a combination of natural disaster and political indifference—was far more important to [Angkor's] transformation than the Ayutthaya invasion [in 1431]. And it stands as a warning to many cities in the U.S. Without a coherent response from local government, cities lashed by climate change will gradually lose their populations. The demise won’t be spectacular, even if the storms are monstrous. Instead, people will leave in dribs and drabs, and the exodus could take generations.

So, I'm going to stay in Chicago, which will likely remain a thriving urban center for hundreds more years.