The Daily Parker

Politics, Weather, Photography, and the Dog

He is serious, and don't call him Shirley

I had a mind-numbing email exchange with a large corporate IT department today.

One of our best customers has a problem: no one has been able to use our software since Friday. We’ve been troubleshooting this problem. But we haven’t been able to fully investigate the issue, despite tremendous effort. We think we've uncovered the main issue preventing us from fixing the main issue.

We couldn't connect to either their production or user-acceptance test (UAT) Web services from inside our office because (we thought) their IP whitelist (a list of Internet addresses allowed to connect to them) didn't have our office on it. We got in touch with one of their developers—let’s call him "Bugs"—and requested the change. He responded that he had forwarded the request to his network team and we should expect whitelisting in 2-3 weeks.

I responded that I would be happy to get anyone else at his company involved to see about resolving this matter more quickly. Anyone, for example, like the division president who happened to be visiting Chicago this week. Or the vice-president whose users couldn't connect. Bugs, in turn, set up a call with their network team for the next morning, India time. So on our 9:30pm call last night (8am for them), we got sufficient firepower from inside their organization to get whitelisted wiki-wiki.

This morning (US time), we again attempted to run our tests against their UAT environment, and again could not connect. We could, however, connect to their Production environment. So their network guys did something, we just couldn't tell what. At least we can start trying to reproduce the production issue.

I sent another email to Bugs: “We can reach Production but not UAT. We are able to hit {production URL}, but not able to hit {UAT URL}. We’re seeing an error message that there is no endpoint listening at {UAT URL}. Is UAT offline or in an unstable state?”

Bugs sent back a message from his network guys showing a screen shot of their whitelist. Sure enough, the network guys did their jobs; we were whitelisted to both environments.

I responded, “OK, but we still can’t connect to {UAT URL}. Is it possible the UAT environment is offline?”

Bugs chewed on his carrot for a moment and responded, “We have stopped the UAT environment.”

O___O

So now I’m weighing responses. I have “Would you please start the UAT environment?” as my opening sentence, but I’m stuck on how to proceed. Options include:

  1. “We anticipate a higher likelihood of successfully testing against it if it’s running."
  2. "Ah, so this is a PEBCAK problem after all. Please address the issue between your chair and keyboard and try the operation again."
  3. "Now you're just fucking with me."
  4. "I don't know who you are. I don't know what you want. If you're looking for ransom, I can tell you I don't have money but what I do have are a very particular set of skills. Skills I have acquired over a very long career. Skills that make me a nightmare for people like you. If you let my UAT environment go now, that will be the end of it. I will not look for you, I will not pursue you. But if you don't, I will look for you, I will find you and I will kill you."

At some point I just expect him to respond with, "A møøse bit my sister once," but I'm also fairly certain he has no demonstrable sense of humor.

Ah, an email just came in from his supervisor's supervisor. Apparently they're restarting the UAT environment. I will now resume my labors.

Update: Once we finally connected to the UAT environment, we got back an error message, clearly indicating that their environment is misconfigured. Not that we were suspicious of Bugs' insistence that nothing at all has changed on their end, but once we got this particular error message, we forwarded the details to his team with the suggestion that they check their logs too. It's just past 2am IST so we expect that nothing much will happen until later tonight.

Comments are closed