Analytics

Tuesday, August 30, 2011

The Lost Weekend in the Baltimore Suburbs

There have been, according to news reports among over 500 email messages piled up during my absence from the Internet, some 4 million or so people without power since the weekend. Given a handful of fatalities, people getting flooded out of their homes, and businesses and employees affected, I shouldn't complain about being without electricity (and the Internet) for a few days, but it did affect my publishing my daily update (for Sunday and Monday) for the first time, I believe, since I started publishing on a daily schedule back in November 2009. I intend to publish my first regular daily update since the interruption in service later this evening.

Occasionally I'll depart from commenting on politics in this blog or weave in some personal stories with my commentary, which I trust regular readers find interesting. I didn't feel this post really fit into a simple segment, so I'm titling the post separately.

I knew about the news coverage of Hurricane Irene before I went to sleep early Sunday morning; Hurricane Irene had touched ground in North Carolina and was heading up the coast. Civil authorities had canceled air traffic to and from the major NYC airports by Sunday. I usually get prolific email notifications from various weather email subscriptions (e.g., Accuweather), and there was nothing of an unusual nature regarding local storms. I heard an occasional splatter of raindrops on my windows a couple of times, but nothing out of the ordinary when I shut down my PC in the early hours Sunday morning.

The point is, I went through Hurricane Alicia while a UH doctoral student back in 1983. The windows of my apartment rattled so hard, I thought they should shatter at any time, and I spent most of my time in my bathroom while the hurricane passed. As I recall, it took about 3 to 5 days to restore electricity and I remember trying to drive around a day or two later, finding things like huge tree branches blocking roadways. There was nothing like this here; from what I see in news reports, the hurricane lost its momentum after hitting North Carolina. I'm not underestimating this was a very bad storm with flooding in New Jersey and New York. As I write, hundreds of thousands of Maryland residents are still without power service.

When the sunlight woke me, roughly 7AM that morning, I automatically checked the time on my digital alarm clock--which was blank. Not good. Looking outside, it looked like your everyday sunny summer day. No hint of any severe damage anywhere. We have lots of trees on the apartment property and there was nothing unusual; I later did see a couple of low-lying branches on a couple of trees, but the landscape workers had been pruning trees. My digital home phone was out, of course; I have a low-minutes cellphone plan with no real Internet capabilities.

Over Sunday and yesterday, I must have called 8 to 10 times trying to get through to BGE (my local utility) and only got through once to report my outage and to get a status report; in hindsight, I think this was the most unsatisfying and unacceptable part of this experience. The only status I heard was a self-serving one talking about thousands of workers on the job restoring service, their first priority was their workers' safety, and residential service will be restored when it is restored.  The point is, what customers expect when these things happen is some degree of status in practical terms--hours, days, weeks? Am I going to lose everything in my freezer? I want to point out, by the way, electricity companies plan for just this sort of thing; there's a long history of weather disruptions, and and if I heard a utility CEO ever say anything as lame as "the world is complex", he or she would be fired so fast, heads would spin. There is NO excuse in this information age for you ever to hear, like I did every time except once. "The circuits are all busy; please try again at a later time." There are ways of extending technologies to handle predictably large call volumes; for computer applications, for example, we can deploy dozens of middle-tier application servers to cope with very large numbers of user connections.

In my articles on usability, I point out a key element of providing feedback to the user, especially when one is dealing with abstract processes. It's important in providing feedback that you provide context, e.g., "as of 7PM Sunday, xx of yy service areas are without power. So far jj service areas have been restored, and we expect zz service areas to be restored by tomorrow. To find out more detailed service area status, try on the line to enter your account phone number..." Why is feedback important? Among other things, lack of feedback only adds to redundant or dysfunctional user activities.

I've probably mentioned this in past posts, but I once worked on an Oracle ERP upgrade process being implemented by an aerospace vendor. There was a database vendor consulting consultant (in fact, I had been a senior principal for the same consulting company about 3 years earlier) whom serves as a figurehead leader on the project but had not participated in a single dry run of the upgrade. There was one abnormally long process during the early phase of the upgrade that took 10 hours. One of my colleagues have provided a method for showing how far along the process was in terms of database object counts. Two colleagues had the beginning shift, I had the late evening/early morning (up to 2 AM), and then the figurehead would take the 2 AM - 8 AM shift. This was a critical path process. Long story short, because the client didn't complete the necessary backups on time, the first shift never started the upgrade; I did, making up for lost time and handed it all, essentially just after kicking off the 10-hour job. I explained in detail: "All you have to do is babysit the process. It's going to take longer than your shift. Here's how you check the process." Fast forward: after sleeping until 7:30 AM or so, I made it there just to find out the figurehead, probably billing the client over $200/hour, had decided that the upgrade had "frozen" (deliberately ignoring or disregarding my very explicit briefing), never bothered to consult any of the DBA's on the upgrade team, and ordered the upgrade restarted from the beginning, blowing nearly 20 hours in the go-live, around-the-clock schedule. The project DBA terminating the upgrade? The guy who developed the method for checking the status on the long job. I asked the figurehead, "On what basis did you decide the upgrade was stuck?" There are a variety of ways to check on process and subprocess times in addition to the utility, but in all our dry runs, we had never had a "stuck" upgrade--just one process that took a long time he had been briefed about in no uncertain terms, that I told him would span his entire shift. When I rebuked my colleague, whom should have known better and pushed back on the invalid request, he simply said, "[The figurehead] is the boss." I went back and reported the incident to my contractor representative, fuming that this guy was probably going to bill the client for his shift time over his gross incompetence and blowing a hole in the go-live project schedule, which had already been delayed by the backup problem. Of all things, the vendor consultant apparently found out and reportedly was most highly offended by the billing allegation (of the client paying him for screwing up, which, of course, nobody was going to tell them). I was told by my contractor rep I was flirting with political fire, but in my view (which remains the same today) both the figurehead and my work colleague should have been fired for cause. It's one thing when a person makes a mistake, but both people had been specifically aware of nature of the job and didn't even consult with me or others before pulling the plug. I didn't get a contract extension for expected post-upgrade work, but I was already in contact with two consulting companies over full-time opportunities.

This is just one example of how problematic or lack of recognizable feedback costs money and adversely affects customer opinions. By not getting through I had to continually call again, which aggravated the call volumes, annoyed me (burning a limited quota of cell minutes) and I"m sure that I was not alone.

There is disaster recovery for just about anything; As a database administrator,  I've written documents on restoring database servers from scratch and have dealt with database availability solutions, cloning and migrations. I wrote an email to vendor management at NASA GSFC criticizing how they deployed a certain database availability technology, pointing out the source and target servers were in the same  building; guess what would happen if, say, the basement flooded out? The contractors were just doing what the clients wanted; if I'm the consultant, I'm jawboning the client, telling them they haven't controlled for geographic risk, which defeated the basic purpose for redundancy.

It may well be that most BGE customers don't have two advanced business degrees or haven't worked in the technology field for a living over 20 years; no doubt some of them are mollified just knowing someone is working on the problem.

Last point of the BGE issue: I did go to the apartment office yesterday (Monday) morning, and the manager on duty said that BGE had told the complex that power was scheduled to be restored Friday.

I did have a plug-in emergency unit (flashlight, radio, etc.,) but I had unplugged it during spring cleaning, and its battery had lost its charge. Great. No candles. My other flashlight was dead. No other radios available. I heard water pipe operations in the building going on, which I inferred were showers. The showers were heated by natural gas; that got me to checking my stove. Sure enough, that was working. I did have some alternatives: some ready-to-eat Nutrisystem food items not requiring heating. I did have a marinated piece of Alaskan wild salmon in my freezer which I fried and ate for dinner.

I couldn't work on a creative writing assignment I was doing because my notebook computer battery was dead, so I decided to do some purging of older magazines that for some reason I never have time to read. No kidding: last year I finally purged a magazine discussing Clinton's plans for his second term. These were among the last 2 or 3 years; I've been letting my magazine subscriptions lapse over the last year or two. Lots of health and fitness magazines for which I'm way out of shape for; you know, it's bad when you outweigh the "before" people in the "before/after" weight loss stories. If there's one way to make use of evenings without entertainment, it's an ideal reason to reignite my long-neglected exercise regimen and walking routine. My biggest concern, though, was the fact that iPod shuffle and cellphone batteries were continuing to draw down.

Yesterday afternoon I decided to take a drive around. None of the traffic lights were working in the area. Traffic-cones were set up to nullify things liked crossover left-hand turns. There are three major malls/strip malls clustered near my apartment complex; all of them were deserted; it was all so surreal: I'm not even sure they're that empty on major holidays (e.g., my once-frequented fitness center). It looked like a ghost city. There really was no need to continue on to my regular supermarket or Sam's Club about a mile down the road.

This morning I continued to work on my new cleaning binge, this time with my car where I found employee benefits information for a former employer which went bankrupt 8 or 9 years ago. After finishing that, I can back into my apartment just before noon, and something was different; I suddenly realized that my desktop lamp was on.

Lessons learned: I didn't realize the gaps in my emergency preparedness and the extent my lifestyle depended on reliable electric power. Even in terms of things like blog posts, I could have had posts scheduled to published at specific times.

Anyway, for better or worse, I'm back and will start on tonight's blog post.