Analytics

Tuesday, June 28, 2011

Miscellany: 6/28/11

Quote of the Day

It has become appallingly obvious that our technology has exceeded our humanity.
Albert Einstein

My Choice for 'The Voice': Javier Colon

Javier Colon The Voice
Courtesy of Poptower

The Resume and Public Sector:
A Preliminary Discussion of Resumes and Hype

My professional resume contains no fluff, no spin, no hype. It's because I don't have to. Some of the most impressive things I've done technically don't even appear on my resume. I once worked for the American subsidiary of a Japanese chip-testing machine producer. Over 13 months, I won a record 3 CEO Awards (in fact, the CEO knew me by sight), but there were more impressive things I did which were not nominated: for example, I did a fix resulting from an unauthorized shortcut made years earlier in setting up Oracle's fixed assets application software. This shortcut involved setting up certain categories of assets; instead of taking 15 minutes of busy work Oracle wanted, the consultants thought they could short-circuit the process and do the setups directly. The accountants and developers never encountered the problem because they had been using an interface Oracle no longer supported. When I put pressure on the accountants to use the authorized interface and they encountered the problem (i.e., no assets were being pulled up to the new interface), they scapegoated my applying (totally unrelated) Oracle-required product patches and without discussing the matter with me directly contacted Oracle Support which tricked them into providing evidence that my company had done something unauthorized, violating the company's Oracle Support agreement. My boss got a price quote and proposal for an authorized fix that required $10,000 and 2 weeks of not being able to use the production database (not a viable option). The nature of the problem required a custom fix I designed, tested and implemented which required detailed knowledge of dozens of Oracle product tables and elaborate table updates. This one fix, which my CEO never knew about, easily paid a few months of my salary (factoring in opportunity costs of an unavailable production database).

I have done tech screens for DBA's. In one case I was the subcontractor production Oracle DBA for a Chicago local government entity. The same vendor had the operations and ERP upgrade project contracts. There was an unrealistically tight project deadline where the agency wanted the upgrade done by mid-November (i.e., before a final end-of-year close) and they started the project 3 months earlier. The project manager, allegedly another PhD, had a team of 3 to 5 DBA's, whom were grossly incompetent. The initial lead project DBA was replaced; the project manager hired a new DBA without having me qualify him. I was introduced to the guy via a cellphone on speaker over a work lunch. The project manager asked him what he would do first; he said, "I'll do an adclone (Oracle database cloning procedure) for creating a copy of the Oracle EBS 11.0.3 production database to a test server." As soon as the conversation, I had a frank conversation with the managers. "Look, this guy does not have the background he claims." There had been custom cloning processes--that I had created and used several times, roughly 3.5 hours long per refresh, but Oracle did not introduce adclone until EBS 11i (the target upgrade version). There was no need for this guy to do a clone at all; I had test and development databases that I constantly refreshed. Well, this new DBA insisted that he couldn't vouch for my cloning procedure but wanted to use an officially supported version. So he joins the project and literally spends the next 2 weeks, billing the client, first verifying the fact EBS 11.0.3 does not have an adclone procedure, then trying to get Oracle to backport adclone to EBS 11.0.3, and then trying to find some third-party Oracle-verified cloning tool. I'm pounding the table at morning status meetings, "The City still doesn't even have a requisition cut for a Microsoft C## compiler Oracle requires for running Vertex (i.e., payroll updates), and we're 2 months before go live." What is the project manager pushing? His DBA's don't like working in the air-conditioned server room; they are trying to get the city to buy software so they can work from the comfort of their cubicles. All of a sudden in my personal email account, I'm finding announcements from an Internet job board that is clearly defining my own subcontract position.

I go into my vendor manager's office and he's got an inches thick pile of DBA resumes on his desk (and I was making a very low subcontract rate). I told him, "Look, this project is failing. We are halfway through the project schedule; we are committed to 2 upgrade cycles for testing; and these guys haven't even gotten through the preinstallation steps of the upgrade for the first cycle. We need these guys running around the clock and the PM is having them all running the day shift. I heard them muttering the other day, and they can't figure out why their database won't go up. It was because they ignored a warning in boldface in the document, run this on a Windows server without a prior Oracle install on it. I can get their database to come up but I have to fix the Windows registry." The DBA somehow incoherently manages to convince my boss and the PM, that Oracle is self-contradictory in its boldface warning, and the reason he wasn't able to bring up the database after doing the install has nothing to do with the post-install Windows registry now pointing to an incompatible version of database software. My boss insists he won't believe me until Oracle Support verifies in writing what I think was said in the boldface warning; I'm pleading with Oracle Support (that's an entirely different bureaucracy to battle): please put down in writing what I believe the warning said; he's saying, "No. We already printed the warning and don't have a need to repeat ourselves. If they don't believe you, it's your problem, not ours. If the City of Chicago ignores our installation instructions, we simply won't provide any technical assistance for any resulting upgraded database because the City will have violated their support contract." Why are two grown men siding with the mentally ill, incompetent head project DBA over a plain English warning? It was all organizational politics.

This guy (the second head project DBA) was a piece of work. To give an example, we once went to a McDonald's for lunch and he went around collecting abandoned receipts for expense reimbursement. Now to be honest, I'm not a licensed psychologist. Maybe Dr. Phil McGraw would conclude this guy was perfectly normal and I was the one with the problem. One weekend I am finally asked to replace this guy; he caused his own database to crash in an unrecoverable state and didn't have a usable backup, losing 3 weeks worth of project work. In the single worst act of bad timing I've ever seen, the city agency IT manager finally took the initiative to crack down on the PM and refused to allow the vendor DBA's fly home.  (The IT manager didn't ask me--what he didn't know is that I would literally have to spend a full third of my time over the next few weeks having to fend off political attacks by the PM and rogue DBA. I was restricted to working 8 hours a day--but the PM refused to have the other DBA's work past my shift and I had to catch the last early evening Metra train out of town. Oh, and I was still doing the production DBA work. The only good deal the city got out of this is they were getting two FTE's for the price of one. My boss, however, was upset, because he wanted some of my hours charged against the project budget.) The plan had been for the subcontractors to let Mr. Wonderful go as soon as he was home in Florida. (He showed up, I remember he argued vehemently over some arcane issue he was wrong about, and I ignored him, working with a third staff DBA.) By the time Monday came around and I made more progress in 2 days than 3 DBA's in 6 weeks, the PM refused to let Mr. Wonderful go because he didn't want to break in a new DBA for the rest of the project. He, of course, doesn't ask for my input. Yeah, that's all I needed working with me: a former lead DBA upset I've now taken over his role working on an upgrade whom I can't trust around a real database because he can't even do a simple backup.  Isn't it wonderful what the city of Chicago was getting for its money? Anyway, we have an impromptu project team meeting with maybe a half-dozen techno-functional consultants. Since the guy didn't go home, he claimed he was unable to launder his business casual clothes. (I don't know; maybe Chicago hotels don't have washers and dryers or are unable to recommend a solution, Chicago dry cleaners or laundrymats aren't open on weekends, or he couldn't find a store that sold shirts open on a weekend. I'm sure he considered these options, because he's a really smart guy, a proven problem solver.) Anyway, he felt it necessary to point out to all his co-workers the status of his dress shirt, raising up his arms, pointing out his huge underarm sweat stains, and repeating, "Stinky! Stinky! Stinky!"; he repeated the stinky triplet about 5 times until Ms. tall blond functional consultant said, "That's enough, [Mr. Wonderful]!" I must live a sheltered life; I never knew pit stains were a spectator sport.

Before the DBA caused his own database to crash, I was in deep political trouble. Since this is not an Oracle DBA blog per se, I won't go into all the technical details here, but it was something so awful and off the wall, it's milk-squirting-out-your-nose funny. There was a group of Chicago personnel coming in for training, so I was doing a routine final check, I found the database down and the application processes wouldn't come up (which meant some DBA incompetently brought down the database before the application processes); I had to clean up processes so the applications would run. So after I ensured the training could happen, I figured Mr. Wonderful brought down, without authorization, the training database (this alone was enough for termination for cause), but I didn't know Mr. Wonderful's motive; he explained he needed a copy of one of the training database files to replace a database file missing (due to his own fault) in his database. As I write this, there must be hundreds of Oracle DBA's around the world rolling on the floor laughing.

Before the relevant project database crashed, my boss was totally unsympathetic. I'm saying, "Check with any consultant of your choosing; call your company headquarters in Pennsylvania and talk to their tech group. Give them the facts: we have a real problem here." My boss is reading this all purely politically. He thinks I'm feeling threatened by having a team of other DBA's there. (It can't possibly have anything to do with the fact we are dealing with a multi-million dollar piece of software, and if and when these guys roll off the project, I'm the guy spending the next 6 months or more putting out production fires. I had done the same type upgrade at least a half dozen times on my own, never a problem.) If the project goes south, he doesn't care: it's not his neck on the line: it's the PM's. He's going to give the PM all the rope he needs to hang himself. He doesn't like the PM who is already pointing his finger at ME being the problem, which makes it my boss's political problem. My boss has ZERO problem with terminating me in a heartbeat.

Musical Interlude: My Favorite Groups

Chicago, "Here In My Heart"