One of the topics he points out is that "failure is not an option", i.e., management simply decides that rejection of bad news is a perverse sort of motivating factor, that people under impossible deadlines will somehow pull it off (after all, their jobs may be on the line). Let me give a short series of examples:
- On a Navy project in south Maryland, a prominent defense contractor was tasked with implementing an installation of Oracle Application Server, which basically, among other things, hosted applications previously running on Developer Server, and failed twice. My new employer had negotiated a number of Oracle support hours (basically oddball projects the defense contractor had picked over.) The prime had told the client project manager that they had upgraded the applications in question to 9iAS, and after I installed it, claimed that I had screwed up the installation because their applications wouldn't run on it. Long story short, I discovered that the reason the applications were failing was because they used obsolete printer codes; the application server wanted standard output files (e.g., in pdf or rtf format). The project manager blasted me in front of the full contractor group, claiming that the fact that administrative personnel no longer had to do busy work in setting up MS Word was a material violation of her criteria that the upgrade project be upward compatible and not affect existing operating procedures. I explained that this was an Oracle product decision beyond our control. She called me "incompetent" and said, "We're the US Navy, and Oracle will do what we tell them to do."
- At the American subsidiary of a Japanese computer chip testing vendor in Silicon Valley in 1999, I initially was a temporary Apps DBA, replacing a DBA whom decided to resign after management refused to promote him to the vacant IT manager position. The new IT manager, a former Tyco IT manager, was hired about 3 weeks later. He later pressured me to take the perm position. He had a couple of major project priorities: a failing Clarify project and a Web Expenses project. He cancelled the vendor contract on Clarify and shelved the Web Expenses project. He decided on a phased rollout of Clarify and named the systems analyst colleague on the Web Expenses project the new Clarify project manager. In the meanwhile he recruited former Tyco employees. The first Clarify rollout was a success in October, but he decided to resurrect the Web Expenses project in mid-November, claiming it was critical to his Y2K planning. Web Expenses was not even installed on test/development nodes; even worse, he named one of his Tyco cronies, with no background whatsoever in Oracle EBS or Web Expenses, to serve as PM. I quickly objected and said that he should put my old systems analyst, whom had in fact been in Web Expenses training, when I started; I warned that he was setting MT for failure; he then told me his style of management was throwing people into the deep end of the pool and making them swim; in fact, he was writing another check on my back. Among other things, I negotiated a 3-week deferral of Web Expenses (a 6-month project) after New Year's; accountants were to handle expense reports manually during that time. I had a number of things to configure (the default workflow assumed rejection if the supervisor did not approve in 15 minutes). Everything was on track; my boss agreed to a brief Christmas vacation so long as I was back New Year's Eve. I got a call from MT that employees were told to enter their expense reports into production after New Years, and I didn't even have the software installed in production. A second complication: the company was converting emails from Eudora Pro to Express Server--but the CEO and my manager, without consulting me, had decided to give managers the option to defer conversion to Express Server. But I had configured all the managers to work off Express Server email addresses in Oracle Apps; an expense report to an invalid email address would be "lost" in the system. My boss didn't even realize the implications of his decisions to first discuss them with me. And just a final example, I heard him tell another supervisor after a manager training to go ahead and submit his own expense report in the system--I made a mad dash to check the CEO's email address in Oracle. My boss didn't seem to understand the political implications of employees running into problems filing expense reports. It was bad enough dealing with HR, which was expiring supervisor records without reassigning subordinates to a temporary supervisor--another wrinkle leading to "lost" expense reports. So when I read that the Obama Administration was warned that the website was not ready for prime time, I totally believe it: so many managers fail to be proactive and don't want to hear "bad news".
- Another example I may have discussed in a prior post: the American subsidiary of a duty-free shop franchise had solicited Oracle Apps/EBS DBA work. The HR department wanted the Apps HR module added to the client's existing licensed (accounting) modules. Oracle at the time required a GUI front-end, SmartClient vs. subsequent browser technologies for the HR module. Troublemaking contractors told the accountants, operating in traditional green screen mode (similar to old Lotus 1-2-3 DOS), that SC "broke" accounting technology, which resulted in a political war between the functions; more on that shortly. But I discovered that the client had a malconfigured Apps database because they had hired a non-Apps DBA to migrate databases to a new AIX server in the past. I discovered this because the SmartClient patch had failed--because the database during the migration had been recreated with a default character set--but Apps required the Western European character set. I had to battle the IT VP on this, because her current production DBA, actually a developer promoted into the role, told her that Oracle's supported solution of exporting the database, recreating the database, and importing the data back it was actually my attempt to rack up billable hours at the client's expense. Going back to the accounting-HR dispute, she decided to resolve the issue Solomonic-style by giving accounting and HR their own ERP databases. I had to jawbone her on this, too: an ERP system by design is a unifying system. But she had more confidence in her IT manager colleagues whom had told her they themselves had done it. Just to humor her, I asked for relevant DBA contacts: the DBA in question said (I'm paraphrasing), "Yeah, we did that 3 years ago and have regretted it ever since. Do you know how to put Humpty Dumpty back together again?" So when I read Shirky's account how he was rewarded for simply taking information from the managers' own IT staff, it was clear what disdain the managers had for the input of their own IT staff, and that's not uncommon to other shops as well.
- Then there was an EBS upgrade for a City of Chicago IT department. I had been brought in as a sub to the prime contractor, in part because I had experience with both 11.0.x and 11i versions of EBS, and they had an aggressive 3-month upgrade schedule (mid-Aug-Nov). Now just to explain they were also running it on a Windows-platform server, and there are nontrivial nuances running Apps on Windows vs. Unix/Linux. The sister consulting arm of the prime contractor had won the bid to do the upgrade. I instantly knew the project was in trouble; the consulting team first brought in a DBA with no background in Windows as a back server, and he left the project after I caught him twice hacking into the production server within 2 weeks. The second DBA was both incompetent and mentally ill; I won't go into the second issue, but he refused to accept/trust one of my database clones (before 11i, DBA's had to maintain their own cloning scripts), wanting to get a back port of adclone for 11.0.3. He also ran RapidWiz, an Apps install utility, on a server with Oracle installed against explicit warnings in documentation. (Let's just say the finicky Windows registry can be a problem.) I knew the motivation for doing RapidWiz at that stage was to get preinstall scripts and I promised to run it on another server and give him a network share to the scripts; he refused, daring me to "prove" my workaround via Oracle tech support. It was like pulling teeth, but he couldn't figure out why my test database clone wouldn't come up after he rebooted the server. I later made repairs to Oracle's registry enabling Oracle to come up, but he would constantly complain that any technical issue was due to my "untrustworthy" fixes to the registry. I was being portrayed as the "control freak" operations DBA; technically I should have had both roles from the start, but there were cost accounting issues across the vendor branches (my boss, in fact, wanted to charge some of my hours back to the consulting arm): I was willing to settle for any competent Apps DBA colleague, not these guys. The PM also read my resistance as political in nature--yes, I was worried that I would end up fighting fires for months after these guys were gone. Just to talk a simple example: the payroll function required installation of an MS C++ compiler; I couldn't even get a requisition cut over 3 months, but these guys wanted the city to purchase software so they could work from their PC's, instead of the cold server room: I could not understand why we were burning morning status meetings discussing that kind of crap, which I knew was a futile request and had nothing to do with the project. The weird DBA at 6 weeks made a blunder I've never seen even a rookie DBA do; he caused his own database to crash in an unrecoverable mode, and worse, all his backups were unusable (he did not quiesce certain Windows services, another rookie mistake). At this point, they finally gave me the car keys, but to provide context, the contract required two test cycles, and I had only 6 weeks. I repeatedly told anyone whom would listen as good as I was, that was unrealistic. The department IT manager during this entire period never personally talked to me even once; even worse, the problem DBA was to have been rolled off when he went home to Florida that week, but the IT manager refused to let the DBA staff go anywhere. I spent a third of my time dealing with the rogue DBA trying to regain his spot and the PM, whom was personally vested in the DBA he selected. I had to deal with a number of technical issues, and the PM refused to operate the other DBA's in a swing shift. I was stuck into a cap of 40 billable hours a week and I had to operate by Metra hours because I lived in the suburbs. I did get through the Apps upgrade the first time, running into an unexpected bug setting up the middle-tier server; I filed a problem report with Oracle Support and was working on a workaround when the client decided to make a change. (There was also an obsession of the consulting group which was openly coveting my test server because it had more CPU's. In fact, the servers they had to work with also had multiple CPU's, and I had used them as development database boxes. I didn't want them touching my test box because I needed it to test any interim production issues. The CPU issue was a red herring and more about politics.)
In the above examples, I'm not arguing my experiences are representative of those experienced by other IT professionals, but I have had to deal with a number of bosses whom were not that technical. For example, on a Wisconsin suburb project, the female client DBA's had a non-techie manager. I remember one day I was talking to one of the DBA's at her desk when the clock turned to 5 PM. The guy stopped by and glared stonily down at me, telling her that she had his permission to leave then and there if she wanted to. (Now of course I had to show up at 7:30 AM and couldn't leave until 7 PM, longer if I had any patches to apply... Flat salary, no overtime, no bonuses, and I had an 83-mile drive back home.) She apologized for the interruption after he left. I had heard how he went after my predecessor whom apparently had the audacity to call the county Unix admin over the weekend...
