Analytics

Tuesday, April 2, 2019

Post #4047 Rant of the Day: A Day In the Life

I knew from my brief military service (I got my honorable discharge) that the military was fanatic about personal appearance.  I had started on New Year's; somehow I had put on 15 pounds (over the holidays)?) since my mandatory physical several weeks earlier, but it was enough to violate the weight limits for my height, so I got put on the "fat boy program", which required mandatory  post-duty open air exercise sessions with other overweight personnel, subject to the humiliation of being seen and mocked by other passerby personnel coming off duty.

In another telling incident I got screamed at (for literally several minutes) because my belt buckle wasn't shiny enough.

Now especially during the early years of my IT career, it wasn't business casual like it predominantly is now, more like shirt-and-tie, if not suit. I don't recall if I wore suits as a graduate fellow, but I recall meeting a former student as a salesman while I was shopping for a suit. I do know the one photo that a graduating student took of me at UTEP (embedded some time back in the blog) showed me in a suit.

So some time back, I was working in a multi-faceted role, including logistics application administration in addition to the underlying database. There were some inventory-tracking complications if, say, they pulled an engine from a truck. (Until the developers fixed the underlying bug, we had to implement some workarounds.) So my mentor into the role would attend one of 2 daily maintenance crew all-hands calls to get a heads-up on these type events. Let's be clear: there was no military request or contractual requirement to attend these meetings; we were simply being proactive in our roles.

The contractor management will often overreact to any incidental comments from their military counterparts. I typically showed up in the background of the group gathered around a central desk. I really wasn't part of the meeting unless there were IT-specific status questions. So somehow one morning I got unnoticed, unwanted attention. There really wasn't anything fundamentally different about my appearance that day. But soon after the meeting broke up, my contractor supervisor got in touch with me. The basic issue was one of his higher-ups had been at that meeting, and his military counterpart had pointed me out. The contractor was pissed that some "slob" was representing the company and banned me from attending any future maintenance meetings. (This is like shooting oneself in the foot, because my purpose for attending the meetings was proactive in nature.) I really don't know what the issue was beyond my obesity, and in fact my mentor himself was obese. The point I'm making here (for later in the essay) was the reaction of the higher-up; I'm fairly certain the military guy didn't ban me, but my higher-up was being obsequious, showing the military his company was taking care of the problem, even at the expense of my dignity.

I can't say that I wasn't too surprised. A prior role in WV was at an Army facility. I was actually the only subcontractor person at the client site; my contractor supervisor, a former Marine, actually worked at a northern Baltimore suburb. We would talk weekly over the phone. But he was there to escort me my first day and me me at the hotel. (By the way, I had to eat that cost; I wasn't in commuting range from MD.)  In getting up, 2 or 3 drops from my Styrofoam coffee splattered on my dark shirt. He then demanded that I go and replace my shirt before driving to the client site some 20 miles away. I really thought that was an overreaction; what if this had happened at work? His view was I should bring a spare shirt with me to work for that contingency. I mean, typically over 90% of time I'm working alone in my cubicle. There's a lot to be said about first impressions, etc.; but the response is out of context.

I think possibly geeks have an undue reputation to overcome. I've mentioned in past posts about an eccentric incompetent project Oracle DBA I met at a Chicago government office who I personally witnessed walking around in a business office setting barefoot. I think one of the weirdest unexpected interview questions I've gotten was whether I wore velcro shoes. What the hell? No, never. What motivated this question? Apparently my predecessor wore them. (He also had staged a personal protest during or after his employment on the front lawn of the home of the company CEO.) Now, to be honest, I have enjoyed some of my tech buddies' idiosyncrasies. At my favorite job in the Chicago southwest suburbs, there were long discussions of the latest Star Trek flick. I would sometimes come to the office after working out (maybe 10 PM) to check on some computer jobs, to find the overhead lights off and wolves' howls playing over the intercom. One female Asian American colleague (I think she later left the company to start graduate studies at Northwestern)  hopped up siting on my desk, while constantly swinging her lower legs up and down. (I took it as possible flirtation, but given  sexual harassment policies, I had no interest in tempting fate by misreading the situation; no other woman before or since has hopped up on my desk. I think over the next few days, whenever I closed my eyes, my brain would replay her legs gif.)

It's bothered my Mom (and me) that I've spent my career working for less intelligent people, when I'm vastly more qualified to run companies or organizations I've worked for. (One can quibble about my social skills and how I managed situations in my short academic career.) But I think a classic example was my first IT job for a well-known, widely-respected insurance company based out of San Antonio. (in fact, they still are my insurer, and I started auto coverage with them as a Navy ensign).

The property casualty actuaries were using APL, an interpretive IBM computer language which lent itself to rapid application prototyping, mathematically notated, cryptic syntax, reading right to left.  The company had had a hard time trying to convert, say, COBOL coders into APL programmers. The property casualty division, frustrated by long delays from computing services, had decided to form its own APL support group. They had one guy already, Joe, who had basically washed out of their actuary track. (I think actuaries at the time had something like a 10-exam sequence to certification.) Joe had certain communication issues complicated I think with a hearing disability from the service (maybe the USAF, like my Dad). For whatever reason, the actuaries bypassed Joe for the new unit supervisor. Instead, they hired a supervisor and a senior programmer/analyst from Blue Cross; I got hired as a trainee, largely because I had two math degrees. (As a southpaw, I also took to the right-to-left syntax like a duck to water.) The targeted supervisor decided to take a different position elsewhere. The company told Joe to train me. Joe made it clear from the get-go he wanted no part of it, because he considered it a supervisory role not in his own job description. I basically taught myself.

After a few months, they transferred JC from the computing services division to serve in the supervisory role. Now JC himself couldn't read or write a single line of APL. He was more of a JCL jock who had helped implement APL on our mainframe, which is how the actuaries had come to know him. Almost immediately, JC perceived me as a/the threat to his position. (My senior BC colleague had other issues, beyond this discussion). This is not a perception, but a straight-up, in-my-face discussion. JC (like a number of early IT professionals), didn't have a college degree, and he saw this 22-year-old kid with 2 math degrees as someone whose career could only advance through his new position. He saw me as an illegitimate hire for a job he should have filled as his managerial prerogative. (In fact, after I left, he replaced me with an office gofer without a college degree.) You might have thought the company would have anticipated this, but they didn't.

JC had no intention of assigning me a high-profile assignment but tried to provoke me into quitting by making my job a living hell. To give a telling example, he took a project that had been earmarded for me and assigned it to a summer intern. And when the guy went back to school, he took the guy's shitty code and assigned it to me, sort of rubbing my nose into the piss. I, of course, wanted no part of it. Ir later got assigned to Joe, who told me that I was right, it was shitty.

So I left the company for the leading APL timesharing company (timesharing basically died by the mid-80's, undermined by commodity PC pricing) where I quickly emerged as their best coder/maintainer. One telling anecdote: a colleague asked for a utility to do something, and I came up with something that cost about 79 cents (low cost being better). Nobody else came close. My own branch manager tried multiple times and about the closest he came was about $1.20. He wasn't a gracious loser, complaining my solution wasn't general enough.

But here's key point: he had an insane nepotism policy. When I was hired, our receptionist had just married another programmer on site. She had originally be living with the programmer who asked for the utility above. By all accounts, the couple were no longer in love and living together mostly out of convenience. But my boss PL was worried that the marriage would cause issues between the 2 programmers and result in conflict in the workplace. So he fired the receptionist.

Long story short, there was a single female programmer on staff, Jackie, not particularly attractive; she had some skin issues and a coffee addiction like I've never seen before or after--she would literally shake without her morning joe. Over the months I got to know her better, knew she had a personal Christmas tradition of going to the Nutcracker Suite. So I noticed that there was an upcoming Houston performance and asked her if she wanted to go.

I still don't know exactly what happened; somehow she tried to transform it into a company event. PL calls me into his office.  "Ron, it's come to my attention you and Jackie are in a relationship." Say what? I've never seen her outside the office; I've never even touched the woman. The only reason he could have known was through Jackie herself. Why would she do that? PL continues: "Your relationship with Jackie causes a problem. One of you two would have to leave." Well, Jackie was a key resource for a high-profile Exxon catalog application. Exxon used to cycle its rising star executives through this position. Jackie probably knew the application better than anyone at Exxon. So, even though I was a far better programmer than Jackie and PL knew it, Jackie's departure would cause a problem, and it was clear he was directly threatening to fire me.

In fact, Jackie couldn't handle stress very well, and it was a high-maintenance situation where PL or others would need to calm her down. But it still shocked me that I would have my job threatened over asking someone out on a date. He had to know talented programmers were hard to find; he had recruited me out of San Antonio vs. Houston.

Only in one case have I had a direct report (the Chicago job described above). where he was a data administrator probably a few years away from retirement. He turned out to be blatantly insubordinate, would not do work I had assigned. He ended up being stripped away from me, the manager claiming that I was out to fire the guy. No, I was trying to get him to do his job. Maybe he resented taking direction from someone old enough to be his son.

But over the years I've lost my job for absolutely petty reasons, solving very difficult problems under limited supervision. I can't go into specifics for obvious reasons, but to give a recent example, I was working with a single-vendor (Oracle) solution, including hardware//firmware, storage and software (including clusterware and database). Part of the solution was a hardware/server remote interface that runs on IPs running a specific command-line interface. It turned out that our security software had flagged a common service (on most Unix/Linux servers) with a known past security flaw.

Initially civilian administrators misidentified the server, thinking my production database servers were at issue. I pointed out that Redhat Linux said the flaw wasn't relevant to them for technical reasons, but the civilians argued I wasn't running a close enough clone of Redhat (Oracle Linux). The Oracle Linux folks quickly confirmed the Redhat discussion applied to them, too. Long story short, I eventually discovered that the civilians had misidentified the database servers for the remote device IPs.

By this time, the network team had been notified to disconnect the remove device IPs. This severely compromised my ability to administrate the servers and resolve any interim hardware issues.

Now this Oracle technology does comprehensive patching, firmware to database software. So I knew I wasn't going to get a one-off (short-term) patch. I'm thinking I need to have a short-term fix so I can get my remote device IPs up and running.

I set out to discover how I could bring the vulnerable service in question. Now on a Linux server, that's a split-second command, but I can't directly shut down the service daemon. I need to figure out how to shut down the flawed service. Only the Oracle command-line interface (CLI)  is not documented in full context. I simply find an example syntax for turning the service on. Maybe the antonym works? E.g., disable vs. enable? Bingo (or at least no error message). I ask IA to confirm the scan turns out clean. I then do the same against the paired IP. IA is more worried if I can turn the bad service off, I can turn it on again. The service in question has incidental, not essential functionality.

This was just a workaround; now I had to focus on getting Oracle to fix its firmware for the remote device. I file a service request. Oracle is in a state of denial. I'm reportedly the only customer who has ever encountered this problem. I'm exasperated: have you used XYZ scanners in your test environment? We've replicated the issue against all 4 remote device IP's. I'm going to Oracle account managers and asking for help.

Some Oracle security discovers the SR and immediately clamps a quarantine on it. Basically this means only I and some 15 or so Oracle employees can see the SR. I've filed Oracle bugs in the past. I know Oracle is only going to fix a problem if it replicates the issue. I keep asking if Oracle can replicate the issue. No response. The Oracle account people say they can't track the issue given the quarantine.

I can only speculate Oracle is worried about word leaking out about a security issue. Maybe the scanner was detecting a false positive/finding, but I needed some explicit acknowledgment from Oracle.

So I was in a holding position for a while when Oracle called me and told me that the newest appliances did not have the issue I had discussed and a fix would come out in a bundle patch in about 2 months. I then applied the bundle patch on my development appliance, reverted the service in question back on, had IA run a confirmatory scan, rinse and repeat on the second node; soon thereafter I repeated the same in production and closed the SR.

I could go on with Oracle Support issues. But there were also organizational integration issues caused by feeder servers running on old Oracle Client software which caused problems with my efforts to tighten Oracle networking security. Part of the resistance dealt with my efforts not being communicated by the client which owned my database servers. A reactionary civilian DBA falsely insisted that the STIG allowed me to use the weaker algorithm his older version software accommodated. Exasperated, I said, "KM, it takes 15 minutes to install compliant client software in a different Oracle home; why don't you install it already?" Civilian branch chiefs'' heads proceed to explode; who is this contractor telling civilians (civil servants) what to do?

I've also got irate Oracle Support duty managers demanding to talk to my supervisor. Part of that had to deal with a poorly communicated bundle patch migration which Support falsely asserted had been detailed in a master note. (Basically Oracle hadn't released a bundle patch (including a recent quarterly security patch) in 6 months, so I had inquired when they were going to release 12.2.1.5.0, and the analyst said there would be no 12.2.1.5.0. He mentioned 18.x;. 18.x at the time was not supported for my appliance. So I'm already in trouble with a government audit wanted to know why my security patching isn't current to the latest CPU (quarterly critical patch update, including the latest security patches), while Oracle seemed to suggest I'm at the end of lifecycle support. If I don't have the latest CPU's, my production servers get pulled off the network.

So on top of all this comes the stupid (not the training, but the circumstances) TARP (terrorist awareness) training issue I describe in more detail here.  Briefly, there were a number (up to about a dozen) of online courses/tests/refreshes I had to complete annually, from the get-go (before I could touch a server). In addition, there was initially one which had to be done in an in-person training, SHARP or sexual harassment training, from 1-3 hours long. (The latter also had an online requirement.)

Now these training requirements aren't that difficult to complete, maybe 2-4 hours each. I can't speak for the civil servants; it's possible/likely they have supplemental training. I/we also had area specific training (like about 30 Windows 10 courses), although these didn't require refreshers.

On the contractor world is the shining embodiment of the Peter Principle, the contract program manager, an obsequious functionary who enforces the overall contract. My original PM on this contractor, BZ, was an amiable older guy who in fact gave me an employee award at one of his last monthly contractor meetings (another contractor (MK), not in my work group but a key developer, wrote an unsolicited gushing endorsement of my quick turnaround, accuracy, etc.) He often had his administrative assistant (KS), now a civilian, handle general announcements (like timesheet and training reminders). BZ left for another (DHS?) position in DC and was succeeded by a personality-challenged, self-important humorless toad WB. (I won't mind if you call him "Weasel Bastard".) WB quickly did away with BZ's meetings (which also included charming monthly birthday recognition, with a complimentary package of cookies) and basically took over KS' announcement role, and raised it to a cringe-worthy art of obnoxious nagging. Literally nobody (on the contractor side) liked the guy; we just hoped that he would leave us alone to do our jobs.

I had probably mentioned in a past post that the local base commander had closed down the base for snow (I think after Washington's Birthday). It was ridiculous; I think most of the snow had melted by noon. But WB sent out the word that we wouldn't be able to make up the snow day by the end of the biweekly pay period that Friday. Now the two-week period had a sort of flextime aspect, and I had to come in that Saturday to do 5 hours of work, including the final bundle patching in production because my key client contact wanted to minimize user downtime. Plus the company no longer let us charge 9 hours (like our normal M-Th schedule), so this made us lose 10 billable hours. So the no-makeup rule seemed to imply that I would have to eat those 5 hours and would have to dip into PTO/vacation time (or have our regular paycheck docked)

WB would send out a post-Snow Day email, reversing himself, subject to certain limits on the 2 remaining days of the pay period. (In fact, I was able to submit my timesheet without dipping into PTO.) But WB's impulsive initial email had created a huge morale problem for us contractors

The TARP kerfuffle was a similar issue that WB had nagged over. My online training refreshers had to be completed or the Army could suspend me. WB seemed to think I was making a choice between online and in-person. (In fact, other than the inconvenience of going to another building and fighting for a parking spot and seat, it would be easier--no test, etc.) He was pushing the online because his local civilian counterparts seemed to be pushing it. When I first objected, he came back at me with a typically obnoxious response: "If everyone could do it online, why do you think the Army is putting this training in large monthly auditoriums, dumb ass?" You're seriously asking me to explain why the Army does stupid stuff? You've obviously never served in the military.

I knew the in-person training was basically redundant (conversations with people who did the training), and I made it clear if the Army required me to do duplicate training, like for sexual harassment, I would do it. But my point was my required online refresher fulfilled contract compliance, and WB never grasped the point. I had a verifiable certificate and an online record showing I was good to go until a year from now.

He sends out the remaining monthly in-person schedule through end of fiscal year starting in April. Maybe a couple of days later, a base announcement indicated there would be March in-person training that morning and the TARP requirement could be fulfilled EITHER online or in-person. I forwarded the email to WB, asking whether I should attend the in-person. No response.

So that's why in the earlier cited post, WB finally sent out ANOTHER email, quoting some Army idiot saying online training was basically a poor substitute provided for people who couldn't attend the in-person training, and the Army really, really, really wants you to do the in-person. Sigh. WB just isn't intelligent enough to understand the online training wasn't optional. This asshole refuses to admit when he's wrong. He's like a dog with his bone, never going to let it go.

He actually thinks this training bullshit is more important than the fixing the vastly more complicated Oracle mess I had inherited, including some 200 unfulfilled (by predecessors) STIG compliance issues, little prior documentation. It's almost impossible to list all the issues I had to resolve, battles where I had to take on Oracle Support, client inertia,  etc. One example I had partially discussed in the same earlier post. One of my disks failed. (This happens in the real world all the time.) Oracle sent a replacement per maintenance agreement. (The documentation was obsolete and sucked, but I had to do the repair, which was a pain because I needed to arrange an escort.) Oracle asks for the failed disk back. I can't do that; Army security policy won't allow it. But Oracle points out I have to return it by contract or pay a fee. I end up contacting Oracle Account Support. At first, they argue we have DDR coverage, basically a premium coverage which allows the client to keep a damaged disk. When the appliance was purchased in 2014, it had DDR maintenance coverage. But it turns out that some Army bonehead cancelled the coverage when the maintenance agreement came up for annual renewal in 2015. I'm told by some Oracle guy that the Army would have to backpay interim premiums. My client boss tells me to get a quote. But he's got no funding. The quote expires. In any event, we still have to pay for the replacement disk. I can't get a quote. I get redirected to Oracle Sales. After some time, Oracle Sales comes back and says that it doesn't carry the disks, that it came from Support inventory, and I would have to wait for Support Accounting to issue an invoice, but they were understaffed, so I'll get the invoice when I get the invoice.

My client boss comes back to me a few weeks later. "Quick, I just got funds, but they may go away very quickly. I need that DDR quote reissued and an invoice for that disk." So, long story short, I get a call from a government accountant/auditor who seems to think in getting the disk replaced by our hardware CSI, I had somehow bypassed the government procurement process. Oh, my God; she was totally clueless; it wasn't Oracle's fault the Army had dropped DDR coverage. The Army's Support contract required them to return the disk or pay a fee, basically replacement disk cost. Army security policy wouldn't allow us to return the disk. So we had to pay the invoice (which really wasn't that much) But the civilian accountant couldn't grasp the point. She kept on pointing out the local procurement chain of command.

So this is just a taste of what I've had to go through; I was told my predecessors had run into similar issues, inertia but had backed off. But Weasel Bastard didn't care. It all paled next to attending in-person TARP training. Good lucking finding a replacement. I don't care anymore.