There are things I can't discuss in detail because of contractual requirements. But I can describe certain scenarios that are good examples of what you often run into on government gigs.
In 2001 I was asked by a consulting company to fill in for a project Oracle DBA who had resigned about 2 months into a fixed-bid contract to do an ERP upgrade for a suburban Milwaukee area county.(This was basically a high-profile project that the contractor had used to win similar project bids in Portland and Austin. I had been pursuing the Austin project, and the company president pulled a bait-and-switch.) There were 2 county female DBAs whose primary responsibilities for the project were to ensure the test server and the key upgrade database was backed up. There were a number of immediate challenges, including being weeks behind schedule, no documentation, plus my predecessor's upgrade had over 60 known issues. And apparently my predecessor had been known to call the county Unix system administrator "too many" times (his workday ended at 3 PM).
The client managers really didn't have the necessary knowledge and experience, e.g., the DBA manager wasn't a DBA, but they were zealous in "protecting their employees from overstepping contractors". So, to give an example, I had an obligation to walk the two DBA's through the upgrade process. So in one case, the DBA finally became available around 4 PM. Her boss saw us in her cubicle around 5 PM and says to her while shooting me a look that could kill, "You know, you can go home now." After he finally left, she apologized for the interruption and asked me to continue.
I did not white-box what the county was doing to back up the server. But to provide context I was commuting 83 miles each way from the Chicago northwest suburbs. True, the suburban Chicago consultancy wanted me to move to the project site, but they didn't have any other local customers. I couldn't leave before 7 PM because of testing (any patching had to wait until after 7 PM), but my bosses required me to be there by 7:30 AM to ensure testing was available They really didn't explain why, but I could guess because I had to fix a couple of related problems. It requires understanding a little bit about Unix and the all-powerful root account. The lady DBAs tended to use the root account to sign onto the user oracle account because they didn't know (or care about) the oracle password. With root, you can easily switch to another user account. A problem can be if you think you've switched accounts and you haven't or similar considerations, with the result root takes ownership of files user oracle needs access to start up processes. It's not a difficult problem to fix if you have root permissions, but the bottom line is nobody trusted the local DBA's to ensure everything was up after backups.
I've written before how the contractor VP demanded a new test upgrade literally 2 days before I was scheduled to move from California to Illinois. (This Indian-American owned company was so dishonest; for example, they had promised to compensate me for my 12-hour plus days with a bonus in negotiating a lower salary; and then later they would say the employee handbook said bonuses weren't available on fixed-bid contract work.)
Shortly thereafter, on a Friday afternoon, I started seeing very unusual Oracle error messages, which through experience I've come to suspect involved hardware issues. I contact the county sys admin; he's in a state of denial, but he's not telling me something because he's on site and rebooting things at 7 PM. And what happens on reboot, everything on the RAID/storage, is going to "lost and found", i.e., I've just lost the database. What about those backups? It turns out they were using a static, obsolete backup script, basically unusable. Somehow miraculously I manage to fish out the key missing file out of lost and found. I set up everything to start up the database again for the county DBA, but she doesn't want to take responsibility, so I have to do it.
Whereas we dodged a huge bullet on the database, it turns out we took a direct hit on the ERP software stack, completely missing from backups. That meant 2 days of installing and patching software so they could test Monday morning--once again, totally my effort. My bosses would later provide the county with an invoice for my services that weekend. The county refuses, saying they have a fixed-bid contract and "shit happens". Wait a minute, I had to do this because you guys didn't live up to the terms of the contract. The company threw me under the bus for hopes of a "Phase 2" project. Think I ever saw a single dime from my company for all this work? Nope.
There's probably no better example that illustrates the petty bullshit you put up with as a contractor than this one. I had an appointment with one of the DBAs and she wasn't there. No notification, explanation or apology. I had to keep checking back. It later turned out she was out having her dog's nails clipped. You've got to be shitting me! So the next day I'm joining the other contractors for a fast-food lunch; we stop by the men's restroom on the way out, and no, I don't check the stalls for anyone who might eavesdrop on the conversation. I said, "You guys will never guess why I had to work late last night" and then I mention the nail-clipping incident.
Well, it turns out one of the county IT managers eavesdropped on the conversation, and all hell breaks loose. He tells my boss he wants me immediately walked off the project, because I didn't know my place as a "guest". I'm literally the only reason this project is on schedule, and my boss finally convinces him that my leaving would delay the project.. He's not really protecting me; he's losing money if they staff the project weeks longer. He probably promised I would be gone as soon as possible after go-live. I, of course, got read the riot act.
And I've had my share of "what were they thinking?" moments working with the federal government. I can think of at least 3 or 4 times where drives had turned amber (needed to be replaced), and I'm the one who raises the issue and get the replacements scheduled (in some cases did the replacement myself. Now I don't usually work in the server rooms, which are staffed by civilians and/or consultants, but how much do you need to know to see and report amber lights?)
In one case (not involving classified data), I was working with an Oracle Database Appliance, basically a one-stop hardware/software solution you can basically plug into your network,and patching issues could be traced back to a defective disk. Now many government departments and agencies have security policies requiring retention of failed disk with proper disposal (e.g., degaussing, etc.) Oracle has an extra-cost service policy (I think it's called DTR) which waives the typical "return the defective part" policy. We had a hardware CSI (support policy) which guarantees replacement of parts, but unless you have DTR, you have to return the part. It turns out that the government had bought the ODA for this one business unit, and it had DTR coverage the first year of ownership. But some misguided government idiot made a bonehead decision to DROP DTR coverage in the second year, and it continued to the point I came on-site.
So Oracle fulfills the replacement by shipping out a disk (they don't send field engineers to replace consummables); I'm then told to return the failed disk. NOW all of a sudden the government cares and reminds me that they don't return disks. Well, there's a penalty/fee if you don't return the disk and you don't have DTR coverage. I can't find out what it is; hasically, "you have 5 weeks to return the disk, and we'll cross that bridge when we get to it". I've also got Oracle accounts and/or the original resellers involved, and they are thinking Oracle Support's records are screwed up because the ODA was sold with DTR coverage. (Eventually we discover the government let coverage lapse after 1 year.) Now the client is asking what it would take to reinstate DTR coverage, and I'm originally told the government would need to backfill to make coverage current (several thousand dollars). But the manager doesn't even have the budget to pay for current coverage and/or the currently owed penalty/fee. In the meanwhile I'm getting pressure to hand over the bad disk, which the government doesn't even own under contract. Several weeks later I get the invoice for a few hundred dollars. I'm having to get a DTR quote from resellers, etc. I hate dealing with all this bureaucratic crap because some government idiot made a bonehead mistake years ago.
So finally the client manager contacted me saying his budget had briefly opened temporarily and he thought he could squeeze renewed DTR and the disk fee/penalty, too (the quote I got didn't seem to include back payments). But that wasn't the end of it. The client manager must have gotten some pushback from a retarded government auditor and sent her to me (phone). She is totally unable to grasp the fact we are dealing with a CONTRACTUAL issue, not a PROCUREMENT. She somehow deluded herself into believing I had procured a disk without going through the military's designated procurement process, competitive bidding, etc. No, this was a maintenance contract issue. Oracle certifies components in its ODA's. We had to pay a penalty (probably Oracle's cost for a new compliant disk) because the government failed to maintain DTR coverage and the government wanted to keep the bad disk. Life is too short to argue with incompetent government bureaucrats. I left the consulting gig soon thereafter for unrelated reasons. I don't know if the government auditor retard escalated the issue, if Oracle ever got paid.
A more recent issue: I had an Oracle database which housed a geodatabase. I had a task to register a nonspatial table into the geodatabase. The geodatabase client software required a supported Oracle client. Yet I ran into repeated issues trying to connect through the geodatabase client; my direct Oracle connection works fine. Long story short: in anticipating support request questions, I double-check certified stats: I know we're running Oracle 12.2, but it turns out 12.2 is not certified on this geodatabase version. "OH, SHIT! The client is the least of my problems; maybe there's an interoperability issue with the geodatabase."
Long story short, it turns out none of the civilians had ever connected to the enterprise geodatabase--and wouldn't know what to do with any registered table. (They used more localized datastores.) It turns out I could connect with a more recent (4 versions higher) client--which also means the civilians are running on really old software clients--which is another issue.
When I'm able to connect through the updated geodatabase client, I still run into obscure issues trying to register. Somehow the civilian project lead starts lecturing me on the concept of an ObjectID, and if I don't listen to his rant, I'm disrespecting his geodata expertise. DUDE! The interface says ObjectID is OPTIONAL. You've never even connected to the enterprise geodatabase. You're in MY WORLD NOW--oracle. After he left in a temper tantrum, I eventually discover there are corrupted geodatabase indexes--which, of course, I'm the first person to notice. I rebuild the indexes and find my table registration succeeds. Of course, it's all for nothing, because the civilians don't know what to do with it.
But let's get back to my discovery that we were running the database on an unsupported Oracle version. How did that ever happen? My contractor colleague RC said that "everyone knew we had to go to 12.2". But no one, government or contractor, ever checked to see if 12.2 is supported? It didn't become an issue because no one was using the enterprise geodatabase (which leads to the question of why they had an enterprise geodatabase to begin with). She was in a state of denial, saying 12.2 wasn't an issue. (Why then did the publisher publish 12.2 interoperability patches for other versions?) She also argued one of the civilian developers connected to the database (5 years ago)? First, not through the geodatabase client, and his account was a plain vanilla Oracle account without any geodatabase roles. So this discovery doesn't make me popular with either side.
I had also written some Oracle functions to do some conversions (e.g., among latitude, longitude, and MGRS) and the geodata "expert" project lead was questioning results, even though I could validate against certain online calculator. Having people assert their "expertise" reminds me of a buffoonish IT manager I once worked for in Houston who used to preface everything with "based on my 17 years of IT expertise?" To quote that wise philosopher Shania Twain, "That don't impress me much."
I'm not going to repeat FEE's excellent post here debunking the idea that federal or other government workers are "underpaid". One of my more recent contractor "supervisors" (not a DBA) had bided his time for 8 years before finally joining the civil service. In my most recent independent contractor gig, I was making $10/hour less than I was making 20 years ago. I've never had government job security, pension programs, etc. Benefits are fungible, and as the post points out, nearly twice the comparable level in the private sector. I think I'm now too old anyway to be offered a government position, and I'm not sure I would do interested in spending the remainder of my career fighting the bureaucracy. The fact is if I were to start my own company, I would not be recruiting anyone out of government.
