(But the Dogs Can't Smell It...)
This May will mark my seventeenth year as a professional database administrator. DBA's are obsessed with backup and recovery of production databases. But an incident happened during my first gig, as a government contractor at the EPA regional headquarters in downtown Chicago. I had repeatedly pushed the network administrator group on backups, and I had been told, yes, the backups are occurring, we verify restore procedures, etc. Another government contractor group was rolling out upgrades of the core application software across the region labs; somehow the upgrade failed, and the contractor requested restoration to the most recent database backup, so I requested the network administrators to restore relevant files from tape so I could perform the recovery. Well, the funny thing was--they couldn't restore the files; not only that, but apparently none of the backups over the last several weeks was usable, either (effectively losing any database transactions in the interim). The network administrators couldn't explain it--they insisted the backup software reported that backup jobs had successfully completed; what annoyed me was I had specifically asked them to check restore functionality long before the upgrade incident. (Fortunately, the jaded chemists maintained a separate copy of records; as a former MIS professor who, in fact, wrote articles on usability, I was not surprised by this, but it's beyond the scope of this post. Let's just say I was more worried about it more the chemists, the managers, and the network administrator group.) This kind of breakdown is not the exception to the rule.
For confidentiality reasons, I cannot disclose more specifics, but let me give some real examples of government incompetence I dealt with as a contractor:
- A security audit script used a certain query to check certain "hidden" database parameters; the "view" referenced in the script holds no information about hidden parameters, so it could never flag an problem setting as intended.
- A production database server installed years earlier had 3 power supplies. No one before me noticed that one of the power supplies wasn't attached to a usable power socket. I also discovered that the RAID-5 array (storing the database) on another production server was "running on its spare" (i.e., a key disk had failed long before I came on board.)
- I once caught a grossly incompetent government auditor pulling equipment he didn't recognize out of a production rack in the middle of a workday. (I had to escalate the matter to my civil servant boss in St. Louis, when local IT management told me I had no standing to interfere with the work of a government auditor; in fact, I was attempting in vain to get the auditor to defer his activities outside the production window.) How did I catch him? I got alerted to the fact that the same unidentified guy had mistakingly rebooted a couple of my other servers in a different rack.
- Database replication is a high availability solution. I discovered that the facility security database being replicated was configured wrongly, with faulty switchover scripts, and the replication database was on a server in an adjacent rack (meaning no control for location risk, e.g., a flooded basement). This is typical of how things in the federal government work (or don't work): they paper over compliance with regulations, even if the makeshift "solution" fails to address the intent underlying the regulation. (I ran into bureaucratic inertia trying to get one of the databases rehosted to a different location.)
- The TSA's explosives/drug canines in Philadelphia failed recertification last month. But the dogs were kept on the job until word recently broke out over the past few days.
- Usable explosives detection gear was discarded because of an unacceptable level of false positives and high maintenance costs.
- Metal detectors failed to detect surgical rods.
- When a passenger at Newark managed to get around security, setting off security alerts, it turned out security cameras didn't capture the incident because they weren't working.
While we endure more and more violations of our privacy through Monday morning quarterbacking regulatory empire building (versus preventive care), what good does it do for TSA to audit us, when in fact the TSA itself is not being competently audited? Which is the greater threat to security--passengers or the TSA without effective resources deployed and monitored on a timely basis?
Schwarzenegger: The Chickens Have Come Home to Roost
I remember my enthusiastic support, as a former California resident, when the economics-literate actor swept to power on a reform agenda after his failed predecessor, Gray Davis. I then watched in dismay as the governor was unable to deliver change in the face of union opposition and other Democratic special interests: the governor seemed resigned to pointless political survival, veering left of center on opposing restoration of the traditional definition of marriage, supporting costly additional government intervention on the environment, and becoming Obama's principal Republican shill in supporting the so-called stimulus bill and in particular the federal Democratic Party Health Care Bill.
So I was not surprised when Schwarzenegger started pounding the door in Washington, demanding a state bailout. (Remember one of my favorite quotes, the Sally Brown philosophy ("All I want is what I... I have coming to me. All I want is my fair share...")? So Californians, angry that states like Alaska have gotten more than they pay into federal revenues, think they are entitled to a bailout based on their own net contribution--as if the average American should add to the federal deficit to accommodate California's fiscal policies and gold-plated benefits!) California is in a state of denial over the chickens coming home to roost after years of progressives making promises to government employees they couldn't keep and engaging in fiscally irresponsible spending sprees without first setting up proper reserves/rainy day funds. Who could have known when Schwarzenegger uttered his movie tagline, "I'll be back", he meant a place at the federal teat?
So Schwarzenegger abruptly changed course on the Democratic Party Health Care Bill in his State of the State Address, bitterly unhappy over corrupt deal making, such as the Cornhusker Kickback. His point, of course, is that California will be affected by federal Democratic scope creep in unilaterally expanding Medicaid, where the state shares expenses; given mandatory budget balancing and the state government already tens of billions in the red. Schwarzenegger is probably more angry with himself for not negotiating a Terminator Rebate in exchange for his earlier political support. However, shouldn't Schwarzenegger have objected more to the consistent pattern of Democratic scope creep on Medicaid eligibility from the get-go as fiscally irresponsible, rather than simply griping California shouldn't have to pay for its fair share of the Medicaid cost burden?
In the meanwhile, do we really expect Obama to put California's rich pool of 2012 electoral votes at risk? Expect some sort of sham rationalization to print more federal money in California....
Political Cartoon
Pat Oliphant's work clearly captures that Obama has not brought change to the Executive Branch. (I love the gatecrasher to the Cabinet meeting; nice touch!) Nearly 9.5 years after 9/11, we still have an intelligent community that fails to connect the dots and doesn't believe in accountability and taking initiative. In a country where credit card companies can validate your credit card purchase in seconds, a trip from Yemen about a terror plot involving a Nigerian, and a recent identification of the Nigerian suspect as a potential threat, somehow doesn't impact the government's no-fly list or even enhanced security check. And people, on matters like health care, really believe more government is the answer? What goes unsaid, of course, is Barack Obama talks sternly about holding his administration accountable, but it's "words, just words"... (He'll probably let the parties put on a coat before lightly slapping their wrists.) While arguing that health care should be preventive, how preventive is Obama's handling of national security? And the word to Obama is: NATIONAL SECURITY IS JOB #1.
Musical Interlude: My Favorite Neil Young Hit: "Heart of Gold"
A deceptively simple song and brilliant performance, with a distinctive sound and despondent vocal which perfectly frames the stark lyrics.To paraphrase Shakespeare: "All that glitters is not gold."
"Heart Of Gold"
I want to live,
I want to give
I've been a miner
for a heart of gold.
It's these expressions
I never give
That keep me searching
for a heart of gold
And I'm getting old.
Keeps me searching
for a heart of gold
And I'm getting old.
I've been to Hollywood
I've been to Redwood
I crossed the ocean
for a heart of gold
I've been in my mind,
it's such a fine line
That keeps me searching
for a heart of gold
And I'm getting old.
Keeps me searching
for a heart of gold
And I'm getting old.
Keep me searching
for a heart of gold
You keep me searching
for a heart of gold
And I'm growing old.
I've been a miner
for a heart of gold.
