Predictable Outcomes
Cordyceps all the more in 2024
My daughter still hates it when I say it. There is always a mix of resignation, incredulity and defiance carved on her face as I speak those two words. She’s older now, a Mom of two. So much has changed, but much to my delight, not long ago I heard her using the same two words with her oldest.
It was during a disagreement over how much candy he was permitted to eat. Six-year-olds and sugar and all.
“You can’t keep eating those!” was her urgent command. It was spoken with a mix of surprise and dismay because it was only 30 minutes before bedtime. It paralyzed him. His tiny hand was more than halfway to his open mouth with another unwrapped morsel of chocolate sugary goodness. Hand frozen in place, mental gears whirred and clicked inside that Machiavellian head. At six, it’s already noticeable that he’s running at least a dual-core Pentium-class CPU in there.
“But you said I could…when we got home,” he protests, hand now lowering to drop the unwrapped kiss into the bag.
“I told you we were running late – and now it’s too late to be eating candy.”
“But Moooooom, you said no school tomorrow…” his second objection, brought before the court of Judge Mom, only served to irritate her – a tremor flashed across her eyes before she promptly brought the gavel down.
“Enough! I said no more! You’ll load up on sugar and you won’t be able to go to sleep and you’ll be all cranky in the morning when we’re all trying to go to the zoo!”
“No I won’t,” he whispered, eyes cast down at the bag of forbidden goodies.
“Yes, yes you will! It’s a predictable outcome!”
There are rare few more delicious moments in the annals of parenting, than when your adult child not only enforces the same routines on your grandkids, but then also follows up with your very own words.
I might be forgiven that phrase someday, but it won’t be soon, because truth be told, I still tend to blurt it out occasionally. The only excuse I offer is it was common at work. I might also note that there is a curious connection – a parallel if you will – between my daughter’s frequently insouciant response and that of the other engineers and managers I worked with.
At first, it was a term that they often blithely ignored. Perhaps I over-used it, but so were some of the all-too-frequent and repeated design ideas and decisions being made at the time. They were foolishly and intractably doomed to produce predictable outcomes. Bad ones at that. Like sourcing the cheapest, nastiest DVI video cables to connect an internal PC to a customer display on a kiosk.
“But it’s cheaper!” the dilettante chorus would sing.
The viewpoint of most new managers and company purchasing agents typically only extends as far as their four-column excel spreadsheets and weekly status reports – the ones where they’ll proudly proclaim how much they saved on a production run of kiosks. In one memorable case, this amounted a whopping $3200 savings for an initial batch of 500 units. What they failed to factor in was the labor hours and testing costs of repeatedly failing FCC part 15 testing1.
Halfway through my 43-year career in technology, I started noticing a huge disconnect between those who see themselves as decision makers and technical staff. At times, it got so bad that almost all new hires were people-pleasers and “like-minded” individuals instead of technically competent staff. After working a few years at one very recognizable shoe brand, I came to the inevitable conclusion that placing non-technical managers over technical groups leads to unfortunate yet predictable outcomes.
As you might appreciate, mere whispered utterance of such a phrase produced hackles to be raised. Damn the abysmal outcomes of the latest bad decision, blame everyone else and full speed ahead! More managers! Moar diversity! Hire them all!
For some, the MBA mantra that was drilled into their hapless little heads was that they’re always right and always in charge; carpe diem! And so, floods of these programmed progressive little automatons entered the workplace with visions of CTO, Engineering Director and sugarplum corner offices in their starry eyes. Engineers and engineering teams were just seen as important steps in diversifying one’s management portfolio.
But Engineers are often quirky sorts.
We’re unconcerned with acronyms after names or the titles under them. We’re invested not for the stupid long hours and weekends that we inevitably sacrifice for the projects, but for the outcomes. We want it to work. We want to see the company succeed. Degrees, titles, alma maters. None of that matters to an engineer. In fact, it proved to be a good indication someone was faking it; their emphasis or reliance on their title and/or supposed degree was a red flag, and more than a few times, resulted in frauds outing themselves.
During one particularly blind rush to populate a new project with fresh talent, I sat on an interview team charged with the final round interviews before candidates were offered positions. One memorable moment was when I noticed an applicant claimed to hold a fresh Electrical Engineering degree from Chandigarh University. Even though it was a coding job, I consider anything on a resume as fair game, so I asked a few EE-related questions. Simple ones, like summarizing Ohm’s Law. Blank stare.
“What’s an Ohm’s law?” He couldn’t even draw a diode much less describe what they do. Somehow, I was the bad guy after turning him down – I mean, I understand and support the desire to hire diverse candidates, but at least to my dinosaur thinking, they need to be competent first.
Frauds are bad in just about every profession, but they’re horrifically bad in engineering. People who are close to any kind of design and approval process for a product or system that people will interact with or depend upon have to know what they’re doing. Otherwise, well, it’s a predictable outcome. Bad things happen ( Google Nexus Q etc.,) and sometimes, people die, (Firestone tires and Ford, Chernobyl. Hyatt Regency Walkway ) spectacularly (Challenger. )
Often times, it’s not an oversight or error in the calculus of the engineers, it’s a fumble at some layer of management that is at the core of these sometimes-spectacular failures – often due to public perception fears, pressure to perform and occasionally, due to the sheer hubris of one or two individuals. To be fair, middle management can be like scraping out a living in a tank of piranha. Show any sign of weakness and you get mauled then eaten. This appears to motivate some into taking enormous risks making a name for themselves; often willfully ignoring consequences to get them some measure of recognition. Perhaps they justify risks by thinking that they know better, or because it worked over here it’ll work over there, or that it’s always worked ‘that way.’ Whatever the mental gymnastics, their indoctrinated academic amygdala kicks in – “I’m in charge, and I’m right!” – and they slip in a design change that’ll save $14 and a few pennies for each and every one of those 1200 units being manufactured.
Looks great on paper, until it doesn’t.
This very thing happened at the last place I worked, but I’m not going to name names. It was a truly great company with a lot of really good people. The guilty know who they are – it was a valuable, if not expensive lesson for them that unfortunately altered the company’s future. Managers were fired, people were let go, and things were never the same. Worse, it was bad timing. This happened during the initial product rollout. It represented the company’s very first commercial hardware venture in the private sector.
The outcome was dozens of negative customer experiences that strained even the new partnership between us and a well-known shipping company. There were far too many negative customer experiences – the kiosks collected biometric and biographical information from paying customers seeking a specialized identity credential that would permit them access to certain restricted places.
It took nearly 25 minutes to complete the enrollment process – the last step involved collection of the applicant’s fingerprints and a photograph. This is when the units would fail. They’d blank out, the software would detect the fault, quickly reset and send the customer back to the starting menu. All information gathered up to that point was lost. Naturally, I got a frantic phone call from my manager who also received a frantic phone call from the field operations manager: who’d just got back from a long weekend and was hammered with dozens of complaints about our new kiosks.
It was not a good first week for the new product in the field.
Fortunately, I had the foresight to write some remote diagnostic software that allowed us to examine all of the hardware connected to the internal PC. It only took me 30 seconds to find the problem. The wrong USB hub was installed several kiosks – in fact, in every unit with failure reports. Apparently, I let out a loud enough …um, groan… after throwing my glasses down that nearby engineers gathered around before I opened my eyes again.
In addition to predicting the need for the diagnostic software, I also recognized the all too familiar, “I know better than you” attitude from one of the project managers and his frequent lunch-time buddy, a junior purchasing agent. I knew it meant trouble when they tried to tag-team me into approving a cheaper USB 3.0 hub.
For reasons still unknown, they were very insistent – as was my own manager – that saving $16K (on a spend of just under 2 million) was important for this initial roll out. They were so bizarrely insistent that I recognized the predictable outcome ahead of time. Fortunately, I’d spent several days documenting the test results for all approved hardware in the kiosk. Because of their strange insistence, I wrote a special Engineering Report on the USB hub in question – a carefully written 2-pager with comparative results to the approved hub in an easy-to-read table. The summary stated that the device was, “an inadequately reliable hub that demonstrated repeatable electrical excursions when placed under full device load.”
This text was highlighted in bold with a yellow background, right above the final analysis declaration of: “RECOMMENDATION: UNSUITABLE – DO NOT USE,” which itself was in a pretty clear shade of 36pt red.
It was summarily ignored.
The kiosks were shipped and installed all over the U.S. – largely with the inexpensive hub I’d warned them about. Looking back on the fallout, it gave me no pleasure whatsoever to see my colleagues fall one by one into the crosshairs of our CEO who went on an absolute rampage upon receipt of the complaints during our critical product launch. Our CEO was a rare sighting most of the time beforehand, but after the USB hub fiasco broke out, he was present in almost every product meeting.
At the conclusion of the last meeting, the CEO singled me out in front of everyone. I don’t recall everything he said, but I still have his card with his hand-written personal cell and home phone numbers written on it. He told me to call him the very next time it started to happen. While I appreciated the gesture, it was a pyrrhic moment. We’d already suffered the loss I wanted us to avoid.
We lost something north of $300K and an unknown amount in lost business and lost consumer confidence. Soon after, my manager, the director of engineering was terminated as were the project manager, his manager, and the junior purchasing agent. To be fair, only a few months later we went from having 89 engineers on staff down to 3 – we’d lost a big contract elsewhere and the company was hemorrhaging money over the new fiasco. A year later, they lost additional contracts, so my siren song played all the way home.
That was my last full-time work in engineering. The company was the best place I’d ever worked, and my manager was by far, one of the best people to work for. He was smart, well-educated and had a lot of experience in engineering. In the end, his fatal flaw may be summed up as heeding the desire to appease others and to be seen as a team player over listening to his own team. In engineering as in hand-grenades, you can’t serve personal feelings when doing business. You’re either going to deliver successfully, or you’re going to blow everyone up.
My fatal flaw was pointed out to me before he left. He said, “I was too soft-spoken.”
Apparently, because I don’t use profanity-laden tirades when communicating, I’m not taken seriously when it matters. Despite an otherwise great working relationship, I knew he was deflecting. Unlike so many managers I’ve worked under, he never looked for scapegoats. That’s when I knew that something was about to happen, and to our collective disappointment, he was let go a few days later.
Why this all matters.
Wisdom is an odd thing. It’s not valued at all when we’re young and full of youthful energy – when the world is wide open to us, or at the start of our journeys. Maybe it’s a built-in rebellion module that our personal mental operating system software struggles to cope with – either validating it or working to root it out. Or maybe it’s a defiance module – one that operates at a more visceral level, closer to the metal as it were, that pushes us to prove them all wrong. Pushing us into believing, into convincing ourselves that we can do it, because we know we can do it. Even though we’ve never done it.
That module is not just useful, it’s critical in entrepreneurial venues, but without wise temperance, it is often disastrous. Wisdom appears to only be valued by those who have had the unfortunate life experience of living through some kind of foolishness that ended up costing them dearly. I have many regrettable errors I’d like a chance to do over, and this observation has revealed another module that seems to make itself known here in this twilight of one’s life.
It's the module of hindsight compassion.
This mental software extension drives you to speak up when you see predictable outcomes in the making. It’s a module that won’t let you sleep sometimes – often feeding bad bytes and archetypes to those internal subroutines that play movies when you do manage to pass out. Other times, it behaves like that entomopathogenic insect fungus, Cordyceps, which consumes you to the extent that it takes control, sitting you down in front of a keyboard and forcing you to write.
Here’s to more Cordyceps in 2024.
Happy New Year my friends, and may your life not become a predictable outcome this year.
FCC Part 15 specifies, among other things, the emitted radio frequency emission limits from devices – necessary for securing contracts and installation permits in buildings and locations where such things really do matter.
I am confused as to how one obtains an EE degree without having ever heard of Ohm's Law. I am not an electrical engineer, more like an enthusiastic member of the magic smoke liberation front, but even I know that one.