Lab Disasters And Other Rites of Passage
by Lee H. Goldberg
Editor's Note: After reading Lee's column on his electronics disasters, be sure to learn about our first annual "War Stories" contest and share your own tales of woe with us and your fellow readers.
The random sparks, explosions, and other catastrophes that punctuate our careers provide for amusing war stories best told over a beer but, equally important, they provide us with important lessons about our profession that we never find in a textbook. Most of the best engineers I know have the best "war stories" often dating back from their youth when they delved into the mysteries of electronics on their own, armed only with their enthusiasm, a handful of components, the ARRL Handbook, or one of Don Lancaster's infamous "Cookbooks."
Since most of my high school interests involved the drama club and bumming rides at the local airport, my first technical disaster (unless you count the ice-bearing equipped rocket sled I built in Junior High) happened in my first semester at engineering school during a lab-based final exam. The exercise (which counted for 25% of our grade) was designed by a particularly sadistic professor who was doing his part to maintain the department's policy of having a 60%+ attrition rate for first year students.
The task he set before us was fiendishly straightforward -- to make measurements on two terminals of our lab benches that has been wired up to represent a "black box" power supply. From these measurements, we were to determine the Norton and Thevenin circuits for the network that lay inside. There was, of course, a catch. We had to make the readings on our black boxes using our own instruments; a voltmeter, ohmmeter, and ammeter, all based on a raw meter movement which produced a one-division deflection for every 10 mA that passed through it. We were given a shoebox full of unsorted resistors, a small proto-board, and an hour to make our measurements. We were herded to individual lab benches, and told to start on the instructor's mark. I set to work building the resistor networks I'd need to divide the voltages and currents in my measurements, all the while trying desperately to remember the equations governing current shunts.
About 10 minutes into the lab, I flipped on the power supply switch to take my second set of readings when a large flash of light and a loud "pop" erupted nearby. Expecting the worst, I reflexively dove for the power switch when I saw the puff of smoke wafting from the bench next to me. It turned out that a fellow student had accidentally placed his ammeter circuit across the terminals instead of in series with a current-limiting resistor. The resulting conflagration fried the resistors surrounding the meter in a spectacular way, and immediately ended the exam for my friend. You can bet that I double-checked each set-up after that, a habit I tried to maintain throughout my professional career.
Then there was the time I was fixing TVs, and forgot to discharge the CRT anode before working on a set. I'd been on the job with Ace TV (the best entry-level job I could find when I graduated from school into the middle of the deep mid-70s recession) about a week when I forgot my boss's stern admonition to treat a picture tube like a coiled rattlesnake. One second I was elbow-deep in the guts of an ancient Philco color console, and the next second I was slammed up against the wall six feet away when my muscles jerked as only a 35 kV charge can make them. I never made that mistake again.
Most of my other lab disasters were less spectacular, but equally instructive. Early on in my career, I had built up a small counter circuit from CMOS logic and was having a devil of a time getting it working properly. The counter seemed to divide erratically, and the output signal was distorted and much, much smaller than the 5 V supply voltage would have led me to believe. I spent an hour checking the schematic I'd drawn for errors, and another hour verifying every tie-wrap connection on the breadboard. It was only luck that caused me to discover the open power supply connection that caused my CMOS circuit to feebly attempt to run itself solely on the energy from the input of the signal generator.
Not much later, I learned that LEDs in the little "gumdrop" packages are very temperature-sensitive, and need to be soldered with an extremely light touch, or better yet, a heat sink between the solder joint and the actual device. Unfortunately, I only learned this after having fried around 50 of the little critters when I wired them into the panel of a piece of test equipment I was building.
Soon after that, I discovered that reversing Vcc and Vdd on a chip can cause all manner of excitement as the sudden current rush turns a $25 chip into a rather efficient little heater. Although I tried to triple-check the large breadboards I built carefully before powering them up, I'd still occasionally find myself frantically killing the power and using the "Braille method" to find the one chip that had had been inserted upside down and tripped the power supply current protect circuit -- but not before frying itself. Of course once in a while the offending chip saved me the trouble of fingering every IC on the board by blowing a tidy crater in its package!
Much of the middle of my engineering career involved writing assembly-level programs for 8-bit microprocessors, so most of my mistakes involved overwriting memory, writing to memory that did not exist, or watching a coding error cause a display to slowly degenerate in to meaningless "mung" instead of the fireworks that accompany hardware problems. Unfortunately, there was one final electrical disaster awaiting me that dwarfed all the others.
I cannot take direct responsibility for the incident that crossed the 28 V bus with the 100 V supply on the direct broadcast communications satellite I was working on, but I am haunted by the thought that I may have been the one who plugged in two supposedly identical cables that were part of a chain of events that caused $10 million worth of damage. The two cables carried both the 28 V main spacecraft power and the 100 V bus that ran the high-power Ku-band transponder system. One of the cables had been modified earlier that week to support a new test requirement but, in the rush of the moment, had not gone through the rigorous paperwork and re-labeling that would have alerted inspectors to a potentially fatal misconnection.
We were under considerable schedule pressure to deliver a working satellite to our customer before they could cancel the contract (that's another story), and were on a grueling 24/7 schedule. I was responsible for assembling all the necessary equipment and cabling to perform a test of the shunt regulation system that took the raw power from the solar arrays and fed it to the main power bus. I was told that the two cables were now different, and did my best to make notes in my hookup procedures, but was also too pressed for time to update all the paperwork.
Sadly, I truly don't know if I crossed the connectors when I plugged them in at the end of a 12-hour day, or if they were re-connected in the fatal configuration sometime over the next 48 hours. What I do know is that somewhere around 4 AM, a combination of software problems and a directive from an over-eager manager caused a deeply fatigued test team to override the test system's warnings and send a command that closed a relay and put 100 V across the spacecraft's main 28 V bus. While there were no sparks, or any other visible signs of trouble, the spacecraft died immediately, and it took 6 months, and $10 million worth of disassembly, repair, and re-test to make things right again.
Besides driving home the need to maintain strict documentation on any complex project, I also learned lots about how we all carry a tremendous responsibility for the success of a project, no matter how small our role is. I also learned that sometimes it's better to push back on management when what they ask is unreasonable -- something I sorely wish I'd done when they pushed our workloads beyond a sustainable level. Finally, I learned that failure is a constant companion, especially in large, complex projects, and that the only things that keep it at bay are equal parts of hard work and discipline, plus a large dose of common sense.
Comments? Questions? Tales of your own disasters and "growth opportunities"?
Write me at: lgoldberg@green-electronics.com