Anyone can program for normal

Throughout my career as an Automation Professional, I have noticed two distinct types of programmers.  Those that just take what the spec says and programs for that and those that use the spec as a guide to develop a control system.  Unfortunately, there are no courses on the differences in the two for automation folks.  For some reason, automation programmers feel there systems don’t need the kind of rigor that traditional software development professional follow.  In fact, I was told by one quality manager for a major systems integrator that they did not develop software and therefore, would not receive value from software development tools and methodologies. 

The classic battle lines between Engineering and IT would tell us there is not much Automation Professionals can gain from our IT brethren.  I disagree with this assertion as software development life-cycles and more specifically, software testing is an area that the Automation Profession has much to gain.

I am reminded of these principles when hearing of the one of the Toyota sticky gas pedal fixes.  Some brilliant person decided to add a “feature” that would disengage the throttle when ever the brake pedal is depressed.  Really??  Is that not what every single cruise control device already does?  How could something so simple be missed during design, development and testing of the software controlling that process?

Thinking back to a man-made disaster at Taum Sauk Generation Station in Missouri (http://en.wikipedia.org/wiki/Taum_Sauk_Hydroelectric_Power_Station).  It is a simple enough concept, a pumped storage electrical generating system.  During the day, a reservoir of water at the top of a mountain is allowed to flow through two generators creating electricity.  At night, the generators are turned into motors and the reservoir is refilled with water from the lake.   Over time and through cutbacks, the once fully manned station became remotely monitored and controlled, not that it would have mattered in this instance.  One night, during the pumping cycle, the level measurement system failed and got stuck at a single low reading. Well, the pumps never stopped and overflowed the resevoir which quickly eroded a poorly constructed wall which emptied the full reservoir in a matter of seconds.   The environment damage was significant as it created a trench down the side of the mountain and a lot of flooding.  It nearly killed the Johnson Shut-in’s Park Ranger and his family as they woke up to their house floating down the river.

A lot of emphasis was placed on the poorly constructed walls but what about the poorly developed control scheme.  This “accident” should have simply never happened!!!

We can assume the level control was simple enough.  When level gets low, stop the generators and turn the pumps until level is high and stop the pumps.  Now, this is a test for all of you readers.   What other things could we have interlocked the pumps on.  Well, how about stopping the pumps if the level does not change for a defined period of time thereby indicating something is wrong.  Or, since this system has been running for years, someone should have had some idea how long it would normally take to fill the reservoir.  So, lets stop the pumps after a certain total amount of time has elapsed.  That is the simple stuff and would have cost a few extra key strokes in a program.  Even if the system was still relay based, a PLC could have been implemented just for this piece at the costs of less than $10,000.  It would be very interesting to learn what really happened and how many engineers made the call that this could happen and some bean counter could not justify the additional safety measures.

What other failures do you know of that could have been easily avoided had someone thought through the problem from a failure standpoint instead of everything is normal?

How many of these type of failures go on everyday in our plants?  How many times does a knowledgable operator keep a control system from making a dumb mistake because of an arrogant programmer?  I believe the numbers are much higher than we would like to assume.

So, the next time you are out to start your next programming assignment, take a few minutes and think about what could go wrong and how your design will react to it.  It really is not that difficult to take care of a majority of the issues any system will face.