Ethics In-Class Exercise

Reading/Preparation

Practice

  • Individually, consider the case study below and the ethical issues it raises. Use the ACM code of ethics as a starting framework, but feel free to consider other ethical issues in terms of other classes you have taken or reading you have done.
  • In your small group, discuss the case study in terms on the ACM’s Code of Ethics and the Ethical Frameworks in the reading.
  • Record for a whole-class discussion the sections of the ACM Code that your group thinks are most significant when applying the Code to the case study.
  • Also, write down any questions or concerns about the ACM Code that you would like the class as a whole to discuss.

Case Study from ACM: The Therac-25

In 1982, a computerized radiation therapy machine, the Therac-20,  was upgraded to create the Therac-25. In the process, much of the previous software that controlled the system was reused since it had been extensively tested and used for many years in the Therac-20 and was considered safe.   It was also written in assembly language and for a unique operating system that made it hard to understand and change. Automated testing was not a common practice at this time and therefore there were no automated tests.  

Unfortunately, a combination of actions by the operator could trigger errors. 

The system had two modes of operation:  X-ray mode and electron beam therapy. One error could occur when the operator incorrectly selected X-ray mode before quickly changing to electron mode, which allowed the electron beam to be set for X-ray mode without the X-ray target being in place. A second fault allowed the electron beam to activate during field-light mode, during which no beam scanner was active or target was in place.  Previous models of the Therac line had hardware interlocks to prevent such faults, but the Therac-25 had removed them, depending instead on software checks for safety.  

Operators would receive error messages (ex: “MALFUNCTION 3”) from the system that had no corresponding explanation, warning, or recovery instructions in their documentation.  Often they would simply override the system and start the process over. Reports of these errors to the manufacturer were often overlooked.

Because of intermittent concurrency errors that occurred if the operator switched modes too quickly, it sometimes gave radiation doses hundreds of times greater than was prescribed and programmed, causing massive overdoses.  Some patients reported feeling like they’d been given an electric shock during treatment, would develop radiation burns and in three cases, the patients died.  

Resources

N. Leveson, “The Therac-25: 30 Years Later” in Computer, vol. 50, no. 11, pp. 8-11, 2017. doi: 10.1109/MC.2017.4041349 retrieved from:  https://doi.ieeecomputersociety.org/10.1109/MC.2017.4041349

N. Leveson and C. Turner, “An Investigation of the Therac-25 Accidents” in Computer, pp. 18-41, 1993.  Retrieved from: http://www-inst.cs.berkeley.edu/~cs162/sp03/hand-outs/Therac_25.pdf

Wikipedia.  Therac-25. Retrieved from: https://en.wikipedia.org/wiki/Therac-25

Note: more recent incidents with radiation therapy accidents are discussed in: https://www.nytimes.com/2010/12/29/health/29radiation.html?mcubz=0