Safety of AI and Robots

Safety in AI and robotics means making sure a system does not harm people or damage property while it is working. It covers physical injury, such as a robot arm striking a worker, and harm caused by bad decisions, such as software that works out the wrong dose of a medicine.

Where the danger comes from

An AI system or robot can put people at risk in three ways. It can malfunction, for example a robotic lawnmower that keeps its blades spinning after it has tipped over. It can produce an incorrect output, such as smart traffic lights that show green to two busy roads at the same moment. Or it can meet a situation nobody planned for — a pavement delivery robot arriving at roadworks, a fallen cone and a temporary ramp all at once — and do something its designers never intended.

Testing before use

Systems where a failure could injure someone are safety-critical, so they must be tested thoroughly before they are used, and not only in the situations the designers expect. Testers try rain, gravel, low sunlight, reflective glass and awkward obstacles such as a black cat lying on the lawn, because a system tested only on a smooth dry floor has not really been tested at all.

Security and fail-safes

Cybersecurity is part of safety: a hacked robot or a hacked AI system can be made to behave dangerously, so access to its controls has to be protected. Since no amount of testing covers everything, autonomous systems often need fail-safe features. If the two position sensors on a robot helping in an operating theatre send conflicting readings, it freezes where it stands and hands control back to the surgeon. A system running without a person watching is not automatically safer — there is simply nobody there to step in.