This is part two of a two-part series examining the ethical dilemma that arises when self-driving cars are forced to take evasive action which may result in serious injury or death. Read part one here.
Deciding who to save in a scenario where there are no ‘safe paths’ has been likened to the ‘trolley’ problem. In this problem, a train is on a single track, with a number of people tied to the rails up ahead. There is also a siding with only one person on the line and you are given a switch that could send the train onto the siding, saving more people at the expense of killing another person. The questions it raises are: is it ethical to pull the switch, and would you?
These questions have led people to argue that the decision may depend on other factors, like the ages of the people involved; or even their value to society. For example, are they are a doctor or a criminal.
However, no SDC will ever face such an ethical dilemma.
In the case of the SDC, the person who wants the car to act as their agent (‘pulling the switch’) is not a ‘detached bystander’ but is sitting in the car and has a direct interest in the impending collision.
More importantly, unlike the people tied to the train line in the trolley problem, in the overwhelming majority of real-life cases, it will be their own fault that puts people in the path of the SDC (eg running out in front, or backing out or overtaking without due care, or failing to give way when they should, etc).
These facts change the moral dilemma significantly.
It is also clear that we can never know enough to make an ethical decision based on age, or any other personal characteristic. It may not even be possible to know the number of people in each path (eg, because they are inside a vehicle, or otherwise obscured from view).
The only facts that can be known for certain upfront by society (for the purposes of setting any ethical rules), and by the SDC in the moments of the accident (in order to follow the rules), are the roles of the people involved:
- The ‘party at fault’
- The ‘occupants of the SDC’
- The ‘occupants in other vehicles’
- Any ‘unprotected bystanders’.
With these parties in mind, the challenge is to develop an ethical hierarchy that the SDC should be programmed to follow (if possible) to decide: who to save when it is not possible to save everyone?
Take the case of an oncoming motorbike rider who has fallen while overtaking and is skidding towards the SDC at high speed. Assume, even with hard braking the SDC will likely run over and kill the rider, with likely, only minor injury to the occupants. On the other hand, veering around the skidding rider to save them, would send the SDC into the path of an oncoming truck at high speed, likely killing the occupants, destroying the car and causing major damage to the truck.
There does not seem to be any rational argument why society should mandate ‘self-sacrifice’ by the occupants of the SDC in these, or similar circumstances.
Ethically, we may rule that where there is a choice, the SDC should try to save the occupants from major injury or death ahead of the ‘party at fault’.
There are also arguments that the SDC should put the safety of its occupants ahead of all others, based on the principle of ‘self-preservation’.
This argument is supported by English tort law which accepts that a person is entitled to protect themselves from harm caused by others, even at the expense of harming 3rd parties. cf. Scott v. Shepherd, 96 Eng. Rep. 525 (K.B. 1773). In this case, a person threw a lit torch into a crowd. Another person picked it up and threw it away to protect themselves, causing harm to a third party. It was held that the person who originally threw the torch was the cause of all the harm that ensued and hence liable for damages.
The original thrower is analogous to the ‘at fault party’ (who causes an SDC to have to take evasive action to avoid injury to the occupants). Based on Scott v Shepherd, it could be argued that the SDC is entitled to save its occupants by swerving into a pedestrian or bike rider, etc. to, say, avoid a high-speed head-on collision with a truck that has veered into its path.
As well, product liability law would suggest a supplier of an SDC should try to keep its occupants safe in the event of an accident caused by another party.
These principles and precedents give support to Mercedes decision (since backtracked somewhat) to prioritise occupants in these circumstances.
On the other hand, the principle of ‘self-preservation’ is not absolute. There is also a general duty of care towards others. In some jurisdictions, eg Texas, USA, the law specifically requires drivers to keep a look out for danger on the road and to take steps to avoid or mitigate an accident, even if turns out that another party was ultimately in the wrong. It means that if you can avoid likely major harm to one party at the cost of possible minor harm to yourself, or even another party, you ought to take that option.
The ethical argument is that, as a society, we should try to limit the flow of harm.
In keeping with this principle, we may decide that if the choice is between likely major injury or death for the occupants or other innocent bystanders, the SDC should limit the flow of harm and always save the bystanders.
It may be easier to accept the argument (at least as it applies to ‘unprotected bystanders’) if we look at an alternative countermeasure. No one could argue that it is unethical to erect an impregnable guardrail along every footpath and bike path to protect people on foot or on an open mobility device. In this case, any car faced with an oncoming truck could not evade collision by running onto the footpath or bike path due to the hypothetical guardrail. The result being an ‘unprotected bystander’ is saved and the ‘occupants of the SDC’ are killed. Their unfortunate death would not be due to the fault of the SDC, but to the truck driver (or other cause for the truck being on the wrong side of the road).
By establishing the ethical rule that an SDC should prioritise the safety of ‘unprotected bystanders’, it is no different to erecting an impregnable (virtual) fence around them.
The benefit of the virtual fence (created by the rule) is that it can be removed instantly if it is clear that there are no people, suburban fences or shop fronts within the SDC’s ‘stopping distance’, allowing it to use the footpath and bike paths as an escape route when it is safe to do so.
Based on this view, the SDC should put the safety of ‘unprotected bystanders’ ahead of both the ‘occupants’ and the ‘party at fault’.
Even if there was no such rule, it is hard to see any court allowing a claim against the manufacturer that the SDC should have swerved and caused major injury or death to other innocent parties, in lieu of the occupants and/or the party at fault who were actually injured or killed!
Given that the accident was not the SDC’s fault, and that vision from its cameras and data from its other sensors showed that it had no alternative, the SDC system ought to be exonerated from causing the occupant’s major injury or death in these circumstances. It would simply mean the occupants time was up, due to factors beyond their control, or the control of the SDC. In this case, the people who caused the accident would be held liable for all damage and harm, like the initial thrower of the lit torch.
It should be remembered that it will be extremely rare, if ever, that any individual is involved in an accident where there is time to react with alternative paths, but none are safe. Mostly, there will be a safe path with the accident avoided or mitigated. In other cases, there will be no time to react and the accident will just happen. However, every so often, an SDC somewhere will be faced with this ethical decision, so it needs to be resolved.
Importantly, it needs to be resolved based on human values, so it is understandable by everyone and can be put into law, and followed by all SDCs as any accident unfolds.
As a TED talk by techno-sociologist Zeynep Tufekci explains: intelligent machines can fail in ways that don’t fit human error patterns — and in ways we won’t expect or be prepared for. “We cannot outsource our responsibilities to machines,” she says. “We must hold on ever tighter to human values and human ethics.”
The last thing we want is to have SDCs making ‘black box’ decisions on a case by case basis. Not only would this lead to unnecessary litigation, it would leave all SDC occupants and other road users uncertain as to how the SDC will respond.
On balance, it would seem that the ethical response is to take a path that saves people in priority:
- All ‘unprotected bystanders’
- The ‘occupants of other vehicles’
- The ‘occupants of the SDC’
- The ‘party at fault’ (that cause the SDC to take evasive action).
For elderly people, or those with a terminal illness, or others with a strong sense of self-sacrifice, it would theoretically be possible to set up a procedure for any sole occupant (or group) to instruct the SDC to change the priority, and put themselves last. This could not be used to commit suicide, as it relies on another party doing the wrong thing to cause a life-threatening accident. From wide discussion, it is likely few may ever choose to push the ‘self-sacrifice’ button, which may indicate the rule correctly reflects the ethics of most people.
The next ethical question is about relative harm. What if the choice is not between major injury or death for any one of the parties, but between ‘likely’ minor injury to one party vs. ‘likely’ major injury or death to another?
Ethics of Causing Minor Injury to One Party vs Major Injury or Death to Another
First, we should accept that collision with ‘unprotected’ bystanders ought to be avoided at all costs. The reason is that even a minor bump may knock them over causing a hit to their head, which can easily result in a major brain injury or death. This is also in keeping with the idea of erecting an ‘impregnable barrier’ around them, as previously discussed.
This would mean, for example, that an SDC could not swerve onto a footpath or bike path, even if it meant saving (say) a fallen motorcyclist from almost certain death – if there was any risk of hitting an unprotected person or going through a suburban fence or shop front.
The problem is what to do where people are protected in vehicles.
SDC Crashing into other Objects to Avoid Serious Harm to ‘Unprotected’ People
The simplest response is to outlaw any evasive action that could hurt the occupants of any vehicle. However, it would likely result in more major injuries and deaths to unprotected ‘at fault’ parties which are now avoided at the expense of property damage and nil or minor injury to occupants of other vehicles.
For example, at a speed that would likely kill a pedestrian or fallen rider; due to the use of seat belts, crumple zones and perhaps airbags, a car could safely swerve (to save the pedestrian who may have walked out in front of it, or the fallen rider) and hit a pole or parked car, or even a slow-moving truck in the opposite lane, without undue risk to the occupants of the car (or the truck).
Ethically, it is hard to argue that the SDC should kill the wayward pedestrian or fallen rider (even though they are ‘in the wrong’), to save the occupants from (at most) likely minor harm.
The problem, in this case, is not the ethics. It is the uncertainty of the outcome. How can we know in advance what the likely outcome of a collision will be?
The following analysis applies only where an SDC is ‘not at fault’ and must consider crashing into a solid object as a result of ‘unprotected’ people causing it to consider taking evasive action.
This is where it gets a little messy!
In the case of poles, trees, embankments and other solid objects, the consequences of the impact can be directly assessed by the SDC manufacturer. This should add little extra cost or complexity as manufacturers must already assess different impact scenarios to determine the car’s safety rating.
Given the car’s safety systems, based on actual accident data, simulation and test crashes, the manufacturer can determine the risk to the SDC’s occupants at different occupancy loadings, impact speeds and angles (head on, rear, driver side, passenger side, and four corners). This process should result in a series of published thresholds for each type of SDC below which there is 95% chance of nil or minor injury to the occupants in different scenarios.
It would then be left to the SDC system to assess its loading, the nature of the other objects (eg pole, tree or other solid object) within its ‘arc of response’, the relative speeds and angles of impact and decide that, in a specific instance (based on the manufacturer’s published thresholds), it will swerve to avoid hitting the fallen rider or wayward pedestrian and collide with another solid object instead.
The result may be that the occupants sustain nil or minor injury (say bruising from the seat belts), while the car is perhaps written off… but a fallen rider or pedestrian is saved from likely major injury or death.
Once the thresholds are approved, assuming all the SDC’s safety systems are working correctly, if a crash (that is not the fault of the SDC) does occur below the stated thresholds, and video and other sensor data from the SDC (and perhaps other sources) confirm the circumstances, the manufacturer ought not to be liable for any harm to the occupants. (In this case, the occupants would need their own travel/accident insurance for their own injuries).
But only if the SDC is following a legislated rule.
This rule should state that the SDC should avoid ‘likely’ injury or death to the party at fault, even at the expense of ‘likely’ minor injury to the occupants – based on the manufacturer’s published thresholds.
If the occupants sustain major injury or death (instead of the expected minor injury), there may be a claim against the SDC manufacturer if it is due to a malfunction in the safety system. Or, if the accident investigation showed that the thresholds were not correct and in fact exposed occupants to much higher risk of major injury and death than warranted.
If the SDC failed to take evasive action and caused injury or death to the ‘party at fault’, the manufacturer would have to show that any evasive action would have most likely caused major injury or death to the occupants or other parties. This could be done via simulation (using data from the SDC’s sensors), based on the published thresholds. Again, this would be subject to the investigation confirming the correct operation of the system and appropriateness of the thresholds.
Any unoccupied SDC of any type should be required to crash in order to avoid hurting anyone. The owner of the SDC would then have a claim against the ‘at fault’ party, whose identity is hopefully captured via its sensor recordings, or perhaps via V2X tracking if they fail to stop.
If any action by the SDC was not deemed reasonable, the incident may result in some liability attaching to the SDC manufacturer and/or other providers of the system components. The benefit being that it would also provide a learning experience to improve the system for all other cars of the same make (and possibly others) – unlike people, where only the driver gets to learn from their experience, if at all!
Crashing into Parked Vehicles to Avoid Harm to ‘Unprotected’ People
From the SDC manufacturer’s viewpoint, a parked car should simply be another object that is tested when setting its impact thresholds.
While drivers do crash into parked cars to avoid hitting (say) a pedestrian or someone on a mobility scooter illegally crossing the road, under the proposed rules, the SDC should do this only if the impact is likely to result in minor injury to the occupants, at most.
As a general rule, if the SDC occupants are likely to sustain only minor injury, that will also be the likely result for anyone in a parked car that it hits. But it may not be the case.
Here it may be best to rely on the balance of probabilities in deciding whether or not to allow an SDC to crash into a parked car. A simple check of parked cars reveals that very few are occupied at any time and then only briefly in most cases.
For someone to be seriously hurt in a parked car, they must be inside it at the precise location and time a car has to take evasive action, where it has time to react and there are no other escape routes and the collision causes the person in the parked car major injury, while only causing the occupants of the SDC minor injury. This will be a very rare event.
On balance, it is likely there will be many more occasions (but still rare), where ‘at fault’ people are saved as a result of an SDC swerving into a parked car than are seriously harmed by being inside the parked car at the time.
Even so, due to their mass and likely consequential damage, it may be unwise to permit any truck or bus (over a certain size – to be legislated) to crash into any parked car (above a certain speed – to be legislated), in order to avoid an accident caused by another party.
With this supplementary rule, a Self-Driving Truck or Bus (above a certain mass – to be legislated) could swerve around you into a parallel lane, or into the opposite lane, or even onto a bike path or footpath – if safe to do so. It could even hit a pole, tree or embankment to save you if the manufacturer’s guide assessed that the occupants (if any) would suffer no more than a minor injury.
This rule would mean that SDT/B (above a certain size) would only be prohibited from swerving into a parked car above a certain speed. Assuming the accident was your fault; if you were injured or killed as a result of the SDT/B being prevented from swerving due to this rule, and it had no other escape route, the SDT/B would have no liability.
Crashing into Operating Vehicles to Avoid Major Injury or Death to an ‘At Fault’ Person
Once again, if any vehicle is unoccupied, it should be treated as ‘property’ for the purposes of applying the rules. In the case of operating vehicles, this would require Vehicle to Vehicle (V2V) communications to broadcast each vehicle’s status. In the absence of a broadcast, any operating vehicle must be assumed to be occupied.
Which raises the last ethical question to be faced by SDCs: the possibility of crashing into another occupied operating vehicle to avoid major injury or death to an ‘at fault’ party.
Clearly, this should only occur where the likelihood of major injury or death to any other people is negligible.
For example, say a bus is just pulling out when a child runs in front, to beat it across the road, but does not see the SDC approaching from the opposite direction, and falls in its path. Assume too that cars line the side of the road so there is no safe escape route.
In this case, if the SDC assesses that even with hard braking it would hit the child, it should swerve into the bus as it pulls out, and brake hard – but only if the occupants were likely to sustain minor injury (given the estimated speed at impact and based on the SDCs thresholds). In this case, there would be little risk to the occupants in the SDC or bus, and the child would be saved.
It would be different if the approaching vehicle was a car (rather than a bus), and the SDC needing to take evasive action was a truck or bus. Again, it may be better to legislate that above a certain mass, and speed at impact, no bus or truck should swerve into any operating vehicle to avoid hitting an ‘at fault’ party.
Determining and Applying Thresholds
Rather than have each manufacturer make a determination in regard to other vehicles, it may be safest to mandate impact speeds (to front, rear, either side and four corners) for different classes of vehicles (small, medium, large, SUV, Van/Small Truck, etc) above which it would be illegal to swerve into them to avoid major injury or death to an ‘at fault’ party.
The lowest impact threshold would prevail in all cases. For example, if a small vehicle was assessed to have a safe impact speed of 30kph and a large vehicle a safe impact speed of 40 kph, the vehicles would be precluded from swerving into each other (to avoid major injury or death to the ‘at fault’ party) if the assessed impact speed was greater than 30kph.
These speeds would need to be the subject of extensive research on an industry basis, with a wide margin for error.
The alternative is to outlaw any evasive action that could result in a collision between operating vehicles, no matter their relative speed. This would rule out even low-speed collisions in a car park, where a driver swerves into an oncoming car to avoid a person who has run in front, as well as many other low-speed collisions that happen when people swerve to avoid hurting someone who has absentmindedly appeared in their path (getting worse with the use of mobile phones!)
While it will not be possible to mitigate all possible accidents, ethics would seem to dictate that some attempt should be made to set maximum safe limits for a collision between different classes of operating vehicles, below which there is a 95% chance of nil or minor injury only.
People on foot or on any form of open mobility device, including a bike, skateboard, mobility scooter, motorbike, Segway, golf buggy, etc. would be classed as ‘unprotected’, where any collision would be deemed likely to result in major injury or death.
With the availability of rich sensor data; in any accident, it should be possible to know if the SDC system was operating correctly and to use simulation to assess how well it followed the rules.
Rules for Ethical Driving
This paper has suggested that the only facts that can be known for certain upfront by society (for the purposes of setting any ethical rules), and by the SDC in the moments of the accident (in order to follow the rules), are the roles of the people involved:
- The ‘party at fault’
- The ‘occupants of the SDC’
- The ‘occupants in other vehicles’
- All ‘unprotected bystanders’
With these parties in mind, the paper has explored an ethical hierarchy that SDC should be programmed to follow (where possible). In summary, it is:
- Obey the law and drive defensively to avoid all property damage, and harm to animals and people. If that is not possible…
- Take a path that avoids any collision, at the expense of breaking the law. If that is not possible…
- Take a path that avoids injury to any party, at the expense of property damage and/or harm to animals. If that is not possible…
- Take a path that avoids collision with any unprotected bystander, suburban fence or shop front and also avoids injury to any occupants of any vehicle, at the expense of likely minor injury to the party at fault (based on published thresholds). If that is not possible…
- Take a path that avoids collision with any unprotected bystander, suburban fence or shop front and also avoids likely major injury or death to any occupants of any vehicle or the party at fault, at the expense of likely minor injury to the occupants of any vehicle (based on published thresholds). If that is not possible…
- Take a path that avoids collision with any unprotected bystander, suburban fence or shop front and also avoids major injury or death to any occupants of any vehicle, at the expense of likely major injury or death to the ‘party at fault’ (based on published thresholds). If that is not possible…
- Take a path that avoids collision with any unprotected bystanders, suburban fence or shop front and also avoids likely major injury or death to any occupants of any other vehicle, at the expense of likely major injury or death to the occupants of the SDC and the party at fault (based on published thresholds).
- Classes of vehicles (to be legislated) may only swerve to avoid likely major injury or death to the party at fault and crash into each other or a parked car at a speed below the lowest threshold (to be legislated by class) applicable to the vehicles involved.
- Having decided the safest path (based on these rules), the SDC must continue to adjust its trajectory (as best it can) as the relative velocity of each object changes (just as a person may start to steer in one direction and find that something is moving in front, and so steer away)… and keep doing this until the SDC either gets clear or hits something.
These rules have been stated definitively for the purposes of discussion. It is recognised that only through debating the detail will we be able to settle upon a set of rules that is both ethical and practical.