Digging into How Self-Driving Cars Make Decisions

Self-driving cars are already cruising down roads from San Francisco to Beijing. But if an autonomous vehicle crashes, how will you know if it will choose to save its passengers or the pedestrians in the street? 

If you knew how the car made tough decisions, would you trust it more?

This question is the ethos behind the research project “Ethical,Trustworthy, Autonomous: The Future Vehicles of Tomorrow,” led by Amr El Mougy, associate professor in the Department of Computer Science and Engineering. 

His research team investigates the development of ethical autonomous driving algorithms and uses virtual reality driving simulations to analyze what makes people trust autonomous vehicles.

The project's co-investigators include Khalil Elkhodary, professor in the Department of Mechanical Engineering; Alia El Bolock, assistant professor in the Department of Computer Science and Engineering; Daoud Siniora, assistant professor in the Department of Mathematics and Actuarial Science; and Moinak Maiti, associate professor at the University of the Witwatersrand.

“Ethical, Trustworthy, Autonomous" received $300,000 in grant funding in 2025 through the African Engineering and Technology Network (Afretec), a joint initiative between nine universities seeking to accelerate Africa’s digital growth through research.

“Whenever I give a seminar about autonomous driving, I usually ask the audience who would be willing to ride one?” El Mougy said. “There's always at least a couple of people in the audience who would say I would never get into an autonomous vehicle, and their response is almost always related to trust.”

How Do Vehicles See?

Like human drivers, self-driving cars are constantly synthesizing new information about the world around them. Sensors create a visual scene for the vehicle to “see.” Passengers input their destination and the system takes this new information — plus the visual scene — to choose when to accelerate, turn or hit the brakes. 

As the car enters new environments, it creates new visual scenes and makes new decisions based on this information. This feedback loop determines how the car perceives the world and acts on these factors.

“Ethical, Trustworthy, Autonomous” investigates the mathematical modeling behind how autonomous vehicles perceive the world and evaluates how these visualization systems impact user trust in virtual reality (VR) environments. 

Research participants strap on a VR headset and are placed in the driver's seat of a car navigating a virtual neighborhood. They may encounter the vehicle making a wrong turn or the dashboard providing faulty information on speed limits.

Rinal Mohammed, a AUC computer science master’s student and a research assistant, explained that the project also assesses how humans emotionally respond to various adversarial attacks they may face during their journey. “Throughout the experiment, we're able to tell whether a person is stressed or not,” Mohammed said. The headset tracks eye movement in response to the environment, giving insight into areas of focus and participants are evaluated on their reactions.

“What we're doing is creating situations where the vehicle might actually be intentionally not trustworthy,” El Mougy said. “And then you see how people react. Are you going to give it a second chance? Are you going to just actually trust it blindly? What are you going to do?

Building Trust in the Machine 

If sensors are the eyes and ears of autonomous driving, algorithms are the brain. “You need an algorithm or system that is able to input the data coming from the perception module, and the output will be these driving decisions,” El Mougy said. 

The research project investigates how reinforcement learning can train the AI modules that synthesize this data to behave ethically in different situations. 

Reinforcement learning is how humans and animals learn. It’s when you receive negative or positive rewards based on your actions. A reinforcement learning system allows the vehicle to learn driving policy and ethics by observing the consequences of certain driving actions. 

El Mougy explained that when a vehicle takes a certain action, a digital table saves the state of the vehicle, then assigns it a negative or positive value. For example, a car could be assigned a negative reward if it drives within 5 meters of a nearby vehicle while accelerating past 80 km/h. The car stores this information in its memory and based on the assigned negative reward, learns not to take that action in the future. 

“If the car takes an action that endangers a child, for example, you can give it the worst possible negative reward you can think of. These highly extreme rewards could be things that the vehicle really remembers to avoid at all costs,” El Mougy said.

The team also tests how different ethical frameworks like utilitarianism and deontology can influence these reward systems. In utilitarianism, every action is assigned a value, and the best course of action is that which maximizes utility. Deontology explains there is only one right action in any situation, and every other action besides that is wrong.

The goal of the project is not to answer the trolley problem — the famous ethical dilemma where the conductor of a runaway trolley must choose whether to let the cart continue on its path and strike four people, or switch to a different path to hit one person. 

The team hopes to show how ethics can be applied to a car's algorithms and potentially make these technologies more approachable to the average person. The research project also seeks to create a black box for autonomous vehicles. Similar to a plane’s blackbox, this system could show inputs and outputs of a vehicle’s AI modules to explain the car made specific decisions.

“We’re very careful when trying to publish our work, not to say that this is what we think is right. This is just a proof of concept,” El Mougy said.

Safety and Security from Hackers

“Percepti-check,” is a closely related research project led by Alia El Bolock, assistant professor in the Department of Computer Science and Engineering, with El Mougy collaborating as a co-principal investigator. The project explores the security of advanced driver assistance systems (ADAS) and what traits make people respond differently to attacks on these systems.

ADAS features, such as blind spot detection, rely on radar and other sensors to operate. If a hacker altered the sensors, the ADAS system could misinform drivers about whether there is a car in their blind spot, potentially resulting in a crash.

“Percepti-check” develops multi-layered attacks on ADAS systems and deploys them in a VR environment. A research participant may be driving, when they suddenly receive strange messages on the simulated dashboard — for example, their virtual speedometer says to drive at 50 K/M per hour but the street sign says to drive at 30 K/M. 

“It tells you ‘watch out there is a bump ahead,’ or ‘there is a stop sign ahead’ — and you either let it decide for you or you have to take over,” El Bolock said. “When would you actually take over, when would you actually trust the system, and why not?”

El Bolock’s research focuses on human-computer interaction and character computing, and the project investigates what character traits can make a person more or less vulnerable to these attacks. Traits such as age, gender, occupation and driving history, may make a person more vulnerable to certain attacks. These same factors can also influence a user’s trust in an autonomous vehicle’s capabilities.

Looking Forward

Currently, fully self-driving cars with an unlimited scope of movement under any condition are not available to the public. Autonomous vehicles such as Waymo are at the fourth level of self-driving: High Driving Automation. At this level, vehicles can operate in self-driving mode in a limited area and mostly do not need human interaction, but still include driving gear for humans to manually override the vehicle. 

El Mougy founded the University’s Autonomous Car Lab in 2023, where students and researchers are developing an autonomous golf cart. This year, they achieved remote operation, allowing users to drive the golf cart around AUC New Cairo using an Android phone and an internet connection. He hopes to expand the project to aid in data collection and research in “Ethical, Trustworthy, Autonomous.”

With new advancements in AI since the project began in 2023, the research team is now testing how large language models can help in the car’s reward system. But as scientists and engineers approach the development of fully autonomous vehicles, the public needs to understand and trust the ethics behind these inventions.

“I think right now it's something that is opaque. It's complex, and people don't really know how it works. So some people will tend to distrust it, but I think this will change over time,” El Mougy said.