The benefits – and risks – of lie-detecting apps

November 20, 2018

Ehsan Hoque outside Wegmans Hall, home of the Department of Computer Science and the Goergen Institute for Data Science at the University of Rochester.

An Early Career Award for Scientists and Engineers from the Army Research Office will enable Ehsan Hoque to develop lie-detecting technology, but also explore ways to ensure it is used responsibly.

Imagine if lie-detecting apps became widely available on everyone’s phones or computers. Merely by processing a video of what you are saying to someone – by analyzing your facial expressions, body movements, auditory details, and the semantic content of your speech --  the apps could accurately predict if you are lying or telling the truth about who you voted for or about your sexual orientation.

“What will that do to society?” asks Ehsan Hoque. “What will it do to our relationships?” 

So, even as the University of Rochester computer scientist creates ground-breaking computer technology to detect lies for a host of beneficial medical and public safety purposes, he also wants to help ensure the technology is used constructively -- and ethically. 

He will further explore both the technology and its proper use with a $1 million Early Career Award for Scientists and Engineers (ECASE-ARMY) from the Army Research Office (ARO). ECASE awards are the highest honor bestowed by the ARO  to outstanding scientists and engineers beginning their independent careers. The awards recognize researchers who pursue  innovative research at the frontiers of science and technology and who also engage in scientific leadership, education, or community outreach.

“This is a big boost that will help my lab continue to pursue blue-sky research,” says Hoque, an assistant professor of computer science and the Asaro Biggar Family Fellow in Data Science. He is also interim director of the Goergen Institute for Data Science.

“The Army Research Office encourages young scientists to take on ridiculously difficult problems and allow ample intellectual freedom to pursue them,” Hoque says. “It is a wonderful opportunity for young scientists like myself to take on high risks early in their career.”

The technology Hoque hopes to develop with this award builds upon his lab’s recent work on understanding deception based on facial and verbal cues. His Rochester Human-Computer Interaction (ROC-HCI) Lab  used data science and created an online crowdsourcing framework called ADDR (Automated Dyadic Data Recorder) to build the largest publicly available deception dataset so far – and to discover, for example, why some smiles are more deceitful than others.

Applications include: 

  • improved security screening of passengers in airports,
  • helping medical practitioners realize whether patients are hiding suicidal symptoms,
  • new ways for law-enforcement agents to assess risk when engaging in self-defense,
  • helping individuals with autism realize when they are being manipulated or deceived.

Making interrogations more objective

With the new award, Hoque and his lab will develop machine learning algorithms that could make the interrogations and interviews used in these settings more objective.

Often an interrogation is conducted in distinct phases, starting with general questions to establish rapport.

“How someone answers in the rapport-building phase could influence how the interviewer looks at that person when the important, relevant questions are asked in the subsequent phases, biasing the interviewer’s ability to tell if someone is lying or telling the truth,” Hoque says.

“So, imagine as I’m interviewing you, I have a computer that is helping me out, that can treat each phase as unique and objective,” Hoque says. “We can build algorithms to help quantify some of nuances that an interviewer might miss – subtle inconsistencies among facial cues, what is said, how it is said.

At the end of each phase, the algorithm, trained with reinforcement learning,  would recommend whether it has “seen” enough to recommend going to the  next phase, or whether additional questions should be asked.

The interviewer is still be in charge, still making the decisions, Hoque emphasizes, but “the algorithm is providing  independent, quantifiable metrics, so that the interrogators can further quantify their decisions. It adds objectivity and transparency to the interrogation process.”

Greater objectivity could benefit both the interviewer and the person being interviewed, especially in security and law enforcement settings, he says.

Legal and ethical issues

But Hoque worries about what could happen when the technology becomes widely available. Worst case scenario: Mandatory use of the technology to conduct job interviews.

“We have talked with people who say they would never consider lying  on job interviews but would be unwilling to do an interview if a camera was trained on them with lie-detecting technology,” Hoque says. “It’s the comfort factor. So what kind of protection would be available for these people?”

To explore the legal and ethical issues that might arise, his team will identify the ways the technoloty could be misused and recommend a framework so the invention would have the maximum net gain for society.

Hoque will collaborate with Mark Frank, a professor of communication at UB Buffalo who has more than 25 years of experience studying deception with US law enforcement agencies.

“I have a fairly well-developed eye for work that would be not just scientifically sound, but also of clear utility, especially in law enforcement and national security settings,” Frank says. “I strongly believe Dr. Hoque is on the right track here to make some practical breakthroughs.”

Technologies that improve human ability 

“Prof. Hoque is considered a pioneer in designing and validating AI technologies that can improve human ability,” a Department of Defense fact sheet notes.

For example, as a PhD student at MIT, Hoque developed an app called   – a 3D character that can automatically play the role of a recruiter, conduct a job interview, and provide feedback. The app provided the first scientific evidence that it is possible for humans to improve their face-to-face interpersonal skills by interacting with a computer.

Hoque has since applied this framework across multiple domains: public speaking, music training, effective conduct of meetings, and end-of-life conversations between cancer patients and their oncologists, among others.

His previous honors include a National Science Foundation CAREER award and selection as one of Science News’s 10 Scientists to Watch and MIT Technology Review’s Top 35 Innovators under 35.

He is also an inaugural member of the ACM Future of Computing Academy, established to guide and empower the next generation of computing leaders.