AI Lip Reading: Eavesdropping Without a Microphone or Laser

 Eavesdropping via lip reading has been around a long time, probably since Og saw Charlie mouth "shhhh" when a bear approached their cave.

My introduction to eavesdropping via lip reading came around 1976. A company president asked for a sweep of his upper floor corner office for bugs. None found. He asked about the possibility of laser eavesdropping. Very unlikely, however, a careful visual scan of nearby buildings directly across on both streets discovered, one floor down, and just off center, there was the glint of a lens, and a darkened room with a desk, and two people. Further investigation established, yes, it was eavesdropping via lip reading.

The concept of AI lip reading actually predated this case. It was 1968, in the movie, 2001, A Space Odyssey. Remember this scene? "Concerned about HAL's behaviour, Bowman and Poole enter an EVA pod so they can talk in private without HAL overhearing. They agree to disconnect HAL if he is proven wrong. HAL follows their conversation by lip reading."

It is now 2024 and real AI lip reading has just arrived from Symphonic Labs. I took advantage of their offer to test the beta app. Of course, it was not 100% accurate, but massively impressive nonetheless.  

For the test, I selected a random YouTube video. (Nate, at the Daily Drop https://www.youtube.com/watch?v=fy2a899yYec) A 10-second video clip was extracted. The clip was processed through Apple's Quicktime to remove the audio track before submitting it to the Symphonic Labs' app. 

This is what Nate is really saying about airport screening: "But if you decide to get Clear and if you do see one of the signs at the airport that has Clear and TSA Precheck then don’t assume that is your fastest option, All right, so hopefully I did a…"

Here is what the AI app reconstructs what Nate is saying:

This is an impressive start, and expected to become much more accurate as the number of training models increases. And, it’s not all for spies. Think about talking to your computer or smartphone without others eavesdropping on you. 

Better start thinking about what you say, where you say it, and establishing a relationship with a technical information security consultant to protect your business.

Sports Espionage at Euro 2012

England's challenge at Euro 2012 almost was drastically undermined after the team's tactics were leaked to rival Sweden by a snooping journalist.

Ola Billger, 40, used binoculars to spy on England coach Roy Hodgson as he outlined his defensive plan to his team at a Kiev hotel ahead of last Friday's Group D clash.

The Swedish hack watched for 40 minutes and compiled detailed notes... (more)

Reflect on this... Visual Eavesdropping

I began warning my city clients about visual eavesdropping back in the 1970's. I still do today. My concern then was not computer screens; clients didn't have them.

My concern was lip reading, and it wasn't theoretical. We had a case where this was the method of eavesdropping. The president had a corner office on Park Avenue. Big glass windows. Scores of vantage point offices across the street. We caught them in the act.

The visual eavesdropping info-target is bigger these days. Lip reading concerns have expanded to concerns about: whiteboard / projection / computer screen reading and now... reading the reflections off of shiny objects in the room. ~ Kevin

via Scientific American...
"Through the eyepiece of Michael Backes’s small Celestron telescope, the 18-point letters on the laptop screen at the end of the hall look nearly as clear as if the notebook computer were on my lap.

I do a double take.

Not only is the laptop 10 meters (33 feet) down the corridor, it faces away from the telescope. The image that seems so legible is a reflection off a glass teapot on a nearby table.

In experiments here at his laboratory at Saarland University in Germany, Backes has discovered that an alarmingly wide range of objects can bounce secrets right off our screens and into an eavesdropper’s camera. Spectacles work just fine, as do coffee cups, plastic bottles, metal jewelry—even, in his most recent work, the eyeballs of the computer user. The mere act of viewing information can give it away." (more)

Smart Spy Cameras

UK - Intelligent CCTV cameras are being developed in Britain that not only see trouble but are able to hear it, scientists said.

The technology allows the sounds of breaking glass, someone shouting, or the noise of a crowd gathering to be 'learned' by artificial intelligence software in the cameras.

The technology could slash the speed with which crimes are caught on camera and responded to by police but will again raise a debate about the extent of "surveillance Britain" and the use of such technology.

The three-year project by the University of Portsmouth aims to adapt artificial intelligence software already being developed to identify visual patterns. (more)

Spybuster's Tip #103 - Spot the Spy

Joe Navarro is an ex-FBI agent. His job was spotting spies. His weapon... he reads body language. These days, Joe writes and teaches poker players how to win.

He can help you spot deceit in your business as well.

Joe says...
"Poker players lie all the time. They pretend they are strong when they are weak or weak when they are strong. The truth is they can all be read. You can have a poker face, but I've yet to see someone with a poker body."

"When you are feeling good _ or have a monster hand _ your body will manifest what it feels. You get happy feet. Your feet begin to bounce up and down like a kid going to Disney World. We squint at things we don't like. ... The involuntary nonverbal mannerisms dictated by the brain will always betray the strength or weakness of a player's hand."

"If your boss asks at a meeting, `Who is not pulling their weight?', the shoulders will rise on those who are not confident. It's called `The Turtle Effect.' You are trying to hide your head inside your shoulders. On the contrary, a person whose fingertips meet like a church steeple with the thumb pointed up indicates a winning hand."

Want to know more?
Joe's website
A list of common spy personality traits.
Joe teaches at the World Series of Poker Academy


What Every BODY is Saying:
An Ex-FBI Agent's Guide to Speed-Reading People
(due out in April, 2008)

Uncle Sam Wants You...r Hearing Aid

Audio Exploitation
Solicitation Number: Reference-Number-BAA-07-05-IFKA
Posted Date: Jul 11, 2007
Classification Code: A -- Research & Development

"The scope of this effort covers a broad range of audio and speech processing technologies not limited to: speaker identification, language/ dialect identification, obtaining the gist of a conversation by recognition of words and phrases, uncooperative speaker audio language translation, whispered speech detection, audio transmission segmentation in continuous speech, background noise identification, channel effect mitigation, usable speech determination, interference (noise and competing talkers) reduction, voice stress analysis, speaker verification, coding to preserve the characteristics of the talker and channel, watermarking, and correlation." (more)

Lip Reading (updated)

Our clients (especially the ones in big cities) have been warned about being eavesdropped on by people who can lip read. Unlikely, but possible. We handled only one case involving lip reading in over 30 years.

This may change...

Researchers at the University of East Anglia (UK) are about to embark on an innovative new project to develop computer lip-reading systems that could be used for fighting crime.

The three-year project, which starts next month, will collect data for lip-reading and use it to create machines that automatically convert videos of lip-motions into text. It builds on work already carried out at UEA to develop state-of-the-art speech reading systems. (more)