Noise-canceling headphones have gotten very good at creating an auditory blank slate. But allowing certain sounds from a wearer鈥檚 environment through the erasure still challenges researchers. The latest edition of Apple鈥檚 AirPods Pro, for instance, for wearers 鈥� sensing when they鈥檙e in conversation, for instance 鈥� but the user has little control over whom to listen to or when this happens.

A 天美影视传媒 team has developed an artificial intelligence system that lets a user wearing headphones look at a person speaking for three to five seconds to 鈥渆nroll鈥� them. The system, called 鈥淭arget Speech Hearing,鈥� then cancels all other sounds in the environment and plays just the enrolled speaker鈥檚 voice in real time even as the listener moves around in noisy places and no longer faces the speaker.

The team presented May 14 in Honolulu at the ACM CHI Conference on Human Factors in Computing Systems. The is available for others to build on. The system is not commercially available.

鈥淲e tend to think of AI now as web-based chatbots that answer questions,鈥� said senior author , a UW professor in the Paul G. Allen School of Computer Science & Engineering. 鈥淏ut in this project, we develop AI to modify the auditory perception of anyone wearing headphones, given their preferences. With our devices you can now hear a single speaker clearly even if you are in a noisy environment with lots of other people talking.鈥�

To use the system, a person wearing off-the-shelf headphones fitted with microphones taps a button while directing their head at someone talking. The sound waves from that speaker鈥檚 voice then should reach the microphones on both sides of the headset simultaneously; there鈥檚 a 16-degree margin of error. The headphones send that signal to an , where the team鈥檚 machine learning software learns the desired speaker鈥檚 vocal patterns. The system latches onto that speaker鈥檚 voice and continues to play it back to the listener, even as the pair moves around. The system鈥檚 ability to focus on the enrolled voice improves as the speaker keeps talking, giving the system more training data.

The team tested its system on 21 subjects, who rated the clarity of the enrolled speaker鈥檚 voice nearly twice as high as the unfiltered audio on average.

This work builds on the team鈥檚 previous 鈥渟emantic hearing鈥� research, which allowed users to select specific sound classes 鈥� such as birds or voices 鈥� that they wanted to hear and canceled other sounds in the environment.

Currently the TSH system can enroll only one speaker at a time, and it鈥檚 only able to enroll a speaker when there is not another loud voice coming from the same direction as the target speaker鈥檚 voice. If a user isn鈥檛 happy with the sound quality, they can run another enrollment on the speaker to improve the clarity.

The team is working to expand the system to earbuds and hearing aids in the future.

Additional co-authors on the paper were , and , UW doctoral students in the Allen School, and , director of research at AssemblyAI. This research was funded by a Moore Inventor Fellow award, a and a .

For more information, contact tsh@cs.washington.edu.

In virtual meetings, it鈥檚 easy to keep people from talking over each other. Someone just hits mute. But for the most part, this ability doesn鈥檛 translate easily to recording in-person gatherings. In a bustling cafe, there are no buttons to silence the table beside you.

The ability to locate and control sound 鈥� isolating one person talking from a specific location in a crowded room, for instance 鈥� has , especially without visual cues from cameras.

A team led by researchers at the 天美影视传媒 has developed a shape-changing smart speaker, which uses self-deploying microphones to divide rooms into speech zones and track the positions of individual speakers. With the help of the team鈥檚 deep-learning algorithms, the system lets users mute certain areas or separate simultaneous conversations, even if two adjacent people have similar voices. Like a fleet of Roombas, each about an inch in diameter, the microphones automatically deploy from, and then return to, a charging station. This allows the system to be moved between environments and set up automatically. In a conference room meeting, for instance, such a system might be deployed instead of a central microphone, allowing better control of in-room audio.

The team published Sept. 21 in Nature Communications.

鈥淚f I close my eyes and there are 10 people talking in a room, I have no idea who鈥檚 saying what and where they are in the room exactly. That鈥檚 extremely hard for the human brain to process. Until now, it鈥檚 also been difficult for technology,鈥� said co-lead author , a UW doctoral student in the Paul G. Allen School of Computer Science & Engineering. 鈥淔or the first time, using what we鈥檙e calling a robotic 鈥榓coustic swarm,鈥� we鈥檙e able to track the positions of multiple people talking in a room and separate their speech.鈥�

Previous research on has required using overhead or on-device cameras, projectors or special surfaces. The UW team鈥檚 system is the first to accurately distribute a robot swarm using only sound.

The team鈥檚 prototype consists of seven small robots that spread themselves across tables of various sizes. As they move from their charger, each robot emits a high frequency sound, like a bat navigating, using this frequency and other sensors to avoid obstacles and move around without falling off the table. The automatic deployment allows the robots to place themselves for maximum accuracy, permitting greater sound control than if a person set them. The robots disperse as far from each other as possible since greater distances make differentiating and locating people speaking easier. Today鈥檚 consumer smart speakers have multiple microphones, but clustered on the same device, they鈥檙e too close to allow for this system鈥檚 mute and active zones.

鈥淚f I have one microphone a foot away from me, and another microphone two feet away, my voice will arrive at the microphone that鈥檚 a foot away first. If someone else is closer to the microphone that鈥檚 two feet away, their voice will arrive there first,鈥� said co-lead author , a UW doctoral student in the Allen School. 鈥淲e developed neural networks that use these time-delayed signals to separate what each person is saying and track their positions in a space. So you can have four people having two conversations and isolate any of the four voices and locate each of the voices in a room.鈥�

The team tested the robots in offices, living rooms and kitchens with groups of three to five people speaking. Across all these environments, the system could discern different voices within 1.6 feet (50 centimeters) of each other 90% of the time, without prior information about the number of speakers. The system was able to process three seconds of audio in 1.82 seconds on average 鈥� fast enough for live streaming, though a bit too long for real-time communications such as video calls.

As the technology progresses, researchers say, acoustic swarms might be deployed in smart homes to better differentiate people talking with smart speakers. That could potentially allow only people sitting on a couch, in an 鈥渁ctive zone,鈥� to vocally control a TV, for example.

Researchers plan to eventually make microphone robots that can move around rooms, instead of being limited to tables. The team is also investigating whether the speakers can emit sounds that allow for real-world mute and active zones, so people in different parts of a room can hear different audio. The current study is another step toward science fiction technologies, such as the 鈥渃one of silence鈥� in 鈥淕et Smart鈥� and 鈥淒une,鈥� the authors write.

Of course, any technology that evokes comparison to fictional spy tools will raise questions of privacy. Researchers acknowledge the potential for misuse, so they have included guards against this: The microphones navigate with sound, not an onboard camera like other similar systems. The robots are easily visible and their lights blink when they鈥檙e active. Instead of processing the audio in the cloud, as most smart speakers do, the acoustic swarms process all the audio locally, as a privacy constraint. And even though some people鈥檚 first thoughts may be about surveillance, the system can be used for the opposite, the team says.

鈥淚t has the potential to actually benefit privacy, beyond what current smart speakers allow,鈥� Itani said. 鈥淚 can say, 鈥楧on鈥檛 record anything around my desk,鈥� and our system will create a bubble 3 feet around me. Nothing in this bubble would be recorded. Or if two groups are speaking beside each other and one group is having a private conversation, while the other group is recording, one conversation can be in a mute zone, and it will remain private.鈥�

, formerly a principal research manager at Microsoft, is a co-author on this paper, and , a professor in the Allen School, is a senior author. The research was funded by a Moore Inventor Fellow award.

For more information, contact acousticswarm@cs.washington.edu.