AustinXiao/voxturn-personal-turn-detector

ShadowMe: turn-taking that listens only to its user (preview)

End-of-turn and barge-in detection that follows one user while other people talk nearby, like a shadow that follows one person. ShadowMe was first released as voxturn-personal: this repository, the Python package (voxturn_personal, VoxTurnPersonal) and the Pipecat plugin keep that name.

A voice agent has to decide, at every pause of its user, whether the user has finished. Speaker-agnostic turn detectors, voxturn v1 included, hear whoever is speaking. When a family member, a colleague or the TV talks near the microphone, they wait while the bystander talks, or cut the user off when the bystander stops. ShadowMe takes ~10 s of the user's voice, recorded in advance or captured automatically from the start of the call, and answers only for that person.

▶ Live replay: the same Pipecat pipeline and the same VAD, with voxturn v1 or ShadowMe as the turn model, on calls where someone else talks. · voxturn v1 (multilingual, no enrollment, 8 ms)

📄 Technical report: method, training data, the path to the model and the full evaluation.

🤗 Try it in the browser, or call it as an API: give the user's voice and a turn where someone else talks, and compare ShadowMe with voxturn v1; no GPU needed (see "Hosted API" below).

Preview. Enrollment-conditioned behaviour in English, one enrolled user per session, Pipecat. The benchmark and the LiveKit plugin come with the full release.

Real multi-party recordings, a corpus it never saw (DiPCo dinner parties)	10–16 points fewer cut-offs than voxturn v1 (far-field device: 61.7 → 46.0%, close-talk 50.3 → 40.1%)
Phone calls, a bystander nearby (0 dB)	20.5% cut-offs with the bystander across the room, 20.2% close by; voxturn v1 47.3 / 51.4%
Everything voxturn v1 is tested on	eot-bench English 10.4% / 563 ms (v1: 11.5% / 669 ms); 14-language mean 17.5% (v1: 19.9%), better in 13 of 14 languages; TurnBench end of turn and interruptions level with v1; Easy-Turn 89.4% (v1: 88.3%); see "Against voxturn v1"
Barge-in: only the user stops the agent	while the agent speaks, stops for 90% of the enrolled user's interruptions and for 2% of bystanders' speech (voxturn v1: 54% / 45%), at 10% of the user's backchannels; see "Barge-in"
Who is speaking	follows the enrolled voice on real recordings at least as well as the previous checkpoint (reference-swap test, 6 real sets)
Enrollment	recorded in advance (enroll) or captured from the first 10 s of speech in the call (call_enrollment); without one it is a normal speaker-agnostic turn detector
Size / speed	2.1B parameters (Qwen3-ASR-1.7B + ReDimNet-B3); 22 ms per decision on an NVIDIA L40S (15 ms without an enrollment), 4.8 GB GPU memory

Quick start

pip install torch==2.11.0 torchaudio==2.11.0 --index-url https://download.pytorch.org/whl/cu128   # the pair for your CUDA
pip install -r requirements.txt     # transformers 5.19, peft, safetensors, huggingface_hub, scipy, soundfile

from huggingface_hub import snapshot_download
import sys, soundfile as sf

path = snapshot_download("AustinXiao/voxturn-personal-turn-detector")
sys.path.insert(0, path)
from voxturn_personal import VoxTurnPersonal

model = VoxTurnPersonal.from_pretrained(path)          # also fetches Qwen/Qwen3-ASR-1.7B-hf at a pinned revision
user = model.enroll(*sf.read(f"{path}/examples/user.wav"))       # ~10 s of the user's speech, once per user
audio, sr = sf.read(f"{path}/examples/mixture.wav")             # the user with a conversation across the room
model.predict(audio, user, questions=("eot", "target_active"), sample_rate=sr)
# {'eot': {'yes': 0.6584, 'no': 0.3416}, 'target_active': {'user': 0.0179, 'other': 0.9804, 'none': 0.0017}}
model.predict(audio, None, questions=("eot",), sample_rate=sr)  # without the enrollment
# {'eot': {'yes': 0.4727, 'no': 0.5273}}

The example ends 0.3 s into the final pause of an AMI user's turn, while a real conversation from another meeting goes on across the room (examples/SOURCES.txt).

Ask while the user is silent and every 100–200 ms while the silence lasts, and answer once eot.yes passes your threshold; keep a timeout as a fallback. predict_batch serves several sessions at once.

No enrollment recorded in advance? Capture it from the start of the call:

enr = model.call_enrollment()                 # once per call
user = enr.add(chunk, is_speech)              # every audio chunk of the user's input, with your VAD's speech flag
model.predict(last_8s, user, ...)             # user is None (speaker-agnostic) until 4 s of speech were heard

While the agent is speaking, ask whether the user is barging in:

model.barge_in(last_8s, user)                 # -> 0.0 to 1; stop the agent above ~0.8

barge_in asks with the "agent is speaking" marker: P(the user is speaking) × (1 − P(backchannel)) × (1 − P(not speech)). Bystanders and the user's own "mm-hm"s score low. Ask once your VAD has heard 0.2 s of speech, then every 100–200 ms.

CallEnrollment keeps the first 10 s of speech in the call and assumes that it is the user's. Start it with the call and give it only the user's input channel, not the agent's own voice. If someone else talks first, the enrollment follows the wrong person; record one in advance where that is likely.

Hosted API (no GPU needed)

The voxturn Space runs ShadowMe (and voxturn v1) on ZeroGPU. Call it from Python with pip install gradio_client; the clips below are this repo's examples/:

from gradio_client import Client, handle_file

client = Client("AustinXiao/voxturn-demo")     # Client(..., token="hf_...") uses your own account's GPU quota

# One decision for the enrolled user, about the moment at the end of the clip (the last 8 s are heard)
client.predict(
    handle_file("examples/user.wav"),      # ~10 s of the user's voice, or the start of the call (its speech is
                                           #   used, up to 10 s); None = no enrollment (speaker-agnostic)
    handle_file("examples/mixture.wav"),   # the audio up to now, other people may talk (English)
    False,                                 # True while the agent speaks: returns barge_in instead of eot
    api_name="/personal_decide",
)
# {'eot': {'yes': 0.6654, 'no': 0.3346}, 'resume': {...}, 'state4': {...},
#  'target_active': {'user': 0.0164, 'other': 0.9817, 'none': 0.0019}}   <- the bystander is talking, not the user
# with True: {'barge_in': 0.0112, 'target_active': {...}, 'state4': {...}, 'invalid': {...}}   <- don't stop the agent

# ShadowMe next to voxturn v1 over a whole turn (up to 60 s), every 100 ms
plot, s = client.predict(
    handle_file("call_start.wav"),   # the enrollment, as above
    handle_file("turn.wav"),         # the turn, with someone else talking
    "8.0-8.4, 10.1-11.6",            # the user's pauses in seconds; "" = estimated from the audio
    0.85, 0.8,                       # reply thresholds of ShadowMe and voxturn v1 (the Pipecat defaults)
    api_name="/personal_timeline",
)
s["reply_personal"], s["reply_v1"]                        # when each would reply (seconds; None: no reply)
s["times"], s["p_eot_personal"], s["p_user_speaking"], s["p_eot_v1"]

Each call takes several seconds (the GPU is allocated per call), so the hosted API is for trying the model and for offline analysis; a real-time agent should run the model itself (Quick start, Pipecat). Calls count against the caller's daily ZeroGPU quota (more when called with a logged-in or PRO token). The Space is also an MCP server (https://huggingface.co/proxy/austinxiao-voxturn-demo.hf.space/gradio_api/mcp/).

Use with Pipecat

pipecat_voxturn_personal.py replaces Pipecat's Smart Turn analyzer, like voxturn v1's pipecat_voxturn.py (Pipecat ≥ 1.12):

from pipecat_voxturn_personal import VoxTurnPersonalAnalyzer
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

analyzer = VoxTurnPersonalAnalyzer()          # one per call; the enrollment is captured from the call's start
strategies = UserTurnStrategies(stop=[TurnAnalyzerUserTurnStopStrategy(turn_analyzer=analyzer)])

Pipecat's VAD reports anyone's speech. While it does, the analyzer asks the model every 200 ms whether the user is the one speaking. When the speech is someone else's, it counts the moment as the user's silence, so the turn can end while a bystander talks. Pass enrollment=model.enroll(...) to use a recording made in advance instead of the call's start.

voxturn v1's Pipecat test, rerun for all three on the same machine: 150 eot-bench English turns streamed through Pipecat's Silero VAD (no bystander; ShadowMe captures its enrollment from the turn itself; a turn counts as cut off if any of its pauses was):

Turn analyzer	Turns cut off mid-turn	Turn end after the final silence starts (median / mean / p90)
Smart Turn v3 (Pipecat default)	41.3%	0.14 / 1.37 / 3.10 s
voxturn v1, threshold 0.8 (its default)	12.7%	0.30 / 0.36 / 0.86 s
ShadowMe, threshold 0.85 (default)	11.3%	0.30 / 0.48 / 1.01 s
voxturn v1, threshold 0.9	8.7%	0.38 / 0.70 / 1.55 s
ShadowMe, threshold 0.9	8.0%	0.48 / 0.70 / 1.38 s

With someone talking nearby (80 phone calls replayed the same way, see the live replay), ShadowMe answered within 1 s of the caller finishing on 54% of calls (voxturn v1: 35%), jumped in early on 33% (35%) and kept the caller waiting over 1 s on 14% (30%).

Barge-in

While the agent speaks, should it stop for what the microphone hears? Only for the enrolled user taking the floor. Test set built from TurnBench dev (real two-party calls, 3 annotators): each speaker in turn is the user, the enrollment is the user's first 10 s of speech in the call, and the events are the user's real interruptions and backchannels (clean, and with a far-field bystander talking throughout) and a bystander who starts talking while the user is silent (another speaker's ordinary speech, real interruption or backchannel; as loud as the user, close by or across the room). Each detector is asked every 100 ms once the VAD has heard 0.2 s of speech; its threshold lets through at most 10% of the user's backchannels (2-fold cross-validation over conversations).

Detector	User interruptions stopped for (median delay)	Bystanders stopped for
VAD alone (0.2 s of speech)	91% (1.4 s)	50%
voxturn v1, interruption vs backchannel	54% (0.9 s)	45%
ShadowMe barge_in	90% (0.8 s)	2.2%

A bystander who says a real interruption close by is the hardest case (4.5% stopped for).

Several users in one session (experimental)

Several people can be enrolled for one session, e.g. two family members who both talk to the agent:

answers = model.predict(last_8s, [alice, bob], questions=("eot", "target_active"))   # one answer per user
model.barge_in(last_8s, [alice, bob])                                               # the highest of the users
analyzer = VoxTurnPersonalAnalyzer(enrollment=[alice, bob])                         # Pipecat; or add_user() in a call

The model is asked once per user (in one batch: 22.7 / 33.0 / 44.8 / 54.7 ms for 1 / 2 / 3 / 4 users on an L40S). VAD speech counts as the users' when any of them is speaking, and the turn ends on p(end of turn) of the user who spoke last. The model was trained with one user; nothing here is trained for several.

Two users in AMI test meetings (16 meetings, every pair of participants in turn; one of the others answers as the agent, the fourth is a bystander; 1,503 turns, 6,415 pauses within turns, 960 of them where the other user goes on):

	Headset mix	Far-field array
False cut-offs @ 300 ms, only one of the two enrolled (the other treated as a bystander)	65.0%	67.1%
False cut-offs @ 300 ms, both enrolled	53.0% (−12.0 [−13.6, −9.0])	57.7% (−9.4 [−11.5, −6.8])
Who spoke last, taken from the model (vs the annotation)	95% right	89% right

Barge-in test with people enrolled who are not in the call (extra enrollments of other speakers), at 10% of the user's backchannels:

Enrolled	1	2	3	4
User interruptions stopped for	90%	87%	86%	86%
Bystanders' speech stopped for (close by / across the room)	1.6 / 2.4%	8.0 / 7.5%	10.4 / 11.1%	13.9 / 14.1%

Each extra enrollment is one more voice a bystander can be mistaken for: enroll only the people who use the agent. When one user stops and another goes on, the agent still answers early about as often as with one user enrolled (about 20% of these pauses at the default threshold): the handover is not learned yet.

What it answers

As in voxturn v1, every question is a multiple-choice prompt over a shared audio prefix, answered in the same forward pass.

Question	Options	Typical use
eot: has the user finished their turn?	yes / no	the endpointing decision
target_active: who is speaking now?	user / other / none	whether VAD speech is the user's (see Limitations)
resume: when will the user go on?	within half a second / later / not this turn	wait a little longer
state4: utterance state	complete / incomplete / backchannel / wait	Easy-Turn style dialog state
intent, invalid	as in voxturn v1	not evaluated with bystanders

How it works

The model is voxturn's multiple-choice speech LLM on Qwen3-ASR-1.7B. A ReDimNet-B3 speaker embedding of the enrollment conditions it through four zero-initialised paths:

1. reference tokens in the prompt;
2. per-token voice matching: the cosine between each audio token's voice and the user's, added to the audio tokens;
3. a target-speaker VAD module;
4. feature-wise modulation inside the audio encoder.

Training mixes real turn-taking data with synthetic user + bystander mixtures (−5 to 10 dB, random rooms, the bystander covering the user's last words) and with real meetings seen from the agent's side (AMI train split: the user's turns as heard on their headset, the summed headsets and the table microphone). It also uses counterfactual pairs, synthetic and real: the same audio with two different enrollments (the user's and another participant's), where the answer must follow the enrollment. Without them a model learns "the closer voice is the user" instead of comparing voices. A clean-input teacher is distilled into the model on mixtures.

To keep everything voxturn v1 does, voxturn v1 is a second teacher: on all non-speaker training rows the model's answers are pulled towards v1's, and two kinds of probe questions carry v1's behaviour where no training data asks: "is this sound someone taking the floor, a backchannel or not speech?" asked under [Agent is speaking] on the audio (interruption handling), and text-only questions on transcripts (TEN-style end-of-utterance, end of turn).

Results

Against voxturn v1, test by test

Every test on the voxturn v1 card, rerun for both models with the same scripts (no enrollment for ShadowMe where the test has none). Text-path tests (TEN, SemanticVAD) are paired per sample (McNemar); "level" = no significant difference (p > 0.05).

Test	voxturn v1	ShadowMe
eot-bench English: cut-offs at 300 ms / latency at 5% cut-offs	11.5% / 669 ms	10.4% / 563 ms	better
eot-bench, 14-language mean cut-offs at 300 ms	19.9%	17.5%	better in 13 of 14 languages (Italian 21.0 -> 21.7)
TurnBench dev, end of turn (commit policy, recall @ FPR <= 0.1)	0.930 (FPR 0.094, p50 565 ms)	0.930 (FPR 0.092, p50 716 ms)	level
TurnBench dev, interruptions (recall @ FPR <= 0.1)	0.960 (FPR 0.100, p50 337 ms)	0.951 (FPR 0.096, p50 506 ms)	level (3 of ~350 events), answers later
Easy-Turn test (zh, state4)	88.3%	89.4%	better
Non-verbal sounds recognised as not speech (SLR99)	99.6%	99.6%	level
Valid speech judged not speech	0.06%	0.0%	better
TEN, text only: English / Chinese	80.5% / 94.4%	94.9% / 93.4%	English better (p < 0.001), Chinese level (p = 0.29)
SemanticVAD, text only: end of turn en / zh	100% / 99.0%	99.5% / 98.5%	level (p = 1.0 / 0.09)
SemanticVAD, text only: interrupt vs backchannel en / zh	80.5% / 88.4%	82.0% / 88.2%	level (p = 0.70 / 0.69)
Speed (L40S, one decision)	8.5 ms	22 ms (15 ms without an enrollment)	slower: a 1.7B backbone and the speaker branch

All numbers are cut300: the share of the user's pauses cut off by the agent at a 300 ms answer-latency budget, lower is better (eot-bench protocol, policy sweep). Brackets are 95% bootstrap intervals; Δ is the paired difference to voxturn v1 on the same rows. The enrollment is ~10 s of the user's speech from elsewhere in the same call (phone) or from another meeting of theirs (AMI, DiPCo).

Real recordings of a corpus never used for training (DiPCo dinner parties, 5 evaluation sessions, 16 speakers, 441 turns; agent view as for AMI below):

Recording	voxturn v1	ShadowMe	Δ vs v1 [95% CI]
user's close-talk microphone	50.3	40.1	-10.2 [-16.6, -4.9]
far-field device (everyone across the room)	61.7	46.0	-15.7 [-18.6, -8.1]

Real recordings, agent view (AMI test meetings, 16 users never seen in training; the agent is the participant who answers next, the other two are bystanders):

Recording (AMI test meetings)	voxturn v1	ShadowMe	Δ vs v1 [95% CI]	decidable-layer AUC, v1 → ShadowMe
user's headset + a real conversation across the room, 0 dB	73.6	59.4	-14.2 [-20.2, -10.0]	0.580 → 0.763
user's headset + a real conversation across the room, 5 dB	72.3	55.4	-16.9 [-20.2, -13.3]	0.648 → 0.799

The real-conversation rows add a stretch of another AMI meeting (other people, another room, recorded by its table microphone) to the user's own headset; 73.6% is what a detector without information scores there. Absolute meeting numbers are high for every model: after a finished sentence, the other participants decide whether the floor passes. The decidable layer (turn ends after a complete sentence vs. pauses after an unfinished one) is the comparison that matters for an agent. DiPCo is never used for training or calibration. The final recipe was chosen among four variants using these test sets, so the DiPCo numbers carry a small selection effect.

Phone calls with a bystander (SpokenWOZ users, 110 calls; bystanders: held-out Seamless Interaction speakers or another phone caller):

Condition (0 dB = bystander as loud as the user)	voxturn v1	ShadowMe	Δ vs v1 [95% CI]
no bystander	15.8	15.0	-0.8 [-2.3, 0.3]
bystander across the room, 0 dB	47.3	20.5	-26.8 [-31.4, -22.7]
bystander close by, 0 dB	51.4	20.2	-31.2 [-35.0, -26.9]
another phone caller as the bystander, 0 dB	41.9	19.6	-22.3 [-28.3, -19.3]

Without a bystander, all 14 eot-bench languages (validation, no enrollment; cut300 / latency at 5% cut-offs):

	ar	de	en	es	fr	hi	id	it	ja	ko	nl	pt	tr	zh	mean
voxturn v1	21.1	15.4	11.5	21.4	13.4	37.5	30.8	21.0	11.1	12.8	15.9	24.1	20.1	22.9	19.9 / 736 ms
ShadowMe	19.1	14.4	10.4	18.9	11.8	28.8	25.5	21.7	7.3	10.2	14.5	22.6	18.5	21.5	17.5 / 701 ms

Other tests from the voxturn v1 card (no enrollment): Easy-Turn test (Chinese, state4, audio only) 89.4% (v1: 88.3%); non-verbal sounds (SLR99) recognised as not speech 99.6% (v1: 99.6%); valid speech judged not speech 0.0% (v1: 0.06%).

Who is speaking (target_active alone, threshold for user recall 0.95 set on calibration mixtures): on phone mixtures 12.5% of the moments when only the bystander speaks are taken for the user's at 0 dB (recall 0.94), 7.0% at 10 dB; the previous checkpoint F4 had 9.4% / 4.6%. Swapping the enrollment for another participant's flips the answer correctly in 49-79% of real DiPCo moments and 69-91% of AMI ones, at least as often as F4 on all six real sets.

Speed (NVIDIA L40S, bf16, 8 s audio, through predict)

	p50	p95
1 question (eot)	22.3 ms	22.4 ms
2 questions in one pass (eot, target_active)	22.6 ms	22.7 ms
4 questions in one pass	23.5 ms	23.6 ms
4 clips in one batch, 1 question each	54.4 ms	55.0 ms
without an enrollment	14.7 ms	14.8 ms
enrollment (10 s, once per user)	19.0 ms	20.1 ms

GPU memory 4.8 GB. The default fast=True merges the LoRA into the backbone and replays one CUDA graph per input layout (batch size, questions, with or without an enrollment), captured on first use (about 1 s each); its answers equal the eager path's. fast=False runs eagerly (60.2 ms; 73.2 ms with merge_lora=False, which reproduces the evaluation computation). voxturn v1 on the same GPU: 8.5 ms (fast) / 40.8 ms (eager). bench_speed.py reproduces the table.

Limitations

• Slower than voxturn v1: 22 ms per decision on an L40S against v1's 8.5 ms (a 1.7B backbone and the speaker branch). On TurnBench, interruptions are detected as reliably as by v1 but later (median 506 ms vs 337 ms).
• target_active alone takes a bystander for the user slightly more often than the previous checkpoint F4 (12.5% vs 9.4% at 0 dB on phone mixtures); following the enrolled voice on real recordings is unchanged.
• Following an enrolled user was trained on English only; without an enrollment the model covers the 14 eot-bench languages. Prompts exist in English and Chinese (lang).
• Heavy overlap at the user's very last words still costs: the agent answers later (by about 0.3 s at 0 dB).
• Multi-party floor control (who speaks next in a meeting) is out of scope.
• The model compares voices. Very similar voices, enrollments shorter than ~3 s and far-field enrollments are harder. call_enrollment assumes that the first speech in the call is the user's.
• target_active is weaker on far-field recordings: with loud bystanders across the room it can take their speech for the user's. If your input has a better user-activity signal (e.g. a close-talk microphone with its own VAD), use it for the pauses and ShadowMe for the turn decision (examples/stream.py --pauses).
• p(end of turn) is calibrated more conservatively than voxturn v1's: tune the threshold on a few of your own calls (the Pipecat plugin's default, 0.85, cuts off fewer turns than voxturn v1's default on eot-bench English at the same median reply time).
• intent and invalid were not evaluated with bystanders.
• Several users in one session are experimental (see that section): bystanders are taken for a user more often with every enrolled person, and handovers between users are not learned.
• The Pipecat plugin handles the end of the user's turn. Barge-in is available through barge_in in Python; a Pipecat interruption strategy comes with the full release.

Responsible use

ShadowMe is a turn-taking component, not a speaker-verification system: it must not be used to identify, verify or track people. Enrol only users who have given informed consent, and store enrollments as you would store other personal data.

Licence

CC BY-NC 4.0 (weights and code), because some training data is non-commercial (see NOTICE for every dataset and its licence). The backbone Qwen3-ASR-1.7B (Apache 2.0) is downloaded at run time from its original source; ReDimNet-B3 (MIT) is included with its licence in voxturn_personal/redimnet/. The example audio is from the AMI Meeting Corpus (CC BY 4.0, examples/SOURCES.txt).

Citation

@techreport{shadowme2026,
  title       = {{ShadowMe}: Turn-Taking That Listens Only to Its User},
  author      = {Xiao, Yang},
  institution = {The University of Melbourne},
  type        = {Technical Report},
  year        = {2026},
  url         = {https://www.alphaxiv.org/abs/2611.shadowme-speaker-aware-turn-detection}
}
@techreport{voxturn2026,
  title       = {{VoxTurn}: Answering Several Turn-Taking Questions in One Pass with One Model},
  author      = {Xiao, Yang},
  institution = {The University of Melbourne},
  type        = {Technical Report},
  year        = {2026},
  url         = {https://github.com/swagshaw/voxturn/blob/main/tech_report.pdf}
}