Publications

2026

1. SLT Phone Recognition

   Phone Segmentation and Recognition through Phonological Activation Mapping

   Shikhar Bharadwaj^*, Kwanghee Choi^*, Stephen McIntosh^*, Chin-Jou Li, Eunjung Yeo, and 6 more authors

   2026

   Abs DOI

   Phone segmentation and recognition are inherently related tasks, yet modern approaches typically model them separately. We argue that phonetic structure is already latent in the representations of self-supervised speech models (S3Ms), and one only needs to steer them to solve both tasks. We leverage S3M-based Phonological Activation Mapping (SPAM), which maps each S3M representation frame to a vector of phonological feature activations, such as voicing and nasality. On top of SPAM, we introduce two simple but effective lightweight, gradient-descent-free prediction heads: a recognition head and a segmentation head. Our method requires less than a minute of phonetic transcriptions, and generalizes to unseen phones during training. Across a diverse range of datasets, our approach attains strong segmentation and recognition performance.

2. SLT Speech & Audio

   HARP: Agentic Hybrid Retrieval and Analysis for Long-Form Audio

   Chin-Jou Li, Masao Someki, Woojeong Jin, Yashish M Siriwardena, Tanmay Laud, and 2 more authors

   arXiv preprint arXiv:2609.14116, 2026

   Abs DOI

   Long-form audio analysis requires systems to localize and integrate evidence distributed across extended recordings. While existing work primarily retrieves semantic content through structured textual representations, many real-world queries depend on acoustic evidence that is better preserved in continuous representations or raw audio. We introduce HARP (Hybrid Audio Retrieval Pipeline), an agentic framework and benchmark for systematically studying retrieval and evidence representations in long-audio analysis. Hybrid retrieval combining keyword and vector search shows the most robust performance. When paired with both metadata and retrieved audio as evidence, average answer accuracy improves by around 10% and rationale accuracy by around 6% over single-modality retrieval and evidence. Fine-grained evaluation shows that answer accuracy alone overestimates system capability and that HARP mostly follows human performance trends across query types. These results highlight the importance of combining structured retrieval with flexible access to audio evidence and evaluating long-audio systems beyond answer accuracy.

3. COLM Speech & Audio

   Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions

   Jinchuan Tian, Haoran Wang, Bo-Hao Su, Chien-yu Huang, Qingzheng Wang, and 12 more authors

   2026

   Abs DOI

   Current audio foundation models typically rely on rigid, task-specific supervision (e.g., speech recognition), addressing isolated factors of audio rather than the whole. In contrast, human processes audio holistically, seamlessly bridging raw audio waveform with abstract cognitive concepts (e.g., all perception details of audio events) to execute complex tasks. Grounded in this philosophy, we introduce Bagpiper, an 8B audio foundation model that interprets physical audio via rich captions, i.e., comprehensive natural language descriptions that encapsulate the critical cognitive concepts inherent in the audio. By pre-training on a massive corpus of 600B tokens, the model establishes a robust bidirectional mapping between raw audio and this high-level conceptual space. During fine-tuning, Bagpiper adopts a caption-then-process workflow, simulating an intermediate cognitive reasoning step to solve diverse tasks without knowing prior task-specific practice. Experimentally, Bagpiper achieves universal generation that can uniformly generate speech, sound effects, music, and their arbitrary combinations. It also maintains comparable performance with the 7B Qwen-2.5-Omni for audio understanding. To the best of our knowledge, Bagpiper is among the first works that achieve open-ended audio understanding and generation on speech, sound, and music. Model, data, and code will be released at Bagpiper Home Page.

4. Interspeech Phone Recognition

   An Empirical Recipe for Universal Phone Recognition

   Shikhar Bharadwaj, Chin-Jou Li, Kwanghee Choi, Eunjung Yeo, William Chen, and 2 more authors

   2026

   Abs DOI

   Phone recognition (PR) is a key enabler of multilingual and low-resource speech processing tasks, yet robust performance remains elusive. Highly performant English-focused models do not generalize across languages, while multilingual models underutilize pretrained representations. It also remains unclear how data scale, architecture, and training objective contribute to multilingual PR. We present PhoneticXEUS – trained on large-scale multilingual data and achieving state-of-the-art performance on both multilingual (17.7% PFER) and accented English speech (10.6% PFER). Through controlled ablations with evaluations across 100+ languages under a unified scheme, we empirically establish our training recipe and quantify the impact of SSL representations, data scale, and loss objectives. In addition, we analyze error patterns across language families, accented speech, and articulatory features. All data and code are released openly.

5. Interspeech Dysarthric Speech

   Adapting Self-Supervised Speech Representations for Cross-lingual Dysarthria Detection in Parkinson’s Disease

   Abner Hernandez, Eunjung Yeo, Kwanghee Choi, Chin-Jou Li, Zhengjun Yue, and 10 more authors

   2026

   Abs DOI

   The limited availability of dysarthric speech data makes cross-lingual detection an important but challenging problem. A key difficulty is that speech representations often encode language-dependent structure that can confound dysarthria detection. We propose a representation-level language shift (LS) that aligns source-language self-supervised speech representations with the target-language distribution using centroid-based vector adaptation estimated from healthy-control speech. We evaluate the approach on oral DDK recordings from Parkinson’s disease speech datasets in Czech, German, and Spanish under both cross-lingual and multilingual settings. LS substantially improves sensitivity and F1 in cross-lingual settings, while yielding smaller but consistent gains in multilingual settings. Representation analysis further shows that LS reduces language identity in the embedding space, supporting the interpretation that LS removes language-dependent structure.

6. ACL Phone Recognition

   PRiSM: Benchmarking Phone Realization in Speech Models

   Shikhar Bharadwaj^*, Chin-Jou Li^*, Yoonjae Kim^*, Kwanghee Choi, Eunjung Yeo, and 11 more authors

   In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul 2026

   Abs DOI

   Phone recognition (PR) serves as the atomic interface for language-agnostic modeling for cross-lingual speech processing and phonetic analysis. Despite prolonged efforts in developing PR systems, current evaluations only measure surface-level transcription accuracy. We introduce PRiSM, the first open-source benchmark designed to expose blind spots in phonetic perception through intrinsic and extrinsic evaluation of PR systems. PRiSM standardizes transcription-based evaluation and assesses downstream utility in clinical, educational, and multilingual settings with transcription and representation probes. We find that diverse language exposure during training is key to PR performance, encoder-CTC models are the most stable, and specialized PR systems still outperform LALMs. PRiSM releases code, recipes, and datasets to move the field toward multilingual speech models with robust phonetic ability.

7. ACL Phone Recognition

   POWSM: A Phonetic Open Whisper-Style Speech Foundation Model

   Chin-Jou Li^*, Kalvin Chang^*, Shikhar Bharadwaj, Eunjung Yeo, Kwanghee Choi, and 3 more authors

   In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul 2026

   Abs DOI

   Recent advances in spoken language processing have led to substantial progress in phonetic tasks such as automatic speech recognition (ASR), phone recognition (PR), grapheme-to-phoneme conversion (G2P), and phoneme-to-grapheme conversion (P2G). Despite their conceptual similarity, these tasks have largely been studied in isolation, each relying on task-specific architectures and datasets. In this paper, we introduce POWSM (Phonetic Open Whisper-style Speech Model), the first unified framework capable of jointly performing multiple phone-related tasks. POWSM enables seamless conversion between audio, text (graphemes), and phones, opening up new possibilities for universal and low-resource speech processing. Our model outperforms or matches specialized PR models of similar size (Wav2Vec2Phoneme and ZIPA) while jointly supporting G2P, P2G, and ASR. Our training data, models, and code are released to foster open science.

8. ICLR Long Context LLM

   Prompt-MII: Meta-Learning Instruction Induction for LLMs

   Emily Xiao, Yixiao Zeng, Ada Chen, Chin-Jou Li, Amanda Bertsch, and 1 more author

   In International Conference on Learning Representations, Jul 2026

   Abs DOI

   A popular method to adapt large language models (LLMs) to new tasks is in-context learning (ICL), which is effective but incurs high inference costs as context length grows. In this paper we propose a method to perform instruction induction, where we take training examples and reduce them to a compact but descriptive prompt that can achieve performance comparable to ICL over the full training set. Specifically, we propose PROMPT-MII, a reinforcement learning (RL) based framework to meta-learn an instruction induction model that can generate compact instructions on the fly for an arbitrary new dataset. We train on over 3,000 diverse classification datasets from the HuggingFace hub, and evaluate on 90 unseen tasks. PROMPT-MII improves downstream model quality by 4-9 F1 points (10-20% relative), matching ICL performance while requiring 3-13x fewer tokens.

2025

1. Interspeech Dysarthric Speech

   Towards Inclusive ASR: Investigating Voice Conversion for Dysarthric Speech Recognition in Low-Resource Languages

   Chin-Jou Li^*, Eunjung Yeo^*, Kwanghee Choi, Paula Andrea Pérez-Toro, Masao Someki, and 5 more authors

   In Interspeech, Jul 2025

   Abs DOI

   Automatic speech recognition (ASR) for dysarthric speech remains challenging due to data scarcity, particularly in nonEnglish languages. To address this, we fine-tune a voice conversion model on English dysarthric speech (UASpeech) to encode both speaker characteristics and prosodic distortions, then apply it to convert healthy non-English speech (FLEURS) into non-English dysarthric-like speech. The generated data is then used to fine-tune a multilingual ASR model, Massively Multilingual Speech (MMS), for improved dysarthric speech recognition. Evaluation on PC-GITA (Spanish), EasyCall (Italian), and SSNCE (Tamil) demonstrates that VC with both speaker and prosody conversion significantly outperforms the off-the-shelf MMS performance and conventional augmentation techniques such as speed and tempo perturbation. Objective and subjective analyses of the generated data further confirm that the generated speech simulates dysarthric characteristics.

2. ACL Long Context LLM

   Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attention

   Emily Xiao, Chin-Jou Li, Yilin Zhang, Graham Neubig, and Amanda Bertsch

   In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, Jul 2025

   Abs DOI

   Many-shot in-context learning has recently shown promise as an alternative to finetuning, with the major advantage that the same model can be served for multiple tasks. However, this shifts the computational burden from training-time to inference-time, making deployment of many-shot ICL challenging to justify in-practice. This cost is further increased if a custom demonstration set is retrieved for each inference example. We present Dynamic Block-Sparse Attention, a training-free framework for retrieval-based many-shot in-context learning. By combining carefully designed block-sparse attention and retrieval of cached groups of demonstrations, we achieve comparable per-example latency to finetuning while maintaining on average >95% of the best method’s accuracy across strong ICL and finetuning baselines. We hope that this will further enable the deployment of many-shot ICL at scale.

2024

1. EMBC CV for Epilepsy

   Epileptic Seizure Classification with Patient-level and Video-level Contrastive Pretraining

   Chin-Jou Li, Chien-Chen Chou, Yen-Cheng Shih, Li-Chuan Kuo, Yu-Te Wang, and 4 more authors

   In 2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Jul 2024

   Abs DOI

   Accurate classification of epileptic seizure types through seizure semiology analysis demands significant clinical expertise. While previous studies have employed various action recognition modules, the scarcity of labeled clinical videos has hindered the deployment of larger models. In this study, we explore unlabeled data to pretrain a transformer-based model with contrastive loss, taking advantage of the information that circumvents the need for additional annotation from medical professionals. We maximize the similarity between embeddings from the same patient and video while minimizing those from different patients and videos. Subsequently, a classification head was finetuned to distinguishing temporal lobe epilepsy (TLE) and extratemporal lobe epilepsy (exTLE), achieving a 5-fold accuracy of 0.93 and an F1 score of 0.88 on the video level (N = 57). Our results outperformed other state-of-the-art seizure classification models, demonstrating the efficacy of our approach. This suggests potential applications in clinical practice, where unlabeled data could serve as a valuable aid in improving seizure classification accuracy and patient care.

2. Interspeech Workshop Speech Enhancement

   Deep Complex U-Net with Conformer for Audio-Visual Speech Enhancement

   Shafique Ahmed, Chia-Wei Chen, WenZe Ren, Chin-Jou Li, Ernie Chu, and 5 more authors

   In 3rd COG-MHEAR Workshop on Audio-Visual Speech Enhancement (AVSEC), Jul 2024

   Abs DOI

   Recent studies have acknowledged the advantages of incorporating visual data into SE systems. We introduce a novel audio-visual SE approach, termed DCUC-Net (deep complex U-Net with conformer network). DCUC-Net leverages complex domain features and conformer blocks. The encoder and decoder of DCUC-Net use a complex U-Net-based framework. Audio and visual signals are processed using a complex encoder and ResNet-18 model. These signals are fused using conformer blocks and transformed into enhanced speech waveforms via a complex decoder. The conformer blocks consist of self-attention mechanisms and convolutional operations, enabling DCUC-Net to capture global and local audio-visual dependencies. Our experimental results show DCUC-Net outperforms the baseline model from the COG-MHEAR AVSE Challenge 2023 by 0.14 in terms of PESQ. Additionally, DCUC-Net performs comparably to a state-of-the-art model and outperforms other models on the Taiwan Mandarin speech with video (TMSV) dataset.

3. CV for Epilepsy

   Face swapping in seizure videos for patient deidentification

   Chin-Jou Li, Jen-Cheng Hou, Chien-Chen Chou, Yen-Cheng Shih, Stephane Dufau, and 3 more authors

   Jul 2024

   Abs DOI

   This study aimed to test different AI-based face-swapping models applied to videos of epileptic seizures, with the goal of protecting patient privacy while retaining clinically useful seizure semiology. We hypothesized that specific models would show differences in semiologic fidelity compared to the original clinical videos. This is the first study evaluating AI face swapping models in epileptic seizure video clips. Optimization of AI face-swapping models could enhance the accessibility of seizure videos for education and research while protecting patient privacy and maintaining semiology.

2023

1. CV for Epilepsy

   Artificial Intelligence-Based Face Transformation in Patient Seizure Videos for Privacy Protection

   Jen-Cheng Hou, Chin-Jou Li, Chien-Chen Chou, Yen-Cheng Shih, Si-Lei Fong, and 5 more authors

   Jul 2023

   Abs DOI

   We aim to investigate the feasibility and accuracy of artificial intelligence (AI) methods of facial deidentification in hospital-recorded epileptic seizure videos, for improved patient privacy protection while preserving clinically important features of seizure semiology.