Publications

2024

1. CoNeTTE: An Efficient Audio Captioning System Leveraging Multiple Datasets With Task Embedding

   Étienne Labbé, Thomas Pellegrini, and Julien Pinquier

   IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

2023

1. Multitask Learning in Audio Captioning: A Sentence Embedding Regression Loss Acts as a Regularizer

   Etienne Labbé, Julien Pinquier, and Thomas Pellegrini

   In 2023 31st European Signal Processing Conference (EUSIPCO), 2023
2. Adapting a ConvNeXt Model to Audio Classification on AudioSet

   Thomas Pellegrini, Ismail Khalfaoui-Hassani, Etienne Labbé, and 1 more author

   In Proc. INTERSPEECH 2023, 2023
3. Killing Two Birds with One Stone: Can an Audio Captioning System Also Be Used for Audio-Text Retrieval?

   Étienne Labbé, Thomas Pellegrini, and Julien Pinquier

   In Proceedings of the 8th Detection and Classification of Acoustic Scenes and Events 2023 Workshop (DCASE2023), Sep 2023

   Abs

   Automated Audio Captioning (AAC) aims to develop systems capable of describing an audio recording using a textual sentence. In contrast, Audio-Text Retrieval (ATR) systems seek to find the best matching audio recording(s) for a given textual query (Text-to-Audio) or vice versa (Audio-to-Text). These tasks require different types of systems: AAC employs a sequence-to-sequence model, while ATR utilizes a ranking model that compares audio and text representations within a shared projection subspace. However, this work investigates the relationship between AAC and ATR by exploring the ATR capabilities of an unmodified AAC system, without fine-tuning for the new task. Our AAC system consists of an audio encoder (ConvNeXt-Tiny) trained on AudioSet for audio tagging, and a transformer decoder responsible for generating sentences. For AAC, it achieves a high SPIDEr-FL score of 0.298 on Clotho and 0.472 on AudioCaps on average. For ATR, we propose using the standard Cross-Entropy loss values obtained for any audio/caption pair. Experimental results on the Clotho and AudioCaps datasets demonstrate decent recall values using this simple approach. For instance, we obtained a Text-to-Audio R@1 value of 0.382 for AudioCaps, which is above the current state-of-the-art method without external data. Interestingly, we observe that normalizing the loss values was necessary for Audio-to-Text retrieval.

4. Multilingual Audio Captioning using machine translated data

   Matéo Cousin, Étienne Labbé, and Thomas Pellegrini

   arXiv preprint arXiv:2309.07615, Sep 2023
5. IRIT-UPS DCASE 2023 audio captioning and retrieval system

   Etienne Labbé, Thomas Pellegrini, and Julien Pinquier

   May 2023

   Abs

   This technical report provides a concise overview of our systems submitted to the DCASE Challenge 2023 for tasks 6a, "Automated Audio Captioning" (AAC), and 6b, "Language-Based Audio Retrieval" (LBAR). In task 6a, we made four distinct submissions. The first submission employed a standard CNN14 encoder paired with a transformer decoder. In the second submission, we replaced this encoder with a ConvNeXt model to enhance audio representation. The third submission incorporated additional training data. We introduced a new task embedding approach to differentiate between different writing styles and audio types. Finally, in the fourth submission, we employed an ensemble method to combine five models trained on different seeds, aiming to improve the quality of the captions. For task 6b, we use the AAC models and we propose a novel approach to accomplish the LBAR task by leveraging the AAC system loss function without requiring any additional training. Our most successful AAC and LBAR systems achieved a SPIDEr-FL score of 0.320 and an mAP@10 score of 0.269. These results demonstrate relative improvements of 22.6\% and 21.2\% compared to the AAC and LBAR baselines, respectively.

2022

1. Comparison of semi-supervised deep learning algorithms for audio classification

   Léo Cances, Etienne Labbé, and Thomas Pellegrini

   EURASIP Journal on Audio, Speech, and Music Processing, Sep 2022

   Abs

   In this article, we adapted five recent SSL methods to the task of audio classification. The first two methods, namely Deep Co-Training (DCT) and Mean Teacher (MT), involve two collaborative neural networks. The three other algorithms, called MixMatch (MM), ReMixMatch (RMM), and FixMatch (FM), are single-model methods that rely primarily on data augmentation strategies. Using the Wide-ResNet-28-2 architecture in all our experiments, 10% of labeled data and the remaining 90% as unlabeled data for training, we first compare the error rates of the five methods on three standard benchmark audio datasets: Environmental Sound Classification (ESC-10), UrbanSound8K (UBS8K), and Google Speech Commands (GSC). In all but one cases, MM, RMM, and FM outperformed MT and DCT significantly, MM and RMM being the best methods in most experiments. On UBS8K and GSC, MM achieved 18.02% and 3.25% error rate (ER), respectively, outperforming models trained with 100% of the available labeled data, which reached 23.29% and 4.94%, respectively. RMM achieved the best results on ESC-10 (12.00% ER), followed by FM which reached 13.33%. Second, we explored adding the mixup augmentation, used in MM and RMM, to DCT, MT, and FM. In almost all cases, mixup brought consistent gains. For instance, on GSC, FM reached 4.44% and 3.31% ER without and with mixup. Our PyTorch code will be made available upon paper acceptance at https://github.com/Labbeti/SSLH.

2. Is my Automatic Audio Captioning System so Bad? SPIDEr-max: A Metric to Consider Several Caption Candidates

   Etienne Labbé, Thomas Pellegrini, and Julien Pinquier

   In Proceedings of the 7th Detection and Classification of Acoustic Scenes and Events 2022 Workshop (DCASE2022), Nov 2022

   Abs

   Automatic Audio Captioning (AAC) is the task that aims to describe an audio signal using natural language. AAC systems take as input an audio signal and output a free-form text sentence, called a caption. Evaluating such systems is not trivial, since there are many ways to express the same idea. For this reason, several complementary metrics, such as BLEU, CIDEr, SPICE and SPIDEr, are used to compare a single automatic caption to one or several captions of reference, produced by a human annotator. Nevertheless, an automatic system can produce several caption candidates, either using some randomness in the sentence generation process, or by considering the various competing hypothesized captions during decoding with beam-search, for instance. If we consider an end-user of an AAC system, presenting several captions instead of a single one seems relevant to provide some diversity, similarly to information retrieval systems. In this work, we explore the possibility to consider several predicted captions in the evaluation process instead of one. For this purpose, we propose SPIDEr-max, a metric that takes the maximum SPIDEr value among the scores of several caption candidates. To advocate for our metric, we report experiments on Clotho v2.1 and AudioCaps, with a transformed-based system. On AudioCaps for example, this system reached a SPIDEr-max value (with 5 candidates) close to the SPIDEr human score of reference.

3. IRIT-UPS DCASE 2022 task6a system: stochastic decoding methods for audio captioning

   Etienne Labbé, Thomas Pellegrini, and Julien Pinquier

   Jul 2022

   Abs

   This document presents a summary of our models used in the Automated Audio Captioning task (6a) for the DCASE2022 challenge. Four submissions were made using different decoding methods : beam search, top k sampling, nucleus sampling and typical decoding.

2021

1. IRIT-UPS DCASE 2021 Audio Captioning System

   Etienne Labbé, and Thomas Pellegrini

   Jul 2021

   Abs

   This document provides a description of our seq-to-seq models used for audio captioning in the task 6 of the DCASE 2021 challenge. Four submissions were made with two different models, a “Listen- Attend-Tell” and a “CNN-Tell”, and two different algorithms for inference, greedy and beam search.