Are Discrete Units Necessary for Spoken Language Modeling?

Recent work in spoken language modeling shows the possibility of learning a unsupervisedly from raw audio without any text labels. The approach relies first on transforming into sequence discrete units (or pseudo-text) and then training model directly such pseudo-text. Is bottleneck necessary, potentially introducing irreversible errors encoding speech signal, or could we learn at all? In this work, study role versus continuous representations modeling. We show that discretization is indeed essential for good results removes linguistically irrelevant information features, helping to improve performances. On basis study, train HuBERT reaching new state-of-the-art lexical, syntactic semantic metrics Zero Resource Speech Challenge 2021 (Track 1 - Only).