DeepMem: A Multiscale Byte-Level Language Model with Invertible Subband Transform
DeepMem is a causal multiscale architecture for byte-level language modeling that separates semantic compression from exact byte preservation using a learned invertible subband transform. It builds a hierarchy of shorter sequences with lossless carriers and detail sidebands, and uses fast weight memory blocks and an attention-based super-resolution decoder. DeepMem outperforms a matched byte-level Transformer on FineWeb-Edu and PG19, achieves best macro-average across FLORES languages, and improves PIQA accuracy while reducing bottleneck FLOPs by ~7,600x.
Why it matters: The paper introduces a novel architecture with concrete performance gains and efficiency improvements, relevant to AI researchers and practitioners in language modeling.