Deep learning, as a vital technique, has sparked a notable revolution in artificial intelligence (AI), resulting in a great change in human lifestyles. As the most representative architecture, Transformers have empowered numerous advanced models, especially the large language models (LLMs) that comprise billions of parameters, becoming a cornerstone in deep learning. Despite the impressive achievements, Transformers still face inherent limitations, particularly the time-consuming inference resulting from the quadratic computation complexity of attention calculation. Recently, a novel architecture named Mamba, drawing inspiration from classical state space models, has emerged as a promising alternative for building foundation models, delivering comparable modeling abilities to Transformers while preserving near-linear scalability concerning sequence length. This has sparked an increasing number of studies actively exploring Mamba’s potential to achieve impressive performance across diverse domains. Given such rapid evolution, there is a critical need for a systematic review that consolidates existing Mamba-empowered models, offering a comprehensive understanding of this emerging model architecture. In this survey, we therefore conduct an in-depth investigation of recent Mamba-associated studies, covering from three main aspects: the advancements of Mamba-based models, the techniques of adapting Mamba to diverse data, and the applications where Mamba can excel. Specifically, we first recall the foundational knowledge of various representative deep learning models and the details of Mamba-1&2 as preliminaries. Then, to showcase the significance of Mamba for AI, we comprehensively review the related studies focusing on Mamba models’ architecture design, data adaptability, and applications. Finally, we present an discussion of current limitations and explore various promising research directions to provide deeper insights for future investigations.
** Key Words and Phrases: **State Space Model, Mamba, Sequence Modeling, Foundation Models, Language Models
这篇综述共有38页！涵盖了235篇文献！作者单位为香港理工大学和范德堡大学。
本文对Mamba进行了全面调查，主要包括三个方面：Mamba模型的进展、使Mamba适应各种数据的技术以及Mamba在不同领域中的优势应用。
深度学习作为人工智能领域中重要的技术，引起了一场革命。Transformers作为最具代表性之一架构，赋予了许多先进模型以力量，特别是那些包含数十亿参数的大型语言模型，在深度学习中扮演着基础角色。尽管Transformers取得了显著成就，但它仍然存在固有限制，尤其是注意力计算所导致二次复杂度会造成推理时间过长。近期出现了一种名为Mamba新型架构，从经典状态空间模型汲取灵感，并成为构建基础模型有前景替代方案。它提供与Transformers相当建模能力，并且在序列长度上保持近线性可扩展性。这引发越来越多研究者积极探索潜藏于 Mamba 在不同领域实现卓越性能之潜力 。鉴于其快速发展态势，迫切需要进行系统综述以整合已有关于赋能模型方面使用 Mamba 方法相关研究内容 , 从而全面认识这种新兴模型架构 。
因此，在本综述中对最近与 Mamba 相关联 的研究进行深入调查，主要涵盖三个方向：基于 Mambas 的模式进步、使 Mambas 适应各类数据技巧和 Mambas 可善用之处。具体而言，首先回顾各类代表性深度学习模式的基础知识和 Mambas 细节作准备工作。然后，为彰显 Mambas 的重要地位，全面回顾相关研究, 特别关注构件设计、 数据适配性和应用方面 。最后，探讨当前局限并探索各类具发展前景的研究方向，提供未来更加深入见解。
版权归原作者 Phoenixtree_DongZhao 所有， 如有侵权，请联系我们删除。