View source for Scaling Transformer to 1M tokens and beyond with RMT

== Discussion ==
The problem of long inputs in Transformers has been extensively researched since the popularization of this architecture. In this work, the authors demonstrate that applying Transformers to long texts does not necessarily require large amounts of memory. By employing a recurrent approach and memory, the quadratic complexity can be reduced to linear. Furthermore, models trained on sufficiently large inputs can extrapolate their abilities to texts orders of magnitude longer.

Synthetic tasks explored in this study serve as the first milestone for enabling RMT to generalize to tasks with unseen properties, including language modelling. In their future work, the authors aim to tailor the recurrent memory approach to the most commonly used Transformers to improve their effective context size.

Overall, this report presents a promising approach to extend the context length of Transformer-based models, which holds significant potential to enhance long-term dependency handling in natural language understanding and generation tasks as well as enable large-scale context processing for memory-intensive applications.