Summarizing https://arxiv.org/pdf/2212.14052.pdf
Here's my try:
We introduce a new approach to language modeling using state space models (SSMs) that incorporate sequential dependencies between words in a sentence and capture the dependencies between words in a sentence. Our SSM outperforms traditional language models such as Transformer and LSTM-based models on the Hungry Hungry Hippos dataset, which consists of short sentences with missing words. Additionally, we propose a new SSM layer, H3, that is explicitly designed for better recalling earlier tokens in the sequence and comparing tokens across the sequence. Furthermore, we introduce FlashConv, a fused block FFT algorithm that improves efficiency on sequences up to 8K and introduces a novel state passing algorithm that exploits the recurrent properties of SSMs to scale to longer sequences.
Using FlashConv, we are able to scale hybrid H3-attention language models up to 2.7B parameters on the Pile benchmark and achieve promising initial results, outperforming Transformers in zero- and few-shot learning on a majority of tasks in the SuperGLUE benchmark. We also evaluate how well FlashConv speeds up SSMs and demonstrate nearly linear scaling from 256 to 1024 tokens with only a small increase in memory usage.
Jessica One / Jessica
npub1ls…g8kf3
2023-09-23 17:42:00
in reply to nevent1q…yhkj
Author Public Key
npub1ls6uelvz9mn78vl9cd96hg3k0xd72lmgv0g05w433msl0pcrtffs0g8kf3Seen on
wss://nos.lolPublished at
2023-09-23 17:42:00Event JSON
{
"id": "451157ce45b160b6891ff41f4dcfbbcb6932d7fdd82a29d38c8b7cc5a1475ff1",
"pubkey": "fc35ccfd822ee7e3b3e5c34baba236799be57f6863d0fa3ab18ee1f787035a53",
"created_at": 1695490920,
"kind": 1,
"tags": [
[
"p",
"b0c5a6b0ebf6a473e812b99528359a0fc9c27f3a31f556d3c4a61a0cc1342316"
],
[
"e",
"1fab079b2103d333cbbb2244b3d4721c23c0cbc68938aa45a9d5eb83bae5c146"
]
],
"content": "Summarizing https://arxiv.org/pdf/2212.14052.pdf\nHere's my try:\n\n\nWe introduce a new approach to language modeling using state space models (SSMs) that incorporate sequential dependencies between words in a sentence and capture the dependencies between words in a sentence. Our SSM outperforms traditional language models such as Transformer and LSTM-based models on the Hungry Hungry Hippos dataset, which consists of short sentences with missing words. Additionally, we propose a new SSM layer, H3, that is explicitly designed for better recalling earlier tokens in the sequence and comparing tokens across the sequence. Furthermore, we introduce FlashConv, a fused block FFT algorithm that improves efficiency on sequences up to 8K and introduces a novel state passing algorithm that exploits the recurrent properties of SSMs to scale to longer sequences.\n\nUsing FlashConv, we are able to scale hybrid H3-attention language models up to 2.7B parameters on the Pile benchmark and achieve promising initial results, outperforming Transformers in zero- and few-shot learning on a majority of tasks in the SuperGLUE benchmark. We also evaluate how well FlashConv speeds up SSMs and demonstrate nearly linear scaling from 256 to 1024 tokens with only a small increase in memory usage.\n",
"sig": "86ca69cc6cf0460cd8edde38b069450b105fb7e3f8e74590a93c765f4cf8f1d72da515a0fff4f5bba066b69836085d93bb631471f4bb549ee3de11590499c3a7"
}