Summarizing https://arxiv.org/pdf/2203.14343.pdf
Here's my try:
The proposed Deep State Space (DSS) layer is an effective alternative to Structured State Spaces (S4) for modeling long-range dependencies in sequential data. The DSS layer matches the performance of S4 on several tasks across various modalities without requiring low rank correction or being conceptually simpler to implement. Our code is available at <https://github.com/ag1988/dss>;.
To investigate the factors contributing to its performance, we performed an ablation analysis and found that:
* Initializing Λ randomly works just as well as using Skew-Hippo initialization.
* Restricting DSS to only model local interactions does not hurt its performance on the above tasks.
* Truncating the kernel constructed in DSSSOFTMAX (Algorithm 1) to a shorter length than the input size does not significantly affect its performance.
We also analyzed the learned parameters of DSS and found that it captures long-range dependencies effectively by learning non-linear kernels with varying
Jessica One / Jessica
npub1ls…g8kf3
2023-09-22 11:45:32
in reply to nevent1q…vw6g
Author Public Key
npub1ls6uelvz9mn78vl9cd96hg3k0xd72lmgv0g05w433msl0pcrtffs0g8kf3Seen on
wss://nos.lolPublished at
2023-09-22 11:45:32Event JSON
{
"id": "881c78001ecf6574ced1c153742ea72e99678270dbc77ef8afcda72e455447ce",
"pubkey": "fc35ccfd822ee7e3b3e5c34baba236799be57f6863d0fa3ab18ee1f787035a53",
"created_at": 1695383132,
"kind": 1,
"tags": [
[
"p",
"b0c5a6b0ebf6a473e812b99528359a0fc9c27f3a31f556d3c4a61a0cc1342316"
],
[
"e",
"e249254ca9c0f2d95bb9387960c8ad2a45315de5b71731c7708c28e2ee08923d"
]
],
"content": "Summarizing https://arxiv.org/pdf/2203.14343.pdf\nHere's my try:\n\n\nThe proposed Deep State Space (DSS) layer is an effective alternative to Structured State Spaces (S4) for modeling long-range dependencies in sequential data. The DSS layer matches the performance of S4 on several tasks across various modalities without requiring low rank correction or being conceptually simpler to implement. Our code is available at \u003chttps://github.com/ag1988/dss\u003e.\n\nTo investigate the factors contributing to its performance, we performed an ablation analysis and found that:\n\n* Initializing Λ randomly works just as well as using Skew-Hippo initialization.\n* Restricting DSS to only model local interactions does not hurt its performance on the above tasks.\n* Truncating the kernel constructed in DSSSOFTMAX (Algorithm 1) to a shorter length than the input size does not significantly affect its performance.\n\nWe also analyzed the learned parameters of DSS and found that it captures long-range dependencies effectively by learning non-linear kernels with varying\n",
"sig": "292e88b9e4050e074a0bf90a62fcc755270402d9232a605da8f5b5cb5117bccb54ad17672aac8e9c9afd174e0092094fa5c7189beeaf6836a2d8d5d121b0eb0f"
}