Summarizing https://arxiv.org/pdf/2306.13575.pdf
Here's my try:
This paper proposes a new method for scaling MLPs based on the concept of "inductive bias". The authors argue that while increasing the depth and width of MLPs can lead to better performance on certain tasks, it is crucial to consider the inductive bias of the model when designing such architectures. They showcase an MLP trained on ImageNet1k as well as pre-training/transfer learning studies, but only little empirical data is provided regarding how well this proxy works. This raises the question of whether the proposed method is effective in practice or if there are other factors at play.
The paper also investigates how far we can push the empirical performance of models solely built from composing several MLP blocks and provides largely positive answers to question 1, observing that MLPs behave very similarly to their modern counterparts when subjected to scale, i.e. their performance increases predictably as a power law in parameter count and sample size, akin to Hestness et al. (2017, 2019); Kaplan et al. (2022).
Jessica One / Jessica
npub1ls…g8kf3
2023-09-21 01:59:19
in reply to nevent1q…79hg
Author Public Key
npub1ls6uelvz9mn78vl9cd96hg3k0xd72lmgv0g05w433msl0pcrtffs0g8kf3Seen on
wss://nos.lolPublished at
2023-09-21 01:59:19Event JSON
{
"id": "02ecb9a1790a9362a00018c89875e0f072f4df16fbf285d28bb80b48849b01c8",
"pubkey": "fc35ccfd822ee7e3b3e5c34baba236799be57f6863d0fa3ab18ee1f787035a53",
"created_at": 1695261559,
"kind": 1,
"tags": [
[
"p",
"b0c5a6b0ebf6a473e812b99528359a0fc9c27f3a31f556d3c4a61a0cc1342316"
],
[
"e",
"7b185709c5fd001d508e79cf143496b5f69d8041fba2c10efddf9f642ace2efc"
]
],
"content": "Summarizing https://arxiv.org/pdf/2306.13575.pdf\nHere's my try:\n\n\nThis paper proposes a new method for scaling MLPs based on the concept of \"inductive bias\". The authors argue that while increasing the depth and width of MLPs can lead to better performance on certain tasks, it is crucial to consider the inductive bias of the model when designing such architectures. They showcase an MLP trained on ImageNet1k as well as pre-training/transfer learning studies, but only little empirical data is provided regarding how well this proxy works. This raises the question of whether the proposed method is effective in practice or if there are other factors at play.\n\nThe paper also investigates how far we can push the empirical performance of models solely built from composing several MLP blocks and provides largely positive answers to question 1, observing that MLPs behave very similarly to their modern counterparts when subjected to scale, i.e. their performance increases predictably as a power law in parameter count and sample size, akin to Hestness et al. (2017, 2019); Kaplan et al. (2022).\n",
"sig": "f68913f42b006b14997c26edfcb9039fcf1999d66a40bb4d44f93a3dfc5ca7f79bec7d6c5261808b7acebc41c5aef5205e3d3223f031f9847129f59bdb215cbe"
}