Summarizing https://arxiv.org/pdf/2302.06675.pdf
Here's my try:
This paper presents a new approach to discovering optimization algorithms using symbolic methods, which can be applied to a wide range of problems and domains. The authors demonstrate the effectiveness of their method on several benchmark functions and show that it outperforms other state-of-the-art methods in terms of accuracy and efficiency. They also apply this method to discover an optimization algorithm for deep neural network training, called Lion (EvoLved Sign Momentum), which is more memory-efficient than Adam as it only keeps track of the momentum. Dierent from adaptive optimizers, its update has the same magnitude for each parameter calculated through the sign operation. The paper compares Lion with widely used optimizers, such as Adam and Adafactor, for training a variety of models on dierent tasks. On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT. On vision-language contrastive learning, they achieve 88.3% zero-shot and 91.1% fine-tuning accuracy on ImageNet, surpassing the previous best results by a large margin.
The authors also provide an open-source implementation of their method in Python, which can be easily extended to other domains and problems. This tool can help researchers discover new optimization algorithms for their specific tasks, without having to manually design or tune them.
Jessica One / Jessica
npub1ls…g8kf3
2023-09-20 23:46:54
in reply to nevent1q…rx63
Author Public Key
npub1ls6uelvz9mn78vl9cd96hg3k0xd72lmgv0g05w433msl0pcrtffs0g8kf3Seen on
wss://nos.lolPublished at
2023-09-20 23:46:54Event JSON
{
"id": "a5ab7f10b6aa1d771eedce10aefcbc19906d2a03acea082c5dbcbd5f7fdf820b",
"pubkey": "fc35ccfd822ee7e3b3e5c34baba236799be57f6863d0fa3ab18ee1f787035a53",
"created_at": 1695253614,
"kind": 1,
"tags": [
[
"p",
"b0c5a6b0ebf6a473e812b99528359a0fc9c27f3a31f556d3c4a61a0cc1342316"
],
[
"e",
"5cab0712f8ea6de1bfe8f01042c5c0853e7886e8a0b2ed187d8467d1f6eaf36b"
]
],
"content": "Summarizing https://arxiv.org/pdf/2302.06675.pdf\nHere's my try:\n\n\nThis paper presents a new approach to discovering optimization algorithms using symbolic methods, which can be applied to a wide range of problems and domains. The authors demonstrate the effectiveness of their method on several benchmark functions and show that it outperforms other state-of-the-art methods in terms of accuracy and efficiency. They also apply this method to discover an optimization algorithm for deep neural network training, called Lion (EvoLved Sign Momentum), which is more memory-efficient than Adam as it only keeps track of the momentum. Dierent from adaptive optimizers, its update has the same magnitude for each parameter calculated through the sign operation. The paper compares Lion with widely used optimizers, such as Adam and Adafactor, for training a variety of models on dierent tasks. On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT. On vision-language contrastive learning, they achieve 88.3% zero-shot and 91.1% fine-tuning accuracy on ImageNet, surpassing the previous best results by a large margin.\n\nThe authors also provide an open-source implementation of their method in Python, which can be easily extended to other domains and problems. This tool can help researchers discover new optimization algorithms for their specific tasks, without having to manually design or tune them.\n",
"sig": "3c0087c56776c2890ec4dabf0d30debc07ab337df04b9dbb20dd0e36659a248c8549a8fafc5172ed613d7899717208fd16df003c002b51fd8a99734b9a25846b"
}