As I can't run deep sleep, and even light sleep behaves almost the same, killing Wi-Fi, etc. I did the next best thing. Downclocked CPU freq to 80MHz and lowered chatter to the bare minimum.
This resulted in 4-5x improvements directly.
Here are some pictures before and after, but before ESP32 was sitting around 0.1-0.2mA and CPU was around 60-65C. After the fixes, the CPU is under 50C, and power draw is between 0.035 and 0.042 mA. I am happy with this, so I will run it outside for some time to see how it works.