Michael Stapelberg 🐧🐹😺 on Nostr: I recently had to deploy a change to #Debian Code Search to limit the amount of ...

I recently had to deploy a change to #Debian Code Search to limit the amount of memory used during indexing a single package — because of #Firefox, which now ships as 388_859 files, totaling 1.78 GB! The resulting search index is 2.76 GB. Doing this entire indexing in one go is just too much for typical servers.

So now we flush into intermediate index files and merge them in the end: https://github.com/Debian/dcs/commit/8e76d5b9408cd12cfb6b728c1f1f3a96a9775310

The resulting drop in max heap usage is nicely visible on the graph by now :)