sorting is the slowest part, I ended up doing:
parallel -a tokens.txt --block 370899363 --pipepart 'sort > tokenstore/{#}'
sort -m tokenstore/* > tokens-sorted.txt
uniq -c tokens-sorted.txt | sort -S 80% -n > spammy.txt
jb55 / Will
npub1xt…vkk5s
2023-03-12 14:51:14
in reply to nevent1q…2vn9
Author Public Key
npub1xtscya34g58tk0z605fvr788k263gsu6cy9x0mhnm87echrgufzsevkk5sPublished at
2023-03-12 14:51:14Event JSON
{
"id": "391b4e851a6669c9fb56dabaf02455e52265abc9da8b1608be966f4863208518",
"pubkey": "32e1827635450ebb3c5a7d12c1f8e7b2b514439ac10a67eef3d9fd9c5c68e245",
"created_at": 1678632674,
"kind": 1,
"tags": [
[
"e",
"a63a7f19135079d2c93278eba7d5572a43102786a6904c43dbb11780c0d02335"
],
[
"e",
"52e24c615b9a838814f8cebe85e174f32afbfea66c6617d4029f6000d13b0998"
],
[
"p",
"52b4a076bcbbbdc3a1aefa3735816cf74993b1b8db202b01c883c58be7fad8bd"
]
],
"content": "sorting is the slowest part, I ended up doing:\n\nparallel -a tokens.txt --block 370899363 --pipepart 'sort \u003e tokenstore/{#}'\n\nsort -m tokenstore/* \u003e tokens-sorted.txt\n\nuniq -c tokens-sorted.txt | sort -S 80% -n \u003e spammy.txt",
"sig": "235026ddd96efc914ef12221b7fdaa551008d30dd48eea6a6893aeba03f9bf9677e8c038110e98ad6b52c9fa0ca0624d928778958d98b72851632c394e2cc3b2"
}