quoting=======
nevent1q…rpne
MINITRU
=======
Why Do We Need This?
What Does MINITRU do?
Limitations
How To Install
How To Build MINITRU
Integrating MINITRU Into Your Client App
WHY DO WE NEED THIS?
--------------------
Spam all but destroyed Usenet, and forced email to retreat behind KYC (DMARC and DKIM, both dependent on DNS). Web forums were able to implement CAPTCHAs, filters and moderators, but most of their traffic is now on Reddit, Xitter, LinkedIn and Facebook, behind venture-capital funded AI filters.
More recently, AI-assisted engagement farming bots have been flooding even those with spam, and now they're showing up on Nostr.
Threat Matrix
-------------
Motives:
Stalkers and Trolls
Spammers (commercial and/or ideological)
Phishers (commercial and/or ideological and/or government)
Sophistication:
Standard client only
Procedural automation
Intelligence-simulating automation
Targeting:
Individual
Group
Non-selective
What Is Nostr Doing Already?
----------------------------
Spam versus Nostr today:
- NIP-05 adds capacity for KYC.
- NIP-13 adds PoW.
- NIP-57 adds paywall.
- Not a NIP, but some clients support WoT moderation based on Kind-1984 reports by follows
The first three forms of defence are proven technologies that raise the cost of reach for spammers. Sadly, this is only unaffordable for those truly non-selective in their targeting, the most common form in the past. Engagement farmers are using paywalled NIP-05 services openly. NIP-13 is an even easier hurdle to clear. NIP-57 paywalls, if high enough, could deter a higher percentage of spammers, but at an even greater cost in new users.
The fourth, Web of Trust moderation based on reports, is purely reactive, and less useful for new members with small Webs of Trust.
The current state-of-the-art for large centralised platforms is moderation with a mix of procedural, intelligence-simulating, and paid human staff. This system appears to work adequately, but requires huge centralised computing resources, salaried staff and vulnerability to state and non-state coercion and censorship. By having centralised algorithms, the platforms also provide trolls, scammers and phishers with a single target to train against, with all the classic disadvantages of the defender.
MINITRU seeks to provide something close to state-of-the-art moderation, but in a transparent and decentralised way that avoids all the weaknesses of contemporary centralised platforms.
WHAT DOES MINITRU DO?
=====================
PlusGood your ThoughtFeeds with MINITRU, a metafilter for Nostr, and OCEANIA, a method for automated individual re-training of MINITRU for an individual's preferences and threats.
MINITRU is intended to do for Nostr what Apache SpamAssassin does for email, to the extent of using the same intermediate format for rules.
While I intend MINITRU's workflow to be accessible to power users of Nostr, not just developers, most users will use one of a small list of free MINITRUs downloaded with their Nostr client, or a custom one produced by a marketplace of paid NIP-90 data vending machines.
MINITRU's architecture is very similar to the models developed in https://ar5iv.labs.arxiv.org/html/2303.08792 but slightly heavier and with many more layers exposed for joint human-machine decision-making. The filtering and moderation system around it is inspired by SpamAssassin (https://cwiki.apache.org/confluence/display/SPAMASSASSIN/SpamAssassin).
Moderation functions, used in tandem:
1. Apache SpamAssassin capabilities ported to Nostr
- Keyword filtering, similar to existing clients, but used to construct as score a la Spam-Assassin rather than a simple pass-fail.
- Regular Expression filtering - can import Spam Assassin content rules directly, including those from existing subscription services
Extended with:
2. Similarity filtering (using word vectors).
3. Part of Speech and Named Entity filters
4. Auto-Moderator: BoW + CNN model, custom trained for each individual npub's threat environment using Web of Trust Kind-1984 reports
All in ~60 MB, and able to be run on any modern popular consumer computing device, even a Raspberry Pi Zero.
MINITRU is hundreds of times smaller than many of the LLMs used by spammers, and millions of times faster to train.
[Possible further use: MINITRU's PoS and NER capabilities may be very useful for NIP-50 searches)]
With MINITRU, we can all control the narrative as Big Sibling. If only on our Nostr feeds.
LIMITATIONS
===========
MINITRU is designed to work with human users' good judgement and personal choices, not to replace them.
You are the teacher. MINITRU needs your Kind-1984 reports, and your follows, to learn your preferences. The main limit on MINITRU's capabilities is not its architecture but rather the quantity and quality of Kind-1984 data available to teach it. You will also need to write some manual rules, but existing SpamAssassin tools and front-ends will work.
Being a quite small LLM, MINITRU will make classification mistakes from time to time, but being individually trained on your npub's threat environment, most of the "collateral damage" will be to notes you wouldn't have interacted with anyway.
A Heads-Up On Bias
-------------------
AIs, like politicians, respond to incentives. Unlike politicians, they are rewarded only for getting the job done, and they will take whatever shortcuts they can evolve to do so.
MINITRU _will_ become as racist, ageist, ableist, heterophobic or elitist as you need it to be to best meet your revealed filtering desires.
There are countermeasures, but, like popular climate change solutions, these are ineffective by design and are more likely to create problems than solve them.
HOW TO INSTALL
==============
1. Prepare infrastructure
You will need a computing device - a desktop, laptop, tablet or possibly mobile phone.
Linux or *BSD: you're good to go!
Windows: you will need to install https://cygwin.com/ (free and Open Source)
MacOS: you will need to install https://brew.sh (free and Open Source)
Android: you will need to install https://termux.dev/en/ (free and Open Source)
iOS: forget it
2. Set up build environment
You will need to install python-pip, virtualenv, pkg-config, git and torsocks.
Linux or *BSD: open a terminal window, and enter "apt-get install python3-pip python3-virtualenv git torsocks pkg-config" (or as per your package manager)
Windows: run the Cygwin install program again, and select python3-pip, python3-virtualenv, git and tor
MacOS: open a terminal window, and enter "brew install python" then "brew install git torsocks"
Android: open a Termux terminal, then as per Linux
3. Open terminal
Open a terminal window (Cygwin's terminal for Windows users)
Create a folder for MINITRU:
mkdir minitru
cd minitru
4. Create a "virtual environment" to install libraries in
virtualenv env_minitru
Activate this virtual environment for this session:
. env_minitru/bin/activate
5. Clone this project:
torsocks git clone git://uehkilylfklvvrx7rj7pua2piprkoc3l26hwk4w5wyeeingbwg6fkpid.onion:/minitru.git
6. Install libraries
cd minitru
python3 -m pip install -r requirements.txt
Done.
HOW TO BUILD MINITRU
====================
Open a terminal window, as before.
Return to your project folder:
cd minitru
Activate your virtual environment for this session:
. env_minitru/bin/activate
Change to the code folder:
cd minitru
Give a short nickname to your existing Nostr npub
python3 oceania.py npubnew -a-n
Test that you can connect to Nostr
python3 oceania.py global -a
Start building and training your own MINITRU
python3 oceania.py build -a
This may run for a while. It is very "chatty" - don't worry if you have no idea about the data it is displaying on your screen, its just letting you know it hasn't frozen and giving you an inside peek at the sorts of data it deals with.
If it crashes (usually a network timeout), just re-run it.
When it completes, test that MINITRU works and can filter and highlight by color:
python3 oceania.py global -a
Blue is the most trusted, green slightly less, then magenta, then red. In a client app, red notes should be filtered out as sedition against the doctrines of IngSoc. And IngSoc is whatever you want it to be!
Test that MINITRU isn't filtering and highlighting any of your own posts:
python3 oceania.py personal -a
You may then start creating custom rules for yourself / your friends/ everyone on Nostr in your favourite text editor.
Look at minitru/rules/examples.txt, and save your own rules in minitru/rules in a file with a .txt ending.
Any front-end or tool that can create / import SpamAssassin rules should be compatible with MINITRU.
Apache SpamAssassin is a mature ecosystem, with many examples, tutorials, subscription services etc.
Many webmail clients allow users to build SpamAssassin rules through a graphical user interface. Nostr clients in future could too.
Since most SpamAssassin rules are just "regular expressions", you can build them with this website: https://regex-generator.olafneumann.org
INTEGRATING MINITRU INTO YOUR CLIENT APP
========================================
MINITRU is built on SpaCy, which is written in Python.
https://spacy.io/
There is a native port of SpaCy to C++
https://spacy.io/universe/project/spacy-cpp/
Other languages can use SpaCy as an embedded Cython module:
https://cython.readthedocs.io/en/latest/src/tutorial/cython_tutorial.html
https://cython.readthedocs.io/en/latest/src/tutorial/embedding.html
Javascript cannot, but look into embedding with Brython:
https://brython.info
Oceania.py is also written in Python, but is much less portable.
If your app is also Python, you might be interested in reusing oceania.py's score_text() function on events in your app.
If your app is written in anything else, you might want to look at my code and implement the functions you need in your preferred language.
Most of the magic is in the MINITRU model, anyway, and SpaCy is how you interact with it.
MINITRU is designed to be used in conjunction with SpamAssassin-format rules. These can be created with GUI front-ends, or downloaded as subscriptions (possibly paid if you want to go that route).
LICENSE
=======
This project reuses some code from Monty888's monstr examples:
https://github.com/monty888/monstr
Monty888 deserves the credit.
All code added by me is released under the GPL2:
https://www.gnu.org/licenses/old-licenses/gpl-2.0.html
Copyright is assigned to the FSF:
https://www.fsf.org/blogs/licensing/FSF-copyright-handling
mplorentz on Nostr: An interesting experiment in Nostr moderation: a personal LLM that will run on your ...
An interesting experiment in Nostr moderation: a personal LLM that will run on your device and tag events for you.