I've just released Tikentoken, an Elixir toy library and CLI tool for experimenting with AI tokenization using Ollama models. It can tokenize text, compute embeddings, and even chat - with a convenient fallback on a basic mock when Ollama is offline or not installed/available.
The documentation is much more extensive than the code itself, as I mainly published it to keep my dev notes organized.
Since English isn't my first language, I used ollama and a few free models to help refine some of the comments and the docs. Sorry for that.
Itβs on GitHub: https://github.com/florinpatrascu/tikentoken, in case you youβre curious or need anything like that. HTH
#elixirlang #Ollama #OpenSource
ππππππ π¨π¦
npub158β¦enu4c
2025-10-19 18:59:24 UTC
Author Public Key
npub158rsa422vjpr8xzlzarx643x7h59f99g4fmnzwsyle2py85ngv5qdenu4cSeen on
wss://relay.momostr.pinkPublished at
2025-10-19 18:59:24 UTCEvent JSON
{
"id": "9ae3a41c8a25baaeff79ac6e07dfc826c2bfc6b7f61550b92f3f7c943ccc3344",
"pubkey": "a1c70ed54a648233985f17466d5626f5e85494a8aa77313a04fe54121e934328",
"created_at": 1760900364,
"kind": 1,
"tags": [
[
"t",
"elixirlang"
],
[
"proxy",
"https://techhub.social/@florin/115402366261654779",
"web"
],
[
"t",
"opensource"
],
[
"t",
"ollama"
],
[
"proxy",
"https://techhub.social/users/florin/statuses/115402366261654779",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://techhub.social/users/florin/statuses/115402366261654779",
"pink.momostr"
],
[
"-"
]
],
"content": "I've just released Tikentoken, an Elixir toy library and CLI tool for experimenting with AI tokenization using Ollama models. It can tokenize text, compute embeddings, and even chat - with a convenient fallback on a basic mock when Ollama is offline or not installed/available.\n\nThe documentation is much more extensive than the code itself, as I mainly published it to keep my dev notes organized. \n\nSince English isn't my first language, I used ollama and a few free models to help refine some of the comments and the docs. Sorry for that. \n\nItβs on GitHub: https://github.com/florinpatrascu/tikentoken, in case you youβre curious or need anything like that. HTH\n\n#elixirlang #Ollama #OpenSource",
"sig": "88329dccded7ca9766a5c979248c89b65993b0fd7748e9c507cc1108c5ba4c2207a499276152cbd030876e6dd0f83794f7f4270780779a45ffee3c8d3e13d0e4"
}