Tiota Sram

#ComputerScience instructor at Wellesley College who researches #games & #AI (he/him; cis). Current #research is on exploration in #Metroidvania games. I build #PCG demos like:

https://cs.wellesley.edu/~pmwh/labyrinfinite/

and games like:

https://cs.wellesley.edu/~pmwh/chlorophyll/

The intertwining problems of racism, ablism, queermisia (including specifically transmisia), sexism, classism, capitalism, and colonialism must be opposed everywhere.

Profile picture: colored lines converging along different paths on a square grid.

tfr

Public Key

npub129f0qyxuyud6qec8jykcppn4ehant3fmrwm8a56pvnpz2m0ue76s2rqe6x

NIP-05 Address

tiotasram_at_kolektiva.social@momostr.pink

Profile Code

nprofile1qqs9z5hszrwzwxaqvureztvqse6um7e4c5a3hdn76dqkfs39dh7vldgpz3mhxue69uhhyetvv9ujuerpd46hxtnfduqs6amnwvaz7tmwdaejumr0dszac2gu

Publishing to

relay.damus.io

nos.lol

Author Public Key

npub129f0qyxuyud6qec8jykcppn4ehant3fmrwm8a56pvnpz2m0ue76s2rqe6x

Show more details

Last Notes

2026-04-09 19:46:04 UTC

- reply

you're right. Focusing on the positive examples makes things clearer, and I agree the negative examples shouldn't/can't be expected to be pure.
By my definitions "memorization" is still a poor negative example for the means axis because it is much more about task than means, even if it blurs across the boundary a little. But I think if you just ignore that example the rest make sense. Thanks for helping to clarify things!

2026-04-09 18:16:16 UTC

- reply

that's fair. I guess a simpler way of saying that I'm getting at here is: "I don't understand your distinction between task and means based on the examples you provided." That's why I said that definitions would be helpful. Perhaps more examples would also be helpful. We're already butting up against category theory here so it's fine to say "these are exemplar-based categories" but in that case my complaint boils down to "I don't understand which salient features of your examples should be used to categorize things as being closer or further from your exemplars."
I do think that a task/means distinction is possible, I'm just not sure what you mean when you make that distinction.
Trying to define those axes in the way that makes the most sense to me would be something like:
The task axis is concerned with the material/sensory inputs and goals of a behavior, while the means axis is concerned with the process used to pursue some goals given some inputs. The attainment axis relates to the specific measurement techniques and the results of using those techniques. The task/means parts of a definition may constrain what kinds of measurement are seen as accurately measuring achievement for a specific behavior vs. measuring confounding information attributable to multiple distinct behaviors.
But these definitions don't line up with your exemplars as I pointed out above.

2026-04-09 16:01:48 UTC

- reply

yes. I did read that example in detail. The bone I'm picking is whether memorization should count as "not reading" because the means is inappropriate, or because it represents a different task entirely. Your example of "different task" is:
(2) “he didn’t really read what I said, he just responded to the tone” (not the same task)
To me, reciting memorized words is clearly a different task than speaking words based on visual recognition of symbols. To you, those are simply two different means of accomplishing the same task, which *some* means-based definitions of reading would exclude but which other definitions of reading might include. I'm fine with the notion that I might not agree with everyone about the definition of reading and that some people might use one that does include memorization. But is that liberalism along the means or the task axis?
Re-reading that section, I don't see a formal definition of either "means" or "task": you define them by giving a few examples in both the reading and then reasoning cases. I guess a more formal definition of what you mean by "means" and "task" would be helpful. For example, I could imagine saying "the means axis encompasses all parts of a definition which hinge on the *process* used, irrespective of outcomes or goals." However, your example definition of the reading *task* states:
‘extracting meaning from written text'
By that measure memorization fails to meet the task definition if the cues used for recall are not all within the text itself, (for example, memorizing text based on the question number or picture associated with it), which I guess is what I supposed was meant by "just memorizing." Furthermore, "from written text" *could* be interpreted/expressed as a constraint on the process uses to arrive at meaning (it must make use of the text) which then means that "task" is not totally separate from the "means" using the process-based definition of "means" I just proposed.
So... What are your definitions of the "task" and "means" dimensions?

2026-04-09 11:36:58 UTC

- reply

to branch off here on this specific point (I'll try to respond at the thread tip to your other replies, also thanks for engaging in discussion here!):
You weren't intending to define reading, but in asserting that "memorize the page" is a means-axis variation, not a task-axis one, you're implicitly setting the scope of definitions you'll allow to be treated as "definitions of reading" very broadly. Perhaps you're offering a meta-definition, not a definition? But your choice of meta-definition privileges attainment in the following way.
Let's suppose I give a "Reading test" by setting up a maze with written directions about the path to the center. I claim that anyone who makes it to the center has passed the test, and has "attained reading." One participant is a cat, and climbs over the walls, ending up at the center. I conclude that "cats can read."
I think we'd both agree on this instance the conclusion is not warranted, and that this particular test needs to come with some means-restrictions in order to even count as a "test of reading" rather than a "test of reading or climbing". The cat's solution is a task-axis deviation from "reading", not a means-axis deviation. But in asserting the task-axis bounds I'm willing to tolerate, I implicitly allow various deviations in approach within those bounds to be considered means-variations. I guess this is a long way of saying "I don't think your task and means axes are really orthogonal" and that "your categorization of memorization as a means-axis variation and not a task-axis variation implies that the way you measure attainment is uncomfortably broad to me."
This reflects well on your paper actually, because I'm finding the terms you've set up to be extremely useful in making this argument, even as I'm disagreeing with some of the nuances.
I've been inspired by the writing of this post with a more robust idea, but it will take me more time to get that one firmed up.

2026-04-09 00:13:52 UTC

- reply

this is a super interesting pairs; thanks for linking it. I've skimmed through it, so apologies if I've misunderstood something, but although I find the call to better define what we mean by reasoning in the debate convincing, I don't find all of the arguments here compelling.
Briefly:
The paper privileges a pretty broad attainment-based definition of "reading" in the example it gives of separating the three dimensions, and does the same for "reasoning" later. I don't know very many people who would consider "can recite a page from memory" as "attainment on a reading task"; I think most would in fact call that performance of a very different task. If you are measuring "reading attainment" in such a way that memorization is a means to success, then your definition of reading attainment is much broader than typical, because of a focus on attainment alone to define the task. A shorter way of saying that is that means-differences and task differences blur together depending on how much emphasis one's task definition places on means. Those two axes aren't really separable. You give a nod to this when mentioning that psychologists do exclude certain means as "cheating" but IMO don't dig deep enough.
In the ML context we can view overfitting as both an attainment problem and a means-based problem. Using a statistical prediction system to accomplish the task in the first place permits overfitting and when we observe overfitting, it shows us that because of the means used, attainment will be limited (of course, in some cases it's possible to overcome overfitting). When overfitting is extreme and generalization is basically nonexistent, we should suspect instead a *task* issue, like when performance on cancer prediction turns out to have learned how to tell whether an X-ray was taken standing or sitting down and is using that as the basis of its prediction. That's not just a means problem, but a task problem (and your paper discusses exactly this kind of issue).
What the paper I described shows is not mere overfitting though: it's overfitting on a broader test domain even though overfitting was not observed in the narrower original test set. This implies that either intentionally or unintentionally, the original test set was not a good measure of the task because the model was performing well (high attainment on a too-broad assessment) merely due to having access to the test set ahead of time (assessment wasn't task-specific). The underlying problem has been well described in papers like this one about data cards; we don't know enough about the datasets used to train foundation models, so they could perform well on a wide variety of benchmarks simply through memorization: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C22&q=%22model+cards%22+training+set+%22data+cards%22&btnG=#d=gs_qabs&t=1775692189785&u=%23p%3DopSzYwlmudkJ
The paper I cited demonstrates exactly that, and a lot of the criticisms that your paper dismisses are about both task and means, not just means. I'm not sure I have a complete formal definition of "reasoning" right now, but I can clarify that when I say "LLMs don't reason" in part what I mean is that much of their observed attainment on various benchmarks that claim to measure reasoning is actually due to the correct answers being predicted because they (or very close analogues) appeared in the training data, and that the resulting system will therefore be fragile and perform much worse when given inputs not in (or not as frequent in) the training data. This is a critique about both means and task, because the model's approach does not involve inference in the reasoning domain at all (means difference) and performs well on the (badly-designed) benchmark only because it has a some level memorized the answers.
This critique is supported by the paper I cited because the obvious prediction from my critique is that unlike a system performing some kind of inference, a system that has merely memorized answers will degrade rapidly when small details of the questions are changed, whereas a system that actually used reasoning to solve the problem should not degrade or should only degrade slightly. The cited paper does exactly this experiment and then concludes correctly that there is a task difference between what the LLM is doing and what a reasoning system would have to do.
Another stark example of this is: prompt a chatbot with any simple reasoning question and pick new questions until it gets one right (often it will do so the first time). Now follow up with "that's wrong" (a lie) and watch it attempt to justify the opposite (incorrect) conclusion. If it were reasoning, by any definition In willing to admit, then the correctness of its first answer would be a proposition that it "believed in" and a naked assertion that it's incorrect would not alone be enough to immediately cause it to abandon that assertion. It might argue or otherwise reassert its original logic. However, it doesn't, because it's merely predicting likely answers and has no concept of reaching a conclusion or belief in propositions at all. It's internal states can affect future predictions, but this doesn't rise to the level of epistemic commitment, which I view as one precondition for reasoning (both a means aspect and an attainment aspect). Of course it's possible for a reasoning agent to abandon epistemic commitments out of extreme deference and exhibit similar behavior, and boosters might argue this is what the LLM is doing. But knowledge of the mechanisms at play shows that this isn't true. We can peer into the black box and see that there's nothing even approximating epistemic commitment inside.
I'm actually happy to admit that LLMs exhibit something than can be classified as a (narrow, idiosyncratic, dangerous) form of "intelligence" because I've got a broader-than usual idea of that term. But to claim that they "reason" only works under some truly unusual definitions of that term that pretty much only people with vested economic interests in the continued inflation of the AI bubble claim that subscribe to.

2026-04-08 01:40:00 UTC

Was just made aware of this pretty good but entirely predictable study:
https://arxiv.org/pdf/2410.05229
I find that in 2025 the "Do LLMs reason?!?" Research question is not really a good one to be asking any more. Papers from 2020 explain how they don't. They are not actually black boxes; the hypesters just like to pretend that they are so they can pretend their capabilities are vastly greater than they actually are. But it turns out even when you're credulous and ask that question, the answer is a resounding "no" every time. The detail and depth of the "no" here is great.
#LLMs

2026-04-07 11:23:25 UTC

In the interests of starting a more productive dialogue than yesterday's main character was interested in, let's make a #brainstorm thread about design changes to ActivityPub and/or client UI that could actually help address drive-by (often racist) harassment on the fediverse.
Feel free to discuss pros/cons but don't feel an idea needs to be perfect to suggest it. Also since this is a brainstorm don't worry about complexity/implementation cost. If you have a great-but-hard-to-implement idea someone else may think of a way to simplify it.
Note that the underlying problem *is* a social one, do there won't be a technological fix! But tech changes can make social remedies easier/harder.
I've got some to start:
1. Have a "protected mode" that users can voluntarily turn on. Some servers might turn it on by default. In protected mode, users whose accounts are less than D days old and/or who have fewer than F followers can't reply to or DM you. F and D could have different values for same-sever vs. different-server accounts, and could be customized by each user. Obviously a dedicated harasser can get around this, but it ups the activation energy for block evasion and pile-ons a bit. Would be interesting to review moderation records to estimate how helpful this might or might not be. Could also have a setting to require "follows-from-my-server" although that might be too limiting on private servers. Restriction would be turned off for people you mention within that thread and could be set to unlimit anyone you've ever mentioned. Would this lock new users out of engagement entirely? If everyone had it on via a default, you'd have you post your own stuff until someone followed you (assuming F=1). One could add "R non-moderated replies" and/or "F favorites" options to soften things; those experiencing more harassment could set higher limits. When muting/blocking/reporting someone who replied to your post, protected mode could be suggested with settings that would have filtered the post you're reporting.
2. Enable some form of public moderation info to be displayed when both moderator and local server opt-in. Obviously each server would be able to ignore federated public tags. I'm imagining "banned from X server for R reason (optional link to evidence)" appearing on someone's profile & an icon on their PFP in each post viewed by someone on server Y *if* the mods of server X decide it's appropriate *and* server Y opts in to displaying such tags from server X specifically. Alliances of servers with similar moderation preferences could then have moderation action on one server result in clear warning propagation to others without the other mods needing to decide whether to also take action immediately. In some cases different moderation preferences would mean you wouldn't take action yourself but would keep the notice up for your users to consider. Obviously the "Scarlet Letter" vibe ain't great, but in some cases it's deserved, and when there's disagreement between servers about that, mods on server Y could either disable a specific tag or disable federation of mod tags from that server in general. Even better shared moderation tools are of course possible.
3. Different people/groups have different norms around boosting. Currently we only have a locked/public binary. Without any big protocol changes, adding a "prefers boosts/doesn't" setting which would warn in the UI before a viewer chooses to boost if the preference is "doesn't" could help. This could be set per-post, but could also have defaults and could have different values for same-server or not, or for particular servers. For example, I could say "default to prefer boosts from users on my server but not from users on other servers" or "default to prefer boosting on all servers except mastodon.social." Last option might be harder to implement I guess.
#ActivityPub #Meta #Harassment

2026-04-02 03:02:04 UTC

For those wondering about the bizarre Apache helicopter flyover of a No Kings rally and "spontaneous" visit to Kid Rock's mansion, whether it was intentionally orchestrated (seems likely to me) or not, the effect was to send a public signal to the military that acts of allegiance to Trump and his supporters will get rewarded and shielded with impunity, even if they're flagrant violations of military codes and procedures.
It's another step in the fascist playbook of getting the military in line to support tyrannical acts, so it's very much not a "no harm no foul" situation.
As for why I think it's more likely quite deliberate: those helicopters don't fly without thousands of dollars of fuel, air traffic control (or you're creating some *serious* danger going without), and other logistical support. There's no way that level of risk makes personal sense to some random army helicopter pilots, just for the benefit of pulling a public stunt? These aren't random rookies in basic training, it takes serious skill and many hours of training to successfully pilot a helicopter at all. Why risk your entire military career on a stunt, and how did you convince an entire ground crew and air traffic control tower to get in on it? Having some kind of orders/encouragement/guarantee of impunity from above makes the most sense. I'd be 0% surprised if Hegseth directly gave the orders/go-ahead and the show of an "investigation into insubordination" that gets publicly cancelled by Hegseth personally is an entirely intentional result if not the main objective.
Also buzzing the No Kings rally was 100% intentional.
#NoKings

2026-03-30 15:08:36 UTC

- reply

just look at the failures of Reconstruction that led to Jim Crow and you can see that even those who never actually disavow their most odious beliefs are often let right back into positions of power.
I'm not *exactly* anti-democracy but as an anarchist one of my main criticisms of democracy as a system (not just US democracy in specific) is that it lets political takeovers like this happen, again and again. To borrow a critique social democrats often level at anarchists: "the idea of pluralism and a tolerant social democracy is a nice one, but I simply can't imagine that its actually possible to achieve." :)

2026-03-29 21:36:15 UTC

- reply

Not a plan I hate, but not one I think is good, because to me it sounds like "sure this boat can drive on the highway, you just need to..."
To be clear, when I say "capitalism" I mean the "ism" which places capital as the means by which we should allot power/prestige/worth. As an anarchist I'm against "isms" in general because they're prescriptions for how to set up a hierarchy. But capitalism in specific says that those who have accumulated the most wealth should have the most influence. In practice, this has allowed them to corrupt all of the institutions that were designed to limit their power. Many people right now keep saying "just one more chance, I'm sure we can set up the correct system of checks and balances this time!" But some of us are out of patience for that game.
To the people like you who at least acknowledge the necessity of checks and balances, although I think that project is doomed and a waste of valuable organizing energy, I can at least see where you're coming from and that you might think similarly of the anarchist project as I once did. Even if you aren't anarchist, saying things like "we need to prioritize X over profits" means you're an anticapitalist of a sort, since capitalism necessary necessitates putting profit above all else.

2026-03-29 16:36:58 UTC

- reply

in my opinion what we think of as "human nature" is much more malleable than we're led to believe.
I imagine that if you described states and city life to humans 10,000 or at least 100,000 years ago, they would have told you such things were impossible because humans' independent and free-spirited nature would make such large-scale (and particularly oppressive/destructive) organization and hierarchies impossible. You don't even really need to lean on a "noble savage" myth to believe that at least, today's society would be incomprehensible to them and would seem counter to "human nature."
What I hope for (or more accurately optimistically expect) is that a future society which seems incomprehensible to many of us now will exist and that it will be better than today's world.
I can imagine it will be better because capitalism and the growth cult are inherently unstable. They cannot help but destroy themselves. As long as we avoid extinction (not a guarantee) their successor will likely be at least more stable over long time scales, as their predecessors were (some of which persist on the margins and will hopefully outlive the current era themselves). These currently-persisting social forms aren't by any means perfect, but they're certainly preferable to our dominant imperial-cis-hetero-patriarchal-capitalism/fascism, and they're living examples of how to organize successfully on 10,000-plus-year timescales.
I do admit many people have wildly different experiences with "human nature" and that despite my divergent views, many heads thinking about (and even disagreeing about) ways to the future are better than one dogma. If my presence for non-hierarchical organizing is thwarted but we end up in a different kind of sustainable society, I won't be too mad. The piles of evidence about even the most-liberal "democracies" we have now make it hard for me to imagine this outcome, but difficulties of imagination are something I'm constantly asking others to challenge :)

2026-03-29 14:53:09 UTC

- reply

I'm an anarchist, so my view is the opposite: leaders by definition centralize power and weaken movements.
I'm this view (which could, I admit, be wrong) we do need many people to inspire each other and it's fine if some people are outstanding at that and become notable for it, but what we really need is ideas (and framings that communicate them) that shift away from leadership models to per-individual shared responsibility for collective action. When every individual accepts responsibility for initiating their own plans and actions (even if they ultimately voluntarily delegate that responsibility to someone else who had a great idea), then the movement cannot be stopped by killing or corrupting its leaders, and the drag effect of everyone waiting around for a leader or hero to save them will disappear leading to greater momentum.
The "fact" that humans tend to behave like sheep and cluster behind leaders could easily be specific to "humans raised in and trapped by oppressive propaganda". Compare the behavior of wild to domesticated sheep, for example.
I think we do need people who will inspire each other, people who are brave enough to step out first in defiance, etc. But I don't think these people should lead by giving orders or even setting the agenda. And I don't think it's productive to think "we need someone to do these things" when instead we should be thinking "how can I contribute to these things myself?"
To me, hero mythology might as well be fascist propaganda, because it implies that *only* heroes can change the world, when in the past significant changes have always been accomplished by mass movements.

2026-03-23 20:34:01 UTC

- reply

I think that the "Epstein class" qualifies as a class because although the Epstein ring is a group of collaborators, when I say "Epstein class" I *think* people understand I'm also including the other pedophile billionaires and millionaires that surely exist, who often do things out of class solidarity with each other.
In fact, I'm sure some of the people in Epstein's emails actually were innocent of literal child rape, but to be "part of the club" and get grants or just access to influential figures they cozied up and looked the other way at a literal conviction for sex trafficking. The way they discuss things like the MeToo movement shows a perverse kind of solidarity, for example, and I'm sure others not even in their orbit but who are similarly rich/influential + patriarchy fans have a default sympathy for these people every time one gets outed. They do things like write/sponsor op-ed pieces talking about how MeToo has gone too far or quietly drag their feet on discrimination investigations in their own institutions, etc. Things that influence our entire society to be more conducive to their evil as a class, not just as individuals.
I think perhaps explaining that, which a wide variety of people will agree with, is the start of a conversation that can then turn to "Epstein is a symptom, not a cause. Let me explain how patriarchy and capitalism ensure that even once he dies, not only are his collaborators not held accountable, but *another* Epstein will inevitably surface.
Along the way you gotta (as you rightly point out) be careful of the tendency to blame this on "the Jews" or another group of people rather than a system, but I think starting from the widely acknowledged ongoing travesty of justice and working towards a systemic explanation could be more successful than starting with systems.

2026-03-23 19:28:52 UTC

- reply

I largely agree, but some points out divergence:
"The Epstein class" is actually pretty good rhetoric. They function exactly like a class: have shared interests and defend them; reflexively defend each other out of class solidarity, etc. It's a small and hyperspecialized class, but it is a class, and it's one that large masses of people (taught to believe that all anti-capitalist rhetoric can be ignored out-of-hand) still recognize is a problem. Using "Epstein class" to get people to actually listen to a systemic critique of capitalism is good strategy, even though using "overthrow the Epstein class" as the whole of your message is bad.
Also, capitalism is *not* the whole problem. Patriarchy, white supremacy, colonialism, and other issues intertwine with it and none of them are fully separable but neither is any of them the single source of all the others.
Reducing the problem to "just capitalism" actually invites some of the same failure modes as reducing it to "just this one group of people," and alienates potential allies who are focused on other facets of the multi-crisis that impact them more directly.

2026-03-17 15:58:36 UTC

- reply

oh thanks for this info! That's a really interesting strategy. I teach labs for an into CS class and we're doing pass/fail grading, the idea of moving to A/D/F is appealing but it might be hard to get my colleagues on board.
For my other labs I do self-assigned grades; this semester I'm requiring a small extra project to earn an A grade which I think is going well. The grade they self-assign ends up becoming 10% of a course grade given by the lecture instructor though, so it's not actually that far from standard grading. We'll see if I can convince that instructor to do something different.

2026-03-17 00:13:27 UTC

- reply

an excellent article. I really liked this point:
"It helps people to believe that not only are there right answers to find, but that scientists know how to find them, which covers up the role of violence in determining what counts as truth."
One difficulty I've encountered with ungrading and related stuff is that when yours is the only non-letter-graded class students are taking, they've been conditioned to treat it less seriously and regularly do so. One can simply say "that's on the students" which is a valid response, and some days I'd like to take zero responsibility for their engagement, but ultimately I don't think that's the place I want to end up (a robust model of overlapping/redundant student & teacher responsibility for engagement feels better to me).
I've also played the "cheating cop" role a lot, which I have extremely mixed feelings about.
Some of this just boils down to "I don't have the energy to fight 12+ years of conditioning" even though that kinda should be my job...

2026-03-02 17:13:54 UTC

- reply

maybe they still think like Trump that this will be over in a few days? More likely there are somehow competing pressures at play, although I can't begin to guess what they would be.

2026-02-12 18:30:15 UTC

- reply

great points. An article that transformed my thinking on this from Crimethinc was this about it being safer at the front:
https://crimethinc.com/2025/01/28/its-safer-in-the-front-taking-the-offensive-against-tyranny
Obviously this doesn't always apply (it didn't work for Pretti) but speaking out when the real consequence is a beating and short unfair prison stay after which you understand the need to get the hell out is much preferable to remaining silent for "safety" and then a few years later you get picked up anyways but this time it's a one-way trip to a concentration camp.

2026-02-12 05:29:32 UTC

- reply

yeah, where's the scrupulous both-sides reporting when you actually need it?

2026-02-12 03:58:17 UTC

- reply

sadly under the current (and not just this year's) FDA these things are getting approval, only to have higher complaint rates than non-AI devices... A quote:
"""
Researchers from Johns Hopkins, Georgetown and Yale universities recently found that 60 FDA-authorized medical devices using AI were linked to 182 product recalls, according to a research letter published in the JAMA Health Forum in August. Their review showed that 43% of the recalls occurred less than a year after the devices were greenlighted. That’s about twice the recall rate of all devices authorized under similar FDA rules, the review noted.
"""
From: https://www.reuters.com/investigations/ai-enters-operating-room-reports-arise-botched-surgeries-misidentified-body-2026-02-09/
But yeah, in the long term, if FDR didn't completely dismantle the FDA, we should see some balancing out.

2026-02-12 02:09:40 UTC

- reply

also FYI as someone with a modest understanding of the CS side of things: A few points:
1. This article is not trustworthy. It conflates very different types of AI system to maximize the threat it states, and that kind of dishonest reporting makes me distrust all of its conclusions. Expert diagnosis systems that use machine learning are a completely different kind of beast from a chatbot, even though that also uses machine learning in a different way. Touting the accuracy of specialized diagnosis systems as evidence for the effectiveness of chatbots is like showing off the excavating capability of a Volkswagen excavator add evidence that a BMW racing car will be faster than the competition.
2. Evaluating machine learning systems is very tricky, and it's not hard to get good-looking results in front of press if you have dollars to burn, which don't actually hold up in the real world. For the curious here's a nice thorough paper on issues with medical imaging specifically: https://www.nature.com/articles/s41746-022-00592-y
As a patient, unless you show me in-the-field accuracy numbers from multiple years of deployment that actually rival human performance for a very particular task that this discussing AI is designed to do alone, I'd always rather have a human doctor's opinion than that of an AI system, and even when I'm willing to accept the AI diagnosis, I'd like a human doctor's second opinion and interpretation.
3. For medical-records and patient-interaction applications, including most of the ones listed in this article that chatbots are "good" at, I think doctors/hospitals using them are opening themselves up to a lot of liability and making a mistake. These systems make egregious errors in predictable patterns, which competent human staff do not make. Incompetent staff sometimes make the same errors, but the difference is that they are responsible for their own errors. If your "bedside manners" chatbot that takes over when the doctor is busy with another patient encourages a patient to kill themselves or take the wrong medicine, is that acceptable, since 99% of the time it speaks with a very reassuring manner? I guarantee you ChatGPT will make these mistakes orders of magnitude more often than even the most sleep-deprived RN or resident. We have seen these events happen already in other contexts; the medical/hospital context has many more opportunities for these failures. There is no real technological mitigation for them in the horizon either. Even for "transcribe my patient notes" I wouldn't trust dictation software unless needed as an accommodation, even though it has admittedly gotten pretty good. There's lots of opportunity for a missed jargon word to cause havoc in notes that get shared with someone else, for example.
4. The big AI companies have developed one tool that shows up very well I demos, but which has nasty flaws that make it unsuitable for a lot of what they're pushing. They are trying to sell their stuff as "the future" and say it must be "integrated" everywhere. If only you integrated our marvelous technology, your problems would be solved! This is backwards. A true solution looks first at the problem, and then asks "what tool would be best to use here?" By putting the tool choice first, you end up with ineffective or even counterproductive "solutions." This only makes sense if your goal is to sell the tools. For example, I understand doctors have very limited time but must write up notes between patient visits to refer to in future meetings. Sometimes notes end up inaccurate or illegible, or just muddled. How can we best solve this problem? Simple: hire more doctors to give them all more time to write better notes. Any alternate solution needs to be understood as a compromise. Might there be a way to use technology to help? Sure, there are probably plenty. Let's consider using a system to record the visit, then produce notes automatically by statistically predicting what notes the doctor would write. We'll have the doctor check them off every time. Is this good? No, because in this design, doctors will get lazy in their checking over time, especially if the system is very accurate for most visits. But such systems are going to make big mistakes for unusual visits, which doctors might not then correct. Even worse, by denying the doctor the cognitive task of organizing their thoughts into writing, you're disrupting the doctor's memory formation and chances to see unusual patterns or slight irregularities. Lowering the doctor's cognitive burden takes away the benefits you get from expending cognitive resources on the problem! As an alternate design, what if you had the doctor write the notes unaided, and then had a system try to flag possible discrepancies, misspellings, or illegible writing? Such a system might still be bad, if it creates too much friction (remember Clippy?). But it *might* be good if tuned correctly. It's not a flashy or "revolutionary" as the "we'll take your work away" system, but it avoids some of that system's worst drawbacks. There's probably even better designs I'm not thinking of. My point is that starting with "let's integrate a chatbot" is the wrong approach, and anyone who insists on it is not someone you should trust, because self-evidently they are stating from their own interests (sell/promote chatbots) while completely disregarding yours. They're basically saying "Can you help me think of a way to sell my product to you?" which is downright disrespectful.
Okay that's probably enough ranting from me. TL;DR: trust AI chatbots about as much as the least trustworthy intern you can imagine working with, because it will eventually make exactly the same kinds of disastrous mistakes, and you'll be the one to blame since (when we're not spinning yarns about machine intelligence) it's "just a tool that you chose to use." It's not even capable of learning from those mistakes either, because to let chstbots learn from their public interactions would be dangerous in other ways (see Microsoft Tay).

2025-12-23 14:48:43 UTC

Making this a subtoot so I don't come across as smug or condescending...
My decision to stop using github when they started providing services to ICE back in ~2016 felt awkward at times but has been feeling really good in hindsight right now.
I see a bunch of people now saying "why boycott X company over some "minor" transgression or political capitulation (or over a "neutral" stance on LLM code). The answer is: it shows what their values are, which predicts their future behavior, especially under the tilted playing field of capitalism. I'm by no means perfect at this and I don't think shouting at people to boycott is a good idea for several reasons. People should boycott what they want to, for their own reasons. But I am posting this to try to help others be aware of the upsides of taking action when confronted with "subtle" evidence of corporate unvalues.

2025-12-10 00:11:59 UTC

- reply

I realize you probably already know this, but for anyone who hadn't yet had the "carbon capture" myth busted:
Burning fossil fuels releases carbon dioxide as a reaction result. The molecular reconfiguration from something else like CH4 into CO2 is what produces the energy that was the whole point of burning the stuff, because CO2 is a more-tightly-bonded, stable/low energy configuration. If you are proposing to "capture" this carbon by binding it to something, the laws of thermodynamics ensure that at the very least, you're going to have to re-spend the energy you got out, thus generating net zero energy.
There are two ways around this:
1. Bind it into a form that's energetically between the original molecule and CO2. Here inefficiencies make this mostly unworkable. In any case burning straight to that intermediate would have been better. Nobody I've heard of is researching anything like this so I assume it's just not a thing (although I'm not a climate scientist).
2. You could capture the CO2 without chemically altering it, perhaps by liquefying it and then pumping it underground. Sadly, this all takes energy to do: energy to separate it from other exhaust products, energy to do the phase transition, and then energy to pressurize it or otherwise package it for storage. Turns out when you factor in inefficiencies, you end up spending more energy than you generated burning the stuff in the first place. You know what would have been miles more efficient? Leaving the pressurized liquid carbon in the ground in the first place.
"Carbon capture" is a red herring designed to distract from renewables for long enough to extend profit margins on planet-destroying fossil fuels, nothing more.

2025-09-25 09:51:34 UTC

- reply

In any case, day 2: Ursula K Le Guin.
As I've said elsewhere, part of her science fiction thesis is that "human" can encompass much more than what we mere Terrans think of it as, and that moral standing extends broadly throughout the universe. This is the antithesis of Tokens fantasy, wherein "race" is real and determines moral standing. For Le Guin, it's barely okay to intervene in complex alien politics unless you carefully ensure you're not causing systemic harms; for Tolkien, it's okay to ambush and murder orc children, because they are by nature evil.
Add to her excellent politics Le Guin's masterful worldbuilding and unparalleled range of plots, and you have the one author I loved as a decidedly liberal and naïve teen and love even more now that I'm an adult. She's an absolute legend and deserves a very high place on any list of women authors (or list of authors, period.).
For a short story, try "The Ones Who Walk Away From Omelas" which you can read here: https://www.utilitarianism.com/nu/omelas.pdf
For fantasy "A Wizard of Earthsea" (also has a nice graphic novel adaptation), or for science fiction, "The Left Hand of Darkness" or if you want a more anarchist flavor, "The Dispossessed."
I'll close this with an amazing quote from her:
"""
We live in capitalism. Its power seems inescapable. So did the divine right of kings. Any human power can be resisted and changed by human beings. Resistance and change often begin in art, and very often in our art, the art of words.
"""

2025-09-25 09:30:08 UTC

- reply

Correction: it's the "Broken Earth" series; Stone Sky is one of the books.

2025-09-24 21:11:56 UTC

Via @npub1z9g…08zr
Challenge: "Name 20 female authors you admire, 1 per day"
N. K. Jemisin.
Incredibly powerful writing and writes science fiction and fantasy that I actually enjoy reading after getting disillusioned with the Tolkein lineage for its deep racism & colonialism. Her Stone Sky series is wildly creative in terms of world building, had an excellent epic plot to rival any of the other greats you could name, has complex and compelling characters, and a satisfying conclusion. She's won the right awards, and I hope that that translates into a lineage of people building in her ideas as rich as Tolkien's.

2025-09-05 18:14:06 UTC

- reply

To me beyond obviously #OwnVoices stuff by non-hegemonic authors, Le Guin comes closest to the antithesis of the Tolkien racist worldview (and it's sad that her work didn't inspire nearly as much other stuff as Tolkien).
Whereas Tolkien reifies racial differences as sharp divisions that come with an inedible moral hierarchy, giving us a narrower, white supremacist view of humanity, Le Guin imagines a broader spectrum of what it means to be human than what we have on Earth, and locates differences in culture, not race. Her array of alien groups spread out across the galaxy with vast differences in lifestyle, reproduce strategies, and appearance are nonetheless all human, and make racial differences on Earth seem trivial in comparison.
Her Hainish

2025-09-05 13:09:24 UTC

- reply

Tolkien himself thought about this and regretted his portrayal of the orcs in the end:
https://en.m.wikipedia.org/wiki/Tolkien%27s_moral_dilemma
The entire idea of "racial alignments" is subconsciously satisfying to imperial/colonist cultures because it echoes the same colonialist mythos that justifies living on/benefiting from stolen land. Once you see it, it's super uncomfortable, and since it caught on so broadly, it's everywhere.
Why is it okay to sneak up on and preemptively execute Ganon's minions in Breath of the Wild? "They're invaders, and evil, right?" Well, that's the Hylian story. But they clearly have cultures (no coincidence borrowing aesthetics from real human cultures considered "primitive" and colonized in the real world). Several areas of the world are even pretty clearly their homelands, whatever we might claim about central Hyrule. It's okay to show up and murder them all with no warning because "they'd attack me without warning if they saw me first?" Okay, but you're a known serial killer who stalks and kills on sight, of course they'd attack you. That logic works both ways.
The parallels with the Pope declaring all regions not in Europe okay to colonize because they were only inhabited by "infidels" and "pagans" (who were okay to enslave, even) are strong.
https://en.m.wikipedia.org/wiki/Discovery_doctrine

2025-08-31 23:55:42 UTC

- reply

sadly the extreme levels of glyphosphate use enabled by glyphosphate-resistant GMOs are not just speculation.
I do completely agree with you that more emphasis needs to be placed on the thoroughly wicked tactics Monsanto has used to immiserate farmers, I just don't think the anti-GMO movement is baseless, even if their tendency to focus only on GMO harms is too narrow.

2025-08-31 22:45:39 UTC

- reply

pushing back gently here: IDK, I don't consider myself much of a hippie, but I think there is real danger in GMOs, especially now in the era of CRISPR. Given all the traits you're usually trying to engineer for and the absolute disregard for safety found in the typical corporate lab, it's unlikely but entirely possible to accidentally develop a super-weed with terrible ecological consequences, and once the cat's out of the bag, it's not going back in. In a twisted way, the for-profit neutering of Monsanto crops is safer in this regard.
The other (actually realized) danger is that you can engineer in such high tolerance to specific herbicides like glyphosphate that you end up with a "nuke the local ecosystem from orbit so that only the hyper-resistant crops have any chance" which has a bunch of negative externalities for nearby people and animals. Yes, *technically* one could probably eventually achieve these results through selective breeding, but that's not a sure bet, and the costs involved might have prevented the investments necessary to achieve that.

2025-08-31 10:36:45 UTC

- reply

this is frankly terrible advice. The failures of large language models are infrequent, but they've got an exponential distribution of impacts. If you're using Linux by relying on AI to handle a lot of terminal stuff, you'll eventually end up doing something horrible like deleting a bunch of important documents or screwing up the graphics configuration.
I know I sound like a crusty old curmudgeon here, and I know learning terminal stuff yourself is painful, *especially* in the current age where AI-generated SEO spam poisons search results. But I promise you if you try to vibe-code your way through Linux, you are going to suffer and regret it. I'd estimate max 6 months to a year of daily use with AI commands going into your terminal before you experience a disaster. If you're Ajay knowledgeable about the terminal and try to be careful, you can probably double or triple that.
To address the obvious reply: yes, you can easily screw stuff up yourself without AI, but the difference is that you'll be learning from those mistakes and most will be recoverable given the knowledge you'll be gaining.

2025-08-22 13:52:25 UTC

- reply

I don't exactly blame scientists, who after doing the actual difficult science part shouldn't be held to account for also fighting an uphill communications battle against a swarm of overfunded oil PR firms.
I guess science journalism shoulders a lot of the blame, but maybe also I do wish scientists thought of public communication as a core part of their role (and were taught about it in school). Really all the ways that the education system tries to distance & separate science from politics are systemic factors at play here.

2025-08-22 10:43:35 UTC

- reply

degrees average warming has always been a terrible metric also because the average person thinks "It's going to be 1° warmer? That doesn't sound like a big deal..." and endless threads like this one can't fully compensate for that intuition.
Imagine a soda bottle. Now imagine you shake it vigorously, and then open it. Your shaking introduces energy into the system, possibly enough to warm it by 1° on average. It also, of course, causes some much more noticeable changes because of how the shaking changes the state of the carbonation in the soda. You could instead have put it in an oven very briefly to raise the temperature 1°, without shaking it. These two hypothetical 1° differences actually are very different in practice, and in the former scenario, the temperature difference itself really isn't a salient description of what happened.
In fairness to scientists, the temperature change due to the greenhouse effect is in fact the root cause of a lot of what's going on (minus things like ocean acidification), but still, it's the wrong way to describe what's happening. "Global Climate Amplification" would have been a better term, with descriptions of how much global high/low extreme temperatures were going to be amplified (a lot more than 1°) would have been better.

2025-08-20 11:08:17 UTC

- reply

Edit: pre-symptoms of course you're not "coughing" the virus out, you're just breathing it out.
@npub193h…phpn adding to this: even for symptomatic cases, the infectious period starts 1-2+ days *before* the start of symptoms.
To repeat: When infected, the virus starts replicating in your body, at low levels. As those levels rise, you start breathing it back out, spreading it to other people. As they rise further, they actually start to kill enough cells that you feel the first symptoms like fatigue, and eventually as your immune response kicks in at scale, a runny nose. *You are already spreading the virus well before you feel anything at all.*
This is just as true of the flu and colds as COVID-19. If these viruses could only be transmitted after symptoms appeared, then staying home when you're sick would be very effective. This is one of the biggest failures of health communication even before the current pandemic. I didn't learn this fact until during the pandemic, and I don't see it repeated nearly enough.

2025-08-04 16:03:32 UTC

- reply

the sad part is that the "centaurs" aren't being assisted by just machines, they're being assisted by machines supported by reverse-centaurs. All the people tasked with cleaning datasets, and all the artists whose work was (and is) stolen to train the machines are the human price of the centaurs' fun experience. And it's not *necessary* to train generative systems thus way, it's just *cheap* and *fast*.

2025-07-20 00:24:59 UTC

- reply

Indeed I can now see the statements I quoted are not yours. Your thread and blog post frame the discussion as "I used to abstain from LLMs but this particular system has made me reconsider that stance." You note a situation that actually happened where the site would have been useful to you, and you've responded to every criticism I've seen raised in this thread with some version of "that's not convincing to me." I interpreted this per my assumption above, which I stated so that it could be refuted it it was wrong. Given that you've asked me to be more careful about assuming what you're stance is, but have not made a direct statement of that stance, I'll ask some direct questions here. To first answer yours:
I have no prior experience with civic engagement. My judgment of the potential benefits here is purely theoretical and yours will be much better. Your judgment of the viability of alternatives involving civic engagement will also be better.
My questions about your stance:
1. Do you feel that the benefits of this system in the specific case you started this thread with outweigh the costs you've been considering so far, or not? Or maybe you're still on the fence on this question?
2. Do you feel that Oberoi's encouragement for others to use ChatGPT, including a proposal to talk about business applications, is ethically relevant to this discussion?
3. You mentioned Oberoi's slide about hallucinations. Do you feel that his system adequately mitigates the risk these pose? (Feel free to read ahead for an example before answering this one). For clarity, my opinion on this question is that Oberoi is paying lip service to the issue here but has woefully inadequate safeguards.
I've now read through your blog post on Whisper, but what struck me the most was that in multiple areas, you acknowledge a harm being done, and then state that because nobody has yet called for a boycott due to that harm, you don't feel the harm is a sufficient reason for you not to use the system. You even acknowledge (I don't remember whether there or in a more recent post I just read) that you have friends who are directly experiencing some of the harms that made you shy of Whisper. I've also used automatic transcription systems on my own videos (with manual revisions), so I'm not in any different situation than you with respect to the ethics of my actions, but I'm not sure I'd write a blog post myself saying that I'd found them ethical-enough-to-use given the additional ethical implications of writing such a post. Neither in that post nor in this thread are you exactly cheerleading for AI systems, and I deeply appreciate that you are discussing the ethical issues, but the fact that you (at least in the Whisper post) ultimately conclude the ethical issues are not enough to warrant abstaining means that your post is part of an ecosystem which supports AI use.
To add a bit more substance to my criticism here, I've gone ahead and taken the time to investigate (briefly) my assumption that the city meeting transcripts would contain basic AI transcription errors that would be easy to find. I followed the link at the top of this thread (https://citymeetings.nyc/meetings/nyc-charter-revision-commission/2025-07-07-0500-pm-manhattan-public-input-session/chapter/a-clarification-on-the-term-jungle-primary-and-its-divisive-nature/) and checked the transcripts of the next few "chapters" looking for the transcription errors that I'd expect to find from AI output that was used with just spot-checking as opposed to a full video viewing & verification. It took me just 3 chapters to find a mistranscription of "jungle primaries" as "juggle primaries" in the testimony of Catherine Du, at 3:07:50 on this page: https://citymeetings.nyc/meetings/nyc-charter-revision-commission/2025-07-07-0500-pm-manhattan-public-input-session/chapter/testimony-by-catherine-du-from-fridays-for-future-nyc-against-jungle-primaries/
I don't have the time to wade through lots of video to establish an accurate error rate, but finding an error of this nature within a few chapters worth of video is exactly what I expected from such a system. Note that the transcript correctly uses "jungle" at other points in Catherine's testimony, so it's not the same kind of mistake a human would make. In case this example seems trivial since readers can deduce the correct word from context: as you stated in your blog post, being able to search these transcripts is one of their main benefits, but if that search becomes untrustworthy, that's a big drawback. Here the AI got 6/7 instances of "jungle" in Catherine's testimony correct. But what if her testimony only had one mention of the term, which the AI got wrong? Then if we were searching for speakers mentioning the issue, we'd miss her entire testimony. This kind of issue is why even infrequent transcription errors can have an outsized impact; of course there are numerous other ways for impacts to be magnified.
Maybe this one error is rare, and I'm unlucky to have encountered it so soon? To check that, I decided to watch a few more chapters to find the next substantial transcription error. Turns out I only had to watch one: https://citymeetings.nyc/meetings/nyc-charter-revision-commission/2025-07-07-0500-pm-manhattan-public-input-session/chapter/testimony-by-susan-stetzer-from-manhattan-community-board-3-on-the-land-use-process/
In this chapter, an "I" becomes an ungrammatical "I'm," "CB3" is rendered several different ways (typical AI mistake); NIMBY is rendered as NIBBI, and then I stopped counting issues. These AI transcripts are clearly poor quality for things like text search. On top of that, quality control for the chapter summary (where we got the promise of closer human scrutiny) isn't great: it has two paragraphs which are repeated as bullet points in a longer list of sub-topics, which is confusing to read. Predictably, the quality of the human oversight declines over time, letting more and more AI-generated stuff through as-is, which also predictably will lead to serious hallucinations getting through eventually.
Having had my initial assumptions about the quality of transcription anecdotally confirmed, I'm not feeling bad about my overall assessment of the potential harms of the system, including the ways it might undercut efforts to improve transcript access times for the human-produced transcripts. Interestingly, in Du's testimony that I reviewed, a human interrupted the testimony to ask her to slow down so that could get an accurate transcript. If you delegate to AI for transcripts, that's not going to happen. I have also already mentioned the likelihood that transcript quality will be systematically lower for a variety of already-marginalized groups.

2025-07-18 23:33:28 UTC

- reply

I'm fully aware I've been critical-to-derisive of the system you praised at the beginning of this thread the whole time. I do appreciate the effort you've put into responding at all; my initial engagement was mainly motivated by the fact that you seemed to be on the fence about AI and I felt that a technical criticism of the system you were crediting with changing your mind might be useful to you and/or to others reading along. I intentionally didn't lead with broad ethical criticisms of AI because I felt that a more targeted critique of problems specific to the system you mentioned would be more useful.
Your position at this point seems to be that the value this system has displayed from your perspective outweighs the various ethical & technical concerns, and it's not worth further time debating them. That's definitely not my conclusion, but it's definitely fair for you to decide when it's no longer worth your time to engage here and I don't resent/oppose that.
Near the start of your thread, you said:
"""
I am tremendously excited about how AI can make government data more accessible and transparent.
Tools like this are valuable public goods. I'd like to see cities fund them in the way they do libraries.
Cities should do things like this instead of releasing chatbots!"
"""
These statements followed an example of a tangible civic good achieved with the help of the AI system we've been discussing, which presumably motivated them. The root of my criticism is that the same civic good could have been achieved by means with fewer negative externalities. The specific means for that I've suggested involves more effort but also has additional upsides. Of course, it's a hypothetical while this system is real & already producing results, so an alternative is an extra hard sell. A lot of the calculus probably does depend on how bad you think it is to use ChatGPT (for any purpose) and to encourage others to use it. I think both of those things are pretty bad, although that's not the main argument I've been making until now.
In direct response to your initial statements I just quoted, I think that:
1. Cities are better off spending their money on non-AI solutions, especially in cases like this where the AI duplicates a lot of effort humans are already doing. This system has a great UI design, but the AI parts of it trade off speed & cost against reliability in harmful ways, before even getting into the negative externalities inherent in large-LLM use.
2. The easy-to-cite, easy-to-verify, and well-organized aspects of access to council meetings is the thing that's s valuable public good here. AI might be one means to achieve that good, but it's far from the only one, and for all the reasons I've cited, I don't think it's a very good means to this end. Equating the system with the public good is bad, because it helps people think that the AI is the good thing.
3. It's interesting that you are excited about how AI can help make government data more accessible & transparent, rather than in accessible & transparent government data regardless of the means used to achieve it. I don't think too much should be read into that phrasing given you're stating it in the context of moving away from AI skepticism: you are presumably also interested in government transparency period. But I think there's a big risk here of leaning too much into the AI toolset as enabling cool new things, when many of those things could have been better achieved without AI, given its significant externalities.

2025-07-18 21:05:30 UTC

- reply

I've got the bandwidth for occasional discussions on here, but with childcare duties for a baby plus a young kid and classes starting in the Fall, I sadly don't have time for deeper engagement.
I don't think I have that much special knowledge on these topics though. Like, I've got a basic understanding of how neutral networks work, but I've never dived deep into transformer models (only "deep" learning project I've done was convolutional neutral nets on images with unsupervised learning, and the dataset was small my today's standards).
Some general advice I can give:
- I think it's worth it to use some tensorflow/torch/keras/whatever-is-hot-these-days tutorials to build a little autoencoder. Shouldn't take too long and you'll get some sense of the necessary basics from it. Maybe there are good transformer model tutorials out there too? Maybe you've already done this kind of thing though?
- I don't have great advice for fiddling with prompt engineering, because I don't think it's ethical to dabble using the big easy platforms, especially if you're going to pay them, and their free tiers may have limits that make dabbling hard. If I had the time I'd try to download something to run locally; most people don't have the compute necessary to train anything interesting is my impression, but running queries should be much cheaper than training. I think I'd try DeepSeek for this if I did it.
I have spent some time with Claude free tier in order to make an assignment that AI will do poorly on so we can have students experience that (it wasn't hard). I make sure to give students the option to read through pre-generated example AI sessions instead of using it themselves if they want to abstain.
- Pick some experts you find trustworthy to follow & listen to them. I'm confident you're already doing that, but others reading this thread might not be. For me, Timnit Gebru stands out as someone with deep expertise in AI ethics & who has spoken out about injustices related to AI when others have been silent. In part my mostly-abstain stance just comes from asking myself "why do I feel I can make a better decision than Timnit Gebru about this?" when I get tempted to dive in.
- LLMs are indeed seductive in what they seem to enable, with the price being climate & digital commons externalities, plus a kind of rotten untrustworthiness in their output that tends to shine through in systems built on top of them. If I wanted to turn to a different AI technique instead for solving problems, I might study Monte Carlo tree search (and probably cooler offshoots of it that have appeared in the decades since; I'm not totally up-to-date). Another obscure but theoretically-exciting-to-me topic is logical abduction systems & rule inference systems. None of these are going to give you flashy results like LLMs, and none of them probably fit the bill for "tech that I can quickly plug in almost anywhere to build a cool-looking demo."

2025-07-18 20:36:59 UTC

- reply

to address some further points:
1. I indeed do not live in NY, so this is a system that impacts you much more directly than me.
2. I'll admit the speed of availability is probably more important than I initially assumed. But the problem "transcripts need to be available faster" has a much less glamorous & less technical solution: change the bureaucratic process by which they are released. That's not a solution that a single developer can unilaterally automate, no matter how much carbon they're willing to burn in the process. But it's completely achievable and has a ton of other benefits. I wonder at this point: did the developer of this complicated system at any point even ask for faster transcript release, or offer to work with city officials? There's a design of this tool that starts with a council official uploading video + transcript themselves, uses AI to synch the two, relies on the human note-taker to include chapter headings in the transcript using a special format, and then presents the whole thing using the same very nice searchable & linkable interface. But that design doesn't use LLMs, so it's not cool. I'm guessing designs not involving LLMs were never considered at any stage of this project, because it was a project about applying LLMs to civic problems, not a project about solving civic problems.

2025-07-18 20:18:26 UTC

- reply

I'll try to reply to your various points here to preserve threading. First, on transcription overview, the annotated slides say:
"""
Efficiently and scalably generating useful chapters for multi-hour city council hearings is much harder.
That is where at least 80% of my effort goes into citymeetings.nyc.
"""
Later it says:
"""
Transcript quality varies by things like:
How far away someone is from a mic.
How fast they’re speaking.
How accented their English is.
How much background noise there is.
Deepgram hits a workable price-to-performance ratio for me. I don’t have to fiddle with it.
"""
It seems pretty clear from this that the transcripts are not being double-checked second-to-second, and are instead at best spot-checked. Checking a transcript in a way that will effectively guard against "hallucinations" requires at minimum watching the entire video. It's maybe 50-75% as much work as transcribing the video yourself. AI transcription quality is absolutely not good enough without careful human checks, because of homophones, unusual jargon, and especially homophones which are also unusual jargon (thanks computer science!). (Digression: I agree with you about the pandemic not being over, and should not have used the phrasing I did. "While teaching remotely due to the pandemic" would have been a better choice of words.)
The fact that the author of this system believes that the transcription AI is good enough not to carefully revise its outputs is a red flag to me. If I'm wrong about this I'll be happy to be corrected, but this attitude that rare-but-serious errors are acceptable is a common one among AI advocates (and technosolutionists more broadly) and examples of the very real harm it causes abound (facial recognition errors, to pick a currently-relevant example). The topic at the start of this thread is a good example: serious debate & political implications arising from a specific non-common phrase like "jungle primary." That's a phrase likely to be mis-transcribed by AI, but much less likely to be mis-transcribed by a human in the room who understands it's relevant in the current discussion context despite being rare-in-corpus. And of course it's more likely to be mistranscribed the less training-set-typical the person's speech, which can clearly introduce class/race/gender biases. If my system is making easily-quotable blurbs perfect for posting & reposting on social media, I'd want to be extra careful about this sort of thing. But the author of this system spends only one side on transcription, and doesn't mention more than anecdotal observation of error rates, with a note about a "workable price-to-performance" ratio that's the opposite of reassuring.
Going back to my central point, if our goal is to make city council meetings more accessible, while the linkable chaptered transcript-with-video design is an excellent solution, its value goes way down if the transcript includes occasional hallucinations, because they can lead to our system creating new harms, so if we were designing such a system from scratch, I'd strategize about the best way to get reliable transcripts rather than assuming AI can solve that problem.

2025-07-17 13:03:11 UTC

- reply

I wrote a longer response up-thread, but I think the harm-reduction mindset has to at least start by asking the question: "Was genAI needed or even effective at solving this problem? Is there a better existing (or researchable) tool for this job that genAI hype is burying? Are there non-computer-involving solutions that genAI is pushing out here?"
I think the city meetings transcript site fares poorly on these questions.

2025-07-17 12:55:12 UTC

- reply

thanks for sharing your thoughts on this, which is also something I think about a lot, as I have a big responsibility to my students to have good takes about the ethics of AI. From what you've said here it sounds like I'm quite a bit more in the "abstain completely from using LLMs" camp than you, but I can respect that you're actually grappling with the ethical questions at all, unlike most people I've ever meet who don't abstain.
To me, the environmental + intellectual commons pollution/enclosure issues are significant enough that I think a default stance should be "don't use unless there's a very good reason" along with "definitely avoid the worst players, even though they're most-accessible."
Having read through some of the description of the NYC meetings site you're talking about here, it seems to me an incredibly irresponsible and damaging project:
1. It's an AI-first process with promises of human checks, but the system is designed without adequate time/effort for those checks, basically guaranteeing AI mistakes will go uncaught. At minimum, checking an AI transcript of a 3-hour video is a 4-6 hour task, for example. I know because I did this for my own video lectures during the pandemic. The accuracy of the AI cannot reduce this task to less than ~2 hours (assuming watching the video at more than 1x speed). Time for these kinds of thorough checks is not budgeted, so checks will be perfunctory, and we can extrapolate that someone too lazy/busy to do thorough checking will end up trusting the AI more and more as time goes on.
2. High quality human transcripts of these meetings *are* available, but are not used (!). The justification is that they take a while to release them sometimes. Even if I bought that that mattered, at minimum a responsibly engineered system would include double-checking system output against the official transcript when it became available.
3. An effort like this sucks energy away from better tools, and reduces pressure that would improve others. For example, those releasing official transcripts slowly, if asked to get them out faster, can point to this system and say that because if it we don't need to.
4. It applies an ethically tainted and resource-wasteful tool which is a poor fit for these problem. Given the availability of official transcripts, with the only complaint being that they aren't time-stamped against the video, this whole (admittedly cool as an end-product) website could have been designed using non-generative AI to add timestamps to the transcripts. Chapter recognition is perhaps a harder problem, but "ask the note-taker to include topic headings" is the correct solution to that. It's particularly noteworthy that AI mistakes in transcript timings have order-of-magnitude less potential to cause real-world problems than AI mistakes in transcription (which are frequent, systematic, and biased).
I think the larger point that this helps clarify for me deserves its own post, but in short, it bothers me that so many LLM applications seem to be all about "now that we have new tool X, what can we do with it" while completely ignoring the question "for problem Y, what is the best tool for the job?"
Brb going to make a post about that.

2025-07-06 11:29:34 UTC

- reply

Tactically, the pharma problem merits actions that free their hoarded knowledge back into the commons, and/or that provide drugs for free in contravention of copyrights/patents.
Trying to deny them access to the commons doesn't make sense, especially because if your alternative is pay gates, they'll easily pay those tolls so you're just excluding others without slowing them down. Some kind of "copyleft license for drug research equivalent" might make sense, but I'm not sure there's any legal framework that would support that.
In contrast, denying commons access to LLM peddlers makes sense, including both read and write access. I don't think that legal/copyright frameworks will be especially effective at this (I used to think so but recent rulings have been disappointing; maybe it was always a mistake to imagine a system designed to serve capital would do otherwise). However, I don't think there's anything wrong with trying to use copyright against the LLM peddlers. Things like Anubis and tar pits to frustrate & poison scrapers are good tactics I think.

2025-07-06 11:19:59 UTC

- reply

yeah the commons by definition must be available to everyone, but defending it from groups that abuse it is necessary to preserve it. And I think you've stated my point better than I did:
The pharma companies are not destroying the part of the commons they take from, but by building new knowledge that they don't share and instead hoard in order to profit from misery, they're detracting from what the commons should be.
In contrast, the LLM companies are effectively destroying the commons they take from in several ways: slop pollution, scraping pressure, and competition with creators are probably the top 3.

2025-07-06 11:00:48 UTC

- reply

um... I think that open-access publishing is great, but also that pharmaceutical companies profiting off it is evil? In the same vein, I think publishing stuff for free access is great, and that the LLM companies are evil. I don't think that the people doing the publishing are blamed by many people in either case.
If you're looking to distinguish the cases though, the pharmaceutical companies don't typically compete directly with the academic doctors doing the work. They're making drugs that help people (good) and locking them behind steep fees (evil) whereas the LLM companies are producing slop of the same form as the content they harvest and displacing it, including actively encouraging people looking for such content to rely on them instead of the people they take training data from.

2025-06-29 12:49:51 UTC

- reply

to add a metaphor here: imagine your view of the world were just a 16x16-pixel square, each pixel averaging the colors from its section of your field of view. Fine details are impossible to see unless you're up close, but coming close you can't see the big picture any more. As your head turns, colors shift, and even a small change in pose can provide a very different picture. You try to hard to extrapolate what you might be seeing that your visual memory isn't very good, and you're always guessing at what you might be seeing but never sure. Based on context, you can often guess correctly, but when context is misleading or absent, you get things wrong. Your mental model of the world is always fuzzy & low-confidence, thus prone to sudden shifts as you try to reinterpret things, but even when trying to correct for mistakes you aren't always right.
Now imagine that you're punished any time you don't sound confident or try to hedge your answers. In this situation, you end up spewing a lot of BS, all while trying to play along with whatever you get told since that's where your context comes from. You do have a mental model of the world, but it's not a very good or stable one.
This metaphor is broken in a few ways. Most LLMs have much more limited mental model capacity than the average person, for example, and are even more context- than model-based in their responses than this example. Also they don't have a lot of human reasoning faculties available at all. It's also at it's heart an anthropomorphization. However, I think it's useful for starting to understand LLM failure modes.

2025-06-29 12:21:17 UTC

- reply

disappointingly, this article still anthropomorphizes LLMs, and cites the recent Apple paper but not DAIR or other earlier critics who said exactly the same thing.
The point is also subtly wrong: LLMs do have a world model, but it's an alien one which is both very distorted and incredibly limited in scope. The internal state of a transformer model or other recurrent neural network serves as its world model, but these are typically extremely small relative to the complexity of modeling required for most everyday tasks & games, and their "rules" as enforced by their trained weights are only pseudo-logical. The point that this is a fatal flaw for a lot of applications still stands, of course.

2025-06-28 23:27:03 UTC

- reply

the most obvious illustration of this is that despite having to pay significant healthcare costs per employee, zero companies have ever lobbied for a government-run healthcare system that would let them avoid those costs. They see them as an investment in loyalty & hard work that's well worth the money.

2025-06-19 12:08:45 UTC

- reply

it's fascinating to me that the author feels the need to position themself in the "reasonable middle" of some hypothetical pro/anti divide, or perhaps actually feels they belong there?
Like, the author is presumably not an actual deep learning researcher or expert, but feels it's reasonable to completely dismiss the stochastic parrots paper even though it's written by actual experts & also very accurately predicts exactly the kind of harms AI has given rise to? Or perhaps they've never actually read the paper, but are reacting to those who cite it (sometimes overzealously) in arguments? In which case, perhaps positioning oneself in the "reasonable middle" between positions one of which you haven't seriously tried to understand is not the move the author thinks it is?
It's also interesting that the main criticisms on offer here are that the original essay is over-hyping things (which, duh), with only a passing mention made of ethical objections which are then ignored throughout the rest of the essay. Like, it's one thing to say "this stuff is really only situationally helpful, you're making too big a deal of it" and another to say that while acknowledging that it's significantly exacerbating an existential climate crisis, such that "making too big a deal of it" takes on an entirely different character.
The AI hype crowd is also actively renovating the modern-day Library of Alexandria into a new Tower of Babel, against the wishes of its librarians, yet this somehow isn't worth mentioning in your critique of them? Perhaps the only objection is that the renovation is likely to be over-budget and behind schedule, which seems to be missing the most important point.
I'm definitely a fan of witty criticisms of AI hype, and I can see how this defanged "centrist" version might even reach some ears that richer criticisms now won't, but this was not a satisfying read.

2025-06-03 18:11:10 UTC

- reply

"I can understand the output and see whether the logic is correct" shows a poor understanding of programming. If this were true, you'd simply generate the output itself, and skip the program. The entire point of writing a program to do something instead of just doing it yourself, *especially* in this age of generative systems, is to have the program be robust enough to handle cases that you do not inspect, either because it would be tedious to do so, it because it would be impossible as you can't predict what inputs users will supply. Properly constructed, a program is a tool that has *predictable* and *well-understood* behavior even for many cases that the programmer anticipated but did not check explicitly, and the programmer can be confident that it has this extrapolative behavior because the programmer has a good mental model of *how* the program works. Given your admission that you don't know the languages you're using, you definitely do not have a good mental model of how the programs work, so you're inspecting a few cases and then assuming the process is probably sound based on a poor/shallow mental model of the program, which is a great recipe for eventual disaster, pretty much no matter what you're using the code for (even hobby/art projects have significant reputational failure risks).
To address the larger point about compilers, which I've certainly wondered about myself, the qualitative difference is that compilers translate code from one language with well-defined semantics into another which also has well-defined semantics. This means that we can verify their correctness, make progress by fixing any correctness issues we find, and such fixes can be applied at scale by recompiling code. Many iterations of that process produce high-quality compilers, which give us strong assurances that as long as we understand the semantics of the high-level language being compiled, we can write programs with predictable behavior once compiled without needing to inspect or even understand the assembly code. Compilers enable a division of labor between those writing high-level code and those checking compiler correctness, and this is only possible because the abstraction they provide is not very leaky (in the correctness domain at least; it's quite a bit more leaky when considering performance, which is why we still teach assembly language in school).
LLM-based programming has basically none of these properties:
Nobody will every be able to give any guarantees about the semantics of the programs LLMs spit out, not can we even make progress towards such guarantees. Fundamentally, the translation performed by an LLM is between input prompts and programs, and "input prompts" is not a domain with any reliable formal semantics. Worse, I'd wager most vibe coders probably do not retain the sequence of prompts & hand-edits made to construct their programs, so it's like throwing away your source code as soon as you compile it. Even if someone did retain this stuff, the models used are mostly in the cloud without strong versioning or archiving, and thus with little ability to reproducibly query the same model in the future. Most models are explicitly tuned to *not* provide the same output multiple times for the same prompts anyways.
If all these things were true of compilers, I'd be against them today, and so would a lot of people. On the other hand, I think it would actually be cool (modulo a few other big ethical issues) to work towards LLMs for coding that actually had some of these properties, but I don't foresee those with the resources to do so doing that, because it undermines a lot of their AI hype, and because it sounds like an actually difficult project compared to tuning up an LLM (which to be fair isn't trivial).

2024-12-25 03:59:30 UTC

URL for metroidvania graph viewer is now live:
https://cs.wellesley.edu/~pmwh/mvmap/egviewer/explorationViewer/viewer/
#metroidvania #visualization #graph
It loads with example data but I have several actual game graphs that I need to make accessible...

2024-12-19 12:22:28 UTC

Getting some pretty nice results from my Metroidvania graph visualizer already. This is Haiku the Robot using a force-directed relaxation of an approximate-trilateration-based initial layout. Lots of refinements to add here but I'm pretty happy with it already :)
#graph #visualization #metroidvania
https://kolektiva.social/system/media_attachments/files/113/679/451/917/840/950/original/37db0d9a9bcb8f54.png

2024-08-06 13:40:42 UTC

So... just confirmed an already-reported bug in #Python that some on here may be interested to know about. In at least 3.11 and probably 3.9-3.12+, adding or subtracting datetime.datetime objects which are time-zone-aware and have the same tzinfo will give incorrect results when crossing a time shift in their time zone.
For example, there's a fall-back shift on November 3rd this year in the Eastern US time zone, from EDT (Eastern Daylight Time) to EST (Eastern Standard Time). The zoneinfo module can represent this time zone if you say zoneinfo.ZoneInfo('America/New_York') and you have IANA time zone files on your system (IIRC Windows doesn't). At 1 a.m. on 2024-11-3, an hour repeats, first in EDT and then in EST. Python can represent the repeated hour and convert times from/to UTC mostly correctly, but will give incorrect results when e.g. adding across the transition. For example, 2024-11-03T00:45 EDT plus 1 hour is 2024-11-03T01:45 EDT which python gets right. But plus 2 hours should be 2024-11-03T01:45 E*S*T, which python gets wrong (it returns 2024-11-03T02:45 EST, which is 2 hours later, not 1.
Similarly, subtracting 2024-11-2T00:00 EDT from 2024-11-4T00:00 EST should give a difference of 2 days and 1 hour, but python gives 2 days exactly (it does work correctly if you give the time zones as UTC offsets, but not if you use the same tzinfo object).
Time math is incredibly cursed (see above) so I don't blame the Python devs for this, but I do hope it will get fixed.
My guess is that the code checks if the tzinfo objects are the same and then does simple math if they are, when instead it should have checked whether their utfoffset results were the same. Gotta think whether I have the time to submit a PR for this...
#cursed #bug

2024-06-24 12:28:10 UTC

- reply

yeah this is it. I dunno how I got there last night, but I reboosted a bunch cause it was so good, only partway through realizing it's all 3 years old. Now I'm wondering whether "necroing" is a bad thing on fedi... I guess it might be annoying for one's notifs, and my client, at least, doesn't have great bundling features. I probably should be more judicious with boosting/liking multiple things from the same thread, I guess, but there was so much insight there, on a lot of different topics.