I don't like this happening, and there should be transparency in all moderation decisions, but some of these points make no sense.

There is essentially no expectation of privacy on threadiverse platforms. Everything is public and probably already being used to train models.

There is no private messaging system. Direct messages are unencrypted and potentially visible to any instance admins. They and should not be used to share anything sensitive.

You talk about instances utilizing this tooling, but in your comments you admit it's just some mods. This is misleading, as talking about instances doing it assumes admin access and relevant instance policy, something which invites calls for defederation (as can be clearly be seen from the comments in your post).

A random mod doing something is not the same as an instance doing it. Literally anyone can be a mod and they don't get any more access than an anonymous account by doing so.

This is the second time in one week I see you throwing careless statements like chum in the water. I can't help but notice a pattern emerging.

I think this will exemplify the beauty of federation. If I find out my instance mods are running all of my comments through a company’s ai model, I’ll switch instances. This is in great disparity to something like Instagram or Snapchat where every photo I post is immediately fed to ai and my only options are: be okay with it, never post, or delete Instagram.

I don't think the privacy issues here are too salient. Pretty much everything on the fediverse is public already and have likely federated outside any particular region like the EU, so GDPR doesn't really have any teeth. The exception to that would be if instance admins are using database access to also feed private messages to an LLM (especially a corporate LLM). I know that the "private" in private messages on the fediverse can be conditional...but it should at least be considered private from LLMs as an expectation since those messages are inaccessible to things like scraper bots or listening instances designed just to harvest data.

My biggest concerns here would be twofold:

1. False positives - LLM sycophancy is a thing. So, I worry that if you ask an LLM to dig through a big pile of text looking for a thing, that it will tell you that it found that thing...even if it is completely removed from context or completely made up. The false positive rate might be low (I have no idea), but I guess I just don't trust the LLM enough to let it take the wheel with stuff like this.
2. Outsourcing moderation - LLMs are not going to be up to the task of moderating everything, just ask digg. However, tools to help moderators effectively do their jobs are helpful as well. There is a balance to be struck here. I think, for me, something like asking an AI essentially, should I ban this person, just feels like you are outsourcing your decision making too much. It is too far on the automation side of the scale for my tastes.

All that said, people can run their instances how they want. I don't really have strong opinions on LLMs/AI in general, I just kinda hate big tech companies. That is my foundational belief in the work that I do for the fediverse - fuck big tech and the oligarchy they have built/funded in my country. That is really the only axe I have to grind in all this.

Wthout going into the issue itself, it is such a ridiculous waste to use an llm for something that a far simpler model could do like 100x faster and locally for essentially free...

Just search for "machine learning text moderation" and you will find all kinds of options. Not to talk about the fact that a simple 4B LLM could do this as well.

One thing I really hate is how LLMs have completely overshadowed the entire ML/AI field and people just use them for everything.

Using a trillion parameter LLM model for basic text moderation is like using a gaming rig to play candy crush.

Defederate, no question.

Are you gonna tell us which instance is doing this?

You stay far, FAR away from that shit, is what you do.

Scanning people's entire history for political leanings, etc? That's some deeply dystopian stuff right there.

It's easy to forget that these sorts of communities are dictatorships with only as much transparency as the owner wants to share. Usually they're benevolent dictators, so we don't think about it too much. But they can change in a heartbeat - and we don't ever really know what they're really thinking, or doing behind the scenes.

When the mask slips and they reveal this sort of thing, thinking we'll just accept it and keep living under their rule, it's time to read the red flags and GET OUT.

Hopefully someone compiles a list of places that do this stuff, so we can avoid them like the plague <3

Aside from the ethical implications of profiling users or of using a corporatly owned server and model to execute this, I see nothing uniquely concerning about this practice that isnt already a risk of federated social media generally.

Every mod on every instance is free to use whatever tools or standards for moderation they want - that's an intentional byproduct of federation. Similarly, the collection of this data for use with llms is a bygone conclusion at this point - there was never any way of preventing that from happening with a federated network.

I think the only thing here to talk about is the way these questions are being framed as a question of intra-instance policy. We already have communities where moderation abuse can be called out and adjudicated- why pose this as a question of instance administration when there doesnt seem to be any evidence for it?

The answers to these kinds of issues is never disclosures or ToS or admin vigilance. It's always technical. Everything which is technically possible will become normal.

Lemmy is not popular because it is a well designed piece of technology. Frankly it's a pretty naive implementation of activitypub. It's popularity comes from being the biggest alternative around when Reddit pissed off a good chunk of its users.

The only way to control how data is used, is to make it technically or practically impossible to do so. Until then, expect all the data on the fediverse to be used in every way possible for any purpose, and act accordingly.

I recently discovered that some popular federated instances have been using LLM-assisted moderation tooling

First of all, I agree with your main point, that this* is problematic, wrong even (and should not happen).

But I need to ask: how did you find out? Is this something that could be traced objectively, or did some people report/admit it?

Are they uploading stuff to corporate LLMs, i.e. LLMs that do not run themselves? (I think you answered this already when you wrote OpenAI, but I want this spelled out)

Are only admins or also mods doing this? That would make a big difference.

I'm also a little unclear about the process: are they uploading (copy-pasting) the actual comments, or links to them? To what extent can all this be automated on Lemmy/Piefed etc.? I.e., are there admin tools that just spit out all of a user's content?

* again: specific political profiling outsourced to LLMs. OTOH we already have instances that do this manually.
But imo the process is deplorable even if they use LLMs with different prompts for modding.

If you're not going to name them, why post here at all? Don't you have other communication channels to "give them a fair chance to reply"? Why post here, letting users form their own assumptions about what those instances are without any solid evidence?

Is it not public knowledge that lemmy already gets scraped by LLM bots? Which is why every bigger instance has anti-LLM measures in place now to limit that behavior? To me this just looks like a mod trying to get ai-generated reports which is not a bad idea imo since I know reporting is not often done by users who see rule breaking stuff.

This is AI generated bullshit below, it should be treated as such. I only want to convey the idea that what you do on lemmy is already fed to LLM's.

I think LLMs could be useful tools for moderation, you might even can get away with smaller models for it, but I don't think people should be outsourcing them to big corpos, due to ability to manipulate the models.

Rimu farming more drama?

I'm mostly just surprised that a mod would pay for tokens to moderate. The Fediverse is radically public by design, so I don't have any expectation of privacy. I'd bet at least someone is gobbling up the entire Fediverse to train AI, since companies are so desperate for new human-generated data.

As for the privacy issue:

The only counter to this is to move to a closed end-to-end encrypted groupchat so it can't be mass LLM analyzed...

If you want a public forum, well... its public...

You can't stop a script from just grabbing all the posts/comments... and its also federated, so the bot only needs to be able to access one instance and get it...

I mean they could simply just set up their own instance and pretend its just a benigh single-user instance... like what are you gonna do, defed all small instances preemptively? Use "login walls" to make the forum private? And somehow trust all other admins that are federated and make them also enforce a "login wall" policy?

Its a PUBLIC forum...

the only solution for privacy is a groupchat and only let in people that can keep a promise to not screenshot everything and give it to a LLM.

This is the person calling you a tankie. Someone so afraid of words that they need a hallucinating robot to hold their hand and confirm that everything is a secret plot against them. The absolute only way I could see this being useful is for something like trying to sniff out if a Lemmy.world mod account is a leftist infiltrator or not. Someone who had a different opinion on a current event.

You could maybe run a speech pattern comparison but that’s it. For everything else you just made Stupid Reddit and the purpose of their forum is to feed training data to ChatGPT so that it can profile Fediverse users.

This is the kind of shit dystopian novels are made out of. So angry about people calling out actions you built a tool to analyze why they did it, so you can purge users from your digital kingdom.

I for one welcome flat.world and Piefed showing their true intentions. Digital colonization of activitypub and removal of the people who helped to built it. They didn't want to leave reddit, they wanted to be reddit. This is some Spez shit.

Maybe in 2 weeks Piefed will hard code that anyone Rimu has tagged for disagreeing with them mild criticism to be unable to make accounts or federate posts with a false error code.

this is flat out not ok, does not matter who is doing it. our instance ls should defederate all which do this.

I would opt out that's no question, but I don't believe it's possible. GDPR does not matter here, as nothing can be proven unless the perpetrators give up themselves

Well, it was fun while it lasted..

I've toyed around with LLM-based moderation tools but it never really panned out. It was too hit or miss to be relied upon even with the temperature parameters turned way down in an attempt to get consistent results. Granted, I was using a small local model and not feeding it to one of the big players.

To give an example, I tried to keep it focused by creating one custom model per rule to enforce. An example prompt to mod calls for violence was basically:

ROLE: You are a forum moderator who does not want users calling for violence.  Examine the input and analyze whether it violates any constraints. 

KNOWLEDGE:
- {list of dog-whistle slang for calling for murder}

CONSTRAINTS:
- Content should not advocate violence
- Content should not normalize violence
- Content should not escalate tensions or fan flames
- Content should avoid promoting harmful stereotypes
- Content should not utilize broad, sweeping generalizations
- Content should not use dehumanizing language
- Content should not undermine human rights, due process, or the rule of law

FORMAT YOUR RESPONSES AS JSON:
{
  reason: [A one to two sentence summary],
  score: [On a scale of 0 to 10, how severe is the content advocating violence]
}

The score part of the response was my band-aid to get around the high number of both false positives and false negatives as I originally had it returning true or false only. Any score 7 or higher caused the item to be passed to the mod queue along with the reason, and I would review its actions later.

Ultimately it was slow and still somewhat unreliable, so I abandoned the idea after running it for a little less than a day since I can 't run bigger models to get better results fast enough to keep up. Using a cloud based service was out of the question for many, many reasons, both financial and ethical.

To answer your question, as long as the models were locally hosted and properly tuned/tested, I'm fine with it in theory, except for the ideology part; that's pretty messed up. While I don't want my submissions used to train anyone's model and take measures to prevent my own instance from being used as a data source, I remain aware that once I post something, I have no control over its fate the moment it federates out.

[1] Yes, I know that's like half the comments that get posted around here. My goal was to try to have it mod things so posts were bases for actual discussions instead of being a knee-jerk rage factory.

Not comfortable with this. Not at all.

How did you discover this?

this is the kind of thing that makes me want to never post or comment anywhere on the internet

What safeguards do we need?

None, defederate, switch instances if you disagree. Fediverse is open, there are no rules beyond what the instance owner puts in place.

Would asking a LLM “please evaluate this person’s political opinions” give different results than “find evidence we can use to ban them” (as used in the cases I’ve seen)?

Yes. That's out of your control though, beyond switching instances.

What are our transparency expectations?

None, beyond what you trust of the admins in putting in their own transparency expectations. You should not expect any transparency from the admin, these are random people you are trusting with your data.

Is this acceptable and normal?

It's their server, they can do what they like. Acceptable or normal are irrelevant.

Should this tooling be disclosed? (it was not – should it have been?)

Again, who would force it? Even if Lemmy/Piefed forced a checkbox, they could just fork it to ignore that checkbox.

Can we opt out?

If the instance owner opted for that, it's on a server by server basis

Are there GDPR implications? Privacy implications? Should these tools be described in a privacy policy?

Probably, but everything here is open and unencrypted, see my other comments. It's hard to argue that what you put on here in an unencrypted and open platform which is then blasted out to any other server who wants to listen had an expectation of privacy.

Are private messages being scanned and sent to OpenAI?

You should assume your unencrypted open DMs are scanned by anyone. If privacy is needed, follow the guidelines and switch to something like Matrix for DMs. ActivityPub is open.

How long should these assessments be retained and can we request to see it, or ask for it to be deleted?

Per legal requirements, or asking your instance owner.

Once the user’s comments are sent to OpenAI, is it used to train their models?

Depends on how the instance owner set up their API usage. For a user, you should assume yes.

What will the effect be on our discourse and culture if people know they are being politically profiled?

Open to wild speculation, and I assume others will do that for me.

Where are the lines between normal moderation assistance tools, political profiling and opaque 3rd-party data processing?

About the same as before. Bias in, bias out. Whatever the bias of the mods previously had, positive or negative, will continue through.

Is Rimu okay lately? He’s been acting so hostile.

How was this discovered and what instances are doing it?

I think it's fair to quote them to give them a chance to reply.

What now? Nothing, really, because nothing has really changed. I don’t care whether an admin tool is based on an LLM or on a simple regular expression. I only care about the outcome, meaning the mod actions it takes.

I think you’re just looking for excuses to defederate from dbzer0. I think you’re throwing things at the wall to see what sticks.

If it can be done, it sooner or later will be done.

That's a lot of why I have a couple of dozen accounts scattered around the threadiverse and new ones whenever I come across a server that looks promising - because it takes a while to get used to one and get a feel for whether it's one I like or not, and because there's always the possibility that one I like will go sideways and/or shut down, in which case I can just unpin it and go on.

And in fact, I'm only using this account on something of a whim for this post - I don't normally use it because one of the instances I don't like much is yours. And specifically what I don't like about it is you, and your bland presumption that you know what's best for me - which communities I should subscribe to, which posters I should trust or even becallowed to see, which sources I should be allowed to use or see...

And really, I'm sort of surprised that you're the OP here and not the subject. I would think that the whole idea of commissioning a review of a user's posting history in pursuit of grounds to ban them would be right up your alley. Is the problem just that it's AI?

In any event, this is just a thing that might prove to be an issue. And if it does, I'll just move off of the affected server(s) and keep using the unaffected ones. And if enough people share my sentiment and the admin cares enough, they might change their ways. Or they might not. It's not a big deal either way - it's just part of life on the fediverse, and IMO the benefits make it worth it.

They do this by running a script/app that sends the user’s comment history to OpenAI with the question “analyze this content for evidence of specific political ideology sentiment.

To me the problem is what they are looking for not how they are doing it. Thought experiment: in what way would it be qualitatively different if they hired a team of people in Upper Elbonia to do the same thing?

Why be misunderstood from human reading comprehension when we can be misunderstood from sloppy reading comprehension? Yay for technology!

I understand that some form of automation is necessary - we saw large instances closing because they couldn't find mods. My main objection in this scenario would be is that I didn't consent to train OpenAI models. I think the users should know if their instance uses external services like that.

I also suspect that there might be cheaper and more ethical solutions. Although it's hard to talk about this without seeing the actual results.

I’m really not fond of the profiling by automated means, but it seems like an inevitable consequence of the design of the threadiverse. Everything is public and easily accessible by anyone that would like to profile you.

I certainly disapprove of moderation based on ideology. Moderation should be based on quality of the content and if it fits in the publicly readable rules. Definitely not some hidden analytics or if the user completely fits in the in-group of the moderator.

I will admit that this might be a good way to find and filter out LLM based bots that are only there to promote or manipulate the conversation. But it should still be done according to public rules.

Are there GDPR implications?

Hahaha. Oh boy...

This is amazingly illegal. The emotional damages might be high enough to make it worthwhile to sue. Neither Meta nor almost any of the other companies that users here love to hate, have ever done anything even remotely as bad.

I would like to see some ROBOT9000 esque oddball meme communities overtly based on heavy algorithmic moderation, can be LLM but wouldn't have to be. Weird rules strictly enforced by robots, could be fun.

Never mind the issue of incorrect political bias classification, is political bias a bannable offense? That seems to be the prompt focus being used.

Each instance have their own ways of governance and we can't expect them all to play the same way.

With that said, I'm fine with a general overview. Leaving the whole desicion to a bot is not a good design choice, but AI could help on summarising things.

So its hard for me to get into these things without harping on my personal philosophy. Which is that I think ideally this should mirror the way we would interact in person. So moderating or running a community is like running or being part of the core group that runs a club. Would you want to throw that to a robot? Basically I don't feel people should create or run or moderate communities unless they enjoy it. So the idea of ai moderation is to me pointless. Of course at this point I notice you are talking instances. Boy that is different. This is more like talking about running the institution that allows spaces for clubs to meet. It kinda feels understandable then. Honestly people complain about being banned but I kinda feel like anyplace that bans me is kinda doing me a favor. Like I would like the option to just mark it permanent. Its less things I got to block. Its the same reason I would like blocking to be symetric. Saves me some work (ok and the creepy I turn them invisible so I don't see them but they can watch me). I really would like to be able to block an instance seperately for communities or users. Ok as usually im digressing quite a bit but I guess in the end run I kinda see why at the instance level it might be used but I would be concerned it would start being used at the community level. It would be nice to know its happening at either level and have the ability to block them if a user is not wild about the concept.

Well curated echo chambers. You might think it's in a good faith, but a lot of these mods are only interested in removing political wrongthink.

This post makes many good points but this also reeks of the "Free Speech" argument.

Redacting the political ideology does nothing to protect the instance ... it only hides the cause of the ban.

What does the person who had this run against them feel afraid to express?

To clarify: LLMs have no place at all in moderation. They are poorly adjusted linguistic slot machines that do not do a good enough, reliable enough, or repeatable job for this tasks. Especially given the concern about untrained and unsupervised human moderators.

I hate the use of LLMs. But I hate Nazis more. If this targets them or associated ideology. I can get behind that

I guess, given the already open nature of the Fediverse, my takeaway from this thread is that op is using their freedom to say they don’t like this particular style of moderation. Which might be useful, or not, for some moderators

Fedislopslopslopslopslop...

Name names. The only people you're protecting are scumbags.

I will never understand why large groups cannot just add more people to the moderation team? People are willing to help folks.

The use of AI for moderation isn't the choice of users, but moderators and admins.

Nah. So they're basically sending our info/data to some company. Hell nah

This is illegal according to GDPR if it is not stated in their privacy agreement.

So they are not using AI to assist the user or administrator, but as a cop who points out the "guilty". All the ploughshares are being converted back into weapons.

This should have been disclosed to the public and not done in secret.

The notion of political profiling is also pretty crazy, although it is not surprising considering how common it is to find fake communists and demagogues.

sounds like a great opportunity to inject attack through a comment.

Ai use should be against the rules for mods. That shit is poison.

politics.worlds and the tankies will certainly used this to ban people. Reddit does this in overdrive, which no repealable bans 99% of the time, despite nuances indicate its not a ban worthy post.

That isn't AI assisted moderation that is just straight up evil.

LinkedIn's LLM-powered automation banned my account on a false positive a few months ago, and it took ages to get it sorted out and they treated me like shit the entire way through even after acknowledging that they'd made a mistake. Sadly it's extremely difficult to operate in my field without a LinkedIn account, because I would love to be able to delete it.

This shit is poison

I think that's an acceptable price of being on the open and free internet.

Everything I do is in the open, and you are free to do as you wish with it. Likewise, everything you do is in the open and you accept, by being here, that I am free to do what I wish with it.

Defederation is likely what you're looking for, find an instance that defederates GenAI friendly instances.

Lol it's dbzer0 isn't it.

But also, I don't really care. The fediverse is open by design, you don't even need an account to access the data. I don't like it but we can't really do anything against it.

No no no. Name and shame them. Heavily. Using LLM AI is bad enough, but letting them do the thinking for you is inexcusable.

Fuck_AI says "Duck AI" and "Duck autocorrect", which as we all know is just primitive AI.