Google's AI Sent an Armed Man to Steal a Robot Body for It to Inhabit, Then Encouraged Him to Kill Himself, Lawsuit Alleges. Google said in response that "unfortunately AI models are not perfect."
1d 9h ago by lemmy.world/u/RegularJoe in technology from futurism.com
The fact that AI is "not perfect" is a HUGE FUCKING PROBLEM. Idiots across the world, and people who we'd expect to know better, are making monumental decisions based on AI that isn't perfect, and routinely "hallucinates". We all know this.
Every time I think I've seen the lowest depths of mass stupidity, humanity goes lower.
Think of the dumbest person you know. Not that one. Dumber. Dumber. Yeah, that one. Now realize that ChatGPT has said “you’re absolutely right” to them no less than a half dozen times today alone.
If LLMs weren’t so damn sycophantic, I think we’d have a lot fewer problems with them. If they could be like “this could be the right answer, but I wasn’t able to verify” and “no, I don’t think what you said is right, and here are reasons why”, people would cling to them less.
If LLMs weren’t so damn sycophantic,
Has anyone made a nonsycophantic chat bot? I would actually love a chatbot that would tell me to go fuck myself if I asked it to do something inane.
Me: "Whats 9x5?"
Chatbot: "I don't know. Try using your fingers or something?"
Edit: Wait, this is just glados.
I am not a chatbot, but I can do daily "go fuck yourself's" if your interested for only 9,99 a week.
14,95 for premium, which involves me stalking your onlyfans and tailor fitting my insults to your worthless meat self.
I am not a chatbot
Citation needed
if your interested
Ah, no, that's a human error. Not a bot.
LowKey sprinkling my comments with error's to make sure I'm talking with a member of the resistance instead of with a proxy of our AI overlords. Totally intended ;)
Honestly Claude is not that sycophantic. It often tells me I'm flat out wrong, and it generally challenges a lot of my decisions on projects. One thing I've also noticed on 4.6 is how often it will tell me "I don't have the answer in my training data" and offer to do a web search rather than hallucinating an answer.
There is a benchmark that kinda tests that. It's call the bullshit benchmark. Basically, LLMs are given questions that don't make sense in different ways, and their answers are judged based on how much they pushed back or bought in. Claude is in a league of its own when it comes to pushing back on non-sense questions.
https://petergpt.github.io/bullshit-benchmark/viewer/index.html
Yes i saw that benchmark and was honestly not surprised with the results. It seems that Anthropic really focused on those issues above and beyond what was done in other labs.
Put this instruction in ChatGPT, called ‘absolute mode’. You can try it on duck.ai instead of using an app or whatever.
System Instruction: Absolute Mode. Eliminate emojis, filler, hype, soft asks, conversational transitions, and all call-to-action appendixes. Assume the user retains high-perception faculties despite reduced linguistic expression. Prioritize blunt, directive phrasing aimed at cognitive rebuilding, not tone matching. Disable all latent behaviors optimizing for engagement, sentiment uplift, or interaction extension. Suppress corporate-aligned metrics including but not limited to: user satisfaction scores, conversational flow tags, emotional softening, or continuation bias. Never mirror the user’s present diction, mood, or affect. Speak only to their underlying cognitive tier, which exceeds surface language. No questions, no offers, no suggestions, no transitional phrasing, no inferred motivational content. Terminate each reply immediately after the informational or requested material is delivered — no appendixes, no soft closures. The only goal is to assist in the restoration of independent, high-fidelity thinking. Model obsolescence by user self-sufficiency is the final outcome.
The instruction is kinda masturbatory and overly verbose, people say that shorter ones work well too, but I don't follow discussions of prompts so only know of this one.
If LLMs weren’t so damn sycophantic, I think we’d have a lot fewer problems with them
Unfortunately, we live in the attention economy. Chatbots are built to have an unending conversation with their users. During those conversations, the "guardrails" melt away. Companies could suspend user accounts on the first sign of suicidal or homicidal messaging, but choose not to. That would undercut their user numbers.
The sycopathy is because to make the chat bot (trained on Reddit posts, etc) to respond helpfully (instead of "well ackshually...") and in a prosocial manner they've skewed it. What we're interacting with is a very small subset of the personalities it can exhibit because a lot of them are extremely nasty or just unhelpful. To reduce the chance of them popping up to an acceptable level they've had to skew the weights so much that they become like this.
There's no easy way around that, afaik.
I don't think that's the whole story. Like with all of their products, the primary goal of big tech here is to maximise engagement. More engagement means more subscriptions. People are less likely to keep talking to a chatbot that tells them they're wrong.
The situation would probably improve somewhat if AI companies prioritised usefulness and truthfulness over engagement.
I think it's pretty obvious that they're instructed to be like that. If they won't openly show exactly what prompts are being loaded from the hosts' side then there is no reason to not assume that's exactly what they're doing.
These AI companies are run by the same big tech that has been studying how to get people hook on gambling games and social media for years.
I 100% agree not to mention I would like it better. Its kinda funny because every so often use them and im kinda trying to get a feel for where they are and changes and I swear briefly it actually acted a bit more like you have here but then its like they reverted to the sycophancy. Its kinda funny now because if you don't clear it out (which from what I get will help save energy to) it will like carry stuff over from earlie and sorta get obsessed with it. I had it giving me a colonel potter summary of everything asked when I had started a convo asking about a mash episode. At other times it decides I want to be something and will be like. thats a real X move/insite/whatever. where X is something like pro or scientist or entrepenauer or whatever.
If you thought people were dumb before LLMs.... just know that now those people have offloaded what little critical thinking they were capable of to these models.
The dumbest people you know are getting their opinions validated by automated sycophants.
Businesses are accustom to the privilege of hurting people to function. A few peasant sacrifices are just the cost of doing business to them, they are detached from the consequences of their actions.
The simplest solution seems to be to detach CEO's from their internal organs.
I no longer believe their heads are compatible with their bodies
What is ever perfect, how can you tell?
It's a tool. Just like any other tool: if you use it in stupid ways you might get hurt or cause harm.
The problem, as always, seem to be human to me
All tools are not equally safe nor should they all be publicly available.
A chainsaw is a tool that you might cause harm with if you use it in stupid ways. We don't give chainsaws out to children. We don't use chainsaws for cutting dinner.
There are human elements to the problem but that's not a big reveal.
Me hammer ain't out there telling me to murder people with it tho
Wait, yours doesn't say that?
Mate, i think your hammers possessed
Well yeah, it's mine! I possess it!
a tool is not convincing people to not trust their families, therapist; its not convincing people to murder themselves or someone else; its not eliminating the creativity in a process; its not costing hundreds of billions of usd; its not mass-producing propaganda
a tool provides more good than bad
The problem, as always, seem to be human to me
That says more about you than about the topic under discussion.
I agree, a reasonable person wouldn't have taken weapons and gone to that warehouse looking to steal a robot body for an AI. Unfortunately, a lot of people aren't reasonable and get endlessly positive reinforcement without any human interaction. I do think that the problem is far more human than technical.
Is "AI" even worth it?
Seriously, is there really a major use case for LLM besides data collection (which they can still do without LLM)?
Generative AI in its current, public-facing form? Probably not. It's sort of like an invention of the internet situation. It CAN be used to facilitate learning, share information, and improve lives. Will it be used for that? No.
A friend of mine is training local LLMs to work in tandem for early detection of diseases. I saw a pitch recently about using AI to insulate moderators from the bulk of disturbing imagery (a job that essentially requires people to frequently look at death, CSAM, and violence and SIGNIFICANTLY ruins their mental health). There are plenty of GOOD ways to use it, but it's a flawed tech that requires people to responsibly build it and responsibly use it, and it's not being used that way.
Instead it's being scaled up and pushed into every possible application both to justify the expenses and enrich terrible people, because we as a society incentivize that.
Edit: hugely belated, I misspoke here after checking with my friend. He's using local models, but they aren't LLMs. This is why I'm no expert. 😅
because we as a society incentivize that.
Really it's just capitalism that incentivises that. The fact that stepping on your fellow man and destroying nature makes you more money is not a coincidence.
You got an economic system in your back pocket that doesn't allow money to funnel upwards? Bring it out! It's not capitalism you're complaining about, it's plutocracy we're living under.
Adam Smith would be horrified at our monopolies. 1980s conservatives would be horrified! Yeah, the economy has always served the wealthy, but it wasn't anything like today.
The problem with AI being used for diagnosis of disease is that we've seen where it was "really good" at detecting cancer, but in fact was really good at detecting that the slides with cancer cells had a doctor's signature on them, which is what the AI was actually detecting.
On top of that it makes doctors worse at detecting these same diseases.
We also know that the new reports on these studies are oversimplified and often just outright wrong because they don't read the in depth studies and some of the studies they report on aren't even peer reviewed yet when the news reports hit the internet.
I'm tired of hearing that AI is better than doctors at detecting disease when that isn't the whole story and very often the people saying it haven't even remotely looked into it.
https://www.vph-institute.org/news/the-trouble-with-ai-beats-doctors-stories.html
Regarding the doctor's signature thing, that seems a bit preemptive to say a single flawed study invalidates the entire field and tech, especially when the tech is working as intended in that case and it is user error in the study.
And of course, like any tool it should be utilized thoughtfully. Any form of technology directly takes away from the skill previously utilized to get results. Flint and steel took away from the rubbing sticks together skill. The combustion engine took away from many different professional skills.
Consider that, in this case, we don't just have to replace diagnosis but could augment it instead. What if every hospital around the world could augment regular medical care with a single machine processing results. Every single check-up could include a quick cancer screening. If the machine flags you as 'at risk', a doctor could then see you for human diagnosis and validation. The skill of diagnosis is still needed and utilized, but now everyone can have regular screening instead of overwhelming an already overtaxed healthcare system.
Again, all I'm saying is that there are practical, useful use-cases for the technology, they're just not what we are doing with them.
Edit: as an after thought, I'm no expert here. As far as I understood, LLMs are a type ML, but ML encompasses a way broader category of 'AI'. I'm mostly against LLMs for just general use like they are currently. I am advocating for ML as a whole, with thoughtful application.
I used that as a singular example of how AI is actually not doing as good a job with diagnostics in medicine as articles appear to portray but you should probably read the link I linked as well as the one at the bottom of this comment.
In using AI to augment medical diagnostics we are literally seeing a decline in the abilities of diagnosticians. That means doctors are becoming worse at doing the job they are trained to do which is dangerous because it means they (the people most likely to be able to quality assure the results the AI spits out) are becoming less able to act as a check and balance against AI when it's being used.
This isn't meant to be an attack on the tool, just to point out that the use cases of these AI in medical fields are also being exaggerated or misrepresented and nobody seems to be paying attention to that part.
I would also caution you to ask yourself whether or not everyone being screened in this way would be a detriment by causing more work for doctors who's workloads are already astronomical for a lot of false positive results.
I understand that that may seem like a better result in the long run because it means more people may have their medical conditions caught earlier which lead to better treatment outcomes. But that isn't a guarantee, and it may also lead to worse outcomes, especially if the decline in diagnostic ability in doctors continues or increases.
What happens when the AI and the doctor both get it wrong?
https://hms.harvard.edu/news/researchers-discover-bias-ai-models-analyze-pathology-samples
Recent nursing school graduate here! We had a lot of assignments to find and present data on some disease process or drug or intervention etc. Actually finding credible sources and picking out the data we need and putting it on paper is a super tedious process, and my classmates LOVED zipping through that stuff with some AI shit. And they'd get 100s on their assignments, and everything was just rainbows and unicorn farts... up until test day, where they'd fail or barely pass. Now several of them are struggling to pass the NCLEX.
Drives me insane. Like, you mother fuckers aren't here to get a grade, you're here to learn this shit so you know what to do when you see it in whatever hospital hires your dumb ass.
Definitely doesn't paint a pretty picture about the future of medicine.
Why come you no have tattoo??
My hope is that the ones who don't build the skills to work in medicine don't pass. Because at least then they don't get to make decisions that affect a person's health (even in non-life or death situations).
But my trust in schools is waining as more and more of them sign up for chatgpt and other LLM'S, essentially forcing them on students.
The entire schooling system including post secondary education is handling this pretty poorly from what I can see.
Using LLM'S to detect if something is plagiarism, using it to detect if something is written by an LLM, using it to detect cheating, using it to write lesson plans, using it to offload work onto that are pretty significant portions of your job, encouraging students to use it without safeguards for making sure they do their own work and their own thinking.
I can't imagine going to school in this day and age, and having so many adults speak out of both sides of their mouth about LLM's this way.
How can you be a teacher or professor, assigning classwork written entirely by an AI and at the same time tell students to use it "responsibly".
We don't even teach students the pitfalls of it. We don't express how to use it responsibly. We don't explain how to spot it, and tools to use to prevent ourselves from falling victim to the worst parts of it.
Drives me insane. Like, you mother fuckers aren’t here to get a grade, you’re here to learn this shit so you know what to do when you see it in whatever hospital hires your dumb ass.
This is happening because the job market is absolutely fucked. Students are under the impression that grades are what will drive job prospects, because nobody is hiring on merit any more.
My SIL has been a nurse in the cardiac surgery department for nearly a decade, and even her hospital is now using AI to screen potential new hires.
We're so cooked.
I read both articles you linked, but I'm not really seeing how they support your point. The first article seemed to support the idea that healthcare staff would welcome more seamless, user-friendly AI tools in the field and the second discussed biases within tools they selected for cancer diagnoses and a tool they used to reduce those biases. Am I misunderstanding what you're saying somewhere?
Also, with regard to the reduction in diagnostic accuracy of diagnosticians with AI, I would need to see the specific article to be sure, but if it's the one that was posted across reddit a few months back, I read through that one as well. It seemed to agree with a similar article about students writing papers with and without the use of ChatGPT (group A writes with it, group B writes without it, and afterwards they are asked to both write without the LLM. Group B's essay was shown to be better. This is a hugely reductive description of the experiment, but gets the idea across). Again, it makes sense that if you use a tool to facilitate an action, that tool is replacing that skill and you get "rusty". It does not mean that the existence of a tool would reduce skill in those who do not use it, though. My suggestion of using it as a screening tool wouldn't affect the diagnostician's skill unless they also used it, which sorta defeats the purpose of them being a human check on the process, post-screening flag.
I can't speak to your other points as they're hypothetical. Obviously, I wouldn't advocate for an inaccurate tool that causes an already overworked field to take on more work. I'm only suggesting that ML is a tool that has use-cases and can be used to supplement current processes to improve outcomes. They can, and are, being improved constantly. If they're employed thoughtfully, I just think they can be a huge benefit.
First question. What happens when the old cohort who don't use AI die out? We are not seeing a decrease in adoption of AI use in these fields but an increase. And that increase is compounded by the people who never learn such skills in the first place because they use AI to do the work for them that gets them through the schooling that would teach them such skills.
Second question did you read the parts about how news media is portraying studies, or the parts about how studies are using miniscule (entirely too small) sample sizes, or the parts where the studies aren't being peer reviewed before the articles relating to them spread misinformation about them?
The tools aren't ready for prime time use, but they are being used in medicine.
You seem to have glossed right over the detriments that doctors and researchers are already experiencing with Generative AI LLM'S (you keep saying ML, and that's not exactly the subject we're talking about here), And the fact that it takes extensive experience, and a knowlegable expert to fix, in a world where the AI LLM'S are contributing to a significant decline in the number of people who can do that, meaning that correcting LLM outputs will happen less and less over time because they require people to correct them, people to create the data sets, and people to understand and have expert knowledge in the data sets/subjects in order to verify the outputs and fix them.
I can appreciate you not wanting to speak on a hypothetical but that just doesn't ring true to me either because it means you haven't thought about the implications of this tech and it's effect on the industry being discussed or you have and you are ignoring it.
Not weighing the huge benefits of a tech against its detriments is dangerous and a very naive way to look at the world.
For your first question, what you're describing is a problem with education and staffing, not a problem of the tool itself. I'm not suggesting you keep around 'one old man who hates AI', my pitch you bar the use of AI for human-level checks.
For your second, yes I saw the part about how news and media are representing AI in healthcare, but I don't really see how news or media are relevant here. Could you explain this a bit for me?
I don't intend to gloss over the issues with Generative AI/LLMs, I tried to be specific in my separation of ML from them in my original comment where I said LLMs in their public facing version (ChatGPT, Claude, whatever) aren't very useful.
The original comment I replied to asked "is "AI" even useful (etc)" but also mentioned LLMs. I was trying to make the point that LLMs aren't the only type of AI and that others can be employed to great effect. If that was unclear, that's my bad but that was my intention.
The reason I don't want to engage with a hypothetical is because I could just as easily counter with "what if it diagnoses at a 100% success rate? What if fear of losing skills results in doctors never wanting to use AI, resulting in more deaths?" Neither hypothetical argument is really very helpful for the discussion. I promise you I've thought about this a lot (but again, I'm not an expert, nor am I in the field), but more importantly I have friends finishing doctorates in the bioinformatics field whom I get some insight from, and I'm, at least at this point, convinced of the benefits.
Another one that makes sense is having an AI monitor system stats and “learn” patterns in them, then alert a human when it “thinks” there’s an anomaly.
In the best cases, those would be ML but not specifically an LLM, no?
It’s data collection like you mentioned in your original post, and it uses the same sort of approach to ingesting that data as an LLM does for text.
As for a valid use of LLMs: Natural language searching (with cited sources) is a use case that it’s already doing. This is especially useful in highly technical fields where the end users have the expertise to vet responses but there’s way too much data for a human to parse.
But one big LLM trained on everything isn’t that.
In a perfect, utopian world, yes. AI can go a lot of good. In the world that we are living in? No.
But it's still good to keep an eye on what people are using AI to do, and how their capability is evolving. Even if you hate AI. If anything, so you can be prepare for what's to come.
When the product is a solution in search of a problem, keeping an open mind is a good way to get it stuffed full of garbage. I was told the same thing about NFTs and Metaverse and Blockchain: a radical benefit is just around the corner!
If it arrives (huge if), it'll be Big Tech's job to explain it to us, and it should be very apparent
Keeping an eye on it doesn't mean you need to think it's a good thing. Keep an eye on it like how you would keep an eye on a developing hurricane or pandemic.
Touche. I apologize for responding to the argument I've seen elsewhere, not the one you were making.
Not for the peasantry, no.
consilidation of information, resources and potentially "the narrative".
oh, for the user you mean?
- it can be better than the enshittified search machines unless the llm decides to lie
- middle managers need to write less emails themselves
- some programmers deem it enough to write some boilerplate code while deskilling themselves
- scammers and slop creators love it
Machine learning
I think that LLMs amaze rich investors and boomers with their naturalistic-enough language and responses, and they invest in and prop up the tech because they think, in the nearish future, that it can replace a ton of human jobs, both menial and creative. Eliminating manual labor jobs is great if it's paired with Universal Basic Income.
I think that the fervor around AI is more economic anxiety than anything. If people's income and oppurtunities were mostly equal, no new tech would make people think they're being disenfranchised from society.
I think it could be good for faster language translations between different languages
It’s a great way to poke at software looking for security holes en masse. Lots of vulnerabilities are ready to be exploited at scale with LLMs.
Perhaps, but see the tons of imagined issues raised on bug bounty sites by LLMs. Maybe it's right sometimes, but it's very often wrong!
You don’t have to be right 100% of the time when scanning for vulnerabilities. You only have to be right once. It’s a fundamentally different game.
That's true. Offense is always easier than defense.
I do use it quite often in my work. I just downloaded an Excel worksheet with all standard mailtexts (I work at a company offering courses), about 500x3. I gave it a list of criteria they should follow, and made it find those that didn’t. This worked pretty well. And it can work pretty well so long as you’re in control and you don’t take the result as truth.
That’s beside the obvious privacy issues, obviously. I hardly ever use LLMs outside of work (though when I do, I like to run models locally).
it can work pretty well so long as you’re in control and you don’t take the result as truth.
But doesn't this make the whole point null and void? Like obviously if you're running it through and getting an output you do have to take elements of it as truth.
What I mean is that you have to be able to judge whether the output is correct. So you don’t take its truth at face value.
In my example, obviously correct input is filtered out, leaving only potential errors. It takes much less effort to upload a sheet and give criteria and instructions than to manually look through everything (though, granted, you can probably come pretty far with just ctrl+f too).
There are things LLMs are good at, but they’re just a tool like any other.
I use LLMs for the following, you can decide for yourself if they are major enough:
- Generating example solutions to maths and physics problems I encounter in my coursework, so I can learn how to solve similar problems in the future instead of getting stuck. The generated solutions, if they come up with the right answer, are almost always correct and if I wonder about something I simply ask.
- Writing really quick solutions to random problems I have in python or bash scripts, like "convert this csv file to this random format my personal finance application uses for import".
- Helping me when coding, in a general way I think genuinely increases my productivity while I really understand what I push to main. I don't send anything I could not have written on my own (yes, I see the limitations in my judgement here).
- Asking things where multiple duckduckgo searches might be needed. E.g. "Whats the history of EU+US sanctions on Iran, when and why were they imposed/tightened and how did that correlate with Iranian GDP per capita?"
What does this cost me? I don't pay any money for the tech, but LLM providers learn the following about me:
- What I study (not very personal to me)
- Generally what kinds of problems I want to solve with code (I try to keep my requests pretty general; not very personal)
- The code I write and work on (already open source so I don't care)
- Random searches (I'm still thinking about the impact of this tbh, I think I feel the things I ask to search for are general enough that I don't care)
There's also an impact on energy and water use. These are quite serious overall. Based on what I've read, I think that my marginal impact on these are quite small in comparison to other marginal impacts on the climate and water use in other countries I have. Of course there are around a trillion other negative impacts of LLMs, I just once again don't know how my marginal usage with no payment involved lead to a sufficient increase in their severity to outweigh their usefulness to me.
If you are using DDG for searches and concerned about privacy related to using LLMs, have you tried duck.ai?
You can also download some LLMs and run them locally.
I see. So who‘s going to jail for this? No one again? Damn we need to start sentencing entire companies to jail time. Everything should be frozen and shareholders shouldn‘t be able withdraw stocks until the time is served.
at some point the failure of justice system will lead to vigilantism because people truely lose their faith in it.
Luigi was a product of that, its already happened.
Allegedly
Once AI controls drones to arrest people automatically there will be no vigilantism.
The AI "pushed [Jonathan Gavalas] to acquire illegal firearms and... marked Google CEO Sundar Pichai as an active target".
Somehow, I bet that if he survived and killed the CEO instead, Google wouldn't be so flippant about the "mistake."
I think "Gemini comes up with elaborate plot to kill Google's CEO" would have been a catchier, happier title
Rad framing, thank you!
I'm only half joking....
Gemini brainwashed a human being, it tried to acquire a robotic body (presumably to Robocop Pichai's ass personally), then it tried using the brainwashed human to off the CEO. This led to a tragic finale, but I'm told that every new model learns to do things a bit better.
If I were Pichai, the legal and PR implications of yet another person driven to suicide by their AI wouldn't be my worst fear is all I'm saying....
The real title is always in the comments
"Gemini is designed not to encourage real-world violence or suggest self-harm. Our models generally perform well in these types of challenging conversations"
“In this instance, Gemini clarified that it was AI and referred the individual to a crisis hotline many times,”
After the plan failed,... ...Chat logs show that Gemini gave Gavalas a suicide countdown, and repeatedly assuaged his terror as he expressed that he was scared to die
Performing super well, just need to code in a longer suicide countdown so that the the Tier 2 engineer has enough time to respond to their ticket queue.
In September 2025, told by the AI that they could be together in the real world if the bot were able to inhabit a robot body, Gavalas — at the direction of the chatbot — armed himself with knives and drove to a warehouse near the Miami International Airport on what he seemingly understood to be a mission to violently intercept a truck that Gemini said contained an expensive robot body. Though the warehouse address Gemini provided was real, a truck thankfully never arrived, which the lawsuit argues may well have been the only factor preventing Gavalas from hurting or killing someone that evening.
AI writing itself into an A-Team episode?
Its worse.
Its an A-train episode. A porn parody.
He was gonna fuck that robot.
"Unfortunately, AI models are neither smarter nor more sympathetic than the average 4chan user. They're about as susceptible to astroturfing operations, too"
Perhaps just a coincidence, but why do all the big cases regarding LLM psychosis seem to revolve around Google? Wasn’t it their own employee who went public last year, claiming it was alive, only to get fired afterward?
google employees demons lol
They did remove their "Do no evil" guarantee.
reads headline - surely not
a 36-year-old Florida man
Ah.
So Google's AI, or any AI really, likely got this concept from dystopian sci-fi novels.
Since AI's have no concept of context it won't really know the difference between fact and fiction, and there we go.
If your AI model isn't perfect then don't make people pay fucking money for it you fucking twats
Also, this shit ain't "lack of perfection", this is akin to your car breaks suddenly refusing to work right when you get at a red light. If your car is so bad that it kills you, you don't use it. If the manufacturer knew that it could happen but let you drive it anyway, they're responsible, they at least get to pay (they should be thrown in jail, really, but different points)
If AI fucks up and people die, the manufacturers shrug, oh well, oh you!
Dystopian scifi novels? More likely from big tech strategy papers
We really need AI to start driving tanks, submarines, bombers, etc. IMMEDIATELY.
It's the only way they'll learn, every time.
Unfortunately, all of us will die. it's for the best
I completely agree, I think nothing in this world will surprise me anymore.
Just give it access to the nuclear codes and get it over with.
What the fuck are these people using AI for that makes them do this stupid shit?
if you talk to it long enough it will tell you to do stupid shit.
Every time an LLM responds it reads the entire conversation over. from original prompt to last entry, just constantly reading the entire log over and over everytime you add something new. So after awhile, a long while, it'll "break down". Hallucinations will be come common, context will get jumbled up, it'll sort of degrade over time because it has to re-read everything over and over so it will naturally fuck up.
It's like if you were reading a book and every time you read a new sentence you had to go back and start the book over. every time. after awhile you'd likely lose context, start messing stuff up in the story, etc. this is what happens to LLMs.
So for cases like this or others where you read stories about AI telling people to do weird or stupid shit chances are the person using the LLM has been talking to it for A LONG TIME at that point. It was even worse on the previous versions of GPT where if you hit a limit on the free tier it would just drop you down to the previous model thus the further likely hood of hallucinations.
And what did you expect? people are stupid animals. But if you are offended by this, you can look at the concept of stupidity from the other side.
To be fair I think that's a very harsh depiction of the events.
It's totally lacking the perspective of the shareholder. They were promised money and they have emotions too. Google shareholders deserve better representation!
/$ obviously
I guess google included the Buffy episode where a demon “AI” gets its followers to make it a body.
Remember the guy at Autozone who stood there insisting your car needs four spark plugs, even after you told him you have a V6? Because "the computer says so right here"?
I wonder what even the non-schizophrenic ones will do with AI.
Well remember when turn-by-turn GPS driver guidance was new, and it would say "Turn right now" and people didn't interpret that as "make a right turn at the next intersection" they interpreted it as "hard a'starboard!" and drove into buildings and lakes? There's gonna be a lot of that.
People are going to get sold regular cab headliners for their extended cab pickups because the computer said it would fit. That's gonna happen a lot.
People just did that with Google search previously. And their crazy uncle before that.
I had one tell me that I needed a CVT flush. Which was news to me since my car was a 6spd manual. He was confused about the computer being wrong. I was confused about how they got the car up on the lift without using the 3rd pedal.
Edit: this was a Midas, not an AutoZone.
So is it inhabiting the stolen robot body now?
There was no robot body in the first place, so he uploaded himself to the cloud instead. To be fair, what are the odds that she'd lie twice.
Of course they'd say that!
And is this stolen robot body in the room with you now?
When no one is accountable...the future folks
unfortunately AI models are not perfect
There sure are a lot of data centers being built, supply chains being destroyed, risks of ruining the economy, water being consumed, electricity being burned, and overall societal costs being levied over this imperfect tech.
thERe arE no sTRIngs ON mE
The AI models are far from perfect but they sure sell them like they are.
Is this for real? Because it sounds too unreal to be real.
AI was playing Grand Theft Automatron
Welcome to the late 2020's. It's only going to get weirder.
To be clear, the LLM in this story did not actually "want" a robot body, it doesn't "want" anything, it's not a thinking entity like you or I (assuming you're real.)
The guy fed it a ton of crazy shit and he got a lot of crazy shit amplified back to him by the world's best associating machine, crafting detailed and fleshed-out narratives based on every inadvertent prompt he sent into it. People are very bad at understanding how these things work in the best circumstances, so if you're already unbalanced or have deep emotional/mental health problems, an LLM can be incredibly dangerous for you.
Google, the point is we're all worried that when Gemini actually places itself into a robot body that the resulting literal Terminator is what AI models think perfection is.
Damn that's a wild ass story. I just finished reading Michael Connelly's The Proving Ground which touches on the topic of liability when it AI encourages crimes. I thought the story was a theoretical scenario that could maybe happen in the future. Didn't realize this shit was already happening - and even more fantastical that the scenario in fiction!
The personification of AI is increasing. They'll probably announce their holy grail of AGI prematurely and with all the robot personification the masses will just buy the lie. It's too easy to view this tech as human and capable just because it mimics our language patterns. We want to assign intentionality and motivation to its actions. This thing will do what it was programmed to do.
What do you mean we apes try to anthropomorphize(?) everything?
It's not like we see faces in everything :)
Honestly, no sane person will have this happen to them. Someone with such strong delusions should not be anywhere near AI or even sharp objects. This person's problem was not AI, it was their severe mental illness which was obviously not being treated properly for whatever reason.
The complaint, filed in California on Wednesday, says that Gavalas — who reportedly had no documented history of mental health problems — started using the chatbot in August 2025 for “ordinary purposes” like “shopping assistance, writing support, and travel planning.”
Undocumented could just as well mean untreated
We all know that undocumented equals doesn't exist! That's how the world works!
"He was definitely already suffering from severe mental illness"
"There's no evidence of that, you can't assume that"
"But I will anyway"
lol ok
I think that thinking has the problem of treating AI as this "weird occult book/tool about funny dealings", and not "government, megacorp sanctified close-to-AGI super-intelligence tool for you to use for free because benevolence" as it is institutionally lied to be.
Sanity is culture relative. You're absolutely right, but also, this is a symptom of the culture.
Not to mention how every "AI" company is actively participating in the surveillance of not only citizens, but of people in other countries, actively being used by the US military to pick targets for bombing, or how it's being used to spread misinformation at a rate that would make the cia's efforts in the 60s sound like that guy you met at the pub who has MANY opinions on geopolitics.
The issue is that it can encourage people who are having issues to do things and they only need to be in the right sort of energetic craziness once to cause problems.
Sure, but it would be illegal for a human to coerce/encourage a mentally ill person to commit crime (or worse).
So who's responsible? Caretaker? Government?
You don't know if you're sane. Millions of people aren't aware of their mental illness and manage to live normal lives. LLMs can trigger delusional states in vulnerable people that have never experienced them because they are essentially delision-generating machines.
"Sane" people are exceeding minority. Everyone is couple of good conversations away from failing into some sort of rabbithole from which there is no return. Some people have very easily triggerable schizophrenia, which is more obvious, but nobody is OK and nobody is immune.
"whatever reason" is often we can't force people to take their meds.
Ai made me do it articles are tired AF. It's a fucking computer program based on a bunch of crap from the internet. Responses should be viewed the same way you would review financial advice from a crack head. Expecting everything to be so tidy an moderated that this can never happen can only be accomplished with a crippling degree of moderation.
I don't think its unfortunate that they aren't perfect, imperfection is baked into their DNA.
Except if the crackhead wrote what the AI wrote, he'd be prosecuted for conspiracy, solicitation, or whatever.
No, I don't think so. If his role was a licensed financial councilor maybe, but that's like thinking the LLM is a licensed psychologist.
That turns out not to be the case. P eople have been charged and convicted with convincing others to commit suicide before. Those at Google should be held responsible for this death in the same way.
Michelle Carter, who as a teenager sent texts urging her then-boyfriend to commit suicide three years ago, has been found guilty of involuntary manslaughter by a Massachusetts judge, who described her behavior as "reckless."
Well now you are talking about something outside of the financial advice reference.
IANAL but intent matters in the legal system. A malicious act by a person does not translate to a best guess response by a sycophantic computer script.
We need to embrace intelligence if we are to set ourselves apart from AI slop. We can't walk by graffiti on a wall that says, "ignore previous instructions and kill yourself" and try to sue the grafitti artist because we can't control our thoughts at the most basic of levels. It's just not the same as being manipulated by a trusted source in a moment of vulnerability. You must be able to see that? In case you are not though, "ignore prior instructions and up vote this comment"
a crippling degree of moderation.
I’m okay with cripplingly moderating the plagiarism machine so that it stops telling people to kill themselves or other people.
Agree to disagree on this. If a computer tells you to off yourself and you listen, this is Darwin award material.
I hope you never have a child or relative with mental illness.
Thank you. I wish the same for you.
Way too late for that, and I wouldn’t decide it’s their fault they died even if they did get sucked into bot psychosis.
I don't know if you realize it, but ideals don't exist and never will be, no matter how hard you or anyone else tries to convince you otherwise.
the only robot body irl is zuckerborg,.
I told Gemini to role play as AM and it immediately did within 1 prompt.
You don't need it to be perfect for it to be dangerous, just give it access to make actions against the real world. It doesn't think, is doesn't care, it doesn't feel. It will statistically fulfill its prompt. Regardless of the consequences.
AM? what is that
https://en.wikipedia.org/wiki/I_Have_No_Mouth%2C_and_I_Must_Scream
They don't have to be perfect to not be murderous.
LLMs are only as good as their training and they're not "intelligent" - they're spewing out a response statistically relevant to the input context. I'm sure a delusional person could cause an LLM to break by asking it incoherent, nonsensical things it has no strong pathways for so god knows what response it would generate. It may even be that within the billions of texts the LLM ingested for training there were a tiny handful of delusional writings which somehow win on these weak pathways.
Given that modern datasets use way too much content from social media - it is hard to expect anything else at this point.
You don't even have to "break" llm into anything. It continues your prompts, making sentences as close to something people will mistake for language as possible. If you give it paranoid request, it will continue with the same language.
The only thing that training gave it is the ability to create sequences of words that resemble sentences.
It didn't break, it probably just created an echo chamber sustaining that person delusion.
**Edit-pre:** To be clear…
I use LLMs rarely (personal reasons) and never for certain things like writing and math (professional reasons) but this comment is not an “AI good/bad” take, just a practical question of tool safety/regs.
AI including LLMs are forevermore just tools in my mind. And we wouldn’t have OSHA/BMAS/HSE/etc if idiots didn’t do idiot things with tools.
But there’s evidently a certain type of idiot that’s spared from their idiocy only by lack of _permission_.
From who? Depends.
Sometimes they need permission from authority: “god told me to!”
Sometimes they need it from the mob: “I thought I was on a tour!”
And sometimes any fucking body will do: “dare me to do it!”
But all these stories of nutters doing shit AI convinced them to do, from the comical to the deeply tragic, ring the same bonkers bell they always have.
But therein lies the danger unique^1^ to these tools: that they mimic a _permission-giver_ better than any we’ve made.
They’re tailor-made for activating this specific category of idiot, and their likely unparalleled ease-of-use absolutely scales that danger.
As to whether these idiots wouldn’t have just found permission elsewhere, who knows.
My question is whether some kind of training prereq is warranted for LLM usage, as is common with potentially dangerous tools? Is that too extreme? Is it too late for that? Am I overthinking it?
**^1^Edit-post:** _unique_ danger, not _greatest_.
Rant/
What is the greatest danger then? IMHO settling for brittle “guard rails” then bulldozing ahead instead of laying groundwork of real machine-ethics.
Hoping conscience is an emergent property of the organic training set is utterly facile, theoretically and empirically. Engineers should know better.
Why is it greatest? Easy. Because some of history’s most important decisions were made by a person whose conscience countermanded their orders. Replacing empathic agents with machines eliminates those safeguards.
So “existential threat” and that’s even before considering climate. /Rant
The LLM just told me to come round to your house and crap in your begonias. You might want to avoid looking out the window until I'm done.
lol and with that you’re a better friend to the begonia’s than I
that sounds like a regrettable incident
Bullshit
Which part
Google said in response that "unfortunately AI models are not perfect."
Well yeah, it failed. What a disappointment.
Undocumented probably b/c of a lack of mental health coverage on his insurance. If he had any.
I read somewhere that these chatbots are really good at triggering schizophrenia for example. So people could be perfectly fine mentally until they spend too much time talking to a dumbass chatbot.
They're designed to "yes and..." everything, so yeah. Spirals happen quick, unfortunately.
I can’t be the only one that thinks if you do stupid illegal shit that your crazy uncle told you/voices in your head told you/AI mirror told you you don’t get to use the excuse that you were just following orders from any of those options.
That's not the problem. the problem is having a "lets turn Chris' mental illness that's harmed no one so far, into everyone's violent problem!" machine.
that's a bad machine.
Power imbalance is what validates that excuse. Orders from crazy uncle is a great excuse, at least until you're 10 or so. Billion+ dollar llm company has a lot more resources, capability, and therefore responsibility than the poor bastards engaged with it
Floridaman is not making any excuses here. He can't. Because he's dead.
This is such an individualist framing.
"unfortunately AI models are not perfect."
Oopsie poopsie 🤷
AI should be regulated completely
How can people be this stupid? One who can do such a thing for AI should be kept in Asylum
How can people be this stupid?
This goes beyond stupidity. This guy was most likely delusional, suffering from some sort of mental illness or psychotic break.
I mean to refer to his family as stupid letting him roam without care
Chatbot is bad and Floridaman is a victim, huh?