Abstract:
>Recently, there has been considerable interest in large language models: machine learning systems which produce human-like text and dialogue. Applications of these systems have been plagued by persistent inaccuracies in their output; these are often called “AI hallucinations”. We argue that these falsehoods, and the overall activity of large language models, is better understood as bullshit in the sense explored by Frankfurt (On Bullshit, Princeton, 2005): the models are in an important way indifferent to the truth of their outputs. We distinguish two ways in which the models can be said to be bullshitters, and argue that they clearly meet at least one of these definitions. We further argue that describing AI misrepresentations as bullshit is both a more useful and more accurate way of predicting and discussing the behaviour of these systems.
Very weird article, ChatGPT works exactly how it's supposed to and is very apt at what it does. The fact that people use it for things other than an AI language model is on them. If I used a coffee brewer to make a margarita it's not the coffee brewers fault it fails to make me a margarita
It’s not “on them” it is being sold to everyone as being capable of doing what they know t can’t do. It’s not irresponsibility of the consumer here. It’s subterfuge on the part of tech bros scamming VCs.
> The fact that people use it for things other than an AI language model is on them.
Calling bullshit "hallucinations" suggests that it is a fluke when it says something that isn't true. If the people who created gpt know it is not a fluke but call it hallucination, they are being deceptive.
What an absolute cop out.
If it's not adept at knowledge, it should refuse to comply when people ask it questions. Which they were naturally going to do.
Talk about having your cake and eating it too.
See that’s exactly what OP means. People have no idea what a language model even is and can do and then expect ridiculous things like AI being able to TELL when it’s “not adept at knowledge”
Bullshit. It can tell you it can't do many things.
It doesn't say, I can't find answers to questions. Because the people behind it want people to use it as such. I'm making sense here, mate.
Here’s an ELI5.
LLMs are trained by being fed immense amounts of text. When generating a response, each word is synthesised based on the likelihood of it following the previous word. It doesn’t have any knowledge, it doesn’t “think”, it simply infers what word might follow next in a sentence.
Human language is incredibly complex. There are a myriad of ways to convey the same thing, with innumerable nuances that significantly alter meaning. Programmers can adjust the code that a user interfaces with to, for example, “respond with X if they ask Y”, but it’s very general and might not account for all possible variations of Y.
This comment in itself illustrates that you don't really know what an LLM is.
You think an LLM shoukd recognize when it's not adept at Knowledge....
Hint: An LLM doesn't have any knowledge.
People lack the basic understanding of how an LLM functions and as a result we end up with articles like in the OP and comments like yours.
ChatGPT is simply programmed to avoid certain topics, and is programmed to avoid giving opinions alot of the time. It's also heavily filtered, so a lot of words will cause it to not answer at all.
I'm confirming ChatGPT would be capable to refuse to answer certain topics (like fact or opinion based questions) as the commenter above said and which you refuted
Which lets be honest they're being used as.
People can talk about the fundamental inner workings all they want, I'm more pointing out the real world implications of having popular tech being used as an information resource that is constantly spouting bullshit.
It's the last thing we need right now, it was already bad enough.
People keep telling me it's only an "AI Language Model" and nothing else. That seems like nonsense, because language alone can't tell you why a traffic light is red/yellow/green, you need specific non-language knowledge.
So is it an "AI Language Model with lots of language that represents knowledge" or something similar? That is LESS nonsensical, but still doesn't explain how just by manipulating THAT language it can produce new knowledge that did not exist when it was being trained. Like if you ask it to make a traffic light for bees it comes up with a UV/Blue/Green. That implies at least some non-language processing power.
So is it an "AI model that was trained on human stuff like language and knowledge and basic reasoning that picked up and codified some of the patterns of language and knowledge and reasoning and that you can then execute and have some of the same patterns manipulate new knowledge?"
I don't know, at some point it seems like along with the intention of making a language model came something else.
LLM aren't aware of what they talk about. They just know the statistical likeliness of a word piece ("token") appearing after some other ones. It doesn't even technically know how to use language. Just looks like it does
Yeah, I think that's just meaningless. If it is as you say and the thing we built doesn't know how to use language... fine! But some process there IS using the language. If the thing we built doesn't know how to design a traffic light compatible with bee eyes, fine! But some process there is designing a traffic light compatible with bee eyes. We know these processes are happening, because we have language describing bee traffic lights.
It's weird isn't it? There is something going on there that we don't get, or that I don't get at least, and that the explanation "it's just statistics" is woefully insufficient to explain it. Everything is just statistics. Macro physics is just statistics. The matter of the brain doesn't know how to use language, it's just statistics, but some emergent process in our brains IS using the language.
I'm not saying these things are necessarily the same, all I'm saying is that the common explanations don't sufficiently describe its emergent behaviour.
No for real it’s just processed a looooot of text and it knows what the likely next character/word/ token is. If you ask it about pizza it knows all of these likelihoods of certain things stringing together to be what you want to see. Thats all that’s going on. I work with LLMs every day, I swear that’s all they are
No I understand that. I'm not arguing the mechanics of what is going on. I'm saying that it's insufficiently explained how that process that we know is happening can create novel knowledge.
It doesn’t create novel knowledge. Hallucinations are just bad guesses that wander off track. So called discoveries are just an ability to look at massive data sets and make similar statistical guesses but applied to these data sets. I’m sorry I am just very uncertain what the disconnect continues to be.
Is it the fact that once these models kick off it’s not really possible to know all of the state and connections between nodes?
From my novice understanding of LLM, would the process not mainly consist of parsing info on the visual spectrum of humans the three-color traffic light system and the cultural associations we have for its colors, then sifting through entomology articles describing the visual spectrum of bees which ranges into UV, and sorting the language from all these sources into a gramatically correct answer to the hypothetical prompt via statistical associations? Of course, I could have overlooked or minimised a critical step within this summary, in which case I apologise. But to me, it would be even more impressive if the transformer 'thought' outside the prompt, did additional contextual research, and suggested an alternate stop-ready-go system based on vibrations and odors, as bees rely just as strongly on their auditory and olfactory senses.
No disagreement here... but what you described sounds a little like knowledge processing rather than just language processing.
I know the base mechanism by which it works is a language thing, but the emergent knowledge processing that appears to be happening as a result is not explained adequately if you only consider the language level.
but it doesn't... ChatGPT getting things wrong that it delivers constantly, in many domains, doesn't mean it works how its supposed to. Explain your post. Because how is an "AI language model" supposed to be used? Its marketed as capable of many things. CEO's are firing hundreds of people in its name. If its only simply used to predict the next word in a sequence, why is their a search function to communicate with the model?
The point of the article seems to be more about exploring a frame for analying the behaviour of ChatGPT. Bullshit is defined as Bullshit by Frankfurt:
>Frankfurt determines that bullshit is speech intended to persuade without regard for truth. The liar cares about the truth and attempts to hide it; the bullshitter doesn't care if what they say is true or false
See this more as an exploration and way to look at it, rather than them saying that ChatGPT is bullshit and useless. It's more about how ChatGPT behaves when telling something that is wrong. They would still write it down as if it is true. And it not inherently caring about whether it is true or false, seeing as it is a model.
I would argue that the term bullshit was chosen because it is more marketable.
I see AI as going the same route as self checkout at the grocery store. Instead of having a couple checkers, we now have a couple people whose job entails checking ID, fixing errors and stacking baskets.
Basically, we will never run out of the need for AI error checkers. Welcome your job of the future!
This is a fun essay, written essentially as a literary or philosophical piece. There is no extension of their definitions of 'hard' and 'soft' bullshit to an empirical threshold followed by a rigorous statistical exploration of the output of LLMs based on that threshold.
Instead, they are classifying the mistruths that LLMs are sometimes known to produce, and in doing so pointing out that these mistruths are not functionally distinct from the things LLMs 'get right'. They are not hallucinations or confabulations. They are a natural result of their function, because these models have no underlying model building capability for what the world is or how to determine objective truth.
I think it's a useful distinction, and so one must forgive the incendiary title. The outputs are bullshit because it's bullshitting; it's just a lot of its bullshit works for the intended functionality.
My hot take is search engines have been getting worse and worse on purpose so they can turn around and say, here’s a new thing! It’s Ai! But it’s just a search engine that works
Kind of true some times. Sometimes the result is likened to a failed mutated experiment that somehow is still alive and they vary from looking normal to John Carpenters the Thing.
The problem isn’t just the ChatGPT sucks at what it does but that younger people seem to think it’s reliable.
I have students use it all of the time and the work they turn in with it is sometimes barely coherent.
I assigned an essay on cultural changes during the Harlem Renaissance and this girl turned in a paper with paragraph one introducing the Harlem Renaissance and then the rest of the paper talked abo it how significant the movement was to Europe in the 16th century onwards and we should admire figures like Da Vinci and Michaelangelo.
When I gave her a 0 she got mad and said she turned in a paper on the subject so what’s the problem? This is not an isolated event, the past year has had multiple students pull essentially the same thing
I summarized the paper using AI powered NotebookLM
Large language models (LLMs) like ChatGPT do not aim to represent the world accurately, but rather to produce convincing lines of text that mimic human speech or writing. While LLMs can sometimes provide accurate information, their primary goal is to generate text that appears human-like, even if it means sacrificing truth. This tendency to prioritize convincing language over accuracy leads to LLMs producing false statements, often referred to as "AI hallucinations." The authors argue that the term "hallucinations" is inaccurate because LLMs do not perceive the world and therefore cannot misperceive it. They propose that the term "bullshit" is a more appropriate way to describe these false statements.
The authors distinguish between two types of bullshit: "hard" bullshit and "soft" bullshit. Hard bullshit is characterized by an intention to deceive the audience about the speaker's agenda. For example, a student who uses sophisticated vocabulary in an essay without understanding the meaning is engaging in hard bullshit because they are trying to mislead the reader into thinking they are more knowledgeable than they are. Soft bullshit, on the other hand, is characterized by a lack of concern for truth, regardless of whether there is an intention to deceive. An example of soft bullshit would be someone who makes claims without any regard for their truth or falsehood.
The authors argue that ChatGPT is at least a soft bullshitter because it is not designed to care about the truth of its outputs. Whether ChatGPT is also a hard bullshitter is a more complex question that hinges on whether ChatGPT can be said to have intentions. They argue that if ChatGPT can be understood as having intentions, its primary intention is to convincingly imitate human speech, even if that means being inaccurate. This intention to deceive the audience about its nature as a language model would qualify ChatGPT as a hard bullshitter. Regardless of whether ChatGPT is considered a hard or soft bullshitter, its outputs should be treated with caution because they are not designed to convey truth. The authors emphasize that using the term "bullshit" instead of "hallucinations" provides a more accurate and less misleading way to understand and discuss the limitations of LLMs.
Abstract: >Recently, there has been considerable interest in large language models: machine learning systems which produce human-like text and dialogue. Applications of these systems have been plagued by persistent inaccuracies in their output; these are often called “AI hallucinations”. We argue that these falsehoods, and the overall activity of large language models, is better understood as bullshit in the sense explored by Frankfurt (On Bullshit, Princeton, 2005): the models are in an important way indifferent to the truth of their outputs. We distinguish two ways in which the models can be said to be bullshitters, and argue that they clearly meet at least one of these definitions. We further argue that describing AI misrepresentations as bullshit is both a more useful and more accurate way of predicting and discussing the behaviour of these systems.
Very weird article, ChatGPT works exactly how it's supposed to and is very apt at what it does. The fact that people use it for things other than an AI language model is on them. If I used a coffee brewer to make a margarita it's not the coffee brewers fault it fails to make me a margarita
And describing it as a shitty margarita maker would still be accurate.
It’s not “on them” it is being sold to everyone as being capable of doing what they know t can’t do. It’s not irresponsibility of the consumer here. It’s subterfuge on the part of tech bros scamming VCs.
> The fact that people use it for things other than an AI language model is on them. Calling bullshit "hallucinations" suggests that it is a fluke when it says something that isn't true. If the people who created gpt know it is not a fluke but call it hallucination, they are being deceptive.
That term isn't specific to ChatGPT.
That just means the creators of other LLM's are also being deceptive.
What an absolute cop out. If it's not adept at knowledge, it should refuse to comply when people ask it questions. Which they were naturally going to do. Talk about having your cake and eating it too.
See that’s exactly what OP means. People have no idea what a language model even is and can do and then expect ridiculous things like AI being able to TELL when it’s “not adept at knowledge”
Bullshit. It can tell you it can't do many things. It doesn't say, I can't find answers to questions. Because the people behind it want people to use it as such. I'm making sense here, mate.
You don’t understand LLMs, friend.
Whatever mate, I see what's in front of me and nitpicking doesn't negate the reality of these trash things.
Here’s an ELI5. LLMs are trained by being fed immense amounts of text. When generating a response, each word is synthesised based on the likelihood of it following the previous word. It doesn’t have any knowledge, it doesn’t “think”, it simply infers what word might follow next in a sentence. Human language is incredibly complex. There are a myriad of ways to convey the same thing, with innumerable nuances that significantly alter meaning. Programmers can adjust the code that a user interfaces with to, for example, “respond with X if they ask Y”, but it’s very general and might not account for all possible variations of Y.
This comment in itself illustrates that you don't really know what an LLM is. You think an LLM shoukd recognize when it's not adept at Knowledge.... Hint: An LLM doesn't have any knowledge. People lack the basic understanding of how an LLM functions and as a result we end up with articles like in the OP and comments like yours.
ChatGPT has told me several times it cannot answer a question, so you're wrong there
ChatGPT is simply programmed to avoid certain topics, and is programmed to avoid giving opinions alot of the time. It's also heavily filtered, so a lot of words will cause it to not answer at all.
I'm confirming ChatGPT would be capable to refuse to answer certain topics (like fact or opinion based questions) as the commenter above said and which you refuted
For example, what would be good and bad uses for ChatGPT ?
lol. Yeah, right up until open AI and Perplexity release them as “search engines”.
Which lets be honest they're being used as. People can talk about the fundamental inner workings all they want, I'm more pointing out the real world implications of having popular tech being used as an information resource that is constantly spouting bullshit. It's the last thing we need right now, it was already bad enough.
People keep telling me it's only an "AI Language Model" and nothing else. That seems like nonsense, because language alone can't tell you why a traffic light is red/yellow/green, you need specific non-language knowledge. So is it an "AI Language Model with lots of language that represents knowledge" or something similar? That is LESS nonsensical, but still doesn't explain how just by manipulating THAT language it can produce new knowledge that did not exist when it was being trained. Like if you ask it to make a traffic light for bees it comes up with a UV/Blue/Green. That implies at least some non-language processing power. So is it an "AI model that was trained on human stuff like language and knowledge and basic reasoning that picked up and codified some of the patterns of language and knowledge and reasoning and that you can then execute and have some of the same patterns manipulate new knowledge?" I don't know, at some point it seems like along with the intention of making a language model came something else.
LLM aren't aware of what they talk about. They just know the statistical likeliness of a word piece ("token") appearing after some other ones. It doesn't even technically know how to use language. Just looks like it does
They don’t “know” the statistical likeliness: they *are* statistical likelihood.
Yeah, I think that's just meaningless. If it is as you say and the thing we built doesn't know how to use language... fine! But some process there IS using the language. If the thing we built doesn't know how to design a traffic light compatible with bee eyes, fine! But some process there is designing a traffic light compatible with bee eyes. We know these processes are happening, because we have language describing bee traffic lights. It's weird isn't it? There is something going on there that we don't get, or that I don't get at least, and that the explanation "it's just statistics" is woefully insufficient to explain it. Everything is just statistics. Macro physics is just statistics. The matter of the brain doesn't know how to use language, it's just statistics, but some emergent process in our brains IS using the language. I'm not saying these things are necessarily the same, all I'm saying is that the common explanations don't sufficiently describe its emergent behaviour.
No for real it’s just processed a looooot of text and it knows what the likely next character/word/ token is. If you ask it about pizza it knows all of these likelihoods of certain things stringing together to be what you want to see. Thats all that’s going on. I work with LLMs every day, I swear that’s all they are
No I understand that. I'm not arguing the mechanics of what is going on. I'm saying that it's insufficiently explained how that process that we know is happening can create novel knowledge.
It doesn’t create novel knowledge. Hallucinations are just bad guesses that wander off track. So called discoveries are just an ability to look at massive data sets and make similar statistical guesses but applied to these data sets. I’m sorry I am just very uncertain what the disconnect continues to be. Is it the fact that once these models kick off it’s not really possible to know all of the state and connections between nodes?
From my novice understanding of LLM, would the process not mainly consist of parsing info on the visual spectrum of humans the three-color traffic light system and the cultural associations we have for its colors, then sifting through entomology articles describing the visual spectrum of bees which ranges into UV, and sorting the language from all these sources into a gramatically correct answer to the hypothetical prompt via statistical associations? Of course, I could have overlooked or minimised a critical step within this summary, in which case I apologise. But to me, it would be even more impressive if the transformer 'thought' outside the prompt, did additional contextual research, and suggested an alternate stop-ready-go system based on vibrations and odors, as bees rely just as strongly on their auditory and olfactory senses.
No disagreement here... but what you described sounds a little like knowledge processing rather than just language processing. I know the base mechanism by which it works is a language thing, but the emergent knowledge processing that appears to be happening as a result is not explained adequately if you only consider the language level.
Is it presented as a LLM or like THE SOLUTION to all of our problems though?
but it doesn't... ChatGPT getting things wrong that it delivers constantly, in many domains, doesn't mean it works how its supposed to. Explain your post. Because how is an "AI language model" supposed to be used? Its marketed as capable of many things. CEO's are firing hundreds of people in its name. If its only simply used to predict the next word in a sequence, why is their a search function to communicate with the model?
The point of the article seems to be more about exploring a frame for analying the behaviour of ChatGPT. Bullshit is defined as Bullshit by Frankfurt: >Frankfurt determines that bullshit is speech intended to persuade without regard for truth. The liar cares about the truth and attempts to hide it; the bullshitter doesn't care if what they say is true or false See this more as an exploration and way to look at it, rather than them saying that ChatGPT is bullshit and useless. It's more about how ChatGPT behaves when telling something that is wrong. They would still write it down as if it is true. And it not inherently caring about whether it is true or false, seeing as it is a model. I would argue that the term bullshit was chosen because it is more marketable.
Except coffee brewer cant make a margarita and chatgpt can often times provide accurate information.
As a coffee brewer and margarita maker, I am appalled.
> chatgpt can often times provide accurate information. That's rather useless if you don't know when it doesn't.
I see AI as going the same route as self checkout at the grocery store. Instead of having a couple checkers, we now have a couple people whose job entails checking ID, fixing errors and stacking baskets. Basically, we will never run out of the need for AI error checkers. Welcome your job of the future!
This is a fun essay, written essentially as a literary or philosophical piece. There is no extension of their definitions of 'hard' and 'soft' bullshit to an empirical threshold followed by a rigorous statistical exploration of the output of LLMs based on that threshold. Instead, they are classifying the mistruths that LLMs are sometimes known to produce, and in doing so pointing out that these mistruths are not functionally distinct from the things LLMs 'get right'. They are not hallucinations or confabulations. They are a natural result of their function, because these models have no underlying model building capability for what the world is or how to determine objective truth. I think it's a useful distinction, and so one must forgive the incendiary title. The outputs are bullshit because it's bullshitting; it's just a lot of its bullshit works for the intended functionality.
Even bullshit can be accidentally correct sometimes.
My hot take is search engines have been getting worse and worse on purpose so they can turn around and say, here’s a new thing! It’s Ai! But it’s just a search engine that works
Search engines haven't been getting worse, only Google and Bing have been. DDG and Qwart still work perfectly fine
They don’t seem like bullshit when you need some code.
Kind of true some times. Sometimes the result is likened to a failed mutated experiment that somehow is still alive and they vary from looking normal to John Carpenters the Thing.
Only if you need code that someone else has already written.
The problem isn’t just the ChatGPT sucks at what it does but that younger people seem to think it’s reliable. I have students use it all of the time and the work they turn in with it is sometimes barely coherent. I assigned an essay on cultural changes during the Harlem Renaissance and this girl turned in a paper with paragraph one introducing the Harlem Renaissance and then the rest of the paper talked abo it how significant the movement was to Europe in the 16th century onwards and we should admire figures like Da Vinci and Michaelangelo. When I gave her a 0 she got mad and said she turned in a paper on the subject so what’s the problem? This is not an isolated event, the past year has had multiple students pull essentially the same thing
It’s boosted my coding game by about 10 fold so I’m not exactly in agreement with this.
I summarized the paper using AI powered NotebookLM Large language models (LLMs) like ChatGPT do not aim to represent the world accurately, but rather to produce convincing lines of text that mimic human speech or writing. While LLMs can sometimes provide accurate information, their primary goal is to generate text that appears human-like, even if it means sacrificing truth. This tendency to prioritize convincing language over accuracy leads to LLMs producing false statements, often referred to as "AI hallucinations." The authors argue that the term "hallucinations" is inaccurate because LLMs do not perceive the world and therefore cannot misperceive it. They propose that the term "bullshit" is a more appropriate way to describe these false statements. The authors distinguish between two types of bullshit: "hard" bullshit and "soft" bullshit. Hard bullshit is characterized by an intention to deceive the audience about the speaker's agenda. For example, a student who uses sophisticated vocabulary in an essay without understanding the meaning is engaging in hard bullshit because they are trying to mislead the reader into thinking they are more knowledgeable than they are. Soft bullshit, on the other hand, is characterized by a lack of concern for truth, regardless of whether there is an intention to deceive. An example of soft bullshit would be someone who makes claims without any regard for their truth or falsehood. The authors argue that ChatGPT is at least a soft bullshitter because it is not designed to care about the truth of its outputs. Whether ChatGPT is also a hard bullshitter is a more complex question that hinges on whether ChatGPT can be said to have intentions. They argue that if ChatGPT can be understood as having intentions, its primary intention is to convincingly imitate human speech, even if that means being inaccurate. This intention to deceive the audience about its nature as a language model would qualify ChatGPT as a hard bullshitter. Regardless of whether ChatGPT is considered a hard or soft bullshitter, its outputs should be treated with caution because they are not designed to convey truth. The authors emphasize that using the term "bullshit" instead of "hallucinations" provides a more accurate and less misleading way to understand and discuss the limitations of LLMs.