That would look good as a title on a nice business card. Letterpress, maybe.
32 days ago [-]
theptip 33 days ago [-]
[flagged]
bsimpson 33 days ago [-]
So the euphemism treadmill is spinning the other direction
SideburnsOfDoom 33 days ago [-]
The euphemism treadmill has not been unalived.
ivan_gammel 33 days ago [-]
This doesn‘t sound like AI. LLMs do not use dictionary definitions for translating a paper this way. It does look like some translation engine like Google Translate or worse, maybe with some text obfuscator trying to avoid plagiarism detection.
https://arxiv.org/abs/2107.06751 found that a lot of the outputs were characteristic of an online tool called SpinBot. (The tool no longer seems to work the same way that it did at the time.)
Bombthecat 32 days ago [-]
Maybe deepl gone wild?
wavemode 33 days ago [-]
LLMs are not the only form of AI.
anamexis 33 days ago [-]
Simple substitution of phrases doesn’t sound like any AI.
xgulfie 33 days ago [-]
Autocorrect unleashed doesn't sound like any AI either but that doesn't stop anyone from calling it that
stymaar 33 days ago [-]
Aren't most autocorrect transformer-based?
sebastiennight 33 days ago [-]
Historically, no.
The simplest way to build autocorrect is to have a dictionary, look it up for each word, and suggest the "closest" dictionary word if a typed word is absent from the dictionary.
It has existed way before transformers and on platforms that couldn't dream of running even the smallest transformer.
sublinear 32 days ago [-]
What do you mean by "look it up" if not measure the closest dictionary word by edit distance?
I think you'll find the broader idea of word vectorization to be remarkably similar to what text transformers do.
Yes, I meant, find the closest word by edit distance. I'm unsure where vectorization would factor in here?
sublinear 31 days ago [-]
When I made this comment, I was thinking of edit distance embedding (like everyone else in this reply chain) while you're thinking of BK-trees.
We're just talking past each other. I agree that a tree is the historical solution. I didn't catch that part.
1over137 33 days ago [-]
Google Translate uses an LLM, last I heard.
scotty79 33 days ago [-]
It switched very recently I think if at all. It was way worse at translations than Gemini even at the beginning of this year. Especially between a pair of languages other than English. It was mixing two completely different words that are the same in English.
krige 32 days ago [-]
I've seen it make very LLM mistakes, esp. "hiccups" and palilalia this month.
kevin_thibedeau 33 days ago [-]
Only since 2023.
stymaar 33 days ago [-]
WDYM? It used an encoder-decoder transformer model long before 2023 (and also long before decoder-only transformers à la GPT became the new standard).
Most people seem to believe LLM only describe these decoder-only architectures, but the term predate them by a big margin. BERT (encoder-only) was a large language model, and there were even language models before transformers were even a thing.
pseudohadamard 32 days ago [-]
Either that or it was an AI trained on conversations between parents and teenagers.
gmerc 33 days ago [-]
That sounds like Anthrophics watermark explanation though.
LPisGood 33 days ago [-]
Anthropic’s watermark is just using a known collection of random seeds during output token generation. That’s it. The models always relied on some randomness, here the randomness is fixed and remembered for querying later.
thaumasiotes 33 days ago [-]
> Anthropic’s watermark is just using a known collection of random seeds during output token generation. That’s it.
Wouldn't that mean that checking the watermark...
(1) Required knowing the prompt used to generate the text; and
(2) was just as expensive as generating the text?
hdjdbckcnfn 31 days ago [-]
wouldn't it mean reversing the output text into prompt and seed?
rcxdude 32 days ago [-]
It's gotta be a bit more than that, otherwise it would fail if you trimmed even one token off the start of the whole session, let alone removing the prompt entirely.
(One approach that's been described is creating a bias in certain token pairs or short sequences: by hashing the previous n tokens, potentially with a secret seed, you create a list of essentially random allowed and disallowed next tokens, then bias the token selection to prefer allowed tokens, so long as they are plausible enough. Then you can check any long-enough subset of the output, if it deviates significantly enough from random chance on following this rule, it was almost certainly generated from a system following these rules.)
indrora 32 days ago [-]
When many of the core datasets were being produced, a lot of these companies mangled the text to be "friendlier" to try and avoid certain issues.
The changes they made often resulted in this kind of weird nonsense: Swapping the word "Failure" for "Disappointment" in a vain attempt to keep it from saying "you're a failure", but also just stripping out words such as slurs and insults, resulting in at one point the hilarious situation in which Gemini for quite a while would INSIST that the lyrics for the Dire Straits song "Money for Nothing" contained "That little has his own jet airplane/that little is a millionaire" -- which in certain contexts changes the meaning of the song incredibly.
These hack-and-slash text manglings have resulted in some of these clbuttic sort of writing styles being used during LLM-assisted writing. A more famous version of this was caused by one paper: https://world.edu/a-weird-phrase-is-plaguing-scientific-pape...
kazinator 33 days ago [-]
That reminds me of an anecdote that my high school English teacher used to tell, as a way of cautioning against use of thesauruses: some student substituted "flatus" for "wind" in their composition, writing something like "a gentle flatus was blowing from the bay".
I also notice that "was opposed to 20 mL" and "unprotected to light" are also not phrases a human would likely produce for that sentence.
bombcar 33 days ago [-]
I've heard of microdosing Hitler but this is a whole new level of clbuttic.
pjc50 32 days ago [-]
Titrating the Hitler particles out of solution.
(remarkably, the phrase "particle of Hitler in every bourgeoise" comes from Trotsky rather than being original internet slang)
defrost 32 days ago [-]
IIRC Feynman had a proof of this, atoms of Socrates, Hitler, and Shakespeare passing into every lung on earth.
Topfi 33 days ago [-]
I had sent this an hour ago as a quote with a few others (lactose bigotry, of course the kidney disappointment) to someone via iMessage. The Apple Intelligence summary was unfortunately not screenshotted by them, but I am told it was a doozy...
<quote>
Have you ever heard of the Joined Together States? Or bosom peril? Kidney disappointment? Fake neural organizations? Lactose bigotry? These nonsensical, and sometimes amusing, word sequences are among thousands of “tortured phrases” that sleuths have found littered throughout reputable scientific journals.
They typically result from using paraphrasing tools to evade plagiarism-detection software when stealing someone else’s text. The phrases above are real examples of bungled synonyms for the United States, breast cancer, kidney failure, artificial neural networks, and lactose intolerance, respectively.
</quote>
33 days ago [-]
apexalpha 33 days ago [-]
During the whole BLM thing a few years ago my emplyer wanted "to do something" I guess so they added one of those "inclusive language" SaaS buttons.
This seems more plausible to me. "Bosom peril" is not in any way a synonym for cancer. But if you think 'cancer' as a word will trigger people then softening it with 'peril' makes sense.
mike_hearn 33 days ago [-]
It's not a woke sensitivity thing. They're using spinners, tools designed for spammers years ago (long predating LLMs) that just randomly substitute words from a thesaurus.
It works because so many academic journals are write-only media. The only things that actually "read" the papers are machines that check for plagiarism, compute citation counts and do other things used to decide on academic promotions. So it doesn't matter if the text is nonsensical.
And the tools work for spam, the original use case, because people who want to buy Viagra from spam mails don't care if they have to read weird English, they just want to find the web shops. Works great for bypassing keyword and Bayesian filters though.
Some papers show signs of being generated by context-free grammars or random phrase combination as well.
These papers are often being produced by "paper mills" that bulk generate fake studies and sell them to academics, who use them to get promoted without having to do any real research.
I wrote an article about the problem years ago if you want to learn more:
Your description of the completely apathetic chain of actors, starting with the journal editors, was depressing to read.
You did not explicitly comment on it, but the situation also implies that even researchers don't really care about research? Like no advisers, or employers or colleagues are reading through these papers?
It feels like more people should have noticed that these journals will print garbage and having an article in a journal like this would not be a positive signal anymore. Maybe it just takes a while?
mike_hearn 31 days ago [-]
Yeah, nobody is reading these journals. They all have dozens of "editors" but it's just CV manipulation.
People are noticing, that's why academic credibility is in free fall but it's all tied up with ideology. Academia purged everyone not on the far left a long time ago, so the left find it useful to promote universities as still having the highest rigour.
isityettime 33 days ago [-]
You sure it isn't a euphemism for "heart attack"?
csense 33 days ago [-]
"Bosom peril" sounds like what someone in the 1800s would write in a spicy novel describing a character's experience being unable to breathe while being hugged by a well-endowed lady.
jordemort 33 days ago [-]
I thought it was booby trap
kps 33 days ago [-]
It's all gone tits up.
avalys 33 days ago [-]
Most of these are authored by what appear to be non-native English speakers. So, the most likely explanation is a translation issue.
In engineering literature from Russia from the 1960s, one sometimes finds references to a ‘water goat’ in papers that are otherwise about heavy machinery. It turns out this is a twice-translated rendition of ‘hydraulic ram’.
kmoser 33 days ago [-]
That's on par with this (apocryphal) translation of "The spirit is willing but the flesh is weak" <=> "The vodka was good, but the meat was rotten."
ben_w 33 days ago [-]
> engineering literature from Russia from the 1960s, one sometimes finds references to a ‘water goat’
Oh, that's where it comes from!
My dad told me something close when I was a kid, but according to him it was the early EU doing dictionary translation between English and French and getting "water sheep". I never could find it.
Through an unrelated anecdote, in my head this is now bound to the music of "smoke on the water", with a sheep impression for the guitar (bass?) — "Sheep on the water, ba baa baaah, ba ba ba baaah"
Joker_vD 33 days ago [-]
Hey, it's not that easy to chose between "водяной баран" and "гидравлический таран", you know.
My favourite example of troubled blind translation is still "naked conductor runs under the carriage".
fc417fc802 33 days ago [-]
Nice double entendre but it seems to be a perfectly valid description of electrical wiring in a machine? What was the original intent?
Joker_vD 33 days ago [-]
It is a perfectly valid description of electrical wiring... but you need to have that context to translate it properly. Technical English translation skills vs. literary English translation skills, if you will.
JustARandomGuy 33 days ago [-]
What was "naked conductor runs under the carriage" supposed to be originally? Open driveshaft? Differential?
Joker_vD 33 days ago [-]
It's supposed to mean "a naked wire is stretched under the car". But it very well can be interpreted as, well, as a cloth-less train conductor having to quickly move under the carriage.
lostlogin 33 days ago [-]
It’ll be a reference to a wire surely? Bare conductor?
> It is widely believed that Carolino could not speak English and that an unknown French–English dictionary was used to translate the earlier Portuguese–French phrase book O novo guia da conversação em francês e português, written by José da Fonseca.
It became so popular for comedic reasons in the English-speaking world that it was reprinted many times and reviewed by Mark Twain.
stymaar 33 days ago [-]
All your base are belong to us.
cestith 32 days ago [-]
Someone set us up the bomb.
supriyo-biswas 33 days ago [-]
This is a really confusing one, since neither the "AI generated paper" nor "translation issue" explanations hold.
I found [1], created all the way back in 2021, which seems to first introduce this "kidney disappointment" term, and I would have to assume that perhaps the authors preferred to translate it from their native language.
LLMs did not exist in their current form in 2021, so couldn't have used it to write down a coherent thought, let alone a paper, at the time.
At the same time, I do live in the same country as the authors, and while we don't speak the best English, being a student at a college without some familiarity of the English language to come up with the term "kidney disappointment" is, let's just say, hard to believe for me.
Edit: This other comment[2] made me realize what it might have been: plagiarism avoidance.
Unfortunately, many colleges here do not have the best reputation and often run like degree mills, and the aforementioned comment made me remember a conversation with a friend, who mentioned writing a review paper.
Again, LLMs were not really a thing at the time, so they mentioned how they had to run their paper through a plagiarism detection tool, and then they substituted words with their synonyms. I guess someone really did search for the word "failure" and found "disappointment" as an alternative.
It fits the "naive deplagiarization" explanation perfectly, however. Many of them are pre-LLM, when people had to plagiarize things the good old-fashioned way, and subsequently hide that fact via synonym replacement.
notahacker 33 days ago [-]
Before LLMs, SEO content spammers used something called an "article spinner" which could rewrite lots of versions of the same article (which may or may not be someone else's article) by substituting synonyms, producing just enough variation to not get flagged as duplicate content by search engines. Some of the substitutions produced truly bizarre writing, but those pages were written for Googlebot to count the back links, not for human readers. This looks like what you'd get if you turned random substitutions down to a minimum to avoid plagiarism detection...
adrian_b 33 days ago [-]
LLMs were not available in 2021, but "Google translate" is much older.
Such translations were not unusual.
Around 2017 to 2019 I used a lot "Google translate", because it saved time despite its flaws, but there never was even a single sentence that I could let exactly as generated by "Google translate".
In each sentence I had to change at least a word or two, and many were hilarious. This was for translations between English and German. I assume that for other more distant languages the translations must have been much worse.
FeepingCreature 33 days ago [-]
LLMs were absolutely available in 2021; the architecture is from 2017 and stuff like GPT-2 and BERT had been around for years. Conversational assistants, ie. ChatGPT, started in 2022.
33 days ago [-]
bhouston 33 days ago [-]
Could it just be neural translation? It existed since the mid 2010s.
GenerocUsername 33 days ago [-]
That's my suspicion. I used Grammerly in 2021 and those are the type of substitutions I could see
ssivark 32 days ago [-]
Perhaps the authors might have originally been going for "rejection" (of transplant) which then got translated into "disappointment".
tha_hnrain 33 days ago [-]
Google Translate has existed since way back then. I remember seeing hilarious posts about bizarre translations from Chinese and other languages in the 2010s.
RobotToaster 33 days ago [-]
I wonder if something similar is happening, except people are using AI detection tools and then substituting words.
weird-eye-issue 33 days ago [-]
Never heard of a content spinner?
lq9AJ8yrfs 33 days ago [-]
"Kidney transplantation is a surgery to eliminate a sound, working kidney from a living or cerebrum dead giver and embed it into a patient with non-working kidneys.
Kidney transplantation is performed on patients with persistent kidney disappointment, or end-stage renal illness (ESRD). ESRD happens when a sickness or confusion harms the kidneys with the goal that they are presently not able to do satisfactorily eliminating liquids and squanders from the body or of keeping up with the legitimate level of specific kidney-directed synthetics in the circulatory system. Without long haul dialysis or a kidney relocate, ESRD is deadly."
Wolters Kluwer seems to be a real multinational company?
thaumasiotes 33 days ago [-]
> Wolters Kluwer seems to be a real multinational company?
Funnily enough, your first link reports that the author of the book is "Kluwer, Wolters".
Being a real multinational company doesn't mean they've read the books published under their imprint.
userbinator 32 days ago [-]
I wonder if anyone else felt a somewhat "late 19th/early 20th-century" style in that paragraph; "sickness or confusion" and "liquids and squanders" stood out in particular.
elric 33 days ago [-]
I hope the synopsis is the only thing that got mangled by the thesaurus bot...
It seems like, particularly in scholarly works, we should cite AI contributions the same way we cite a third party, with similar consequences around plagiarism. If you’re copying AI whole hog, not quoting and citing it, and not catching basic failures. Is it really “your” work?
In the spring semester this year my teacher friends said they were getting a lot of student work that was unusually well written compared to the student’s other work, but also contain unnatural typos throughout.
The students were using ChatGPT and thought they would trick the teacher by editing some typos in before submission.
The cheating market is moving fast. There are now tools you can use to rephrase LLM generated output until the Pangram score drops low enough.
Versions of this cat and mouse game have been going on for a long time. Before ChatGPT was a household name there was a market for simple paraphrasing tools that spammers would use to send variations of email text to try to delay pattern matching spam filters.
SoftTalker 33 days ago [-]
Cheating detection technology will be a cat and mouse game, but a good teacher who knows his students will spot cheating almost immediately (proving it might be more difficult).
The only solution is to evaluate students in ways that they can't cheat. Hand-written exams in class, oral exams, etc. but that is quite time-consuming and most teachers at least in K-12 are already overloaded.
logicallee 33 days ago [-]
I accidentally did this myself not long ago. Like everyone else I'm tired of cookie banners so I had ChatGPT make a law against it, it used "act" in its version, I search-and-replaced it with "Law" before getting Claude to ratify it. Claude pointed out the search-and-replace replacement, oops. The conversation links are in my comment here: https://news.ycombinator.com/item?id=48585778 and you can see Claude found these mistakes:
"
At some point an earlier find-and-replace swapped "act" for "Law" and garbled three real words. As written, the text says:
Section 1: "cookie-banner Lawions" — should be actions
Section 3(2)(b): "next ordinary feature update after enLawment" — should be enactment
Section 5(5): "a site-specific Law that the signal cannot satisfy" — should be act (or action)
These aren't stylistic quibbles; "Lawions" landing on the desks of four legal departments would cause real confusion about intent. The fix is a single reverse find-and-replace. With that correction, I'm comfortable signing.
"
It understood what happened correctly.
rafram 33 days ago [-]
I’m having a hard time figuring out if this State of Utopia thing is tongue-in-cheek or serious.
logicallee 33 days ago [-]
it's serious but still a small project, there isn't much on the site.
I personally would like the law I linked to be followed. I sent a copy to the major browser makers and left this suggestion:
where it got 1 kudos from a community manager but no other upvotes or comments.
The announcement of the law here on HN (linked in my gp comment) got 2 upvotes and no comments. So a couple of people liked the idea, but it's not a big hit.
rafram 32 days ago [-]
I have to be honest, to someone who knows nothing about you, this is all a little concerning.
logicallee 32 days ago [-]
ok. I'd like to discuss your concerns with you privately for medical privacy reasons, as it could be related to my mental health (which I guess is what you said was concerning about this).
- Could you please email me at the email address in my profile? (or add an email to yours.)
I have some followup questions about your concerns. Thank you.
32 days ago [-]
33 days ago [-]
api 33 days ago [-]
I’m giving my kidneys a participation trophy.
drcode 33 days ago [-]
yeah lots of kidney shaming in these comments
cubefox 33 days ago [-]
By the way, Google Translate is really good nowadays. It would certainly not make such mistakes anymore. In my experience, for long text it is even outperforming conventional LLMs in terms of faithfulness: Some LLMs tend to produce overly loose translations on long prose. Also, sometimes they just refuse to translate text they deem offensive.
jsphweid 33 days ago [-]
Reminds me of "Complicated Hilbert Space" or "Quantum Door" from Siraj Raval's days of plagiarism.
sourweasel 33 days ago [-]
My favorite so far:
>”…in 2015, 1.2 million people kicked the bucket from kidney dissatisfaction, a development of 32% since…”
nickdothutton 33 days ago [-]
Once had a lengthy battle with a marketing person. They attempted to change "uninterruptible" (as in Uninterruptible Power Supply) into non-interruptible, and then finally into interruptible. Took the best part of a week to get it fixed.
SideburnsOfDoom 33 days ago [-]
An earlier example of something similar was "Vegetative electron microscopy"
As others have pointed out, this phenomenon is pre-AI; but what perplexes me about LLMs is how unnatural their output can be, despite being trained on real human English. A recent example I saw here: https://news.ycombinator.com/item?id=49235242
Many of us are old enough to remember “medireview”, which emerged from different automatic replacement reasons.
Molitor5901 33 days ago [-]
Huh.. Ok I'll play along. Could this be a translation issue? I confess the first thing that leaped to mind was.. could it be AI written? But I didn't want to leap to that but.. it's .. such a strange word that is not expected in .. I guess formal medical training.
amelius 33 days ago [-]
AI changing words for watermarking purposes.
dash2 33 days ago [-]
Isn't it more likely to be changing words to disguise plagiarism?
Update: looking at some of these, I think it is probably just poor machine translation.
Molitor5901 33 days ago [-]
Yeah some kind of translation issue. The papers seem to know what they're talking about, and I don't entirely get the AI gut feeling, only on the key words.
Molitor5901 33 days ago [-]
Good point, but I see lots of articles are using this "kidney disappointment" so maybe but.. holy toledo look at Impacts of Renal Transplantation and its Mental (2023) by Renan Kamath : “organ disillusionment,” “kidney relocate dismissal,” and “excusal of the moved organ.”
dash2 33 days ago [-]
"waste can build to exorbitant level inside the blood, which harms both
kidney and closures in kicking the bucket... "
"Regularly, it's more irritating than it is not kidding..."
(Pal and Gautam 2018)
flexagoon 33 days ago [-]
Its like when you write a school essay and look up every second word in a thesaurus thinking it makes the essay better
Molitor5901 33 days ago [-]
Interesting! I started reading and there are so many sentences like “gauges an individual's gamble of kidney disappointment.”
Changing words for watermarking .. I understand why it may do that .. but .. what an unfortunate choice of where to do it.
TheOtherHobbes 33 days ago [-]
The substitutions are transparent and don't change the meaning. What can possible go incorrectly?
akoboldfrying 33 days ago [-]
What tin possible leave incorrectly?
ReaLNero 33 days ago [-]
It's very clear what's happening: they're passing it through a writing tool that has an "Auto-improve" feature the users click. This would do the naive, pre-LLM, trick of taking "repetitive" vocabulary and varying it using synonyms.
freehorse 33 days ago [-]
I assume it is due to machine translation? I do not see mentions of "failure" at all, which would be expected to exist if it was to increase variety.
I think (heard) some used to translate an article back and forth a few times to get a "new" article, but maybe it could have been some kind of "paraphrasing" software like you describe (but for avoiding plagiarism detection rather than repetitions).
A few of my favorites are "counterfeit consciousness" (artificial intelligence), "discourse acknowledgement" (voice recognition), and "arbitrary get right of passage to memory" (random access memory).
FloatArtifact 33 days ago [-]
I find this kind of free-phasing completely ridiculous. Just ask anyone that's had a kidney transfer that did not work out. I'm pretty sure they considered a failure, not just a disappointment.
croisillon 33 days ago [-]
i kid you not, yesterday i was wondering what's the english word for bathroom scale, so i googled "pèse personne in english", and got answered "person weighs"
heaney-555 33 days ago [-]
Searching the same thing gives me "The French phrase pèse-personne translates to bathroom scales (plural) or bathroom scale (singular) in English."
freehorse 33 days ago [-]
In my google search, it gave me directly a translation box where it was translated as "personal scale" (which I think is also used).
debugnik 33 days ago [-]
The translations from that Google Search widget are painfully literal given how searches are usually phrased. I can't believe they've put that in production.
Similarly, I find its AI tab useless because it has a hard time interpreting search phrasing. If I need to reword it I might as well switch to a chatbot.
dilyevsky 33 days ago [-]
this reminds me how in the late aughts russian government tried to create some national, lib-free wikipedia version so what they did was rip entirety of wikipedia, deleted some articles and then search-replaced "wiki" with their name in the entire corpus. hilarity ensued within articles about vikings (most slavic languages only have single variant for v/w, except polish) and more.
omsta 33 days ago [-]
"renal disappointment" yields even more papers, in all its awkwardness
"Tortured phrases" is the name of the phenomena, papers have been written about it since 2021.
tingletech 33 days ago [-]
Google scholar thinks I'm a robot and won't let me look at the link
33 days ago [-]
33 days ago [-]
paul_milovanov 33 days ago [-]
"I'm not even angry, I'm just
pmpmpmpm 33 days ago [-]
Well…it is a disappointment when your nephrons don’t work.
WhyComboNadir 33 days ago [-]
We were staying at a nice ocean-side resort in Dana Point, California. My wife, our daughter, and I, were chilling by one of the pools.
Suddenly security was announcing "There's been a sanitary inconvenience, everyone must leave the pool!!! We apologize, but there's been a sanitary inconvenience!".
A moment later we saw them scooping up some floating poop that had escape some baby's swim diaper.
cryptoegorophy 33 days ago [-]
In some languages disappointment does mean malfunction or problem. Like disappointment of stomach. Maybe they are using English words in papers instead of English language? Common for people whose English is not their first language.
33 days ago [-]
aorth 33 days ago [-]
So far all the comments discuss whether this is due to LLM slop, plagiarism avoidance, or translation issues. I want to know how these passed the peer review process.
xyst 33 days ago [-]
This is a post-meritocracy world, unfortunately.
Plagiarism has turned the to pushing out blatant slop. People gaming the system. Diluting the pool of real research.
thesdev 33 days ago [-]
> persistent kidney disappointment (CKF)
Guess why the acronym doesn't match. It's called chronic kidney disease (CKD). Yeah the acronym is wrong too.
33 days ago [-]
hazrmard 33 days ago [-]
Turns out the balance of my PhD thesis was coming up with ways to cheer up disappointed drones >:(
Semiapies 32 days ago [-]
"Heart disappointment" also shows up.
kjimlau 29 days ago [-]
[flagged]
ZeroDayDreamer 32 days ago [-]
[flagged]
rizky05 33 days ago [-]
[dead]
exabrial 33 days ago [-]
I’m guessing this has something to do with the EU mandate for AI watermarking content
exabrial 33 days ago [-]
Well for the ding dong that downvoted me, here's a technical explanation of why this is a plausible theory
* current LLMs work by choosing next likely token, with a small randomization built in (emulating how humans sometimes substitute words)
* one implementation of the text marking is to "rig" the dice roll above, but do that on based upon a certain key
* so by looking at word substitutions in text, you can use a key verify it was generated by an llm.
* the algo is designed not to substitute proper nouns and some other asterisks... no idea if it's deterministic or not.
So yes, this is completely possible and even fits what happened.
duskwuff 33 days ago [-]
The phenomenon of tortured phrases in published papers has been going on since at least 2020 - long before LLMs were a significant factor, and certainly before watermarking their output was a consideration.
The general working theory is that these phrases are the result of a "text spinner" tool being applied to a manuscript and blindly replacing random words or phrases using a preset list of synonyms (which are often imprecise and/or inappropriate). https://arxiv.org/abs/2107.06751 found that many of the commonly observed phrases were characteristic of an online text-spinning tool.
“Subsequently, 1 mL of the mass killing of an ethnic group was opposed to 20 mL of the skin sample and unprotected to light for 7 min.”
From: https://bsky.app/profile/forbetterscience.bsky.social/post/3...
The simplest way to build autocorrect is to have a dictionary, look it up for each word, and suggest the "closest" dictionary word if a typed word is absent from the dictionary.
It has existed way before transformers and on platforms that couldn't dream of running even the smallest transformer.
I think you'll find the broader idea of word vectorization to be remarkably similar to what text transformers do.
https://en.wikipedia.org/wiki/Edit_distance
We're just talking past each other. I agree that a tree is the historical solution. I didn't catch that part.
Most people seem to believe LLM only describe these decoder-only architectures, but the term predate them by a big margin. BERT (encoder-only) was a large language model, and there were even language models before transformers were even a thing.
Wouldn't that mean that checking the watermark...
(1) Required knowing the prompt used to generate the text; and
(2) was just as expensive as generating the text?
(One approach that's been described is creating a bias in certain token pairs or short sequences: by hashing the previous n tokens, potentially with a secret seed, you create a list of essentially random allowed and disallowed next tokens, then bias the token selection to prefer allowed tokens, so long as they are plausible enough. Then you can check any long-enough subset of the output, if it deviates significantly enough from random chance on following this rule, it was almost certainly generated from a system following these rules.)
The changes they made often resulted in this kind of weird nonsense: Swapping the word "Failure" for "Disappointment" in a vain attempt to keep it from saying "you're a failure", but also just stripping out words such as slurs and insults, resulting in at one point the hilarious situation in which Gemini for quite a while would INSIST that the lyrics for the Dire Straits song "Money for Nothing" contained "That little has his own jet airplane/that little is a millionaire" -- which in certain contexts changes the meaning of the song incredibly.
These hack-and-slash text manglings have resulted in some of these clbuttic sort of writing styles being used during LLM-assisted writing. A more famous version of this was caused by one paper: https://world.edu/a-weird-phrase-is-plaguing-scientific-pape...
https://en.wikipedia.org/wiki/Euphemism_treadmill
(remarkably, the phrase "particle of Hitler in every bourgeoise" comes from Trotsky rather than being original internet slang)
<quote> Have you ever heard of the Joined Together States? Or bosom peril? Kidney disappointment? Fake neural organizations? Lactose bigotry? These nonsensical, and sometimes amusing, word sequences are among thousands of “tortured phrases” that sleuths have found littered throughout reputable scientific journals.
They typically result from using paraphrasing tools to evade plagiarism-detection software when stealing someone else’s text. The phrases above are real examples of bungled synonyms for the United States, breast cancer, kidney failure, artificial neural networks, and lactose intolerance, respectively. </quote>
This seems more plausible to me. "Bosom peril" is not in any way a synonym for cancer. But if you think 'cancer' as a word will trigger people then softening it with 'peril' makes sense.
It works because so many academic journals are write-only media. The only things that actually "read" the papers are machines that check for plagiarism, compute citation counts and do other things used to decide on academic promotions. So it doesn't matter if the text is nonsensical.
And the tools work for spam, the original use case, because people who want to buy Viagra from spam mails don't care if they have to read weird English, they just want to find the web shops. Works great for bypassing keyword and Bayesian filters though.
Some papers show signs of being generated by context-free grammars or random phrase combination as well.
These papers are often being produced by "paper mills" that bulk generate fake studies and sell them to academics, who use them to get promoted without having to do any real research.
I wrote an article about the problem years ago if you want to learn more:
https://dailysceptic.org/2021/10/03/436-randomly-generated-p...
Your description of the completely apathetic chain of actors, starting with the journal editors, was depressing to read.
You did not explicitly comment on it, but the situation also implies that even researchers don't really care about research? Like no advisers, or employers or colleagues are reading through these papers?
It feels like more people should have noticed that these journals will print garbage and having an article in a journal like this would not be a positive signal anymore. Maybe it just takes a while?
People are noticing, that's why academic credibility is in free fall but it's all tied up with ideology. Academia purged everyone not on the far left a long time ago, so the left find it useful to promote universities as still having the highest rigour.
In engineering literature from Russia from the 1960s, one sometimes finds references to a ‘water goat’ in papers that are otherwise about heavy machinery. It turns out this is a twice-translated rendition of ‘hydraulic ram’.
Oh, that's where it comes from!
My dad told me something close when I was a kid, but according to him it was the early EU doing dictionary translation between English and French and getting "water sheep". I never could find it.
Through an unrelated anecdote, in my head this is now bound to the music of "smoke on the water", with a sheep impression for the guitar (bass?) — "Sheep on the water, ba baa baaah, ba ba ba baaah"
My favourite example of troubled blind translation is still "naked conductor runs under the carriage".
https://en.wikipedia.org/wiki/English_as_She_Is_Spoke
> It is widely believed that Carolino could not speak English and that an unknown French–English dictionary was used to translate the earlier Portuguese–French phrase book O novo guia da conversação em francês e português, written by José da Fonseca.
It became so popular for comedic reasons in the English-speaking world that it was reprinted many times and reviewed by Mark Twain.
I found [1], created all the way back in 2021, which seems to first introduce this "kidney disappointment" term, and I would have to assume that perhaps the authors preferred to translate it from their native language.
LLMs did not exist in their current form in 2021, so couldn't have used it to write down a coherent thought, let alone a paper, at the time.
At the same time, I do live in the same country as the authors, and while we don't speak the best English, being a student at a college without some familiarity of the English language to come up with the term "kidney disappointment" is, let's just say, hard to believe for me.
Edit: This other comment[2] made me realize what it might have been: plagiarism avoidance.
Unfortunately, many colleges here do not have the best reputation and often run like degree mills, and the aforementioned comment made me remember a conversation with a friend, who mentioned writing a review paper.
Again, LLMs were not really a thing at the time, so they mentioned how they had to run their paper through a plagiarism detection tool, and then they substituted words with their synonyms. I guess someone really did search for the word "failure" and found "disappointment" as an alternative.
[1] https://www.researchgate.net/profile/Suprodip-Mandal/publica...
[2] https://news.ycombinator.com/item?id=49319764
Such translations were not unusual.
Around 2017 to 2019 I used a lot "Google translate", because it saved time despite its flaws, but there never was even a single sentence that I could let exactly as generated by "Google translate".
In each sentence I had to change at least a word or two, and many were hilarious. This was for translations between English and German. I assume that for other more distant languages the translations must have been much worse.
https://blackwells.co.uk/bookshop/product/Handbook-of-Kidney...
https://books.google.com/ngrams/graph?content=kidney+disappo...
Wolters Kluwer seems to be a real multinational company?
Funnily enough, your first link reports that the author of the book is "Kluwer, Wolters".
Being a real multinational company doesn't mean they've read the books published under their imprint.
Conterfeit Consciousness is gonna be the name of my new post-liquid analog modular synth IDM project.
I feel like these text spinners could actually be an untapped gold mine for creatively naming things.
And SCIgen was generating bullshit papers that actually managed to be published in 2005: https://en.wikipedia.org/wiki/SCIgen
https://scholar.google.com/scholar?hl=en&as_sdt=0%2C38&q=%22...
In statistics: "tendency score matching" https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22t...
The students were using ChatGPT and thought they would trick the teacher by editing some typos in before submission.
The cheating market is moving fast. There are now tools you can use to rephrase LLM generated output until the Pangram score drops low enough.
Versions of this cat and mouse game have been going on for a long time. Before ChatGPT was a household name there was a market for simple paraphrasing tools that spammers would use to send variations of email text to try to delay pattern matching spam filters.
The only solution is to evaluate students in ways that they can't cheat. Hand-written exams in class, oral exams, etc. but that is quite time-consuming and most teachers at least in K-12 are already overloaded.
I personally would like the law I linked to be followed. I sent a copy to the major browser makers and left this suggestion:
https://connect.mozilla.org/t5/ideas/comply-with-choose-cook...
where it got 1 kudos from a community manager but no other upvotes or comments.
The announcement of the law here on HN (linked in my gp comment) got 2 upvotes and no comments. So a couple of people liked the idea, but it's not a big hit.
- Could you please email me at the email address in my profile? (or add an email to yours.)
I have some followup questions about your concerns. Thank you.
>”…in 2015, 1.2 million people kicked the bucket from kidney dissatisfaction, a development of 32% since…”
https://www.sciencealert.com/a-strange-phrase-keeps-turning-...
https://www.sciencebase.com/science-blog/vegetative-electron...
Update: looking at some of these, I think it is probably just poor machine translation.
"Regularly, it's more irritating than it is not kidding..."
(Pal and Gautam 2018)
Changing words for watermarking .. I understand why it may do that .. but .. what an unfortunate choice of where to do it.
I think (heard) some used to translate an article back and forth a few times to get a "new" article, but maybe it could have been some kind of "paraphrasing" software like you describe (but for avoiding plagiarism detection rather than repetitions).
It's just an old plagiarism method
https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22f...
A few of my favorites are "counterfeit consciousness" (artificial intelligence), "discourse acknowledgement" (voice recognition), and "arbitrary get right of passage to memory" (random access memory).
Similarly, I find its AI tab useless because it has a hard time interpreting search phrasing. If I need to reword it I might as well switch to a chatbot.
https://scholar.google.com/scholar?&q=%22renal+disappointmen...
Suddenly security was announcing "There's been a sanitary inconvenience, everyone must leave the pool!!! We apologize, but there's been a sanitary inconvenience!".
A moment later we saw them scooping up some floating poop that had escape some baby's swim diaper.
Plagiarism has turned the to pushing out blatant slop. People gaming the system. Diluting the pool of real research.
Guess why the acronym doesn't match. It's called chronic kidney disease (CKD). Yeah the acronym is wrong too.
* current LLMs work by choosing next likely token, with a small randomization built in (emulating how humans sometimes substitute words)
* one implementation of the text marking is to "rig" the dice roll above, but do that on based upon a certain key
* so by looking at word substitutions in text, you can use a key verify it was generated by an llm.
* the algo is designed not to substitute proper nouns and some other asterisks... no idea if it's deterministic or not.
So yes, this is completely possible and even fits what happened.
The general working theory is that these phrases are the result of a "text spinner" tool being applied to a manuscript and blindly replacing random words or phrases using a preset list of synonyms (which are often imprecise and/or inappropriate). https://arxiv.org/abs/2107.06751 found that many of the commonly observed phrases were characteristic of an online text-spinning tool.