Hey reader(s),
I’ve always found the spread of misinformation to be an interesting concept to think about, and I suspect that’s for a couple reasons. For one, thinking about misinformation means thinking about things that appear true but aren’t, which means wondering what qualities make something feel true and how mistaking those qualities for indicators affects our ability to actually know things as humans. More specifically though, I think the spread of misinformation intrigues me because the language we use to describe it is also the language we use to describe the spread of diseases and contaminants. This sort of applies to how we talk about the spread of information in general—when an image/video/meme gets really popular on the internet we say it goes “viral”—but I think the framing that’s applied when we say that misinformation is “infectious” or that we need to “contain” it is really interesting. It’s easy enough to see the similarities behind this borrowing of terms, but that particular perspective of knowing there’s this adversarial thing out there that transmits from person to person and you have to figure out how it catches on because, like an epidemiologist, you know that the more it spreads the more it’ll be capable of causing harm to people everywhere it is—I feel like that’s a really engaging perspective to think from.
I’m not sure how I feel about that previously somewhat-fun-and-engaging thought exercise now that developments in generative AI have made it significantly more relevant, but I guess it does give me an excuse to write about it.
I saw this article from 404Media at some point over the past few months covering a research paper which found certain names to appear as recurring parts of large language model generated text output, and it inspired me to start writing this post because of just how much it reminded me of this one piece of science fiction horror writing instead of a real thing with real consequences that we should be introducing into our real world. The specific piece of sci-fi writing in question is SCP-922, one of the many files that make up the online collaborative writing project known as the SCP Wiki. For those unfamiliar, the SCP Wiki is a writing project set in a shared universe where there exist paranormal objects, entities, and phenomena which defy normal science, and a secret "SCP Foundation" which seeks to study and contain these anomalies* so they don't disrupt normal civilization. The anomalies—occasionally referred to within the Foundation as simply "SCPs"—each have a numbered foundation file which outlines the Special Containment Procedures** in place to contain them, gives a description of their characteristics, and often includes a log of experiments or incidents which divulge how those characteristics have been observed. It's usually within this description and appended log information that the ramifications of each anomaly are explored, and the horror of them is revealed.
SCP-922’s file in particular (which you should go read, because it’s very good, includes interesting details which I will be omitting, and isn’t very much longer than the summary I’m including in this post) begins by