Technology, and the way in which human beings have come to hone it, will soon pass a number milestones that likes of which would make our ancestors’ brain melt in their skulls. Just think: In the last 50 years alone, (less than a lot of lifetimes), we’ve gone from the 8-track tape with 10 songs on it in our gas guzzlers getting 10 miles per gallon, to ludicrously fast electric cars that can pluck any song ever written from a satellite and instantly play it for you.
It’s incredibly impressive, sure, and our lives are so much nicer because of it.
But there are lines that must remain in the sand lest we wish to find ourselves under the thumb of some great technological beast with sentience far sharper than our own.
This week, a milestone has arrived that should put some of us on our heels.
Computer Science PhD student Giannis Daras noticed that the DALLE-2 system, which creates images based on a text input prompt, would return nonsense words as text under certain circumstances.Trending:
“A known limitation of DALLE-2 is that it struggles with text,” he wrote in a paper published on pre-print server Arxiv. “For example, text prompts such as: ‘An image of the word airplane’ often lead to generated images that depict gibberish text.”
Here’s where it gets wild:
“We discover that this produced text is not random, but rather reveals a hidden vocabulary that the model seems to have developed internally. For example, when fed with this gibberish text, the model frequently produces airplanes.”
A terrifying Twitter thread explained the threat.
A known limitation of DALLE-2 is that it struggles with text. For example, the prompt: "Two farmers talking about vegetables, with subtitles" gives an image that appears to have gibberish text on it.
However, the text is not as random as it initially appears… (2/n) pic.twitter.com/B3e5qVsTKu
— Giannis Daras (@giannis_daras) May 31, 2022
This could actually be quite dangerous.
If Daras is correct, he believes that it would have security implications for the text-to-image generator.Advertisement - story continues below
“The first security issue relates to using these gibberish prompts as backdoor adversarial attacks or ways to circumvent filter,” he wrote in his paper. “Currently, Natural Language Processing systems filter text prompts that violate the policy rules and gibberish prompts may be used to bypass these filters.”
If the A.I. we create doesn’t have guardrails preventing it from creating these things in secret, there is truly no telling what hidden paths these computer citizens could be taking behind our backs.