So-called AI slop, which means LLM-generated low-quality pictures, movies, and textual content, has taken over the web within the final couple of years, polluting web sites, social media platforms, at the least one newspaper, and even real-world occasions.
The world of cybersecurity just isn’t resistant to this drawback, both. Within the final yr, folks throughout the cybersecurity trade have raised issues about AI slop bug bounty stories, which means stories that declare to have discovered vulnerabilities that don’t truly exist, as a result of they have been created with a big language mannequin that merely made up the vulnerability, after which packaged it right into a professional-looking writeup.
“Persons are receiving stories that sound cheap, they appear technically appropriate. After which you find yourself digging into them, making an attempt to determine, ‘oh no, the place is that this vulnerability?’,” Vlad Ionescu, the co-founder and CTO of RunSybil, a startup that develops AI-powered bug hunters, informed iinfoai.
“It seems it was only a hallucination all alongside. The technical particulars have been simply made up by the LLM,” stated Ionescu.
Ionescu, who used to work at Meta’s crimson crew tasked with hacking the corporate from the within, defined that one of many points is that LLMs are designed to be useful and provides optimistic responses. “In case you ask it for a report, it’s going to offer you a report. After which folks will copy and paste these into the bug bounty platforms and overwhelm the platforms themselves, overwhelm the shoppers, and also you get into this irritating state of affairs,” stated Ionescu.
“That’s the issue individuals are operating into, is we’re getting numerous stuff that appears like gold, nevertheless it’s truly simply crap,” stated Ionescu.
Simply within the final yr, there have been real-world examples of this. Harry Sintonen, a safety researcher, revealed that the open supply safety challenge Curl acquired a pretend report. “The attacker miscalculated badly,” Sintonen wrote in a publish on Mastodon. “Curl can scent AI slop from miles away.”
In response to Sintonen’s publish, Benjamin Piouffle of Open Collective, a tech platform for nonprofits, stated that they’ve the identical drawback: that their inbox is “flooded with AI rubbish.”
One open supply developer, who maintains the CycloneDX challenge on GitHub, pulled their bug bounty down completely earlier this yr after receiving “nearly completely AI slop stories.”
The main bug bounty platforms, which basically work as intermediaries between bug bounty hackers and corporations who’re keen to pay and reward them for locating flaws of their merchandise and software program, are additionally seeing a spike in AI-generated stories, iinfoai has discovered.
Contact Us
Do you might have extra details about how AI is impacting the cybersecurity trade? We’d love to listen to from you. From a non-work machine and community, you’ll be able to contact Lorenzo Franceschi-Bicchierai securely on Sign at +1 917 257 1382, or through Telegram and Keybase @lorenzofb, or e-mail.
Michiel Prins, the co-founder and senior director of product administration at HackerOne, informed iinfoai that the corporate has encountered some AI slop.
“We’ve additionally seen an increase in false positives — vulnerabilities that seem actual however are generated by LLMs and lack real-world affect,” stated Prins. “These low-signal submissions can create noise that undermines the effectivity of safety packages.”
Prins added that stories that include “hallucinated vulnerabilities, obscure technical content material, or different types of low-effort noise are handled as spam.”
Casey Ellis, the founding father of Bugcrowd, stated that there are positively researchers who use AI to search out bugs and write the stories that they then undergo the corporate. Ellis stated they’re seeing an total enhance of 500 submissions per week.
“AI is extensively utilized in most submissions, nevertheless it hasn’t but precipitated a big spike in low-quality ‘slop’ stories,” Ellis informed iinfoai. “This’ll in all probability escalate sooner or later, nevertheless it’s not right here but.”
Ellis stated that the Bugcrowd crew that analyzes submissions critiques the stories manually utilizing established playbooks and workflows, in addition to with machine studying and AI “help.”
To see if different firms, together with those who run their very own bug bounty packages, are additionally receiving a rise in invalid stories or stories containing non-existent vulnerabilities hallucinated by LLMs, iinfoai contacted Google, Meta, Microsoft, and Mozilla.
Damiano DeMonte, a spokesperson for Mozilla, which develops the Firefox browser, stated that the corporate has “not seen a considerable enhance in invalid or low-quality bug stories that may seem like AI-generated,” and the rejection charge of stories — which means what number of stories get flagged as invalid — has remained regular at 5 – 6 stories monthly, or lower than 10% of all month-to-month stories.
Mozilla’s staff who evaluation bug stories for Firefox don’t use AI to filter stories, as it will probably be troublesome to take action with out the chance of rejecting a authentic bug report,” DeMonte stated in an e-mail.
Microsoft and Meta, firms which have each guess closely on AI, declined to remark. Google didn’t reply to a request for remark.
Ionescu predicts that one of many options to the issue of rising AI slop will likely be to maintain investing in AI-powered programs that may at the least carry out a preliminary evaluation and filter submissions for accuracy.
The truth is, on Tuesday, HackerOne launched Hai Triage, a brand new triaging system that mixes people and AI. In keeping with HackerOne, this new system is leveraging “AI safety brokers to chop by noise, flag duplicates, and prioritize actual threats.” Human analysts then step in to validate the bug stories and escalate as wanted.
As hackers more and more use LLMs and corporations depend on AI to triage these stories, it stays to be seen which of the 2 AIs will prevail.
