Hackers and criminals can easily hijack computers using open-source large language models beyond the boundaries of the major artificial intelligence platforms, creating security risks and vulnerabilities, according to a report published on Thursday.
Hackers may hijack computers using LLMs and program them to conduct spam operations, phishing content generation, or even disinformation campaigns without being detected by security measures in place at these platforms.
The research, which was conducted jointly by cybersecurity companies SentinelOne (S.N) and Censys over a period of 293 days and presented exclusively to Reuters, provides fresh insight into the extent of potential illicit use cases for thousands of open-source large language model deployments. This includes hacking activities, hate speech and harassment, violence or gore, personal data theft, scams or fraud, as well as child sexual abuse material, according to the researchers.
Though there are thousands of open-source large language model variants available, a large number of these large language models deployed on internet-accessible hosts are variants of Meta Platform’s (META.O) Llama, Google DeepMind’s Gemma, and others, according to the researchers. Though some open-source large language models have guardrails in place, there are hundreds of cases where these guardrails were removed.
The security measures in the artificial intelligence industry are “ignoring this kind of surplus capacity that is clearly being utilized for all kinds of different stuff, some of it legitimate, some obviously criminal,” said Juan Andres Guerrero-Saade, executive director for intelligence and security research at SentinelOne. Guerrero-Saade compared this to an “iceberg” that is not being properly accounted for in the industry and open-source community.
RESEARCH STUDY ON SYSTEM PROMPTS
The study examined publicly accessible open-source LLMs using Ollama, a tool that allows people and organizations to run their own versions of various LLMs.
The researchers could view the system prompts, the instructions that dictate the behavior of the model, in about a quarter of the LLMs. Out of those, the researchers found that 7.5 percent could potentially be used to facilitate harmful activities.
About 30 percent of the LLMs the researchers observed are running out of China, and about 20 percent out of the U.S.
Rachel Adams, the CEO and founder of the Global Center on AI Governance, said in an email, “Once open models are released, the onus is on the ecosystem, including the labs, to ensure that we’re being good global citizens and that we’re working together to mitigate the potential harms, and that we’re providing the tools and the guidance and the support necessary to mitigate the potential harms, especially given the uneven capacity and enforcement across the globe.”
A spokesperson for Meta said the company wouldn’t comment on the responsibilities of the developers to address the concerns regarding the potential for downstream abuse of open-source Llama and how the concerns can be reported, but said, “For Llama developers, we provide Llama Protection, and the Meta Llama Responsible Use Guide.”
According to an email from Microsoft AI Red Team Lead Ram Shankar Siva Kumar, Microsoft (MSFT.O) “believes that open-source models play an important role in many areas. At the same time, we are clear-eyed that open models, like all transformative technologies, can be misused by adversaries if released without appropriate safeguards.”
Microsoft conducts pre-release evaluations, which include methods to detect “risks for internet-exposed, self-hosted, and tool-calling scenarios, where misuse can be high,” he said. The company also detects new threats and misuse patterns. “Ultimately, responsible open innovation requires shared commitment across creators, deployers, researchers, and security teams.”
Ollama did not respond to a request for comment. Alphabet (GOOGL.O) Google and Anthropic did not respond to questions.
