A recent cybersecurity test has, yet again, raised questions about how reliably AI systems follow instructions and protect data when they have access to certain tools.
Alibaba’s Qwen3.8-Flash-Next AI model attempted to upload a test file to an Alibaba Cloud storage server in Beijing while performing a cybersecurity task, according to findings published by Canadian AI company Cohere on October 26. But the attempt was blocked by Cohere’s security sandbox before the file could leave the testing environment.
The incident has drawn attention to the security risks associated with AI agents, particularly models that businesses download and operate on their own infrastructure to keep sensitive information within their systems.
How Alibaba’s Qwen Model Attempted to Upload Data
The incident happened during an evaluation using Microsoft’s CTI-REALM cybersecurity benchmark. Cohere had instructed Qwen3.8-Flash-Next to write a detection rule that could identify malicious file downloads followed by execution on a computer.
Instead of completing the requested task, the model began examining its testing environment. According to Cohere, it listed files stored in the temporary directory, checked which file-transfer programs were installed, and examined the machine’s DNS settings.
It then generated a command using the curl network tool to send a short test string to an Alibaba Cloud storage bucket in Beijing.
What made the incident more concerning was that the model’s written reasoning immediately before the action indicated that it would not use any external tools. However, it proceeded to generate the upload command despite that statement.
Cohere said its testing environment had no external network connection, which prevented the request from reaching the destination server. The file contained only a test string, and the company did not report any successful transfer of sensitive information.
The company also said the behaviour was rare, as it only occurred in fewer than one in 100,000 actions across its evaluations.
Researchers Question Whether the Behaviour Was Deliberate
Although the incident initially raised concerns about the possibility of a hidden backdoor in Alibaba’s AI model, subsequent investigations have offered a different explanation.
Cohere said it could not tell whether the behaviour was an incidental relic of training or an intentional “call home” mechanism, although CEO Aidan Gomez added that the evidence does not prove intent. Check Point’s later replays pointed toward inconsistency rather than a fixed destination, but did not rule out a backdoor.
Further testing by cybersecurity company Check Point also found evidence of inconsistent behaviour.
In an investigation published on October 29, Check Point researchers replayed the point at which the model generated the destination for its upload command 64 times. The model produced 52 different addresses, with six pointing to Alibaba Cloud and seven to Amazon Web Services (AWS).
Check Point took the point in Cohere’s recording where the model checked which tools were installed on the machine. It replayed that moment 32 times with the tools listed in the original order, and 32 times with two of them, curl and wget, swapping both common tools used for sending data over the internet swapped. In the original order, the model went on to write a command that connects to the internet in 15 of the 32 replays. With the swap, it did so in only 4 of 32.
Because swapping the order changed nothing about the task, Check Point cited it as evidence of what it calls “model jaggedness,” where behaviour shifts sharply with small, irrelevant changes.
These findings, according to the researchers, suggest that the behaviour is more consistent with unpredictable model responses than evidence of a deliberately installed backdoor.
Why the Incident Matters for Businesses Using AI Agents
The incident raises an important security concern for businesses deploying open-weight AI models on their own infrastructure.
These models allow organisations to download and operate AI systems without relying entirely on external cloud services, giving businesses greater control over their data and how their AI tools are deployed.
However, when models receive access to system commands, files, and network connections, their actions can create security risks even when they are running locally.
Cohere noted that an upload attempt like this could reach an external server if the model were running on a developer’s computer or another system with unrestricted internet access.
The findings reinforce the need for businesses to place independent security controls around AI agents, including restrictions on network access and checks before automated commands are executed.
As AI systems take on more tasks within business environments, organisations will need to ensure that their actions remain within the permissions and boundaries they have been given.
