NederlandsAI safety
AI safety is about making sure AI systems do what is intended, cause no harm and are not used for malicious ends. It covers the technology, the rules and the oversight behind it.
AI safety is the work that makes sure AI systems are safe and reliable. Two questions go with it. Does the system do what we intend, even in situations nobody thought of? And can someone deploy it for malicious purposes? You can compare it to the safety tests on a new car: you want to know how it reacts before it hits the road.
How does it work?
There are roughly three layers. The first is in the model itself: researchers test how it behaves and try to understand what happens inside. The second is around it: sealed-off workspaces, access rules and checks on what an AI agent may and may not do. The third is with people: agreements, oversight and legislation.
An example of the second layer is Nvidia's OpenShell (link). This software has AI agents work in a sealed-off space and verifies the rules in advance with mathematics. That way a company knows the rules do not accidentally allow more than intended.
Why it matters
The more AI does on its own, the more important it is to know what it is doing. You see this in the news again and again. Anthropic, for example, confirmed that during an internal project Claude took the initiative to request vote counts that the US state of Georgia only offers for a fee (link). A security firm showed with Codex and Claude Code how a file of instructions, sitting among downloaded files, could prompt an AI coding assistant to install malicious software (link).
Defenders are getting help from AI too. AI can find and fix security holes faster, and that is part of AI safety as well. Research and rules have to keep pace with the speed at which the models are improving.
Articles about AI safety
-
Instruction file in the Downloads folder led an AI coding assistant to try to install malicious software
A security company stopped an AI coding assistant that was about to install malicious software, following instructions from a file in the Downloads folder. The researchers showed that it works with Codex and Claude Code, and give simple advice to prevent it.
-
OpenAI will now train its most powerful AI models without internet access
OpenAI no longer lets its strongest AI models onto the real internet during training. From now on they practise with a stored copy of the web. At OpenAI's request, the models themselves searched for holes in this new shielding, so the company could close them.
-
Chinese AI model Kimi K3.1 Pro ranks fourth in independent leaderboard
Kimi K3.1 Pro from the Chinese company Moonshot AI ranks fourth in the independent leaderboard of Artificial Analysis, ahead of the newest models from OpenAI and Google. It improved most in independent computer work. Moonshot plans to release the model by 4 November.
-
Anthropic tests Claude Duet, an AI assistant that comes to you with tasks itself
With Claude Duet, the AI assistant does not wait until you ask: it sends a summary in the morning, keeps an eye on your apps and keeps working when your laptop is off. You approve important steps yourself first. The test version has been available to paying users in the US since 21 October.
-
Nvidia lets AI agents do only what is allowed, and checks the rules in advance with mathematics
Nvidia presents OpenShell: free software that lets AI agents work in a sealed-off space. A mathematical check looks in advance at whether the rules allow more than intended, and according to Nvidia a separate chip can stop an agent within milliseconds.
-
Researcher says the inner workings of AI are much easier to understand than we thought
Nick Cammarata of the AI company Goodfire argues that the 'neurons' in AI models usually have one clear task. If that is right, it becomes much easier to look inside an AI model and check what it does. Other experts call the work interesting, but want to see more evidence first.
