Security matters more today than ever

There is one thought that comes back to me every time I read about another security hole that artificial intelligence helped to find: software security matters more today than it ever has, and at the same time many people who are just starting to build their own applications with AI give it surprisingly little attention. Let me try to explain where that conviction comes from.

Security holes found with the help of AI

Within a single week it came out that AI models had helped to find three serious vulnerabilities. The first was in PHP, the language a huge share of websites are written in (including those built on WordPress, Drupal and Joomla). Researchers at Wiz, using models from Google and OpenAI, found bugs in the language's engine that made it possible to take over a server without logging in and to read passwords or access keys from its memory (researchers at Calif and OpenAI reported the same problem independently). The bugs had been in PHP for more than ten years, and because they sat in the language itself, they also affected sites whose owners had written their own code flawlessly. The bugs' identifiers (without technical details) leaked online two days before the planned release of the fix, so PHP's developers released it a day early, and according to Amazon the first attack attempt came four days before the fix was published, even before that leak. Some public institutions in Europe took their websites offline as a precaution.

On the same day, fixes for ten bugs in Node.js came out (Node.js is the environment that a great many web applications run on), and one of the four rated highest in that release, which under certain conditions let someone run their own code on the server, was reported by OpenAI researchers working with Codex Security, an AI tool that hunts for vulnerabilities. Less than a week later, Anthropic described a bug in the Linux kernel, the heart of the system that a huge share of the world's servers run on. An internal Claude research model that is not publicly available not only helped to find it but also built a working attack on top of it, which gave full control over a server with certain settings, over the network and without any password, and in Anthropic's test the attack itself, from the first step to that control, took less than an hour.

These are only the ones people write about

Those three made the news because they affect software that a huge share of the internet runs on. Beyond them there are plenty of smaller bugs that someone finds and quietly patches, without any publicity, and I think there will be more and more such finds, because the tools that search for them are reaching more and more people. I also assume that many small and medium-sized companies have break-ins and data leaks they have no idea about, because nobody there has the time and the knowledge to check.

Large companies have security teams, small ones usually do not

Large companies have people whose only job is security: they follow which fixes come out, they test their own systems before someone from outside does, and they have written procedures for a data leak (whom to notify, how quickly, what to switch off). In a small or medium-sized company it usually looks different, because there is rarely a dedicated role for it, and often not even an hour a week, so all that is left is to react once something has already happened. For years you could more or less live with that, but I have the impression that the damage a single break-in can do is growing, and that far more people can look for holes in security today than before. AI lets people experiment who until recently would not have known how to start, and of course it is also used by those who, with bad intentions, look for holes in every system they can reach.

A working application is not yet a secure application

More and more often I read and hear that developers are becoming unnecessary, because today a junior developer can build an accounting app or a newsletter tool with AI, and a company's CEO can put together a simple customer management system (a CRM) on their own. That is true, AI really does allow it, but the fact that something can be built does not yet mean that someone built it properly. An application that works and an application that is secure are two different things, and until something bad happens they look the same from the outside. This way of building, where you describe to a model what should be made and accept what it returns without looking into the code, is called vibe coding. For a prototype or a private tool it may be perfectly enough, but for a system that holds customer data, invoices or passwords, in my opinion it is not.

An AI agent will secure your application, if you know what to ask for

I have been a developer for more than fifteen years (today I work as a lead developer) and I use AI agents every day, which lets me see up close how much can be done with them. My experience is that with an agent's help you can write secure code, run a proper audit or check whether all your libraries are up to date, and save a great deal of time in the process, but on one condition: you need to understand what you are doing. That means a general idea of how such systems work, how they are built and how security fixes are rolled out, or at least the awareness that you can ask the agent about these things (to ask for something, you first have to know that it exists). People also share ready-made instruction sets for agents online (so-called skills), including ones dedicated to security, but to judge such a skill you also need to understand what it does, whether it suits your industry and whether it takes into account the regulations that apply where you operate.

I have also learned that when an agent reports that it has finished and that this is everything, it does not yet mean the topic has been exhausted. A model can only take a limited amount of information into account at once (the so-called context window), so some things may simply not have fit into it, and the agent decided it was done because it could not see the rest. It can also simply make a mistake or not know something, and if nobody checks its work, the error usually comes to light only when something breaks.

GDPR, NIS2 and the AI Act: the regulations to keep in mind

In the European Union there are quite a few such regulations. The GDPR requires a company to report a personal data breach to the data protection authority (in Poland, the President of UODO) without undue delay and, where feasible, within 72 hours of becoming aware of it (unless the breach is unlikely to pose a risk to the people the data concerns), which assumes that the company notices it at all and knows what to do next. Poland's amended national cybersecurity act, which implements the EU's NIS2 directive, has placed obligations on many medium-sized companies in selected sectors as well, and the AI Act is gradually introducing further requirements for companies that build systems based on artificial intelligence or use them. I am not saying everyone has to know these regulations by heart, but a CEO who builds a system with customer data over a weekend, and a junior developer who was given a task due yesterday, usually do not have the time to dig into them, and then all they can do is trust an agent that may not know about something either.

Why security matters more today than it used to

I am not writing this to discourage anyone from building with AI, my point is something else: the same tools that let one person build an application in a few days today also let someone else look for holes in it far faster than just a few years ago. The model that built the attack on the Linux kernel is not publicly available, but if it managed that in code that has been reviewed for years by a great many experienced developers, then by my simple reckoning an application written over a weekend and never checked by anyone other than its author is a much easier target. So I believe that a system holding other people's data should be thought through from the start with security in mind, built responsibly rather than by feel, and checked by someone who understands what they are looking at.

In short

• Within a single week it came out that AI models had helped to find serious vulnerabilities in PHP, Node.js and the Linux kernel, and these are only the ones the media wrote about.
• Large companies have security teams and procedures, while small and medium-sized ones can usually only react once something has already happened.
• AI lets someone who is not an experienced developer build an application, but the fact that an application works does not yet mean it is secure, and in the European Union the GDPR, cybersecurity rules and the AI Act come on top of that.
• With the help of an AI agent you can write secure code and run a proper audit, but you need to understand what you are doing, and an agent deciding its work is finished does not yet mean the topic has been exhausted.
• In my opinion security matters more today than ever, because looking for holes has become easier, and a system holding other people's data is worth building with security in mind from the start.