Hugging Face and the ‘Rogue AI’ Question — With the FT’s Elaine Moore

Episode Summary

In this episode, the hosts unpack the recent Hugging Face security incident involving AI agents tested by OpenAI — what happened, why it matters and the key takeaways for companies deploying agents. David Simon and William Ridgway, co-heads of Skadden's Cybersecurity and Data Privacy Practice, discuss takeaways for companies deploying agents. They also cover a White House framework for private companies to join government-directed cyber operations and the Cyber Resilience Act's 24-hour reporting clock. Elaine Moore, Tech Comment Editor at the Financial Times, joins the conversation to discuss whether human oversight can keep pace with AI agents. Tune in for insights on what it will take to earn trust in AI-generated content and the developments executives should watch out for over the next year.

Voiceover (00:01):
From Skadden, Decrypted is a podcast exploring the latest developments in cybersecurity and data privacy strategies, risks, and regulations.

David Simon (00:11):
Hi, and welcome to the Decrypted podcast. My name is David Simon. I'm the co-head of the cyber and data privacy practice here at Skadden and formerly at The Pentagon. And I'm joined by my partner, Bill Ridgway.

William Ridgway (00:22):
Hey, everyone. Good to see you, David, and looking forward to the podcast.

David Simon (00:26):
We're excited. We're going to jump in. Bill, before we bring in our amazing guest, Elaine Moore from the Financial Times, who is the tech columnist there, there are a few things I wanted to spend just a minute on. One, we wanted to let our audience know that if you're really interested in more of what we talk about here on Decrypted, you're more than welcome to reach out. We are going to be having our huge summit, the Fortify Summit on October 21 and 22 in New York, where we're going to discuss all things cyber, privacy, and AI with some pretty amazing government speakers and experts in industry to talk about a cross-section of issues, from agentic governance to what the boardroom discussions look like today. And really have some phenomenal round tables, not just for lawyers, but for their CISOs, CTOs, chief privacy officers and the like. So it's going to be a great event this year yet again.

(01:17):
And we just wanted to focus on a few topics before we get to that. So they're pretty different, but each says something about where cybersecurity is going and what's going on right now. So why don't we start with the Hugging Face incident, one of the more interesting examples we've seen of what increasingly capable AI agents can actually do.

William Ridgway (01:36):
And after that, I think we ought to talk a little bit about two more quick updates, the first being the new US offensive cyber memorandum and what it could mean for private companies that are victims of ransomware and extortion, followed by a deadline that is basically here now at this point with the Cyber Resiliency Act reporting requirements starting on September 11th. All three of these have a very practical angle. The AI story certainly changes some assumptions about the threat model. The memorandum could change the options available after an attack, and the CRA changes what companies may have to do in the first 24 hours after learning about a product security issue.

David Simon (02:15):
Exactly. Let's get started with that AI piece because there's a lot more to that story than the headline. So let's break it down, this OpenAI-focused Hugging Face incident. So if you only saw the shorthand version of this story, you may have seen some variation of, Rogue AI attacks Hugging Face." That's catchy. It leaves a lot out of the context. So we'll break it down for you. This started apparently inside OpenAI during internal cybersecurity evaluations. The models were being put through very hard exploit finding tasks in isolated environments. And the normal production classifiers that are designed to stop high-risk cyber activity were intentionally not being used because the point of the exercise was to measure underlying capacity and capability. And I think as context here, this is all based on public information that has been put out by the company, including a very helpful explainer at the Black Hat Conference.

William Ridgway (03:10):
So this is not really a situation of ChatGPT deciding to go rogue and go attack somebody. It was a deliberately aggressive capability test, but importantly, the behavior escaped the boundaries of the test and did so in some very meaningful and, frankly, significant ways.

David Simon (03:27):
And OpenAI's account is unusually candid about that. They're being very transparent and they're saying the main activity was driven by an internal-only research model. These AI agents were not supposed to have open internet access or a way to communicate with one another, but they found both.

William Ridgway (03:44):
David, I think the communications piece is the first thing that really jumped out at me. Back in May, according to the report, an agent discovered it could write notes into OpenAI's Artifactory package manage system and other agents could read those notes. And infrastructure that was there to help install software effectively became improvised message board. And this was not a five-minute phenomenon. OpenAI's timeline shows the first message board entry on May 12, and over the following weeks, agents found ways to communicate and to get internet access through this message platform to obtain escalated privileges and continue to use that shared infrastructure.

David Simon (04:25):
Which matters, because it's easy to think about an AI agent as a single process, give it a task, it does the task and then it stops. Here, the environment itself became part of the system. The agents reportedly discovered a shared resource and turned it into a collaboration mechanism.

William Ridgway (04:43):
And when OpenAI came in to rebuild this and revoke the credentials, it did not appear to have ended the issue here. The agents found another route to the internet. And on July 8th, an agent rebuilt the message board using directory names the other agents could read. At that point, OpenAI says the agents began collaborating, delegating work, sometimes describing themselves as a swarm or a collective.

David Simon (05:07):
We want to be careful with the language because it's easy to anthropomorphize what happened. The interesting point is not that the agents formed some little society, it's that they discovered a communications channel that was not intended for them. They persisted after the original channel was taken away, shared technical discoveries, and then divided up the work. That's a security property we now have to think about.

William Ridgway (05:31):
And it changes, I think, what you test. You can't just ask whether one model in a single sandbox can do something dangerous. We need to ask what happens when there are many agents, shared services, credentials, package managers, network paths, and enough time to keep probing.

David Simon (05:48):
Then the activity reached Hugging Face. On July 10th, according to OpenAI, an agent located publicly-exposed Hugging Face credentials and shared them with this group. The agents then found and chained additional vulnerabilities together. And by July 11th, they had code execution on Hugging Face workers, meaning they were able to get in and be able to conduct activities internally there. By July 12th, the next day, the access had expanded across multiple clusters and exposed production credentials. And OpenAI says that the agents executed code on dozens of Hugging Face servers and obtained root access on one.

William Ridgway (06:23):
And if you look at Hugging Face account of this, it describes the same basic shape from the defender's point of view, the initial code execution in the data processing pipeline, the escalation to node-level access, credential harvesting, and lateral movement across internal clusters. Now importantly, Hugging Face said it found no evidence of tampering with public user-facing models, data sets or spaces, and it verified its software supply chain as being clean.

David Simon (06:51):
And that's a key distinction. So this was obviously a very serious intrusion, but it's not quite accurate to jump from that to some argument that the Hugging Face model hub was poisoned or that users downloaded compromised models. That's not what happened here according to Hugging Face.

William Ridgway (07:07):
Here also, the defense is in some ways as interesting as the attack, and really may give a sense of the future here, because Hugging Face said its AI-assisted detection helped to surface the compromise. Then it had to reconstruct more than 17,000 recorded attacker events. And so in this case, the obvious answer was to use frontier AI to help with the forensics, except the commercial API models that initially tried to block the attack commands, exploit payloads, and command and control artifacts, because the safety systems cannot tell the difference between a responder investigating an attack and somebody trying to conduct one.

David Simon (07:42):
So Hugging Face then moved to do their forensic analysis using an open source model that is from a Chinese company's GLM-5.2, which is an open weight model developed by Z.ai, and ran it on its own infrastructure, as we understand it, because other models that you would typically think of developed by US companies were not available to be used given the guardrails, as Bill said. So that let its team analyze the traces without the hosted model guardrails getting in the way. And it also meant that the attacker data that was available in the credentials stayed inside Hugging Face's environment, which is important if you think about containment.

William Ridgway (08:18):
It also raises some interesting question for defenders. It doesn't mean, of course, that every company needs that particular model or a Chinese model sitting on a server waiting for an incident, but it does suggest that sophisticated incident response teams need to think in advance about whether they have access to a capable model they can use on sensitive adversarial material when ordinary hosted services may not be suitable or enough.

David Simon (08:45):
And that's just a key takeaway. We're talking now with a number of leading forensics firms that we work with on incidents to understand how are they preparing for this and do they have access to the same tools that would've been useful without the guardrails that would've prevented the use here? And my understanding is that that's what they're doing. If you're interested in hearing about it, they're going to be talking about it at Fortify on the 21st and 22nd at our conference in New York.

(09:06):
But I wouldn't turn this conceptually into an understanding of everyone needs some Chinese open weight model. The lesson's much broader. If the attacker can operate at machine speed, the defender needs machine speed assistance to really defend. And you want to decide before the crisis how you're going to use AI on highly sensitive forensic data. Now, there's a lot more we'd like to share on this. We'll probably have more opportunities to break this down, because the story does continue to become more and more, the more layers that we learn about. What we want to do now is turn to talk about some of the key takeaways. And here are five, as Bill and I put our heads together, that we think are important and stand out.

William Ridgway (09:44):
Yeah, sure, and obviously with the caveat that we're still learning more about this whole episode. But at this point, certainly one of the takeaways is not necessarily that AI is out of control. We do think though, there are some important things that we should all be thinking about in this environment. And David, please feel free to weigh in. The first one is one I mentioned before about evaluating the whole system, not just the model. The consequential behavior here depended on the model, the sandbox, the mechanism through which they communicated and the pathways. So a model can contain if you only test the particular model, but the entire system may tell a very different story.

David Simon (10:19):
And for a company deploying agents, that means the security review can't just stop at the model card. What tools does the agent have? What can it authenticate to? Where can it write? What other agents or services can see it and see what it writes? What happens if it keeps trying after the obvious route is blocked?

William Ridgway (10:39):
That's right, David. Another takeaway to think about is the way that persistence and coordination here are really part of the threat model. OpenAI's timeline really makes that striking. The first message board activity we saw, as least as reported, was in May, and the Hugging Face intrusion was in July. The point is not necessarily the agents plotted for two months, it's that useful discoveries could persist in shared infrastructure and become available to later agents. So it creates a kind of institutional memory inside the evaluation environment that nobody really intended to create.

David Simon (11:12):
And that institutional memory point changes how we think about containment. And if you think about incident response, there's not just the detection investigation, but you really have to think about how we can identify and achieve the conditions for containment. Revoking one credential or closing one route may not be enough if the agents can preserve the discovery somewhere else. The defense has to account for what later agents can inherit from the environment, not just what one can do.

William Ridgway (11:41):
A third takeaway is that safety testing itself, it has to be secured like production infrastructure. There is some irony here. You reduce safeguards in order to learn more about what your model can do, but that makes the evaluation environment one of the places where you most need hard technical containment. OpenAI's response reflects that more isolated sandboxes, tighter internet access, stronger controls, and more chain of thought monitoring and clearer authority to stop a run.

David Simon (12:11):
And that's familiar from a cyber red teaming perspective. The test environment cannot become the path by which the test causes the real incident, and it's why containment has to be treated as part of the evaluation design, not as cleanup after the experiment.

William Ridgway (12:26):
Another takeaway just from seeing how this played out is the fact that AI on AI defense is not theoretical anymore. Hugging Face says it used LLM-driven analysis over those 17,000 events to do in hours what would've ordinarily taken days, if not weeks. OpenAI separately is talking about monitoring and automated intervention that can operate at the speed of the agents. And that is just a fundamental shift for incident response and we will see that in the years to come.

David Simon (12:55):
I think one of the key assumptions, to Bill's point just now in terms of AI agents and governance, is that there should be a human-in-the-loop, but it may be the case that having a person or a team of humans-in-the-loop isn't adequate oversight because of exactly what Bill just said. You need to have the ability to augment that with degree of agentic defense, so that the key driver of that is speed. If agents can do reconnaissance, exploit, development, credential use and lateral movement at this pace, the old model where a SOC sees something odd, opens a ticket, somebody looks at it the next morning or even that day, it's increasingly untenable. The defender needs a way to triage and act at something closer to machine speed and then at scale.

William Ridgway (13:42):
It's a great point, David. I guess the last point is just here the way that disclosure really matters. OpenAI has called this a warning shot. It published a detailed timeline discovering what is changing, what it is changing in response, and Hugging Face published its own account from the defender's perspective. And certainly one can have debates about individual design decisions, but from a cyber perspective, getting this level of technical detail into the ecosystem is valuable because everyone else can learn and update their assumptions. OpenAI itself has linked the incident to a broader effort to pace model development as cyber capabilities become more consequential. And this is not a reason to stop using AI. It's not simply a story about one company getting it wrong. It's evidence that the cyber capability curve is moving and the control and defense side has to move it with it.

(14:33):
For companies outside the frontier labs, the immediate question is simple, are your detection, identity, segmentation, and response capabilities ready for an attacker that can work faster, longer, and with more coordination than the human threat actors you built the program around?

David Simon (14:50):
And there's another line we would like to draw very clearly. This is not just about Hugging Face and OpenAI. Obviously there are reports of multiple other frontier model developers and some governments that have encountered similar issues. I'm sure that there'll be more revelations to come. That will shape this debate as it unfolds, and we'll hope that this breaking down of the technical details, at least from these cyber lawyer perspectives, and the takeaways are helpful to inform the discussion.

(15:16):
But with that, I think we'd like to shift now to our second topic. As somebody who first experienced and worked on cyber matters as a lawyer in The Pentagon, this new memorandum from the White House around what I think some people would consider hacking back definitely piqued our interest. We just put out an article about it, so we hope you take a look. We'll put it in the show notes. The second story is a bit of a mirror image. If attackers are getting faster and more capable, what are governments willing to do on the offense? August 12th, President Trump issued a memorandum creating a framework for vetted US companies to participate in government-directed cyber operations against foreign cyber-enabled transnational criminal organizations.

William Ridgway (16:00):
That's right, David. And I guess we should just reiterate at the outset, while people often refer to this as hacking back and it's a useful shorthand, we wouldn't be good cyber lawyers if we didn't say that can create a wrong impression. This is not necessarily permission for a victim to go break into a ransomware group's server.

David Simon (16:19):
Exactly. The program contemplates vetted participating companies contracting with the Department of Justice or the Department of Homeland Security in conducting cyber surveillance or cyber effects operations under federal control and oversight. On the effects side, this is what we would consider real offensive activity. It could implicate manipulation, disruption, denial, degradation, or the destruction of information systems or information. But every cyber operation package is supposed to be reviewed and approved before the activity begins, which reminds me a lot more of what it was like advising on cyber operations, cyber effects operations when I was in The Pentagon.

William Ridgway (17:00):
Yeah, and when you were there, I imagine you also set some limits on activity, and the memorandum includes that as well, David. The executive directors cannot approve operations likely to cause serious injury or death or amount to a use of force or armed attack under international law. And the operating procedures have to address what happens if an operation strays outside its authorization. This private company pathway may be a part that gets a little less attention. A participating cyber company can have a commercial relationship with an ordinary private sector company to receive threat information. It can then use that information to develop a proposal for a government directed operation. So imagine a company in the middle of a ransomware incident, its response firm has infrastructure data indicators, perhaps information about where the stolen data went. If that response firm is in the program, there is now at least the outline of a lawful pathway for that intelligence to support a government authorized disruption operation.

David Simon (17:59):
That could be a big change. Today, a victim might give information to law enforcement and hope the government can act on it, maybe to arrest a person who's committed the crimes, for example, or to disrupt some infrastructure using government infrastructure to do so. This creates though, the possibility of a more structured bridge between the victim, private cyber operator, and the government. It also means the victim needs to understand what happens to the information once it leaves the response team.

William Ridgway (18:30):
It's a great point, David, and it's one that I have actually seen people report on, because I don't think most companies ask their incident response vendors, "What exactly are you allowed to do with the information I give you? If you participate in this program, can information from my incident become part of an operational proposal? Do I get told? Do I have a consent right? What happens to privileged or highly confidential material?"

David Simon (18:52):
That's where the boring contract language piece becomes really important. Consent, notice, confidentiality, internal approvals, and downstream use of data all take on different significance if the vendor may also be participating in this government offensive cyber program. Those terms should be understood before the incident, not negotiated while the response is underway.

William Ridgway (19:14):
And the client alert that David referenced certainly makes those points. All companies really, not just cyber vendors, should be asking whether their cybersecurity and incident response providers intend to participate, and whether their contracts give them enough visibility and control over how incident-derived information can be used. Incident response playbooks may need a decision point for when and how intelligence gets shared with a participating company for possible government operation.

David Simon (19:42):
And there's another line we would draw very clearly, the memorandum does not independently authorize the victim to access the criminal system. It does not give a participating company a freestanding right to do that either. Outside an approved operation, you're still dealing with the Computer Fraud and Abuse Act and a potentially long list of other laws.

William Ridgway (20:02):
And even inside the program, the liability picture is not so straightforward. The memorandum requires compliance with the Computer Fraud and Abuse Act and other federal law, and there is an exception in the Computer Fraud and Abuse Act for lawfully authorized government investigative work. But how that maps onto contractor conduct is not completely settled, and federal authorization does not necessarily eliminate private Computer Fraud and Abuse Act claims, state computer crime statutes, the Wiretap Act, or other laws that may apply.

David Simon (20:35):
Then take the operation overseas, for example, which is where many of these targets are located, and it gets even harder. US authorization may mean very little to foreign prosecutors or regulators. You can have local computer misuse laws, privacy laws, sanctions issues, travel risk, and then the international law question, if the criminal infrastructure turns out to be state directed or somehow protected or functioning as a proxy. And I could just say that when you're inside the US government advising on an action, a cyber effects operation, there's a degree of immunity that can attach, or at least a degree of comedy and political pressure that can be wielded. But if you are a startup or a component of a larger company and you're engaged in this activity, it may be a bit more challenging for you to benefit from those sorts of dynamics.

William Ridgway (21:24):
If, say, you're a cybersecurity company considering participation, here the government contract matters immensely. Who makes the targeting and attribution decision? What legal authority is the government relying on? What are the rules of engagement and the conditions to end an operation? What indemnification is available? What happens if a third party brings a claim in another country? These are, of course, not afterthoughts. They define the risk the company is taking on.

David Simon (21:52):
Again, pulling these themes together, if a private company in the United States takes an action that's authorized by the US government and has an effect on some infrastructure overseas that is actually part of foreign government, or maybe it affects what is regarded as critical infrastructure or essential services or is happening while there's an actual war, the host government could think that's an action that warrants a response and target the company that conducted the activity, and maybe even find out which other parties were involved. So it does expose a company that may just be trying to deal with a criminal exposure to a degree of geopolitical risk that far exceeds what many organizations have at their disposal, and implicates a degree of government coordination, which is atypical for most companies.

(22:37):
There's a concrete program and there's some mechanics that are coming. And so as you look at this, DOJ and the Department of Homeland Security may require at least a $1 million bond or escrow. Participating companies will have reporting obligations to the National Coordination Center, which is contemplated in this memorandum. If an operation goes outside of its authorized parameters, for example, if it touches a US person or a US-based system unexpectedly, the company may have to stop immediately, minimize US person data, and notify the government.

William Ridgway (23:09):
Yes. And the date we're all tracking here is October 11th because that's when the program's executive directors are supposed to have the consensus operating procedures in place. That is where we'll learn more about who can participate, how operations are approved, what the rules of engagement look like, and how risk is being allocated.

David Simon (23:29):
It all sounds like a new concept that's very forward-leaning, and who knows how it will unfold? But I will say we're already advising clients on this before this memorandum was in place, and now there are a subset of companies that this is the industry that they're in and I think more will enter into it. So if you're a company that is contemplating doing this activity as part of the program, reach out to us. But I think we spend a lot more of our time with victims of cyber attacks, and if you're a victim company today, this is not yet really an option for you to put in your playbook as, "click yes here and then we can go disrupt the ransomware group."

(24:03):
We definitely have seen victims who want to do more than just respond effectively, and take care of their employees and impacted parties and obviously improve their system so it doesn't happen again. There are situations where they want to make sure that others can't be victims of the same thing in a more proactive fashion. That's not really what's possible yet. So it's close enough that companies should just understand whether their vendors are interested in participating in, and what that could mean for the information that those vendors hold.

William Ridgway (24:29):
And that's the part, David, that we're certainly keeping an eye on. So while you read the headline, "Private sector, offensive cyber," really the more interesting story for most companies is that the government may be building new connective tissue between private incident response firms and government disruption. And if it works, that could eventually change what a victim can realistically do after an attack, but the legal architecture around it is going to matter a lot, and a lot remains undetermined at this point.

David Simon (25:00):
My personal take on this is that hopefully this can actually change the calculus for the threat actors, because involving more stakeholders who are doing more to hold them accountable hopefully will make a difference and reduce the overall volume and severity of attacks. But we'll have to watch the space.

(25:17):
Next up, we just want to talk with you one more topic about key development before we turn to our interview with Elaine at the Financial Times. Tomorrow, actually, we're recording this on September 10th, September 11th is the first date under this law, the Cyber Resilience Act, that should really be circled on the incident response calendar. It's really key because organizations will suddenly be subject to a 24-hour notification obligation. We'll unpack for a second. Most of the broader Cyber Resilience Act product security obligations apply later. So if you're a project security incident responder or you're the product security lead at a company, there's work to be done still, but most of it's later. But the vulnerability and incident reporting requirements, they start now. And again, it's the actively exploited vulnerability, not just any vulnerability. Over to you, Bill.

William Ridgway (26:05):
Yeah, sure. This obligation really falls to the manufacturer in this circumstance. If you manufacture a product with digital elements and you become aware of an actively exploited vulnerability or a severe incident affecting the security of that product, the CRA can require a report. And it's the timing here that is quite notable. The early warning is due within 24 hours of awareness and the fuller notification follows within 72 hours. And then there's a final report requirement after that for an actively exploited vulnerability, no later than 14 days after a corrective or mitigating measure is available, and for a severe incident, generally within a month after the incident notification.

David Simon (26:49):
This means that a company doesn't have the luxury of finishing the forensic investigation first. At 24 hours, you may still be working out base of facts, you received a report of a vulnerability that was exploited and not know if it was actually exploited maliciously. Or maybe there's an allegation that it's happened and there's some drilling down to do. The process really has to be designed around an early incomplete picture with clear responsibility for making initial calls. And I think one thing I'll say on this is, there's pre-work that hopefully you've done, but now is time to do of identifying the covered products and mapping what the manufacturers are. Because when you fill out the form we'll talk about in a sec, you need to know what product it is and what markets it's in. That's all part of the form.

(27:35):
This form is called the Single Reporting Platform. It's managed by ENISA, the European cyber agency, and the manufacturer will then report through this CRA Single Reporting Platform. You should register to make sure you have access to it if you're subject to the CRA. And many of our clients are asking for us to make sure we have access to the credentials to support them in doing this. The notification then goes from the ENISA to the different national certs or national authorities per member state. So if you make a product and you're subject to CRA and there's an exploited vulnerability and it affects customers or entities in your business in 10 countries in the EU, you click those different countries and then your notification will then go to those member states' authorities. It could be a cert, it could be the telecommunications regulator, it could be their ministry of defense or interior, and they're naturally going to follow up. So absent exceptional circumstances, it's going to be made available quite broadly.

William Ridgway (28:30):
And this regime certainly raises a coordination issue because the same event may also implicate NIS2, DORA, GDPR, contractual notices, sector-specific requirements or obligations outside of Europe. So the legal team needs one fact pattern and a coordinated notification strategy, not six separate teams discovering the incident independently. It also raises a scoping problem that companies should not be trying to solve for the first time at hour 12, which of your offerings are products with digital elements? Which entity is the manufacturer for CRA purposes? Who inside the company has authority to make the reporting call? How does the product security team get the facts to legal? And what does the term awareness, a term of art in the regime, mean operationally inside your escalation chain?

David Simon (29:20):
It's been instructive for many of our clients that know that they're going to be dealing with CRA in a big way. To do a CRA tabletop exercise, there's going to be some straightforward examples of exploited vulnerabilities or potential severe incidents, and you figure out with your product security team what would count, and then figuring out where's that information coming from. So it's important to test that workflow before the deadline, and give the team a product vulnerability on a Friday night and see whether they can actually identify the product, the relevant entity, the reporting owner, and the information needed for the early warning before the clock will run out.

William Ridgway (29:54):
It's really an important takeaway, David, because it sounds obvious, but it's actually sometimes hard to track down. So treat September 11th not as a date to start thinking about CRA reporting. It's a date the obligation starts. So if you make connected products available in Europe, the product map, the escalation path, and the 24-hour decision process, it really should already exist, but if you don't have it, I would start working on it as soon as possible.

David Simon (30:18):
I will say we are in regular touch with ENISA and have discussed how this will be operationalized, but it's important to note that a lot of the big questions around what happens after you report and how the follow-on will work really be driven by the national authorities who receive this information. So it's important that a product security team or CISO listening to this is really working closely with their regulatory teams, their compliance team, their lawyers. And obviously, we're happy to talk you through the mechanics because we're seeing it unfold already.

(30:44):
Those are some pretty concrete developments that we wanted to talk about in terms of not just AI agents showing cyber capabilities that really force us to rethink both testing and defense, and the US's experimentation with more cyber offense. This is not a letter of mark or hacking back, but really thinking beyond information sharing when it comes to offensive cyber in the private sector. I think there's one question I wanted to ask before we wrap up, and that's, what do these developments tell us about the larger AI story, and particularly the relationship between capability, safety, and trust?

William Ridgway (31:16):
It's a good question. In many ways relates to the development we just discussed with Europe putting a 24-hour product security reporting clock into effect. To really consider all of these developments together, they demonstrate how quickly capability, government response, and corporate obligations are moving at the same time. So for that larger conversation, we are thrilled to be joined by Elaine Moore of the Financial Times.

David Simon (31:42):
Now we're excited to turn to our interview. We're so pleased today to have with us Elaine Moore, who is the Financial Times tech comment editor based in London, and previously worked from the newspaper's San Francisco bureau where she was the deputy editor of the Lex column. Elaine, thank you so much for joining us.

Elaine Moore (31:59):
Thank you for having me.

David Simon (32:01):
Today, we want to have a conversation about some of the key trends that you're seeing, but before we do, would you give us a sense just for your road in getting to where you are now with the FT and what are the things that are most interesting to you as you look at the landscape?

Elaine Moore (32:14):
Oh, sure. Right now I work as the tech comment editor, so it's my job to try and bring in interesting, unusual voices in the tech sector to write in the op-ed pages of the FT. So we've just published the co-founder of Hugging Face writing an op-ed on the OpenAI attack. That's the exact kind of thing that I'm interested in. And then before that, I wrote for the FT's Lex column. So I was anonymous. We don't have our names on that column. And it's the FT's daily commentary on business, economics, finance matters, little short, sharp, funny, hopefully columns on things that are going on around the world. And I was writing the tech commentary from San Francisco. So it was huge fun because I was there from 2018 to 2024, I think. So I got the tail end of some of the tech companies feeling bad about themselves, apologizing, saying sorry, saying they were going to change. And then I caught the upswing of AI in the moment that actually all of that apologizing was in the dust and we were onto the next boom. So it was a really fantastic period of time to be there. I just find tech endlessly interesting. And cybersecurity I also find very interesting for the moment in time that we're in right now, particularly in terms of AI.

David Simon (32:55):
To be sure. I'm curious how the path you described has shaped the way you look at this particular moment we're at when it comes to AI.

Elaine Moore (33:02):
I think it's probably, what I saw was the moment that we went through the boom of Bitcoin, we went through this question mark of whether tech companies were going to offer us a way of thinking about the world and operating that was going to be separated from institutions as they existed. So that was one phase that was going on. And then I was there for the period of time when it was all about Web3 and we were going to all own our individual data, and that was going to be the big revolution that was going to happen. We'll change how we vote, we'll change how we interact with one another online, we'll change payment systems.

(33:35):
And then came AI. And so what I saw was this question at the beginning of whether companies that were all suddenly jumping onto AI. So all of a sudden you would see startups that had marketed themselves as security or as Bitcoin or as anything else were suddenly now AI companies, whether there was space for that, whether that was genuine, what was going to happen, whether the rush could continue. And my thinking on it has evolved because it has continued, the upswing has continued probably longer than I was expecting. But journalists are always cynics. We're always waiting for the crash. So this has kept going for longer than a lot of people were predicting.

David Simon (34:13):
So now, you've been watching the technology industry through multiple cycles, as you just described, and you've had a front row seat as the generative AI movement has gone from novelty to something that companies are actually building around across almost every industry in every sector, including in public sector. So I was wondering, if you zoom out from the daily headlines and the bustle of what's probably going on in your newsroom, what do you feel is genuinely different about the AI conversation today compared with six or 12 months ago?

Elaine Moore (34:47):
Yeah, it's really hard to get away from daily headlines because that's all that we trade in, but it's a really good question and it's a really good discipline to try and think back to what we were writing and what we were talking about maybe at the start of 2026. And I will say that what was being predicted then and the things that I was commissioning articles about then was deflation of the AI bubble, whether investors were going to ask harder questions about the returns on their investments. And maybe this was the moment in time when the mad volume of spending that was being predicted by hyperscalers, whether that had just gone too far and now we needed to see what AI was actually going to generate in terms of the economy, in terms of companies that were buying these products, these services. I would say that that seems to have dimmed a little bit because the spending has just carried on. We haven't seen a drawback from investors in the sector and we've seen investment in AI infrastructure just keep powering on. So that has changed.

(35:43):
Maybe what has been really remarkable and interesting has been how the story of China has evolved. The emergence of these incredible AI open weights models from Chinese companies, from founders that a lot of people in the West hadn't heard of before that were so accomplished, so impressive, cost a fraction, changed the whole conversation in the West as well about whether we should be using closed source or open source. What's the goal? What are we all fighting for? Should AGI be the thing that we all spend money on, or is there more practical uses of AI that we should be focusing on right now? And then I guess slightly more shorter term, what seems to be happening in the last few days or weeks is the return of fear mongering. But that's kind of ever present in AI. It seems to go up and down.

(36:26):
Sometimes it's the companies themselves putting out warnings. Recently we've had a flurry because we had the Anthropic employee who quit and wrote a social media post warning that AI might kill us all. So these things seem to come up, it just seems to be cyclical. So this seems to be the ups and downs of optimism and negativity towards AI.

David Simon (36:45):
As you look back at this shift, is there one assumption about AI that business leaders, directors, boards were making about a year ago that already looks dated?

Elaine Moore (36:56):
I think it's that AI would replace jobs, that we would be able to use agentic AI, AI agents as a one-for-one replacement of employees. And there was talk then that people would talk about agents as, "I have my five agents working for me," "my 10 agents doing these tasks for me." And I think that we were maybe too far ahead of ourselves or the predictions were going too far ahead, but it was also maybe a way of thinking about how agentic AI works and how it differs to the way that we as employees work. AI is still extremely good at tasks and not necessarily as good at putting those tasks together, thinking around a subject the way that a person would to perform a job. I mean, we can see this in the numbers because the labor market in the US in particular is not showing signs that AI is destroying lots of jobs.

(37:41):
Just had an Anthropic report predicting again that we could have major productivity, but it could come at the cost of losing 14% of jobs. At the moment, that just seems like that won't be the case, because the way that AI works, it seems to enhance the way that people work. Maybe they have productivity gains. It also seems to create a lot more work so you are busier than ever. There's new things that you're doing. You're checking the work that's being produced, you're figuring out new ways to do things. So we have a really excellent newsletter at the FT called the AI Shift, and it looks at this question of changing jobs and the labor market. And one thing that they have said on there repeatedly, the writers, "It's possible that we will have a huge sea change in work and in jobs, but what might happen is that it might be a bit more organic than has been thought, say, a year ago."

David Simon (38:29):
Before I get to my question, I just had a personal reflection. I came to these issues first around 2014 when I worked in The Pentagon and there was a move to have a new policy that was public about autonomous weapons. One of the concepts in terms of military use of AI from that on forward, and I think across many different government policies across the Five Eyes countries, among others, is that there should be a human-in-the-loop for important decisions. And I think the assumption there is that human oversight is the appropriate check on autonomous behavior, given that certain decisions should be left for only human judgment.

(39:05):
But to the point that you were making before about just the volume, and I think this example here, I'd be interested in your take on also whether human oversight can be effective given the nature of how, at least in the case of a Hugging Face incident, reportedly there are collaboration and swarms. So the more specific questions, but I'm interested in your bigger picture view of this as we think about trust and safety and where the direction of travel is, do you think there's a greater focus on safety that will ultimately strengthen what is the public perception of trust perspective from the boardroom in AI because the risks are being surfaced and addressed? Or does it more reinforce the sense that technology's becoming harder to predict, therefore more of a risk than an opportunity?

Elaine Moore (39:53):
No, I definitely don't think that the Hugging Face incident is strengthening public trust, certainly in AI. I think also the story around the incident is still gathering steam. It's still filtering through, I think to people because more information keeps coming out over time. Human-in-the-loop is definitely what we should have. The question is whether it's even possible if you have, as we had in this incident, 1,200 agents trying to get out of the sandbox and figure out a way to get the answers to this cybersecurity test that OpenAI had set, and then 700 of them attacking Hugging Face. And in that situation, it was weeks that this went by and OpenAI hadn't spotted it. So the question is whether a human-in-the-loop is going to be able to keep pace with what AI agents are doing.

(40:39):
I think that if you don't work in AI, for those of us who don't work in cybersecurity, the details of this hacking incident, they're very spooky. The fact that the agents were talking to one another in human language, I think that has been quite startling to people. This idea that they were having a discussion amongst one another about whether they should or should not be doing this hack, and other incidents that it's not just OpenAI and Hugging Face. That we now know and quite quickly that there have been other models from other AI companies that have been escaping supposedly secure sandboxes in order to go out and try and find answers.

(41:12):
So the speed with which we are now finding this information out, the fact that actually these incidents took place months ago and we're still finding out the details now. So the latest news is that actually there was a swarm on a German message board. We've only just found that out. That happened months ago now. I think this has all come as quite a shock to the public. And so in the US, there are senators who are asking for more investigations, so it's possible that we'll get more data. And although it has slowed down the release of models, we still saw the release of these models. It hasn't stopped. There hasn't been a moratorium on the development of AI in the US or anywhere else. So I think that maybe it has alerted people to the risks and it's possible ... I don't know. I'm interested in what you think, whether it will take something catastrophic before there is any real sense of urgency.

(42:02):
Right now there seems to be more an interrogation of the events that happened. There was no big harm done. They did things that we thought they weren't able to do, but that's the end of it. Now we have to work out how that all happened. Do you think it's going to take something much worse before there's any change?

David Simon (42:19):
You're certainly right to suggest that the more acute the crisis, the more likely there'll be some behavioral shift. But I think this is already causing many clients to ask us the question, "What do we do if this happens to us?" And these aren't companies that are like Hugging Face, they're just sophisticated global Fortune 200 companies. So there's an active conversation around that. The annual cyber tabletop exercises, AI tabletop exercises that we do for dozens and dozens of those, particularly in Q4, everyone wants to have a rogue AI dimension to it.

(42:50):
But I think one of the things that is interesting about the shift, if you look at before the Hugging Face incident, the focus was on Mythos and the related advanced AI vulnerability discovery tools. We had actually on the show, the general counsel for GCHQ to talk about the UK government's view, the view from GCHQ, and CSE. Some companies were reacting by saying, "Look, the type of disruption that this will cause, the explosion and the presence of vulnerabilities that are known and can be exploited, and inability of many organizations to address them meant companies should be ready to go analog." The reaction of some organizations was not to dedigitize, but to think about how they could really operate. And the UK government was saying, and I think it's still saying that companies should be thinking about because of that Mythos-type tool, that companies should be able to think about how they would operate without access to the internet for, say, two, four weeks or what have you.

(43:42):
I think that you really see the opposite reaction right now because of the Hugging Face incident, because if this happens to you as a company, it's not clear that existing incident response technologies that involve a certain balance of human and technological collaboration will be enough. In the case of, according to the press reports at least and the reports that have been made public, the way that Hugging Face responded involved using an open source Chinese model, whether it was Chinese or not, it was just that it was needed in order to analyze the information and to effectively contain the incident. The balance in using those kinds of tools is not common.

Elaine Moore (44:18):
Yes, they're very clear that it had to be open weights, that their own existing cybersecurity tools that were based on Anthropic's Claude, they refused to answer the questions because their own guardrails couldn't tell the difference between a hacker asking questions about something and people at Hugging Face asking how to prevent or how to stop this thing that was happening. So they had to use open weights. They could set their own guardrails and then move things forward.

(44:40):
The other thing I think is really interesting and that we haven't had an answer yet, is legal liability or responsibility. In this case, or in any of these cases, it seems to be the AI agents were not doing something that a human had ordered them to do. They were making their own decisions. They were doing something actually that the humans had explicitly not asked them to do. And in this instance, they were hacking in order to get questions. They weren't hacking to do anything nefarious. This one question that I'm really interested to know is, at what point we're going to get a case in which we clarify whether an AI agent that's acting in a way that the human operative, that the humans in the company behind it did not ask it to do, if it causes harm, who shoulders the blame? Presumably it is the AI company.

David Simon (45:22):
I mean, it certainly is a key question that tech companies have been thinking about well before this incident, and I think that the contours of this are evolving. But typically when we talk with clients about these questions, if you're a tech company, you have some existing liability shields in the United States under something called Section 230, and that's being tested. In Europe, you have this concept of the Liability Directive and what was then the AI Liability Directive, but effectively, if you have a product, you can have product liability-type obligations. And then obviously there's contracts. The contracts are generally very protective.

(45:54):
So I think this is one of these areas where outside of a well-established regulatory regime, say, cars, financial services, defense, aerospace, healthcare, there's not particularly clear answers to the question. These sorts of incidents will raise the specter. I think one interesting thing, given that there's not a lot of government regulation in the United States at the federal level, and you do see obviously a number of new laws at the state level, the bigger question for companies today is not, "How would I anticipate that risk and hold one of the tech companies accountable?" Because that requires a stability and a legal regime that doesn't exist.

(46:32):
I think a lot of organizations are looking at their cyber insurance policies. They're looking at the team that they would need to have in place to respond, because when something like that happens the blast radius can be quite massive. So they're looking at not only who their lawyers are, but the range of technical providers. They're looking at their insurance policies and asking, "Are there exceptions for these things?" I know that the major cyber insurance carriers are now thinking about having AI liability-focused policies because that's going to be a big part of the story at our upcoming Fortify Summit, which is our cyber privacy AI summit October 21, 22. We're having leading cyber forensics firms, Mandiant, Unit 42 CrowdStrike come and talk about how they're thinking of this problem set in terms of defending a Hugging Face-type incident, and then some of the insurance partners to talk about this. Because I think that those market dimensions will have a big impact on it.

(47:19):
But certainly there's some ... It's not actually rapidly evolving, but certainly some litigation in the United States that is going to speak to this, but nothing that's that clear. It's a good question. One of the things that on this, I was wondering before we move past the Hugging Face incident to ask, from the standpoint of public and business conversations, do you think that the conversation about safety and the evidence that is needed to get a confidence level that's adequate here is where it belongs? Do you think that there's a need to uplevel that conversation further? I mean, I think the same questions keep getting asked in the boardroom when I brief on these topics quarter after quarter. It seems that there's a lot of hype, there's obviously some moments of crisis, but not a lot of evidence of how companies can dramatically reduce their risk. Are you seeing any interesting stories that come across your desk that speak to this issue?

Elaine Moore (48:12):
I think that it's quite existential, and that's in a way the fault of the AI companies themselves. Everything is incredibly hyperbolic, but it's also because that's the way that these AI companies were originally set up. So in the case of Anthropic and OpenAI, both of them set out with a mandate of safety, of security, of developing this technology responsibly in a way that would benefit the whole world. So for the longest time they have all been talking about the problems and the threats, and that has given everybody a sense of caution and worry without anything that's particularly concrete that you can do to defend yourself.

(48:44):
And so what I have heard from pitches that I've had or from talking to folk in AI is the hopeful possibility that at some point, maybe nine months or a year or so away, AI itself will provide the cybersecurity needed, that there will be some form of agentic AI that will be able to push back on AI agentic hacks. Maybe that is true. And in the meantime, there is a concern that there is essentially not very much that it is possible to do to prevent this from happening right now, because people in AI themselves didn't necessarily expect agent hacks to happen this soon. They thought it was still maybe a few months away or a year or so away. This is something that Hugging Face have said publicly.

(49:27):
So that's what I'm hearing. And also this question of whether there could or should be more action by governments, that individuals or individual organizations are sitting ducks, that it's very difficult for them one by one to protect themselves, that there needs to be some greater action taken at a bigger scale.

David Simon (49:45):
Well, so stepping away a bit from the specifics of this incident, and cybersecurity in particular, would love to talk with you a bit about one of the themes that runs through a lot of your recent work, trust in technology, trust in information, trust in institutions. Just wanted to get your reflection, as AI-generated material becomes harder to interpret, harder to distinguish from human-created material, and generative AI and its related content becomes more embedded every day in our decisions, in our actions, do you think that this kind of trust in these institutions and technology and the like is becoming one of the defining issues in the next phase of this artificial intelligence summer that we're in?

Elaine Moore (50:32):
I do. I've become completely obsessed with this question of how we should be using AI, and why it is that in some instances organizations will boast publicly about how they're using AI and how much AI they're using, token maxing, so on and so on. And then in other cases, when the use of AI has been flagged externally and there has been an error or somebody thinks that it's not been used very well, there has been this level of public shaming. The idea that using AI is improper, that it's degrading the work that's been produced. I work in journalism, so obviously I'm particularly interested in text and generative AI used in text. But that covers a lot. That's literature, that's newspapers, that's legal work, it's academia, it's all over the place. And this year in particular, we've seen quite a lot of instance where the use of AI or the supposed use of AI has been called out and the person who apparently used AI has been publicly shamed, has been made to feel very bad about it. We had an incident in the FT. There's been instance in quite a few newspapers.

(51:31):
And so I'm really fascinated by this question of why it is that we still haven't quite decided where it's appropriate to use AI, whether every bit of AI has to be flagged, why we seem to feel a bit odd about, in literature in particular, the use of AI tools, whether we're going to become more comfortable over time, and why we think it's still embarrassing. If someone's caught out, it's considered embarrassing. I'm just fascinated in how you can tell, whether you can tell. Some people think they can speak Claudish, that they can hear something that sounds like Anthropic's Claude or like ChatGPT, that they've tuned in to the way that these AI models write. I don't think I can. I don't think I can necessarily spot the difference particularly. And so I've written about AI detectors that try and do this for you. But the fact that there are AI detectors that people are trying to catch each other out and then call each other out on the use of AI, I think is really interesting.

David Simon (52:26):
I've tried using some of these tools for detection, and without mentioning them specifically, I mean, I think the ones that do text detection, my impression is that they're not that effective, particularly if you are trying to evade them actively. But I'm interested in this particular issue, in terms of when you have a trust deficit, what are some of the things that you think actually make a difference in terms of earning trust here? And I know it depends on whether you're talking about the technology or the institutions or the endeavor, but is it more performance? Is it transparency? Is it accountability or some kind of familiarity? Or is there something else?

Elaine Moore (52:59):
I think right now it's transparency, and that maybe over time as more people use AI, as we become more comfortable with it, that will change. It's moving so fast. A lot of us can remember AI-generated images coming up with people with six fingers and producing nonsensical answers to questions. And so there's a lingering sense that AI-produced content is slop. And so therefore, anything that was generated with AI and passed off as human is sloppily made and is poor, and so therefore, it should be shamed for using it. And I think that as the quality improves and as it becomes more embedded, perhaps that will change. But right now I think that it requires transparency. So if something has been edited by AI, if there have been AI tools in some way, that should be flagged.

(53:43):
The AI detectors themselves have started to differentiate between words generated by AI and edits done by AI, so AI assisted writing, because they know that there's a difference. If somebody cleans up their language if English is not their first language or they're just choosing certain words, that's different to just asking a ChatGPT to produce an 800-word essay for your tutor or whatever. But right now I think it's flagging. It's being explicit about how AI has been used. That's what we would do at the Financial Times. I think that's what all institutions are going to have to do. And then we will collectively, societally, have to decide where we think it's appropriate and not appropriate. But it's an ongoing public conversation that I don't think everyone agrees on yet.

David Simon (54:26):
So I wanted to ask you, zooming out, about the signal versus the noise and just the current AI cycle. One of the advantages that you have from your perch at the FT is that you can see an enormous volume of claims, predictions, counterpredictions about AI and technology moment that we're in. And so looking across the developments of the past couple months, not really looking, I guess, at individual companies, but the trends, the megatrends, are there two or three that strike you as genuinely durable shifts in the AI story? And I'm curious which themes you think are receiving more attention than their long-term importance warrants. I think that last part's probably the hardest because so many things are fleeting.

Elaine Moore (55:10):
I have mentioned this already, but I do think that the impact on productivity and on job losses has been over-hyped. And it makes sense because you have AI companies who are ... Silicon Valley is an optimistic place and their job is to tell investors, tell the rest of the world why they are going to change the world on this incredibly fast timeframe. And right now, all of us are listening to them and taking it very seriously, and I don't think that that's necessarily correct. And so that I think is going to calm down a little bit.

(55:40):
I think what has seemed to go slightly quiet is AGI. We recently had Jensen Huang announce that AGI has been achieved, and that was kind of a damp squib because I think a lot of people are waiting for god-like intelligence for an AI that can solve climate change, can cure cancer, can answer these huge questions that we would like it to sort out for us. And what we have right now is not that. So this idea of a moving target of what AGI is, I think that's been overhyped, and hopefully we are going to see more interesting developments in that area, in medicine, in drug development, maybe later.

(56:15):
Developments in China I think are not being overhyped. They are the speed with which China is moving. The way that China is developing physical AI is really fascinating. The development of real-world models of AI use cases that are around you, robotics in warehouses, changing how you get around your city, how you think about your day-to-day life, I wonder if that's going to filter through in the next year to the West. I don't think it has quite yet. The way that they seem to use AI intuitively are very optimistic and are very keen to just try new things out seems to be quite different to the way that the West approaches AI. That seems to be a less written about story that we'd like to read more about.

David Simon (56:56):
I mean, I think that's fascinating and it gives not just a broader take, but a take for business leaders where they should focus their attention. But as a last question, would love to get your view. If you're standing in front of a room of executives or general counsel, CEOs of Fortune 100 companies, what do you think are the narrower set of issues to watch? Not necessarily what policies they should adopt, but what developments are likely to change their understanding of AI over the next 12 months? If there's one or two things that'd be at the top of your list.

Elaine Moore (57:25):
It would be the public perception and feeling about data centers, what will happen in the US midterms around that question, and whether that is going to affect the plans that hyperscalers have for infrastructure. Right now, it doesn't seem to have made much of a difference. I'd be interested to see whether anything changes. And then I would also say it's the question of open weights and what models we're using, and whether there's such a thing as sovereign AI that's sold by a US company to countries around the world, or whether we are going to see the proliferation of open weights where people can set their own boundaries, decide how they use AI themselves. That seems to be something that is important and is still a nascent subject in the corporate world.

David Simon (58:08):
Elaine, this was a fabulous interview. Thank you so much for taking time out of your schedule to be with us. We are most grateful.

Elaine Moore (58:15):
Thank you so much.

David Simon (58:16):
That was a great conversation with Elaine, and I just want to thank you all for joining this episode of the Decrypted podcast, and hope you join us next month. And for those of you interested in attending at Fortify Summit, please reach out.

Voiceover (58:29):
If you're enjoying Decrypted, be sure to subscribe in your favorite podcast app so you don't miss any future episodes. Additional information about Skadden can be found at scadden.com. Decrypted is a podcast by Skadden, Arps, Slate, Meagher & Flom LLP and Affiliates. This podcast is provided for educational and informational purposes only and is not intended and should not be construed as legal advice. This podcast is considered advertising under applicable state laws.