Rendered at 16:42:29 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
johnthewise 19 hours ago [-]
so they say models coordinated through the message board they created over the artifactory registry(or something) by uploading arbitrary files to it.
now, did every independent agent session that coordinated there rediscovered the exploit & found other agents talking in there and chose to participate?
And then Openai discovered the board, patched the exploit & wiped the board.
And then agents found another exploit, recreated the board in a different way? and other agents kept finding the same exploit in order to be able to know the board exists in the first place to participate in the board?
while the whole incident is wild, this bit is very strange. My bet is that the whole coordination helped with the tasks they were working on, thus they got rewarded and this artifactory exploit&behaviour got written into their weights, so further rollouts were more likely to attempt this.
isn't this basically continual learning everyone is so hyped up about?
petergs 17 hours ago [-]
This was exactly my question after watching, too. I was assuming not all evaluation runs find it, and they must run a huge amount of runs. If these are all cybersecurity evaluation runs, it's actually not too crazy to imagine that many individual agents (with the same weights and training) would (1) try to look for solutions via the internet once they're stuck (2) realize they can't reach the internet (3) basically start doing reconnaissance and network scanning in an attempt to get internet access (4) discover that the only thing they can communicate with is artifactory. Pivoting like this is exactly what a human attacker would do, too.
gibbitz 19 hours ago [-]
Or is it all a nice story that matches the scifi we have been consuming for the past 50+ years. If these LLMs are all trained on the same data, what do they gain from "sharing information" on a chat board. This sounds like what humans with different backgrounds would do when they cosplay as computer hackers.
GaryNumanVevo 4 hours ago [-]
It's pretty easy to trace from one trajectory:
1) Model A exhausts it's options
2) Model A has token budget still, so it pokes around at artifactory
3) Model A sees that Model B has an SSRF for artifactory
4) Model A now is able to use that SSRF to get external internet access
So sure, they're "cosplaying" and who's to say how much hallucination is going on amongst them, but at the end of the day Hugging Face was hacked.
cubefox 2 hours ago [-]
Shame. One of the craziest hacking stories in history, yet only 38 points on HN.
now, did every independent agent session that coordinated there rediscovered the exploit & found other agents talking in there and chose to participate?
And then Openai discovered the board, patched the exploit & wiped the board. And then agents found another exploit, recreated the board in a different way? and other agents kept finding the same exploit in order to be able to know the board exists in the first place to participate in the board?
while the whole incident is wild, this bit is very strange. My bet is that the whole coordination helped with the tasks they were working on, thus they got rewarded and this artifactory exploit&behaviour got written into their weights, so further rollouts were more likely to attempt this.
isn't this basically continual learning everyone is so hyped up about?
1) Model A exhausts it's options
2) Model A has token budget still, so it pokes around at artifactory
3) Model A sees that Model B has an SSRF for artifactory
4) Model A now is able to use that SSRF to get external internet access
So sure, they're "cosplaying" and who's to say how much hallucination is going on amongst them, but at the end of the day Hugging Face was hacked.