False positive malware alerts in OSV

追加: 2026-06-03

[00:00:00]OSV deleted 157 malware entries because they had been found to be false positives. What we understand is there was some kind of LLM generating reports where maybe the reports were not vetted by a human on the outgoing side and then in the incoming side into OSV, maybe nobody at OSV validated them either. There is real harm when false positives make it into the ecosystem, whether it's vulnerabilities or malware. But the reaction from application security vendors was immediate and kind of brutal and honestly kind of not cool.

[00:00:41]>> I think it was really crappy that a bunch of ASPM and AppSec vendors jumped on this.

[00:00:46]First, because [clears throat] most of those vendors have never supported OSV in anyway. They've never submitted a single thing to it. And yet, many of those vendors, and I'm not going to use any names, have used OSV as their primary malicious package detection source. Here's the backstory. AWS is the largest contributor to the Open Source Buffer Foundation, which is part of Linux Foundation, who collabed Google for OSV.dev.

[00:01:11]AWS suddenly moved to an automated submission process. By their own admission, Chai at AWS in the PR says, "Hey, we turned on some new detection rules and they're a little bit too trigger-happy and we pushed some stuff that we shouldn't."

[00:01:25]Now, here's the problem is that some of those are legit. The problem is that when LLMs decorate these things and everybody inside of their malware detection engines are all using LLMs.

[00:01:34]You basically have static findings and your sandbox findings, those all come together and the LLM kind of goes through those things. It's a very gray area, especially with some of these crypto ones where you're seeing wallet addresses and other things like that.

[00:01:44]It's really easy for something to get ticked over from 49% malicious to 51% malicious and it's decorated as malicious and it gets shipped, right?

[00:01:53]And so, this idea that it's really obvious at face value if something is malicious or not is not the case. Bad guys hide stuff.

[00:02:01]So, I just think it's crappy that people are piling on who are not part of the process, who are not part of the solution, right? Moreover, I the vast majority of submissions over the last couple years have been from AWS and from Chai's team at Inspector. So, I just want to say publicly, Chai, I appreciate your work. AWS, I appreciate your work.