fwsgonzo 4 days ago

Don't disable nested, it's still better security than everything else, and having it on means access to real hardware virtualization. Second, the article is AI-written. It really should be rewritten by a human.

  • rootnod3 4 days ago

    > It really should be rewritten by a human.

    That does not seem to be what HN has been thinking for months.

  • xmodem 4 days ago

    Outside of development and CI, I have yet to hear a compelling use case for nested virtualization.

    Agree that the article is downright painful to read though.

    • JoshTriplett 4 days ago

      > I have yet to hear a compelling use case for nested virtualization.

      AWS instances are slow to launch; they take whole seconds. Nested virtualization allows running something like firecracker/crosvm/rust-vmm atop a normal AWS instance, without needing a bare-metal one.

      • tptacek 4 days ago

        If you're segregating internal workloads, by all means use nested virtualization on EC2 or DO or something. Your threat model is different and you probably have a lot more room to maneuver when a KVM escape is announced, because you have some level of trust in all your workloads.

        If you're running multitenant for mutually-untrusting customers, that excuse goes out the window.

        • JoshTriplett 4 days ago

          One distinction I'd make as well is "are you running more than one customer's workload on the same system" (yikes, don't), vs "are you using nested virt as additional protection rather than running the customer workload directly on an instance, but one instance only gets jobs from one customer".

          • tptacek 4 days ago

            Another good point, but this one is maybe a little less dispositive than it looks like: it's very difficult to engineer an orchestration system that reliably segregates worker physicals from each other, so I'd rate the likelihood of a game-over situation after a guest->host escape as very high regardless of how yuo schedule workloads. It's sort of the same situation as how SSRF's are usually game-overs (you're pivoting from a hardened attack surface to a much softer one) but even harder to engineer again.

            • JoshTriplett 4 days ago

              That's fair. You'd want a system designed around the notion that every customer instance must be isolated from each other without the nested virtualization.

              (Aside: my current guess at the appropriate security posture towards Nitro Sandboxes, in the current era of vulnerabilities, is "decent, but probably has security holes too, and is less widely scrutinized, don't count on it either". Thoughts?)

              Any recommendations for good guides on that, besides "design very careful IAM policies and security groups, harden any common services they have access to, and ideally run them under separate AWS accounts"?

          • necovek 4 days ago

            Does this hold for the likes of Amazon?

            Do your lambda functions always get a per-customer dedicated instance or is firecracker distributing them so that multiple customers can have VMs on the same actual box?

            Sounds like it would be pretty expensive for Amazon to dedicate an instance to every customer, especially for those with small workloads.

            If that's the case, why would it not hold for Amazon but would hold for everyone else?

            • swiftcoder 4 days ago

              > Do your lambda functions always get a per-customer dedicated instance

              At some point in the past they did, for pretty much exactly this reason. I'd assume they don't anymore (I last worked there a decade ago).

        • exceptione 3 days ago

          Do you mean that you would run each tenant on separate (physical) hardware? If so, I presume that you enforce this only for workloads where you run code outside your control. I guess the average SaaS does not install hardware boxes for each tenant separately.

          Also, how could VPS hosting still be a thing if KVM is that easily escaped?

          EDIT: I am talking about KVM in general, not nested virtualization.

          • tptacek 3 days ago

            No, we're generally always talking about multitenant shared hardware, and then the different things you're "sharing" within that hardware --- a kernel (no virtualization), a hypervisor (KVM), two hypervisors (nested KVM). Note that the "extra" hypervisor isn't the big problem with nested KVM.

            • exceptione 3 days ago
                > If you're running multitenant for mutually-untrusting customers, that excuse goes out the window.
              

              I still don't understand what you mean by this then?

              • tptacek 3 days ago

                "If you provide services for customers, and the customers don't trust each other (because they are unrelated entities), and multiple customers share the same platform, then the added complexity and poorer security track record of nested KVM becomes a large problem".

                • exceptione 3 days ago

                  Thanks, I didn't get you meant it that way, but that makes sense indeed.

      • secondcoming 3 days ago

        What's the usecase for this? You're already paying for the instance to be up but it's being deliberately under-utilised in case you have to start a VM?

        • JoshTriplett 3 days ago

          Keep a warm pool to serve customer requests.

      • xmodem 3 days ago

        If you need as-fast-as-possible guest start, AND you have to be on AWS, AND you can't use any of their native microVM-based services, AND you can't use the bare-metal instances - whether due to cost or some other reason, then nested virtualisation might be the least-bad option. That doesn't make it compelling.

        • danielheath 3 days ago

          Security primitives which don’t nest are fundamentally a bad API design.

          Sometimes that’s justified in the name of making the security code simple enough to reason about, but making it harder to use has a security cost also.

        • c0balt 3 days ago

          I have been a multiple times been on a guest VM on on-premise infra where microvms were used with nested virtualization for exactly this purpose. The setups don't have a AWS Lambda to fallback too nor do they just get to have access to the hypervisor/bare metal.

          This was gVisor in kvm-mode and qemu inside containers for integration tests. The main reason here is cost, it is very compelling for getting more out of a hypervisor setup.

    • immibis2 3 days ago

      I think the use case is that you buy a VM from someone and you also want to run VMs. You wouldn't design your own infrastructure to run VMs inside VMs, but you might already be inside a VM, and want to run VMs.

      • xmodem 3 days ago

        Oh, sure, there are few limits to the ways you can work around a broken organizational design with a bad technical design. But if nested VMs are the answer then I think you're probably asking the wrong question.

        • shakna 3 days ago

          Running a few containers inside a kubernetes cluster that you don't manage, is not that an usual a setup in SaaS things. You don't get full insight into someone else's stack.

          That's usually nested, and either cri-o or... KVM.

        • LoganDark 3 days ago

          I mean, nested virtualization allows your containers to implement virtualization-based security. Defense in depth -- more layers for an exploit to have to penetrate.

    • FrinkleFrankle 3 days ago

      I personally use it to test virtualization platforms. Not sure if you'd count that as dev.

    • stinkbeetle 3 days ago

      Outside of the compelling use cases I can't say I've heard any compelling use cases either, now that you mention it.

  • TZubiri 4 days ago

    ? If you don't use it, disable it. Simple as that

immibis2 3 days ago

Links to a CVE record. The kernel project marks everything that could conceivably be of any use to an attacker as a CVE. What are we actually looking at here? Sounds like a dangling-pointer type of bug?

  • 3g56g6g6h 3 days ago

    failure of KVM to grok and implement long-standing x86 behavior: INIT invalidates the TLBs

jacobgold 4 days ago

Our infrastructure will become so much more secure but we're in for a bumpy ride through 2027.

  • dvh 4 days ago

    Not necessarily. Only if you assume that no new insecure code is added. If ai found 1 security bugs in old code but ease of code generation with ai adds 2 new security issues (in unrelated code), then we are getting less secure.

    • SOLAR_FIELDS 4 days ago

      Do we have any evidence around the rate that AI emits security bugs vs humans previously? You would think that perhaps AI would be better about this, but perhaps not.

      • dvh 4 days ago

        I'm making no claims. I'm just saying that contributing before ai was difficult so people rarely did it. Now that contributing is easy, more code will be added. More code = more bugs. Always.

        • voakbasda 4 days ago

          Yet the opposite seems to be true. Many projects now shun new contributors due to AI.

          • hparadiz 4 days ago

            I've run Daybreak against my human written pre-AI code and found so much stuff that I don't trust human or AI written code. It all MUST be audited by a security sweep AI.

          • punchmesan 4 days ago

            Eh I'm not sure that's an indicator that the opposite is true. Github's vastly increased commit activity is the best indicator we have of this. https://www.tomshardware.com/software/github-report-highligh...

            Sure, some projects are trying to be responsible or are trying to keep waves of slop PR's from drowning them, but that is by no means an indication of a general distaste for AI code. It's been largely embraced.

          • ranger_danger 3 days ago

            Are any of those large projects used by many people?

      • the8472 4 days ago

        Even if Fable and Astra are superhuman at identifying and fixing bugs that does nothing when projects get bugslopped faster and cheaper by some low-end model.

    • j45 4 days ago

      Existing code will have security holes that haven't been found yet.

    • tptacek 4 days ago

      I hate these meta conversations on crunchy technical topics and feel like we should be digging into the KVM security model more than this general security prognostication stuff. Having said that, I will hypocritically further this debate by pointing out that the determinant of software security moving forward is not so whether AI is drastically better than humans at avoiding common vulnerability patterns, (or instead just a little better). Rather, it's that AI will make it much easier to move subsystems or whole projects to memory-safe programming languages and towards formal verification.

    • tyingq 4 days ago

      Is it a given that it's as good at outputting secure code as it finding insecure code? That leaves a fair amount of room for people not baking in deep review steps. Also mentioned elsewhere, but if velocity is way up...that ramps up bugs anyway.

      • MajesticHobo2 4 days ago

        No, it's not a given. AI always finds security issues in the things I just had it write.

        • hparadiz 4 days ago

          This is true for human written code too. This is why in the future all written code MUST be run through an internal security sweep before being given to your harness for writing. That's just basic due diligence at this point.

          • catlifeonmars 4 days ago

            Yes but LLMs famously produce vastly more code than humans. My hot take: LLMs will produce more insecure code than they can catch, and will also find ways to smuggle insecure code to avoid extra work.

          • vbezhenar 4 days ago

            As long as model providers deny access to security-capable models, it can't be basic diligence.

            • hparadiz 3 days ago

              This is something agents already do as part of their normal work. Providing code that you know is vulnerable is in fact a dereliction of duty and negligence. Don't provide a POC. Simply that as it produces code a background agent just checks it without showing the work and points out potential problems.

            • greyr0ck 3 days ago

              The slop is cheap but checking will cost extra, what a pricing scheme! Modern day Pharo level of lockin!

        • FrinkleFrankle 3 days ago

          But that should be part of your pipeline, no? Personally, the thing I've spent the most time working on with AI is.. a stack to get AI to keep a project on track and follow strict code writing standards that are reviewed. Not everyone is working in that, but I'm sure any serious development will be done via something similar.

          Although, at the pace of improvement, I'd bet it won't even be necessary in a year or so.

  • TZubiri 4 days ago

    Have you ever been told that you are problematically optimistic?

  • InsidiousWombat 4 days ago

    ...if the pace of both fixing and finding bugs got accelerated, all we're in for is machines being on average more vulnerable (because patching takes time and effort and more bugs increases that)

snvzz 3 days ago

I don't get the insistence on using Linux as hypervisor knowing how bad it is, instead of the far more appropriate choice, seL4[0].

0. https://sel4.systems/

  • abeyer 3 days ago

    The situation might improve if they were to spend some time and effort documenting how seL4 can be used that way. Last I looked their docs around the hypervisor use case were pretty sparse, had trivial examples that were far from what I'd expect to see in even small production use cases, and kept implying that using VMs wasn't really the seL4 way, and wouldn't you rather rewrite everything native?

    • snvzz 3 days ago

      Even the most cursory research yields multiple solutions, some of them with commercial support.

      They'd hold your hand through the entire process of avoiding being hit with criminal negligence charges whenever something serious happens as a consequence of making Linux's MLoCs your TCB.

praseodym 4 days ago

The blog post is very difficult to read, I suspect that unfortunately it’s fully written by an LLM.

  • Betelbuddy 4 days ago

    The LLMs find the security issues, the LLMs will write the reports and the LLMs will fix and patch them.

    I guess its time for all of us to step aside, leave space for somebody who knowns what they are doing, and go fishing...

    • praseodym 4 days ago

      And this way only LLMs can learn from these vulnerabilities, making it even more likely that the next one will be found by an LLM.

      • catlifeonmars 4 days ago

        LLMs currently don’t learn, at least in the way most people mean, but I assume you mean the trainers will incorporate the knowledge in future releases.

    • cduzz 3 days ago

      We're all living in the story of mel.

      At first, there were just big logic blocks and storage that did as they were told.

      Later, in hardware, there were all sorts of optimizations built in where the logic would do its own thing -- so you'd get caches and speculative execution and even on-the-fly conversion from one instruction set to another.

      This abstraction was also piled onto the runtime -- you'd write machine code, then assembly, then higher level languages; eventually you'd run in interpreted or just-in-time generated bytecode (hello to Java from UCSD p-System from the 70s and Burroughs machines in the 60s). And of course... operating systems...

      Now, we've got machines reading source code (and machine code, and everything in between) and they've got some ability to thing the thing to make it thing. Sometimes correctly (enough to pass existing tests) and sometimes without the ability to express to the human what the thing they thinged was.

      But, we're all living in the land of mel and for some it's always been better to go fishing than to pretend to participate.

      • immibis2 3 days ago

        The "hardware doing its own thing" can be understood. There was this big ideological movement called ECS in the 2010s that was about writing software in ways that actually run well on hardware.

  • xlayn 4 days ago

    doesn't change the value of what's writen, if true and actionable that's all that should be needed

    • drum55 4 days ago

      If it was valuable a human would have been able to communicate it in a few lines of text. Using a LLM to blast slop devalues everything and communicates little. A drunken twitter rant has more ability to convey information than sloppy blog posts with 3 pieces of information per sentence and a load bearing attitude.

      • cluckindan 4 days ago

        Just ask the clanker to read it buddy, you’re already behind /s

      • Ylpertnodi 4 days ago

        Get hip to it, daddy-o.

        • Pannoniae 3 days ago

          these squares aint hip to the jive, I know that!

    • sylos 4 days ago

      Value is more than just iron something is made from. How it's shaped and presented is just as important

      • tptacek 4 days ago

        No, it is not "just as important" as the underlying information.

        • tsunagatta 4 days ago

          It actually might be far more important. You are a lot more important than the molecules that you are made of

        • mahogany 3 days ago

          It can be, if the intent is for transmission of information or truth. This is at the heart of any propaganda, for example, where the style and medium of messaging are of top importance. Or see the “abc conjecture” and how a gap in communication means that we (or, the math community) are not sure if the conjecture is true or not. The “underlying information” is static, but is essentially null if it cannot be communicated.

    • catlifeonmars 4 days ago

      It kind of does, the OP could have just saved themselves the tokens and posted a 130 word summary on x/twitter that would have been just as useful.

      • Kozmik1 3 days ago

        OP (me) was not the author but I generally agree - a well written or at least better edited summary as a preamble would have been preferable, however it's educational to see the process and a potential repro of the issue don't you think?

        Of course I imagine they used AI to find the exploit, and the documentation may have come as an output from that process.

    • manbash 4 days ago

      It has "potential", not "value".

      Also in this case, a "missed potential" if it eventually fails to reach anyone.

    • orbital-decay 3 days ago

      Like someone have said very long ago, "If."

  • biotechbio 4 days ago

    Glad to see this is the top comment. This is written by Claude, I know because I talk to Claude for 8+ hours a day, and it says stuff like this. I stopped reading the article because of it.

arde 4 days ago

That's a 404 now.

gbrindisi 4 days ago

Thinking about agents sandboxes... it truly seems like the time is ripe for unikernels

  • wmf 4 days ago

    Unikernels that run on KVM aren't going to protect against KVM escapes.

    • eyberg 3 days ago

      As others have stated this is a nested issue. Most unikernel users I know deploy straight to ec2 or gcp. This is going to affect "sandbox" providers more than anything else.

      • fcarraldo 3 days ago

        Nested virt on EC2/GCE are vulnerable. Metal instances wouldn’t be, though using nested virt on Nitro Sandbox capable EC2 instances wouldn’t be either.

        So this is a tricky one. You can (and should) deploy unikernel workloads on metal and avoid nested virt but this is hard to make economically viable in some cases.

        • wmf 3 days ago

          I've always been under the impression that bare metal is cheaper.

          • fcarraldo 23 hours ago

            No, not really. Metal instances from cloud providers typically cost the same as their virtualized counterparts.

            They only come in large sizes, so they’re much more expensive when you aren’t able to fully utilize the capacity on each instance. It’s easier to binpack and saturate a mixed pool of instance types of various sizes than 24xl metal nodes, especially if you care about spreading your workload instances across multiple nodes to prevent single-node instance failure from having impact.

      • cryptonector 3 days ago

        > This is going to affect "sandbox" providers more than anything else.

        This is going to affect "sandbox" providers and their customers more than anything else.

        FTFY

  • Kozmik1 3 days ago

    This was some of my first thinking when I posted the original link - the importance of hardening VMs for agent isolation, and the risk that bugs like this pose in legacy code - especially C codebases.

sadkgasjdfgk 3 days ago

I wonder if AWS' Nitro Sandboxes would prevent this?

  • fcarraldo 3 days ago

    It should? Would be good to have confirmation. I’ll try to POC it but won’t have time until next week.

adverbly 4 days ago

Well this sucks. So this needs a kernel fix then? Looks like I can disable nested passthrough manually, but am I stuck waiting for downsteam fixes at this point then?

Honestly, this gives me pause on the whole "VM is sufficient for any agentic sandbox environment" theory.

cloudie78 4 days ago

Nested virtualisation.

Just don’t.

Ungh 4 days ago
  • kristjank 4 days ago

    There is a Theo quote for everything, isn't there...

  • tptacek 4 days ago

    Theo was wildly wrong about this, but he wrote it pithily so it's never going to die.

    https://news.ycombinator.com/item?id=48883089

    • Ungh 4 days ago

      Well he's not really. It's just useful and shit!

    • cryptonector 3 days ago

      I wouldn't say wildly wrong. I would say virtualization is a lot better than nothing, but that there's a pile of hacks with a pile of vulnerabilities (Spectre/Meltdown come to mind) is undeniable. It's just there's no need to tell people they are stupid or worse for using it.

      • tptacek 3 days ago

        Microarchitectural attacks hit cotenant shared-kernel workloads harder than they hit virtual machines.

    • eddythompson80 3 days ago

      He is right. It’s just that selling bare metal doesn’t scale and we must sell VMs instead, so we can't understand his point.

      Since he made that comment there has been numerous virtualization vulnerabilities in KVM, Xen, HyperV, and all other hypervisors as well as all their device virtualization helpers/ecosystem. I fail to see how Theo's statement is anything but 100% right. It's just that we can't live with it because we must sell VMs.

      There is a reason you graduate to bare metal, then you graduate to air gapped.

      • tptacek 3 days ago

        Bare metal has nothing to do with his point. He's comparing cotenant shared kernel to cotenant virtual kernel, and he's wrong: virtualization drastically reduces the attack surface.

        • eddythompson80 3 days ago

          Is he? I didn't follow the entire thread, but it doesn't seem to discuss shared kernel, just whether the security of x86 virtualization is real or if it's just a convenient way to run multiple kernels on the same x86 host. If you don't trust x86 virtualization as a security boundary, then you're down to bare metal and hoping that your networking cards and switches don't have some vulnerability too.

          Virtualization does reduce the attack surface, but it's still there. This is a thread for a hole in it.

bigbuppo 4 days ago

Looking forward to Ubuntu finally rolling out an updated kernel for this in the year 2057.

  • aeonik 3 days ago

    The 2038 unix time bug is before this date, assuming they don't fix that bug either, it would be back to 1920, just in time for the great depression.

LelouBil 3 days ago

At this point, I'm just hoping the attackers get choice paralysis when thinking about which exploit to use.

ArcHound 4 days ago

Surely, we'll run out of critical CVEs in open-source someday. Seems it's not today.

In the meantime the patching will continue until security improves!

  • j45 4 days ago

    Doubt it, but there will be a big spike for a while and then slow down.

nubinetwork 4 days ago

I guess its time to shut off the internet.

calvinmorrison 4 days ago

what is the mitigation.

  • j45 4 days ago

    The cloud is somebody else's computer.

    Mitigations are layered, first not using a shared VPS, and instead running your own bare metal server with something like proxmox to run the containerization.

    Those containers could be susceptible to issues between them too, but you at least control the environment and can put more checks and balances in place.

    Shared cloud and vps is going to become bumpy and more insecure the next little while. There's no way around it.

  • stilldavid 4 days ago

    Look in the article under the headline "Mitigation".

    • pixl97 4 days ago

      Page has gone 404.

tptacek 4 days ago

Another nested KVM bug. We debated enabling nested KVM for many years, and held the line every time, because the only post-2020 KVM vulnerability we knew of required nested VMs. That decision aged well: we've gotten at least 2 more nested KVM escapes in the last year.

Don't allow nested KVM.

  • rwmj 4 days ago

    Certain Windows features need it unfortunately. Virtualization-Based Security (VBS), Credential Guard, Hyper-V (obviously), and I'm going to say WSL2(?)

    • tptacek 4 days ago

      Right, sorry, that's a fair point (and Josh Triplett made another fair point earlier). I'm sort of implicitly locked in to the "hosting company" use case here: services running mutually-untrusting isolated workloads for customers.

  • bonzini 3 days ago

    KVM maintainer here. On one hand it's true that nested is the single biggest attack surface in the hypervisor, on the other hand it will be over relatively soon. Considering the LLMs have ~15 years of nested virtualization to catch up with, it's not in an awful state. These bugs are extremely complicated to exploit, which means that the past ten years of fuzzing with syzkaller have already found the low hanging fruit.

k0morebi 4 days ago

404 now

  • pixl97 4 days ago

    Confirmed, wonder what happened here.