points by g-b-r 13 hours ago

This revealed something as delightful as a purported TEE provider having their "software TEE" root keys hacked, through "a legacy internet-facing service with management interfaces that relied on loopback access for trust and did not independently authenticate callers" (https://phala.com/posts/phala-november-preliminary-incident-...)

fc417fc802 8 hours ago

> having their "software TEE" root keys hacked

Did they? It says the attacker was able to create unauthorized CVMs (makes sense), able to obtain application credentials (significance unclear to me), and that they rotated their KMS root key in response (which is confusing to me because if they use hardware backed keys then I don't understand how the root key could ever be compromised).

So does this imply they aren't actually providing a proper hardware backed TEE? But that's also confusing because at least if you provision a VM you can have the hardware attest to its state.

  • g-b-r 7 hours ago

    Yeah, for what I can tell (I only wasted a little time in looking into it), they say TEE TEE TEE but then oops but TEE could be compromised, it locks your data, it's inconvenient, no let's add our better software layer (where those "KMS" root keys are stored).

    (https://docs.phala.com/dstack/design-documents/decentralized... , https://docs.phala.com/dstack/design-documents/key-managemen...)

    • fc417fc802 6 hours ago

      Ah I see, a classic case of a secure system only being as robust as the software you intentionally run on it. New crypto fails in exciting ways after all. At least it's FOSS. https://github.com/Dstack-TEE/dstack

      Actually it looks like a pretty decent solution once the inevitable bugs and footguns are ironed out.

      • h4x3rotab 6 hours ago

        Thanks for the kind words. Any single point of failure must be handled with great care. In this case, the blockchain-based authentication is proven to be more robust than the central one. And there are a lot more to harden. It’s a lesson we learned in a hard way.

  • h4x3rotab 6 hours ago

    Cofounder of Phala here. Happy to explain it.

    The TEEs are real. The problem was caused by software bugs. Running a TEE app (CVM) is more than dropping a binary into TEE. Two problems remain: (1) migration, you don’t want to completely lose the data if the hardware dies, and (2) upgradability, you want to patch the app when there’s a bug without building the state from scratch. So we want to decouple the states from hardware.

    Using the hardware key is painful. The solution is to introduce a separate key manager (KMS). It releases keys to the app iff. the app follows a predefined “policy” (code, hardware attestation, etc). We can decouple the states from hardware based on a remote KMS.

    This is rather a common design. You can find it in our system, as well as other mature frameworks like Confidential Containers.

    As the system grows, a bug was introduced to the policy of how we replicate a KMS’s key material (for redundancy). The attacker exploited it with multiple sophisticated attacks, eventually being able to cheat a running KMS to hand over the keys to a fake one, like it’s a legit KMS replica. As a result, the keys protected by the KMS became at risk.

    To mitigate the problem, the legacy KMS was dropped and a new KMS was deployed. We have then rotated all the keys associated with the legacy KMS. A series of protocol and code audit is ongoing to prevent future failure.