We don't test devices; the device makers run the tests and certify compliance. We effectively rely on the security research community to identify cheaters. Note that there have been a few revocations. I expect there are far more that need to be revoked, but don't know which.
Conversation
Sorry, your answer isn't clear to me. In your answer, checking SoC's secure boot is done by "the tests"? If yes, which ones?
1
Of course tests can't check if the implementation is secure, only that it exhibits the expected functional behavior. There are non-functional requirements, too. Device makers effectively self-certify that they pass the tests and meet the untestable requirements.
1
5
It's not possible to write a test suite or list of requirements to impose upon other companies to force their software and hardware to be reasonably secure. They need to care about security themselves and put resources into it, beyond just complying with bare minimum standards.
2
4
Most OEMs are pretty stupid. But in a good way!
If CTS gets automatic testing of default bootrom for Qualcomm, Mediatek, Unisoc's OTPs (including detecting SoC default keys...), they'll probably check it. Issue is that it must be done not just in CTS, but also in production
1
2
CTS runs tests via apps and adb shell within the security model of the OS. It runs on a production build of the OS with the full security model intact and is very limited in how deeply it can probe and test. It can't peak behind the curtain and enforce implementation details.
1
3
CTS is very limited in what it can test. That's why there's the VTS for testing the kernel and vendor via a special system image, rather than on the production OS. However, that's still testing functionality via public APIs, not peaking into the details of the implementation.
2
4
I use "CTS" in the broad sense. Google has CTS-like certification tests that run as root that pokes for known vendor security flaws (like MTK-SU).
With GKIs around the corner, Google not knowing about OEM implementation gets even more improbable.
1
2
How can you test low-level secure boot beyond setting it up and flashing images not signed with the correct keys? The SDK from Qualcomm and other SoC vendors already provides everything that's needed, and Google lists it as something that has to be done, and presumably checked.
1
2
They could spell out a very specific process vendors need to run through to check that this works, if that's what you mean. I don't see how any software they could provide to vendors would be helpful. Testing is just flashing corrupt images or ones signed with different keys.
It's spelled out that this has to be provided, so if they're running through the list of requirements, this is something they need to do. Maybe it needs to be more explicitly spelled out that vendors need to go through the list of requirements one by one and ensure compliance?
1
1
I don't think raising minimum requirements and trying to enforce them is going to result in decent security. If vendors don't have an interest in security themselves, they're a lost cause. For a vendor that cares at all, the recommendations that aren't mandatory help them too.
1
1
Show replies
And even if all of this testing is done, and done well, it still only confirms that the nominal cases work properly, it does nothing to prove that the system is actually secure. That requires inspection of the design and implementation, plus penetration testing, fuzzing, etc.
1
3
It's just not realistic to mandate and certify security. It's great to have clear documentation for how things should be done with tests for as much as possible, but achieving decent security is far beyond just complying with a list of rules / requirements and some test suites.
1
3
Show replies