Sorry, your answer isn't clear to me. In your answer, checking SoC's secure boot is done by "the tests"? If yes, which ones?
Conversation
Of course tests can't check if the implementation is secure, only that it exhibits the expected functional behavior. There are non-functional requirements, too. Device makers effectively self-certify that they pass the tests and meet the untestable requirements.
1
5
It's not possible to write a test suite or list of requirements to impose upon other companies to force their software and hardware to be reasonably secure. They need to care about security themselves and put resources into it, beyond just complying with bare minimum standards.
2
4
Most OEMs are pretty stupid. But in a good way!
If CTS gets automatic testing of default bootrom for Qualcomm, Mediatek, Unisoc's OTPs (including detecting SoC default keys...), they'll probably check it. Issue is that it must be done not just in CTS, but also in production
1
2
CTS runs tests via apps and adb shell within the security model of the OS. It runs on a production build of the OS with the full security model intact and is very limited in how deeply it can probe and test. It can't peak behind the curtain and enforce implementation details.
1
3
CTS is very limited in what it can test. That's why there's the VTS for testing the kernel and vendor via a special system image, rather than on the production OS. However, that's still testing functionality via public APIs, not peaking into the details of the implementation.
2
4
I use "CTS" in the broad sense. Google has CTS-like certification tests that run as root that pokes for known vendor security flaws (like MTK-SU).
With GKIs around the corner, Google not knowing about OEM implementation gets even more improbable.
1
2
How can you test low-level secure boot beyond setting it up and flashing images not signed with the correct keys? The SDK from Qualcomm and other SoC vendors already provides everything that's needed, and Google lists it as something that has to be done, and presumably checked.
1
2
They could spell out a very specific process vendors need to run through to check that this works, if that's what you mean. I don't see how any software they could provide to vendors would be helpful. Testing is just flashing corrupt images or ones signed with different keys.
2
2
It's spelled out that this has to be provided, so if they're running through the list of requirements, this is something they need to do. Maybe it needs to be more explicitly spelled out that vendors need to go through the list of requirements one by one and ensure compliance?
I don't think raising minimum requirements and trying to enforce them is going to result in decent security. If vendors don't have an interest in security themselves, they're a lost cause. For a vendor that cares at all, the recommendations that aren't mandatory help them too.
1
1
In fact, those are a nice litmus test. Check which devices comply with security recommendations that aren't mandatory and use that to gauge their work on security. If they don't bother with anything not mandatory that requires any work, it's clear they'll do a bad job elsewhere.
1