Most OEMs are pretty stupid. But in a good way!
If CTS gets automatic testing of default bootrom for Qualcomm, Mediatek, Unisoc's OTPs (including detecting SoC default keys...), they'll probably check it. Issue is that it must be done not just in CTS, but also in production
Conversation
CTS runs tests via apps and adb shell within the security model of the OS. It runs on a production build of the OS with the full security model intact and is very limited in how deeply it can probe and test. It can't peak behind the curtain and enforce implementation details.
1
3
CTS is very limited in what it can test. That's why there's the VTS for testing the kernel and vendor via a special system image, rather than on the production OS. However, that's still testing functionality via public APIs, not peaking into the details of the implementation.
2
4
I use "CTS" in the broad sense. Google has CTS-like certification tests that run as root that pokes for known vendor security flaws (like MTK-SU).
With GKIs around the corner, Google not knowing about OEM implementation gets even more improbable.
1
2
How can you test low-level secure boot beyond setting it up and flashing images not signed with the correct keys? The SDK from Qualcomm and other SoC vendors already provides everything that's needed, and Google lists it as something that has to be done, and presumably checked.
1
2
They could spell out a very specific process vendors need to run through to check that this works, if that's what you mean. I don't see how any software they could provide to vendors would be helpful. Testing is just flashing corrupt images or ones signed with different keys.
2
2
And even if all of this testing is done, and done well, it still only confirms that the nominal cases work properly, it does nothing to prove that the system is actually secure. That requires inspection of the design and implementation, plus penetration testing, fuzzing, etc.
1
3
It's just not realistic to mandate and certify security. It's great to have clear documentation for how things should be done with tests for as much as possible, but achieving decent security is far beyond just complying with a list of rules / requirements and some test suites.
1
3
I disagree with this, assuming an adequate definition of "certify". If your device is, say, certified to Common Criteria EAL 5+, with an appropriate protection profile that includes AVA_VAN.5 penetration testing... it's good. It may not be invulnerable, but it's very good.
1
1
However, that level of certification testing is extremely expensive and simply isn't cost-effective for a $50 Android device.
2
1
I personally don't think it's very realistic to enforce a decent level of security through certification. Security is also an ongoing process and a certification at a single point in time misses the bigger picture, such as quality of duration of security updates provided for it.
Having formal requirements, testing and certification can certainly be very helpful and I'm not saying otherwise. I just don't think that it can actually force a company to provide decent product security if they only care about meeting the minimum requirements and nothing more.
2
1