Of course tests can't check if the implementation is secure, only that it exhibits the expected functional behavior. There are non-functional requirements, too. Device makers effectively self-certify that they pass the tests and meet the untestable requirements.
Conversation
It's not possible to write a test suite or list of requirements to impose upon other companies to force their software and hardware to be reasonably secure. They need to care about security themselves and put resources into it, beyond just complying with bare minimum standards.
2
4
Most OEMs are pretty stupid. But in a good way!
If CTS gets automatic testing of default bootrom for Qualcomm, Mediatek, Unisoc's OTPs (including detecting SoC default keys...), they'll probably check it. Issue is that it must be done not just in CTS, but also in production
1
2
CTS runs tests via apps and adb shell within the security model of the OS. It runs on a production build of the OS with the full security model intact and is very limited in how deeply it can probe and test. It can't peak behind the curtain and enforce implementation details.
1
3
CTS is very limited in what it can test. That's why there's the VTS for testing the kernel and vendor via a special system image, rather than on the production OS. However, that's still testing functionality via public APIs, not peaking into the details of the implementation.
2
4
I use "CTS" in the broad sense. Google has CTS-like certification tests that run as root that pokes for known vendor security flaws (like MTK-SU).
With GKIs around the corner, Google not knowing about OEM implementation gets even more improbable.
1
2
How can you test low-level secure boot beyond setting it up and flashing images not signed with the correct keys? The SDK from Qualcomm and other SoC vendors already provides everything that's needed, and Google lists it as something that has to be done, and presumably checked.
1
2
They could spell out a very specific process vendors need to run through to check that this works, if that's what you mean. I don't see how any software they could provide to vendors would be helpful. Testing is just flashing corrupt images or ones signed with different keys.
2
2
And even if all of this testing is done, and done well, it still only confirms that the nominal cases work properly, it does nothing to prove that the system is actually secure. That requires inspection of the design and implementation, plus penetration testing, fuzzing, etc.
1
3
It's just not realistic to mandate and certify security. It's great to have clear documentation for how things should be done with tests for as much as possible, but achieving decent security is far beyond just complying with a list of rules / requirements and some test suites.
I disagree with this, assuming an adequate definition of "certify". If your device is, say, certified to Common Criteria EAL 5+, with an appropriate protection profile that includes AVA_VAN.5 penetration testing... it's good. It may not be invulnerable, but it's very good.
1
1
However, that level of certification testing is extremely expensive and simply isn't cost-effective for a $50 Android device.
2
1
Show replies