Android CTS: Google’s Test Suite for Android ecosystem compatibility

Testing applications is an essential task in software development and ongoing maintenance. Developing an extensive custom Android ROM is no exception, but has its own unique challenges. Good news: Developers can use standardized tests to ensure the compatibility of custom developments and their compatibility with Android best practices.

Google offers various test suites to maintain consistency across the Android ecosystem, including the compatibility test suite (called Google CTS or Android CTS), which ensures adherenc to Google’s standards.

This article will detail Android CTS, its benefits, and highlight situations where it’s needed. While not always required, Google CTS can be beneficial in some cases. As Android platform experts, we'll outline different scenarios where testing is or is not sensible and discuss the impact of Google CTS on product development.

What is CTS (Compatibility Test Suite)?

CTS is a set of automatic tests that ensures the correct implementation of Android APIs and core platform features. Google CTS tests are derived from Google's Compatibility Definition Document (CDD); passing them is essential for obtaining a Google Mobile Services (GMS) certification.

What is the difference between CTS and VTS?

Android VTS tests the kernel and HAL, the hardware implementation, whereas CTS focuses on APIs and core platform features, the app ecosystem implementation, for compatibility with Google standards. Then there is GTS Google testing for Google services compatibility, granting GMS certification once passed.

Why invest time in device compatibility testing: three key perspectives

Many developers and manufacturers use Google CTS to ensure devices comply with Google guidelines and obtain GMS certification, which is essential for distributing devices with popular Google apps, like Play Store, YouTube, or Gmail.

GMS is, therefore, somewhat mandatory for consumer devices like smartphones and tablets, as these services might directly affect device sales. In the context of Android automotive, CTS is also useful since Android Automotive OS is expected to show the user's favorite Google apps.

However, not all Android devices need Google services. At emteria, we don't aim for GMS certification. While it is expected to have Google’s proprietary services on consumer devices, there is no real benefit or requirement for industrial and embedded solutions. These devices are mostly task-specific, running a single or set-off custom applications, hence Google’s proprietary services are of no use. If anything, they pose a security risk and inflate the OS unnecessarily. 

A lean OS without bloatware is ideal for customized products. Still, xTS tools like Google CTS are valuable for product development.

Compatibility: A developer's perspective

Android CTS is a test suite that contains numerous unit and functional test cases. As such, it can help meet quality standards by uncovering bugs and other issues, such as side effects. On top of that, Android CTS tests offer decent test coverage by default. Android CTS is free to use, so developers are encouraged to use the Android CTS source code.

In addition, Android CTS includes performance test cases useful as benchmarks during development. With its comprehensive test cases and overall test coverage, it can be beneficial to use in partial testing scenarios, like performance testing.

Our blogs on Android TEE and Android Verified Boot show the importance of security in Android OS. Promptly addressing vulnerabilities is crucial when building solutions for customers. While Android CTS provides security tests, maintaining compatibility with the ecosystem and adhering to global standards across different devices generally helps— following development standards helps with identifying and speeding up recent bugs and vulnerability fixes.

Compatibility: A users perspective

Unique requirements often drive custom Android solutions. Whether or not these requirements originate from user preferences or technical requirements, users expect a consistent experience. Implementing these requirements while adhering to ecosystem standards is often crucial for end-user acceptance. Android CTS is one way to ensure consistency despite customizations.

Compatibility: A professional product builder’s perspective

For consumer devices, compliance with Google's standards is useful, but for professional devices, it is often unnecessary.

For example, imagine running a specific application in kiosk mode on an Android custom ROM. In this case, the effort of going through Google compliance testing is over the top. When building task-specific devices (also called dedicated devices), it is likely not even possible to comply. Why? The hardware running the OS is likely built on a custom or off-the-shelf board. Not all hardware boards are compatible with Google’s ideas of proper hardware.

Keep in mind that those tests are designed for building B2C consumer products mainly consumer phones and tablets, not SBCs, compute modules or custom board designs.

Focusing on app usability and product design is more beneficial than meeting compliance standards, especially if Google services aren't needed. Following the developer’s perspective, Android CTS still has its value for partial testing due to its extensive test library. While mainly for consumer devices, professional product builders can use parts of Android CTS to ensure custom solutions are correctly implemented. Conclusively, the specific use case should guide the compliance effort.

The three key perspectives

Developers and manufacturers can benefit from Android CTS but should assess its necessity based on their use cases. General compatibility may not always improve development and maintenance. Ensuring a consistent user experience is a good thing, but high development costs due to compliance are not. Keep in mind that while apps like YouTube may be essential to end users, GMS services, like the Play Store, have alternative solutions tailored to industrial and embedded use cases.

How to run a CTS Google test case?

To run an Android CTS test case, first, set up the test environment by installing Android CTS on your host machine. Connect the devices under test, then execute the CTS commands through the command-line interface to initiate the test suite. Results are generated for analysis.

How to set up CTS on devices?

Let’s look into the technical details of how to set up and run Android CTS. There are specific requirements for devices under test (DUT): Hardware and Software.

Device specs

Android's flexibility powers many devices, making it challenging to address all hardware and setup requirements. The AOSP documentation specifies requirements for the test environment if the DUT has features like:

1. Bluetooth
2. Ultra-wideband
3. Cameras
4. GPS/GNSS
5. Wi-Fi, IPv6 and Wi-Fi RTT

Google sets precise setup criteria to guarantee compatibility and accurate test results. Feel free to look up the exact setup specifications per your device's features.

Software setup

Setting up the test environment on your device is more linear. There are dependencies for running Android CTS:

1. Correct JDK version: For Android 11 and later, use OpenJDK11. For earlier Android versions, refer to the official docs.
2. Python: Android CTS requires the use of virtualenv.

1. Get the appropriate Android CTS version from AOSP downloads
2. Unzip the file; Make sure NOT to rename it.

1. Install/Verify Android Asset Packaging Tool (AAPT)
2. Install/Verify Android Debug Bridge (adb)

1. Enable USB Debugging
2. Enable Developer Mode
3. Disable Screen Timeout/Lock

With these steps, you have successfully set up the test environment and are ready to run CTS on your device.

Please note that we have intentionally skipped over some details to give you a more general approach. The official docs provide additional setup and test instructions appropriate for your environment.  

Get started: Run a CTS test case

After ensuring the DUT meets the CTS Android testing requirements, we can continue with a first test run.

Position the device on a stable surface to avoid movement, adjusting as needed based on device features like cameras or accelerometers. Proper setup is crucial for consistent results. Navigate to the home screen on our DUT.

Next, launch the CTS console to run the first test plan:

1. Navigate to android-cts/tools
2. Invoke Android CTS console: ./cts-tradefed

The Google CTS console comes with various features. We can continue running the entire test library with the run command:

3. Execute tests: run cts

Running the entire library is time-consuming and cumbersome for resolving test failures. If using CTS Android as a general testing tool, rather than for Google compliance, focus rather on a subset of CTS commands and option flags.

Other than that there are some useful features to enhance CTS Android testing: ⬇️

Test distribution using test sharding

Test sharding is helpful when dealing with extended runtimes, e.g., running the entire CTS or larger CTS modules. Sharding helps reduce test runtime by distributing CTS tests across multiple devices.

Be aware that if the number of test cases is low, some devices might not get a test assigned. This feature can decrease test time when testing multiple identical builds or testing builds on slightly different hardware.  

Targeted approach using test plans

Repeating successful tests isn't helpful for debugging. For this reason, CTS supports test plans. Test plans are synonymous with collections and test libraries, allowing for a more targeted approach. These can be run in the CTS console using the run cts --plan flag, and custom plans can be defined.

Dealing with external dependencies: CTS Verifier

Standard CTS lacks coverage for features that require manual input and external information. However, using external APIs, a supplemental test suite (the Android CTS Verifier) is specifically designed for this interaction. With CTS Verifier, we can test features such as touchscreen interactions in an automated and repeatable way.

How to interpret and use results?

After the test is executed, which can take several minutes or even hours, CTS generates a result report. This report is saved in the CTS directory' s/results.

>> $  ~/android-cts/results/

CTS test results are stored in a zip archive named after the test start time, containing a test_result.html file with details on test modules, case numbers, and their status.

The entire compressed directory is required to pursue an official Android build certification.

While it is nice to see that almost all tests pass, it is not necessarily needed. As we have outlined earlier, when developing custom applications and devices besides smartphones and tablets, we are likely more interested in a specific subsets of Android CTS. As developers, we naturally concentrate on instances when things fail to function as expected. So, what steps should we take from there?

How to investigate failures

When using Google CTS in development, the test results can be used to initiate debugging. While CTS test reports lack detailed debugging info, here are some starting points.

Exclude host interference or CTS issues

Rerun failed test modules. Reruns are set as parameters prior to the initial testing or carried out manually using the run retry command. Reproducing persistent failures on a different host machine and using a clean/earlier version of CTS to exclude any side effects is another approach.

Verify host, CTS, and DUT setup

CTS tests have specific requirements for the testing environment and devices. Verify the correct software versions, setup, and CTS use. Stick to the official setup as CTS has specific testing expectations.

Examine root causes

Once anomalies are excluded and setup is verified, device and host logs can indicate the origin of the test failure. Tools like Logcat can help.

Compatibility despite customization: Leverage Android CTS

Google CTS is a helpful tool for Android development. While it is mandatory to get certification for Google's proprietary services, most industry and application-specific devices do not use these services. They merely need parts of the test library to check for general compatibility. Adhering to Google’s requirements is optional.

CTS Google aids developers in specific testing scenarios, streamlining test processes and ensuring correct functionality implementations.

Great products come from grand visions. At emteria, we are experts in the customization of Android operating systems. We know what it takes to develop a rich, robust Android OS with the flexibility your business needs. As a leading company in customization and maintenance, we provide the customer with advice and solutions they need to make their visions a reality. Reach out to us and discover our product and service portfolio!