I recently was asked to help on a project where the team was trying to get the test coverage for e2e tests also known as component tests. I initially tried to say that component tests run on the actual application process and not in the context of a test/source file. So the app is a black box to the running tests and can't be instrumented for getting coverage.
I was wrong!
Running a java app: basics
The way a Java app runs is quite similar to a NodeJS app. A bundle in Java world is called a jar. You can run an app by just running the jar file which contains built classes and the app dependencies all together.
java -jar <application-code>.jar
That's it, the app server would start and you can run your component/e2e tests using any framework of your liking.
For simplicity I'd choose running manual curls and seeing if the api responds correctly or not. Aim here is to not solve specifically for a test framework but seeing if the app can be instrumented at runtime which would then enable us to get coverage for any framework.
JaCoCo
This is the library which is used by all Java programmers to get coverage for their unit tests. It stands for Java Code Coverage. (I could've never guessed it in my dreams had I not googled it). Usually people use it to generate the coverage for Unit tests but it also supports runtime coverage.
Runtime code coverage
This tells you what code is running once the app starts, which code paths are getting executed and how many times. This is done using a Java Instrumentation API which allows you to write a program which can add some instrumentation changes to the generated bytecode. It is commonly used to:
- profile apps, find the hot methods to see what's eating up your CPUs and taking a long time
- coverage - the current use-case at hand
- monitoring
All these methods require you to use a Java Agent which is attached to the code that you want to run. Below is an example on how you'd attach an agent to profile/monitor your code
java -javaagent:<your-monitoring-code>.jar -jar <application-code>.jar
Good news is JaCoCo already ships a java agent jar which can instrument the code at runtime! Wow, framework agnostic test coverage is here :)
I've created a sample repo to test this out, it's present here on github.com/ankeetmaini/jacoco-e2e
step 1 - build the jar
Build your project and get the jar file ready. To build the sample project linked above, clone and run the following
./gradlew build
The jar file would be created at build/libs/rest-service-0.0.1-SNAPSHOT.jar path.
step 2 - run the jar with java agent
The trick here is to attach the java agent before starting the application code. The java agent jar would add some bytecode to the app jar for instrumenting the code.
java -javaagent:jars/org.jacoco.agent-0.8.6-runtime.jar -jar build/libs/rest-service-0.0.1-SNAPSHOT.jar
I used the
jars/prefix as I stored it inside ajarsfolder in my codebase. Any valid relative path should work.
step 3 - run your tests
Now is the time to go ballistic and run all your e2e tests in the framework of your choice, or run some curls, or make some requests via Postman.
For simplicity's sake I'm going to run some curls to test the coverage.
curl localhost:8080/greeting
{"id":1,"content":"Hello, World!"}
Once you're done, shut down the process and a jacoco.exec file should get created at your current working directory.
jacoco.exec file contains the codepath and number of times all the classes and methods were run. This is the file that contains raw coverage, but there's just one problem that it's not human readable.
step 4 - generate pretty HTML reports
JaCoCo gives another jar which takes in the .exec file and converts it to a report.
java -jar jars/org.jacoco.cli-0.8.6-nodeps.jar report jacoco.exec --classfiles=build/classes --html coverage
Make sure you point the folder which contains the class files which were generated while building the file.
Closing
The above post takes into account where the raw coverage data is dumped in one go when the JVM exits, but there are other options also where you can:
- use a client to connect to the agent and get the coverage dump on demand
- provide a server which accepts incoming requests where java agent can dump
- imperatively using the CLI jar to get the dump from time to time.
Please see this and the command line interface documentation for more information.