---
# System prepended metadata

title: 'Essay 3: Quality and Evolution'

---

# Essay 3: Quality and Evolution

<!--
With key aspects of the architecture described, the third essay focuses on means to safeguard the quality and architectural integrity of the underlying system, with special empahsis on the rate of change. Aspects to take into account include:

    The overall software quality processes that apply to your system
    The key elements of the system’s continuous integration processes
    The rigor of the test processes and the role of test coverage
    Hotspot components from the past (previously changed a lot) and the future (needed for roadmap)
    The code quality, with a focus on hotspot components
    The quality culture, as evidenced in actual discussions and tests taking place in architecturally significant feature and pull requests (identify and analyze at least 10 such issues and 10 such pull requests)
    An assessment of technical debt present in the system.

If you conduct measurements, you may find that certain numbers point at poor code quality. In such cases, offer an explanation why the code ended up like it is now, assess whether trade-offs have been made, and demonstrate leadership by offering constructive proposals how to resolve the problems you found.

-->
The quality behind software systems is an interesting topic to study, as it is a very broad concept. One important factor is, for example, the quality of documentation, which plays an important role in understanding the system. Also, the quality of software itself should be assured, which is done, for example, by writing test suites or by making use of continuous integration. In this post, we discuss the libGDX project in terms of quality and evolution.

First we discuss the documentation that libGDX provides. This is followed by a further look at its test suites. Afterwards, we consider the continuous integration of the project. Finally, an overall analysis is presented of the quality of the past and future of libGDX.

## Quality of documentation
When using a framework such as libGDX, it is essential for the developer to get a deep understanding of how the framework is organized and how they should use it. libGDX has different types of documentation which are easy to find. On their website they have an entire page dedicated to referring the developer to the documentation[^libgdxdoc].
libGDX's documentation is spread over three different sources; each of these sources is targeted at the different users libGDX might have:
1. **The documentation section on their website**[^libgdxdoc].
Here they give short guides that explain to developers how to get started with libGDX; the topics include  "Set Up a Dev Env", "Generate a Project" and "Demos & Tutorials". This is also the documentation a beginning developer will stumble onto first, since they might already have experience with using Github or reading Javadoc. This page also contains some guidelines regarding reporting issues and contributing.
2. **The Javadocs**[^javadocs].
This documentation is provided at a programming level and annotates classes, methods and functions within Java itself. This documentation helps developers understand what certain functions do while they are working on code inside their project. Lots of feature-rich IDEs like Eclipse[^eclipse] and JetBrains IDEA[^intellij] enable hints that help the user with understanding the description of functions or classes that they are using. Assuming that methods and classes are well-annotated with Javadoc, this can be a very useful tool in understanding and navigating the libGDX codebase.
3. **The documentation on the Github Wiki**[^githubwiki].
The Github Wiki contains documentation that goes into more technical depth compared to the documentation on libGDX's website. libGDX themselves call this documentation their "Developer's Guide". Developers who want a more thorough understanding of libGDX and its various components can read up on it here, which is most useful to people that seek to contribute. 

It is nice to note that the overall quality of documentation is good. The documenation for beginners on the website is quite extensive and easy to follow along with the wiki on the Github page, although it might lack some architectural choices. We also analyzed the code in terms of Javadoc. We have determined that a total of 34% of production code (which excludes extensions and tests) has Javadoc attached to them. This might seem like a low percentage, but we must keep in mind that this also includes classes that override methods or implement interfaces. However, we have found numerous sparsely documented classes; the choice of when a method contains Javadoc is sadly not a consistent one. When we attempted to add Javadoc ourselves as a contribution we were told that this was "overdocumentation". This is a rule that libGDX follows that we respect, but might not entirely agree with.


## Quality of testing
<!-- - Testing:
    - Amount of testing done
    - Testing 'workflow'/'philosophy' (test-driven development or no?)
    - Minimum test coverage required?
    - Do new contributions require tests? -->

Each platform supported by libGDX has a dedicated folder for testing platform-specific code. The folder structure is shown below[^platformtests].

{{<image file="test-folder-structure.png">}}

Each of the platform-specific tests spin up their own "application" in which certain functionality is "tested"; they do not use an actual testing framework. Therefore this platform-specific code does not have actual test coverage, even though the folder is populated with many different classes and scenarios. In total the tests folder contains 828 files, some of the files are resources and others utility classes, while the majority of the content is used for scenarios. These are mainly used for manual testing the merge request[^contributing].

The cross-platform tests are structured differently:

{{<image file="maven-test-structure.png">}}

These tests are actually tested run jUnit and thus a coverage can be retrieved of the files that make up the core code of libGDX. The following is the code covered of libGDX by test coverage.

{{<image file="test-coverage.png">}}

It is clear from the test coverage that the core code of the project has not seen much testing. Only a staggering 45 of almost 1000 classes are located inside the test coverage. The actual number of classes that have a test implementation is lower, as some of these 45 classes are implemented once for an intregation test. The awful test coverage is likely a result of contributors that are not forced to fulfill any test coverage criteria for their implementation or changes to the project.

{{<image file="test-execution-time.png">}}

The positive effect of a poor test coverage is that it is extremely fast to run, as can be seen above. The libGDX project cares more about the platform-specific code than the core code. It also applies integration tests over unit tests. Which is an odd choice, as unit tests should serve as a foundation[^testing].

{{<image file="test-pyramid.png">}}

The above is the testing of a project in the ideal case: unit tests form the base, integration tests should be less and manual testing should rarely be used and even avoided where possible. Not only does the project not follow the ideal case of the testing pyramid, it follows an anti-pattern of testing as seen below.

{{<image file="test-ice-cream-cone.png">}}

The anti-pattern fits the project perfectly. It relies heavily on manual testing and integration testing and very little on unit tests. This anti-pattern follows when unit testing is not embraced from the start of the project[^testing]. Which supports the claim that proper testing was not applied from the start.





## Quality of continuous integration
The libGDX project has continuous integration in place. The pipeline that is used, is shown below.

{{<image file="continuous-integration.png">}}

One thing that becomes clear after reviewing the pipeline, is that the continous integration is mainly focused on releases of the framework. There is room for improvement in terms of quality checking.

One point that would be a good addition for the continuous integration process, is to run tests. This gives the opportunity to see if the codebase still functions as intended after the change.

Another possible addition is that of Checkstyle [^checkstyle]. This would be a good addition, in order to keep an eye on the general style of the codebase.

## Analysis of the past and the future

<!--
- Analysis of quality and history (nice graphs)
    - past and future
-->

We have analyzed the codebase of libGDX by inspecting the repository and calculating common code quality metrics over the code itself. This can give insights into the evolution of the codebase and its development process.

The analysis of the git repository indicates the development intensity for each file. We iterated over the commits from the master branch on March 16, 2021 (commit `95fc6c7`[^latestcommit]) all the way to the first commit on March 6, 2010 (commit `69c97c5`[^firstcommit]). For each file we recorded the number of lines added and removed and calculated the size of the changes relative to the length of the file at that commit. We also counted how often a file was referenced by a commit. For additions, we took the length of the file after the commit, while for removals we took the length before the commit. This caps the relative change to 100%.

| Filename | Times referenced |
|---|---:|
|tests/gdx-tests/.../tests/utils/GdxTests.java|172|
|gdx/.../graphics/BitmapFont.java|150|
|tests/gdx-tests-lwjgl/.../tests/lwjgl/LwjglDebugStarter.java|135|
|gdx/.../graphics/g2d/SpriteBatch.java|128|
|gdx/.../scenes/scene2d/ui/TextField.java|127|
|gdx/.../scenes/scene2d/ui/ScrollPane.java|114|
|gdx-backend-android/.../backends/android/AndroidGraphics.java|112|


As can be seen in the above table, the file that was commited to the most was `GdxTests`. Instead of using a common testing framework, libGDX has its own framework and any test classes that should be run are registered in this class. The list contains 210 test classes, with one disabled via a comment, which largely explains the 172 commits touching this file.

The next file is for a non-test class `BitmapFont`, referenced by 150 commits. Font handling is a large part of libGDX and there are still outstanding issues[^bitmapfontoobissue][^bitmapfontidentglyphissue], so it is likely that more changes will follow.

|Filename|Cumul. rel. changes|
|---|---:|
|gdx-backend-android/.../backends/android/AndroidGraphics.java|18.9|
|gdx-backend-android/.../backends/android/AndroidApplication.java|16.0|
|tests/gdx-tests-lwjgl/.../tests/lwjgl/LwjglDebugStarter.java|16.0|
|backends/gdx-backend-lwjgl/.../backends/lwjgl/LwjglInput.java|13.0|
|gdx-backend-lwjgl/.../backends/desktop/LwjglGraphics.java|12.6|
|gdx/.../graphics/BitmapFont.java|12.5|
|backends/gdx-backend-lwjgl/.../backends/lwjgl/LwjglApplication.java|12.5|

The classes that saw the most intense changes, where commits often changed large portions of the file, are `AndroidGraphics` and `AndroidApplication`. This should come to no surprise, since Android is one of libGDX's major target platform. Both files have, in total, many times more changed lines than they are long and can therefore be seen as the most intensely edited files in the project.

Although there is a sharp dropoff in intensity after these two files, files that come close are `LwjglInput`, `LwjglGraphics` and `LwjglApplication`, mirroring the patterns seen for the Android backend. It is safe to conclude that the `Graphics` and `Application` interfaces are the most active parts of libGDX and that they will continue to be so in the future.

<!-- Unsurprisingly, `BitmapFont` also scores high in this metric, although nearly half as much as the `Graphics` and `Application` classes. -->

A special case here is `LwjglDebugStarter`. The developers use this class to quickly start certain test cases and because many commits touch on different parts of libGDX, which test is started and with which parameters this is changes often. The file itself is rather short, but it changes often and on many lines, which gives it a high development intensity score.

### Code quality

Code quality metrics were calculated over every git tag. We will look at a few key metrics.

{{<image file="graph-file-count-over-time.png">}}


In the first two years of development, there was a rapid increase in the number of .java files in the repository. After that, it started plateauing, which hints towards feature stability as no additional modules are added. With libGDX 1.0.0, released in April 2014, the codebase was cleaned up and demos were moved to separate repositories[^libgdxdemos], causing a noticable drop. 

{{<image file="graph-test-production-ratio.png">}}

The test-to-production ratio is also climbing, but at a much faster rate. It should be noted that automatic detection of test classes is error-prone and the percentage shown here may not reflect the actual proportion (it is likely the ratio is greater). However, it shows that the project is moving towards more standard methods of testing, which are easier for the metrics software to detect.

{{<image file="graph-cyclomatic-comp-1.png">}}
{{<image file="graph-cyclomatic-comp-separate.png">}}

The mean cyclomatic complexity of the project is fairly high. We took the average cyclomatic complexity of all methods and later some methods, depending on a lower bound. Because idiomatic Java uses many getter and setter methods, we tried to filter these out by only including methods from a certain complexity and up.

The global cyclomatic complexity is dropping, which is likely due to the addition of more getters and setters. When these are excluded, the overall complexity is still decreasing, as can be seen by the yellow, upside-down triangle "cycl. cxty >= 2" line. However, the larger complexities remain worryingly high, with methods surpassing 20 and one even reaching 170. For the extremely complex methods, with a complexity of at least 10, the median is at 14. These complexities may severely impact the testability and maintainability of these components.

{{<image file="graph-method-lines-of-code.png">}}
{{<image file="graph-file-lines-of-code.png">}}

The number of lines of code in each method is, on average, decreasing, likely also due to the growing numbers of getters and setters. Meanwhile, files are growing in size, caused by the same phenomenon as well as general additions to the codebase.


<!--
{{<image file="graph-percentage-of-commented-lines.png">}}


Finally, we look at the number of commented lines in the codebase. This includes both Javadoc and regular comments, both of which are important for the evolution of the project itself: Javadoc documents the public interface, while regular comments describe the details of the system. The moderate ratio indicates that the project is fairly well-commented, even though our experience felt otherwise. The ratio remains roughly the same over time, only decreasing slightly, which means that the documentation is kept up to date good enough to track the evolution of the codebase.
-->

These metrics show that the codebase is fairly healthy, save for some complexity issues. It is growing at a moderate pace and maintenance is keeping up. Analysis of the intensity of development shows that there are few classes that form the core of the project. We also showed that the project is using strange testing methodologies, but also that this is improving.

[^libgdxdoc]: libGDX. (n.d.). Source & Documentation. Retrieved March 19, 2021, from https://libgdx.com/dev/

[^githubwiki]: libGDX. (n. d.). Wiki Home. Retrieved March 19, 2021, from https://github.com/libgdx/libgdx/wiki

[^javadocs]: libGDX. (n. d.). Javadoc Overview (libgdx API). Retrieved March 19, 2021, from https://libgdx.badlogicgames.com/ci/nightlies/docs/api/

[^eclipse]: Eclipse Foundation. (n.d.). Eclipse homepage. Retrieved March 22, 2021, from https://www.eclipse.org/ 

[^intellij]: JetBrains. (n.d.). Intellij IDEA. Retrieved March 22, 2021, from https://www.jetbrains.com/idea/
[^platformtests]: libGDX. (n.d.). Platform-specific test. Retrieved March 18, 2021, from https://github.com/libgdx/libgdx/tree/master/tests

[^contributing]: libGDX. (n.d.). Contributing. Retrieved March 18, 2021, from https://libgdx.com/dev/contributing/

[^testing]: A. Bowles. (2017). Willow Tree. Retrieved March 18, 2021, from 
https://willowtreeapps.com/ideas/test-early-test-often-automation-testing-with-the-test-pyramid

[^bitmapfontoobissue]: geertjan-apptornado. (2016). BitmapFont draws out of bounds when first character has negative xoffset. Retrieved March 20, 2021, from https://github.com/libgdx/libgdx/issues/3755

[^bitmapfontidentglyphissue]: J. Piegsa. (2015). Problems with bitmap fonts with pixel identical glyphs. Retrieved March 20, 2021, from https://github.com/libgdx/libgdx/issues/

[^libgdxdemos]: M. Zechner. (2014). removed demos from main repo, jippie. Retrieved March 22, 2021, from https://github.com/libgdx/libgdx/commit/940fa1a22d4a63da86558cf4cd0614c35c25a039

[^latestcommit]: obigu. (2021). Fix #6454 (#6455). Retrieved March 22, 2021, from https://github.com/libgdx/libgdx/commit/95fc6c7

[^firstcommit]: (no author) (2011). Initial directory structure. Retrieved March 22, 2021, from https://github.com/libgdx/libgdx/commit/69c97c5

[^checkstyle]: checkstyle – Checkstyle 8.41. (n.d.). Checkstyle. Retrieved March 22, 2021, from https://checkstyle.sourceforge.io/