This is a story about how we at SpareBank 1 Utvikling over the years have always sought a fast feedback loop when we code, test and deploy our solutions. The article describes how we believe the use of Fakes today (2024) has contributed to a faster feedback loop, and pushed a more domain-driven design with hexagonal architecture, also known as ports and adapters.

History and background

Feedback-junkies

In SpareBank 1 Utvikling, more and more of us have become dependent on quick feedback loops. But how did we end up there?

When we experience a slow feedback loop, the result is that we create more code before it is deployed to production. As more code is deployed, it becomes harder to test, find errors, make changes, and the overall quality decreases. The same applies to developer motivation.

We appreciate small changes, and we want to see the results as quickly as possible — both locally and in production. With short iterations we can modify code and direction faster, leading to a better product over time and increased learning.

In recent years, we have made significant changes in this area.

Monoliths

Not long ago, the entire codebase was in larger monoliths. 

Code was written with a few unit tests here and there, where deemed necessary. Then it was manually tested during longer test periods. It was also not uncommon to have lengthy end-to-end tests in the testing environments — written by dedicated testers, not the developers themselves. And when everything was ready, it all went into production simultaneously. Teams worked across the monolith, navigating its complexitie.

Microservices

The monolith was then divided into microservices. These microservices were distributed across different teams, making it easier to manage the code. Cognitive load decreased, and the microservices could be tested individually, reducing the need for both end-to-end testing and dedicated testers. Now, developers could test more themselves, faster.

Integration tests and docker containers

With microservices architecture, running integration tests became easier. And everything could be executed locally.

Here’s how we did it:

1. App Initialization with Custom Test Config: We started our apps with their own test configurations using shell scripts.
2. Simulating Backend Systems with Docker Containers:
   We simulated backend systems using Docker containers, launched via docker-compose.
3. Local Testing Environment:
   The code in this local testing environment was tested by our own integration tests within dedicated test modules.

As a result, we became less reliant on external test environments. This increased our pace of change, and we deployed more frequently.

But we were not entirely satisfied. 

• Starting backend systems with docker-compose was slow. 
• Integration tests were a separate module outside the app. 
• Code changes required building with Maven, stopping and starting Docker, and so on.

The feedback loop was still too long.

The Revolution: Local App IT-testing without docker-compose

The App IT-test

We adopted SpringBootTest and created our own App IT-tests (application integration tests) that reside alongside unit tests in the test suite. By annotating classes with SpringBootTest, the app starts up in a test context when the tests run. We replaced docker-compose with technologies like WireMock and TestContainers. These tests made HTTP calls to the app using RestAssured. Everything was initiated simultaneously when running the test.

Example: testing the whole application through the App IT-test

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@WireMockTest //startup for wiremock
class AccountsAppIT {

    @Inject
    lateinit var accountsRepository: AccountsRepoDb

    //Setup and boiler plate in helper classes for PostgreSQLContainer
    //and setting datasource in Spring during startup
    companion object : PostgresSpringEnabled {
        @JvmStatic
        @DynamicPropertySource
        fun initializeDb(registry: DynamicPropertyRegistry) {
            initDataSourceInSpring(
                    registry,
                    dataSourceName = "db",
                    schemaOwner = "my_owner",
                    appUser = "my_user"
            )
        }
    }

    @BeforeAll
    fun setup() {
        RestAssured.port = port
        RestAssured.baseURI = "http://localhost/personal/banking"

        accountRepository.cleanTables()
    }

    @Test
    fun `Should return accounts with status 200 OK`() {
        storeAccountsInDb()
        stubAuthenticateUserApi()

        Given {
            accept("application/json")
        } When {
            get("/accounts")
        } Then {
            statusCode(HttpStatus.OK.value())
            body("accounts.size()", CoreMatchers.`is`(2))
        }
    }

    private fun storeAccountsInDb() {
        val accounts: List<Account> = listOf(
                Account.createValidAccount().copy(accountNumber = "123"),
                Account.createValidAccount().copy(accountNumber = "456"),
        )

        accountsRepository.store(accounts)
    }

    private fun stubAuthenticateUserApi() {
        WireMock.stubFor(
                WireMock.get(WireMock.urlEqualTo("/authenticate-user"))
                    .willReturn(
                        WireMock.aResponse()
                          .withStatus(HttpStatus.OK.value())
                          .withHeader("ContentType","application/json")
                          .withBodyFile("authenticate-user-response.json")
                  )
        )
    }
}

This was fantastic! Now you could sit in your IDE and test the entire app locally in under a minute, without connecting to external systems or dealing with docker-compose, scripts, and containers. Hooray!

Hooray?

For every small change, we tested the entire app, from the top layer down to all integrations, simultaneously. Everything was verified in one place — in the new App IT-test at the HTTP layer. We saw that everything was interconnected — a sense of “it all works.”

But is it truly reassuring to always test everything simultaneously? Is checking that everything works the sole reason for writing tests?

As more backend systems were added, the App IT-test became increasingly busy. It also developed a strong relationship with the integration to backend systems and required significant setup to function. Keeping track of everything became challenging.

The App IT-test only cared about the app running and whether the input/output from HTTP calls was correct. In practice, this meant we could write code like a large, unreadable spaghetti in the layers below, as long as the input/output was correct at the HTTP layer.

Further challenges arose:

• As we added more backend systems, the feedback loop slowed down. The App IT-tests had to set up simulations for all these backend systems (including databases, APIs, Redis, and Kafka).
• Simulating all these backend systems led to an abundance of response/request test files for various test cases from the backend systems. These files contained implementation details specific to the app, which the HTTP-level tests didn’t necessarily need to concern themselves with. This complexity made refactoring and changes difficult. Essentially, the App IT-tests became closely tied to all integrations at once.

Are App IT-tests a replacement for regular unit tests?

We found that much of the traditional Domain-Driven Design mindset disappeared when relying on App IT-tests, which simultaneously test all integrations. Developers’ focus shifted toward merely verifying that the HTTP layer delivered as expected. After all, didn’t the App IT-tests cover everything?

We heard statements like “the apps are so small that testing only the top layer is sufficient” and “we’ll add more unit tests later if the codebase grows.”

The result? Complex apps with technical debt, including test debt, became challenging to manage. And once again, the feedback loop was too long.

The Solution: Fakes and focus on the domain

The technical debt reached a critical point in some of our most crucial apps, and we needed to take action. We brought in test-and-Kotlin guru Anders Sveen to help us address the issue concretely.

He advised us to focus on domain-driven development, a fast feedback loop, and simple refactoring. We should think in rapid change cycles, making smaller changes at a time than before. Pair programming and test-driven development (TDD) were encouraged. Focus less on integrations, as well as adopting Fakes.

What is a Fake?

Fakes is a technique from Martin Fowler’s Test Doubles to decouple from external dependencies during testing. Let me explain further:

• Fakes are a type of test double that replaces the actual integration classes, not the backend systems, with custom implementations solely for testing purposes.
• When the tests run, the code never reaches the actual backend systems or the usual simulation of those systems. Instead, we get a different representation of the integration classes.
• Because the purpose of Fakes is to simulate the real system, the test usually doesn’t require special setup.
• Why use Fakes? Because it allows us to test integrations separately from the domain layer while effectively “faking” away the integrations. This concept provides “separation of concerns”.
• The result? It positively impacts the code, design, and development speed.

Example: AccountRepoDb that communicates with a database using standard JDBC and DataSource 

This class is an implementation of the IAccountRepo interface. By using Fakes, we can simulate this integration class without involving the actual database.

interface IAccountsRepo {
    getAccounts(userId: UserID): List<Account>
}

@Component
class AccountRepoDb(private val jdbcTemplate: NamedParameterJdbcTemplate) 
: IAccountRepo {        
    override fun getAccounts(userId: UserID): List<Account> {
        return jdbcTemplate.query(""" SELECT ....

Let’s create an AccountRepoFake that also implements the IAccountRepo interface. However, AccountRepoFake will use only a HashMap for reading and writing. This is what the tests should use instead of AccountRepoDb.

@Component
@Primary //primary tells Spring to choose this implementation during tests
class AccountRepoFake: IAccountRepo {
    private val accountsMap = mutableMapOf<String, List<Account>>()

    override fun getAccounts(userId: UserId): List<Account> {
        return accountsMap.get(getKey(userId.id))
    }

    //needed to add accounts, but is not part of interface
    fun addAccounts(userId: UserId, accounts: List<Accounts>) {
        accountsMap[userId] = accounts
    }
}

In the domain layer, the AccountService takes an IAccountRepo in its constructor. It then calls accountRepo.getAccounts() to retrieve accounts. During testing, AccountRepoFake is used, while at runtime, AccountRepoDb is employed. This design ensures that the domain layer remains decoupled from the implementation in a classic interface-oriented manner.

@Component
class AccountService(private val accountRepo: IAccountRepo) {
    fun getAccounts(userId: UserId): List<Account> {
        val accounts = accountRepo.getAccounts(userId)
        //code to call to other services goes here..
        return accounts
    }
}

AccountsAppIT, that run tests against the REST-layer, injects AccountRepoFake so that is chosen as the implementation during test execution in Spring. 

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
class AccountsAppIT {
    @Inject
    private lateinit var accountRepoFake: AccountRepoFake

    @BeforeAll
    fun setup() {
        RestAssured.port = port
        RestAssured.baseURI = "http://localhost/personal/banking"
    }

    @Test
    fun `Should return accounts with status 200 OK`() {
        addTestAccountsToFake()
        Given {
            accept("application/json")
        } When {
            get("/accounts")
        } Then {
            statusCode(HttpStatus.OK.value())
            body("accounts.size()", CoreMatchers.`is`(2))
        }
    }

    private fun addTestAccountsToFake() {
        val accounts: List<Account> = listOf(
                Account.createValidAccount().copy(accountNumber = "123"),
                Account.createValidAccount().copy(accountNumber = "456"),
        )
        accountRepoFake.addAccounts(UserId("123"), accounts)
    }
}

//Object Mother. Kotlin-file with sharable functions
fun Account.Companion.createValidAccount(): Account {
    return Account(
            accountNumber = "1234 12 123456",
            name = "Test deposit account",
            ...
}

In summary:

• When the app runs in runtime scope, it uses AccountRepoDb, which communicates with the database.
• When the app runs in test scope, it employs AccountRepoFake, which utilizes a hashmap.
• The domain code itself only interacts with the IAccountRepo interface. The integration with the database remains completely decoupled for other tests.
• We create reusable objects using the Object Mother pattern, which are shared across tests and serve as input for Fakes. This simplifies refactoring and changes.

The significant difference from before is that the App IT-test no longer tests database logic. The App IT-test has no knowledge of the implementation details of IAccountRepo; it only concerns itself with domain objects going in and out of AccountRepoFake. This separation aligns with our desired approach. Tests at the HTTP level should not be aware of the database in the lower layers; they should solely interact with the domain layer.

Isolated integration tests for the integrations

App IT-tests no longer directly test integration with the database. Instead, all tests now interact only with Fakes for the integration classes, resulting in significantly faster execution. But how do we test the actual integrations themselves?

Integration tests against backend systems are now exclusively moved from testing the entire app at the top layer to the integration layer. Here, we simulate the backend system that the integration interacts with, using tools like TestContainers or Wiremock, and test the integration with dedicated IT-tests. However, now there is one integration test per integration, rather than a single test covering all integrations as previously done in the App IT-test. This approach is known as separation of concerns.

Example: Integration test of database queries with AccountRepoDbIT

AccountRepoDbIT starts a PostgreSQLTestContainer that is populated with data. It provides a Datasource that can be used in AccountRepoDb, allowing us to run tests directly against the PostgreSQL database. The PostgreSQLTestContainer starts up in a matter of seconds and simulates a PostgreSQL database exactly as desired. This way, we can thoroughly test the integration with the database, completely isolated from the rest of the application. Similarly, we test external APIs using Wiremock.

Previously, the App IT-tests at the HTTP level were responsible for setting up this database configuration. Additionally, they handled other backend system setups. We had AppIT tests that simultaneously started Oracle, Redis, Kafka, and Wiremock. However, this setup was not appropriate at the HTTP level. Now, the App IT-tests exclusively interact with Fakes and domain classes.

Fakes drives us against better domain driven design

We create separate interfaces specifically for testing purposes when using Fakes. You might find it odd that we create interfaces solely to replace implementations with fakes. After all, these interface classes are only present in the implementation code for testing purposes. Of course, this approach can be subject to discussion.

Rest assured, we’ve put a lot of thought into this process.

The domain code will now only interact with IAccountRepo. This means that the domain code is completely decoupled from the integration in AccountRepoDB and only deals with domain objects, which we consider essential in good domain-driven design.

In essence, this shields the code in the integration classes entirely from the domain layer. However, it also hides the logic, as we have a completely separate implementation in AccountRepoFake. If important business logic resides in the integration classes, we view it as a code smell. Integration classes should have minimal logic and focus on CRUD (create, read, update, delete) operations. Business logic belongs in the domain layer. This approach also simplifies maintenance and testing.

We discover this when creating Fakes, as Fakes are meant to replace the integration classes. Business logic should never be placed in a Fake. As a result the business logic in integration classes is isolated and moved to the domain layer, and tested with unit tests there. The introduction of Fakes often leads to beneficial refactoring!

In this way, we find that using Fakes actually drives us to create better domain-driven design, even though it wasn’t the initial goal. Anders Sveen calls this “Testing Through The Domain”.

Hexagonal architecture (ports and adapters)

The integrations were now developed and tested in isolation, with a strong focus on the domain layer, aided by Fakes. Surprisingly, this unintentionally led us to create a Hexagonal architecture.

Fake service classes in the domain-layer from the HTTP-layer

In a Hexagonal architecture, even the HTTP layer is considered an integration point. Here, we have the option to create Fakes for service classes in the domain-layer, effectively decoupling the HTTP layer entirely from the domain layer. 

Consequently, our AppIT tests now interact solely with an AccountServiceFake. These tests focus exclusively on testing the HTTP integration in isolation. Essentially, the App IT-test becomes a top-level integration test, examining only the HTTP input/output without any additional complexities.

Benefits of This Approach:

• By adopting this architecture and testing setup, we find that we spend the majority of our time in the domain layer. And we’re quite content with that! Our domain layer tests remain clean unit tests, blazingly fast, and the feedback loop is optimized.

Summary

While Fakes themselves don’t magically create better code or faster feedback loops, they push us to focus on the domain. This shift has led to positive consequences:

Isolated Testing of the HTTP Layer:

• We’ve tested the HTTP layer in isolation, without any knowledge of business logic or integrations.
• Integration with backend systems is now tested in one place, isolated and decoupled.

Isolated Domain Layer Testing:

• We’ve replaced integration classes with interfaces implemented using Fakes.
• This ensures that the domain layer remains centered around business logic.

Increased Feedback Loop Efficiency:

• All of the above significantly enhances our feedback loop.

Fakes in a Nutshell:

• Remove integration testing when testing the system as a whole.
• Replace integration implementations with Fake implementations that work solely with domain classes.

This approach allows us to make small changes incrementally, with good tests for each modification. We no longer need to consider all integrations simultaneously when replacing them with Fakes and testing in isolation.

• It’s optimal for Test-Driven Development (TDD) and pair programming. Small changes worked on together beeing reviewed and verified super fast.
• We rarely test in dedicated environments, eliminating the need for test infrastructure and test data maintenance. Reduced wait times enable rapid and continuous deployment.
• Consequently, we don’t rely on lengthy end-to-end tests. Dave Farley explains this concept well in this video.

We believe in many small tests, isolated across different parts of the code, and testing with real data in production. Our process involves making small local code changes, running tests and CI, and deploying directly to production (often behind feature toggles). With robust monitoring and alerts, we maintain a calm pulse while developing and deploying critical code for our customers.

Fakes have pushed the feedback loop during development down to milliseconds. The feedback loop from local development to production only takes a few minutes. And we have never had fewer errors.

So what is the next step for us at SpareBank 1 Utvikling? We are pushing further towards “real continuous integration (CI)” .