Kubernetes zero downtime deployment: a Spring Boot use-case

In our first article on Kubernetes zero downtime deployment, we defined the reasons why one would want to achieve zero-downtime deployment, how Kubernetes could help us, and the challenges and solutions of updating a database schema while doing so.

Though we had a thorough explanation on how to do it, we didn’t implement it. In this post, we will prove this is feasible by going down the rabbit hole: we will develop an application that can be deployed with zero downtime using the well-know Spring Boot stack.

As the entire source code is available online, only the relevant snippets will be displayed.

An architectural overview

The mock application we’ll build consists of the following technologies:

• Kotlin, but any JMV language that is compatible with Spring Boot could be used
• Spring Boot
• Spring MVC, for the web part
• Thymeleaf to render the view templates. This is for the sake of completeness, as it plays no role in the deployment
• Spring Data JPA for database access
• Flyway, to handle database updates

The Kubernetes infrastructure on which we’ll deploy the application is composed of:

• A Deployment of 3 replicas of the Spring Boot app above
• A Service to access the app
• A Deployment of a single replica of the H2 database
• A Service to access the database. While not strictly necessary as the database should only be accessed by the app, it allows administrators to use the H2 admin console.
• A Namespace to nicely isolate the above objects. Again, this is for the sake of completeness, as it plays no role in the rest of this post

Leveraging Spring Boot’s Actuator to implement readiness probes

In order to keep Kubernetes up-to-date with the readiness status of one’s app, we mentioned the need for a dedicated HTTP endpoint. Luckily enough, in this example we won’t need to implement one ourselves, as Spring Boot offers the Actuator.

The Actuator is a set of HTTP endpoints. One of them - /health - will collect all application dependencies (filesystem, database, etc.) it’s aware of, verify they are up, aggregate their status, and return the result: as soon as one of the dependencies is down, the application is considered down as well. This translates into a JSON response, with an HTTP 50x status code.

Here’s a sample of the returned JSON:

{
  "status":"UP"
}

Since having a detailed status broken-down by dependency can allow a would-be attacker to get useful information about the application, only the aggregated status is displayed by default. The full report - which requires the relevant access rights or the correct configuration - would look like that:

{
  "status":"UP",
  "details":{
    "db":{
      "status":"UP",
      "details":{
        "database":"H2",
        "hello":1
      }
    },
    "diskSpace":{
      "status":"UP",
      "details":{
        "total":250685575168,
        "free":70284525568,
        "threshold":10485760
      }
    }
  }
}

In all cases, remember that Kubernetes only checks the HTTP status code.

Of course nothing prevents you from adding additional dependencies that Spring Boot didn’t add automatically, e.g. a specific web service.

By default, Actuator’s endpoints are available under the /actuator umbrella context. The Kubernetes readiness probe can be configured accordingly:

spec:
  containers:
  - name: spring-boot-app
    image: zerodowntime:1.0
    readinessProbe:
      initialDelaySeconds: 10
      periodSeconds: 2
      httpGet:
        path: /actuator/health
        port: 8080

Flyway integration

As mentioned in Integrate Evolutionary Database Design in your Continuous Integration Pipeline, Flyway is a great tool to manage database changes. Spring Boot provides Flyway integration out-of-the-box. The referenced documentation is pretty exhaustive, but to sum it up:

• Flyway relies on migrations
• A migration is a SQL script that follows a naming scheme that allows to infer a version number e.g. V1.0.0__foo_bar.sql is version 1.0.0
• Migrations are ordered by their version number
• Migrations are guaranteed to be executed only once
• It’s possible to create rollbacks, to remove the execution of a migration. This is for reference purpose only, as there’s no Spring Boot integration for this, and it’s not necessary for rolling update deployments
• Regarding Spring Boot, Flyway migrations should be stored in the db/migrations folder

As startup time, Spring Boot will try to execute every migration in order. Because of Flyway’s only-once guaranty, migrations that have already been executed will be discarded.

Note that the fastest application Pod will start the migration process. Under the hood, Flyway created a dedicated table and writes every migration in it. Because Flyway locks this table until the migration(s) is (are) finished, other application Pods won’t be able to start migrations at the same time.

Configuration-wise, the first step is to delegate table creation - and data initialization if need be - to Flyway. By default, Spring Data JPA uses Hibernate, and it’s configured to create the tables from the existing JPA entities if they don’t exist, or to validate them if they do. Hence, we need to change that, and just validate whether the schema is synchronized with the JPA entities. This is achieved by simply configuring it in the application.yml file:

spring:
  jpa:
    hibernate.ddl-auto: validate

Setup the Database with an initial migration

Now is time to create the first migration, to create the initial PERSON table. Since it’s the first migration, it should be named accordingly e.g. V1__init.sql. Its content is quite straightforward:

CREATE TABLE PERSON
(
  ID            BIGINT      NOT NULL AUTO_INCREMENT PRIMARY KEY,
  NAME          VARCHAR(50) NOT NULL,
  ADDRESS_LINE1 VARCHAR(50) NOT NULL,
  ADDRESS_LINE2 VARCHAR(50),
  CITY          VARCHAR(50) NOT NULL,
  ZIP_CODE      VARCHAR(10) NOT NULL
);

On the source code side, the corresponding entity is Person:

@Entity
class Person(@Id @GeneratedValue(strategy = IDENTITY) val id: Long,
             val name: String,
             val addressLine1: String,
             val addressLine2: String? = null,
             val city: String,
             val zipCode: String)

Implementation of the Database zero downtime migration

As seen in our last post, three steps are necessary to obtain compatible rolling updates:

• Duplicate data: from the legacy table to the new table
• Duplicate data: from the newly-created table to the original table
• Cleanup

Let’s implement each of them in turn.

Duplicating data to the ADDRESS table aka v2.1

The first step consists of duplicating data from the PERSON table to the ADDRESS table. Obviously, this requires a new table, ADDRESS. Let’s create a new Flyway migration to create the table at application startup:

CREATE TABLE ADDRESS
(
  ID            BIGINT AUTO_INCREMENT PRIMARY KEY,
  ADDRESS_LINE1 VARCHAR(50),
  ADDRESS_LINE2 VARCHAR(50),
  CITY          VARCHAR(50),
  ZIP_CODE      VARCHAR(10),
  PERSON_ID     BIGINT
);

For the duplication itself, there are two approaches: JPA listeners, and Hibernate listeners, which are of course implementation dependent. JPA listeners hardly don’t integrate nicely with Spring dependency injection framework, it’s much easier to directly integrate with Hibernate. Besides, there isn’t much chance we will change the persistence implementation.

First, let’s create an Address entity:

@Entity
class Address(@Id @GeneratedValue(strategy = IDENTITY) val id: Long,
              val addressLine1: String,
              val addressLine2: String? = null,
              val city: String,
              val zipCode: String,
              var personId: Long? = null)

Then, extend the above Person entity with an address property:

val Person.address: Address
    get() = Address(addressLine1 = addressLine1,
        addressLine2 = addressLine2,
        city = city,
        zipCode = zipCode,
        personId = id)

This allows automatically getting the Address associated with a Person.

Finally, implement the listener itself:

@Component
class WriteAddressListener(emf: EntityManagerFactory) : PostInsertEventListener { // <1>

  private val sessionFactory = emf.unwrap(SessionFactoryImpl::class.java)         // <2>

  init {
    val registry = sessionFactory
                     .serviceRegistry
                     .getService(EventListenerRegistry::class.java)               // <3>
    reg.getEventListenerGroup(EventType.POST_INSERT).appendListener(this)         // <4>
  }

  override fun onPostInsert(event: PostInsertEvent) {
    val entity = event.entity
      if (entity is Person) {
        val session = sessionFactory.openSession()
        session.save(entity.address)                                              // <5>
      }
  }

  override fun requiresPostCommitHanding(persister: EntityPersister) = false      // <6>
}

1. A Hibernate listener should implement one of the many hooks available (check the EventType Javadoc for an exhaustive list). Here, we need to insert an Address just after a Person has been inserted, hence we implement PostInsertEventListener
2. Get the underlying Hibernate SessionFactoryImpl from the JPA EntityManagerFactory abstraction
3. Get the EventListenerRegistry from Hibernate’s service registry. As its name implies, the EventListenerRegistry is a component to register listeners.
4. Register our listener
5. When the entity that has been inserted is a Person, immediately insert a new Address with duplicated data
6. Deprecated, but required to implement

For the sake of simplicity, only the insertion listener is shown, but as stated previously, you should also implement an update listener - when the Person is updated - and a delete listener - to remove the associated Address entity.

Because Address entities are not read, the original application version still functions properly, and is thus fully compatible with this change. Rolling back the app to this version is no issue: there will be an unused ADDRESS table, that’s all.

Moving the source of truth to the ADDRESS table aka v2.2

As per the previous section, the first step should be to create a migration. At this time, the migration needs to make sure the data in ADDRESS mirrors the data in PERSON. When a Person has been inserted/updated the original version of the application, but was not updated with v2.1, then it has no associated Address.

Here’s a script using the H2 syntax, you can update it to work for the database you’re using:

MERGE INTO ADDRESS AS A USING PERSON AS P
  ON (A.PERSON_ID = P.ID)
  WHEN MATCHED THEN
UPDATE SET A.ADDRESS_LINE1 = P.ADDRESS_LINE1,
  A.ADDRESS_LINE2 = P.ADDRESS_LINE2,
  A.CITY = P.CITY,
  A.ZIP_CODE = P.ZIP_CODE
  WHEN NOT MATCHED THEN
INSERT (ADDRESS_LINE1, ADDRESS_LINE2, CITY, ZIP_CODE, PERSON_ID)
VALUES(P.ADDRESS_LINE1, P.ADDRESS_LINE2, P.CITY, P.ZIP_CODE, P.ID);

Note that it also updates the Address even if it finds one associated. While not strictly necessary because it shouldn’t happen, the belts and braces strategy is always safer. On the flip side, depending on the database size, executing this statement can take a while. It’s up to you do decide which is more risky.

One the code side, it’s time to use a regular one-to-one relationship between a Person and its Address. The updated model now looks like the following:

@Entity
class Person(@Id @GeneratedValue(strategy = IDENTITY) val id: Long,
             val name: String = "") {

    @OneToOne(mappedBy = "person", cascade = [CascadeType.ALL])
    @JoinColumn(name = "id")
    val address: Address = Address()                                          // <1>

    init {
        address.person = this                                                 // <2>
    }

    val addressLine1: String                                                  // <3>
        get() = address.addressLine1
    val addressLine2: String?                                                 // <3>
        get() = address.addressLine2
    val city: String                                                          // <3>
        get() = address.city
    val zipCode: String                                                       // <3>
        get() = address.zipCode
}

@Entity
class Address(@Id @GeneratedValue(strategy = IDENTITY) var id: Long? = null,
              var addressLine1: String = "",
              var addressLine2: String? = null,
              var city: String = "",
              var zipCode: String = "") {
    @OneToOne
    @JoinColumn(name = "person_id")
    lateinit var person: Person                                               // <4>
}

1. The address property is a regular one-to-one JPA relationship
2. During initialization, set the other end of the relationship
3. Expose properties as delegates - this is an area where Kotlin shines compared to Java. JPA will use them to fill the PERSON table
4. Replace the personId property of type Long with a full-fledged one-to-one relationship to Person

Version 2.1 still functions properly, because it reads data from the PERSON table: this table is updated by JPA because Person exposes the relevant properties - see item #3 above.

Cleanup aka v2.3

The last step is actually the easiest: remove extra columns from the PERSON table, and the associated properties from the Person entity.

This translates into the following migration script:

ALTER TABLE PERSON
  DROP COLUMN ADDRESS_LINE1;
ALTER TABLE PERSON
  DROP COLUMN ADDRESS_LINE2;
ALTER TABLE PERSON
  DROP COLUMN CITY;
ALTER TABLE PERSON
  DROP COLUMN ZIP_CODE;

Also, the Person entity is now much simpler:

@Entity
class Person(@Id @GeneratedValue(strategy = IDENTITY) val id: Long = -1,
             val name: String = "") {

    @OneToOne(mappedBy = "person", cascade = [CascadeType.ALL])
    @JoinColumn(name = "id")
    val address: Address = Address()

    init {
        address.person = this
    }
}

Version 2.2 of the app uses data from the ADDRESS table, so that columns can be safely removed without any issues.

Conclusion

While Kubernetes is a great asset to help an organization achieve zero-downtime deployments, the core issue resides in the state - the database - as always. To handle rolling updates, one needs to split non-compatible schema changes into a series of compatible ones. That not only impacts the deployment, but application code as well! Even with modern application stacks, this is a non-trivial exercise. It’s up to everyone in regard to his own context to decide whether the costs are worth the benefits.