Spring Boot Microservices on GKE

In this blog I’m going to explain step by step how I have deployed Pulse, a responsive, multi-channel, feedback service. Below you can see Pulse’s landing page:

Pulse Landing Page
Welcome to Pulse

Pulse has been deployed following a Microservices-based architecture. In particular, I’ve used Spring Boot, Docker, Kubernetes and Google Kubernetes Engine (GKE) to deploy this application.

From a high-level perspective, Pulse architecture looks like the following:

  • A User enters the URL: https://feedbackpal.org in the browser
  • I’ve purchased this domain from Google Domains and mapped it to a static IP address that I’ve created in Google Cloud (GCP)
  • The static IP address forwards requests to the pulse-ui Microservice. This is a Kubernetes Service of type Load Balancer, which forwards all requests from port 443 to port 8443. Port 8443 is where the Spring Boot pulse-ui micro service is running. This has been configured to run with SSL enabled.
  • The UI appears as above and the user interacts with the application. All data comes from the pulse-backend Spring Boot Microservice, which exposes REST APIs. The UI interacts with the database only through the pulse-backend microservice. The pulse-backend microservice interact with the MySQL database (itself running as a MySQL Docker container).
  • Neither the pulse-backend microservice or the MySQL service are publicly accessible outside the Kubernetes cluster, making this solution quite secure.

Pre-requisites to build and deploy the application on Google Cloud

  • First, I’ve purchased the domain from Google Domains. You can of course choose a different provider. Make sure that your provider allows you to edit the DNS settings for your domain. In particular you will need to map your chosen domain to a static IP address, as explained later.
  • You will also need a SSL certificate. There are various ways of obtaining one: you could either purchase one from a Certificate Authority (CA) or create one yourself as self-signed certificate. For production I’d suggest to get a CA certificate, otherwise users will be prompted that the traffic from your website is not private and they will likely leave
  • I’ve deployed my application on GCP, using Load Balancers, Volumes, Docker Image Registry and GKE. There is of course a cost associated with running things in the cloud, especially the Kubernetes Cluster nodes and the Load Balancer. If you don’t want to spend money, this blog post is probably not for you. The current cost all included is about $70/month.
  • You will need to install a Docker native client on your machine. Instructions can be found here. Make sure that whatever client you install allows you to run a local Kubernetes cluster on your desktop, useful for development.
  • You will need the kubectl, docker and gcloud (for the latter instructions can be found here) command line tools. There are plenty of guides online on how to set these up.

Creating a static IP address on GCP

The first thing you will want to do is to create a static IP address on GCP. Once you have an account and have set up a project, the command to obtain it is straight forward (see below). Instructions can be found here.

gcloud compute addresses create pulse-web-static-ip --region <gcp region>

To verify the IP address that GCP has created, you can run the following command:

gcloud compute addresses describe pulse-web-static-ip --region europe-west2

The result can be seen below:

address: 35.230.133.0
creationTimestamp: '2019-01-06T04:34:40.052-08:00'
description: ''
id: '----'
kind: compute#address
name: pulse-web-static-ip
networkTier: PREMIUM
region: https://www.googleapis.com/compute/v1/projects/----/regions/europe-west2
selfLink: https://www.googleapis.com/compute/v1/projects/-----/regions/europe-west2/addresses/pulse-web-static-ip
status: IN_USE
users:
- https://www.googleapis.com/compute/v1/projects/-----/regions/europe-west2/forwardingRules/---

Take note of the public IP address as you will need it next

Mapping a domain to the static IP address

This step is relatively easy. Below I’m pasting the screenshot from my Google Domain screen that shows the mapping for the feedbackpal.org domain.

Just map the hostname and the DNS A record to the static IP address and the CNAME www to the root of your domain (e.g. feedbackpal.org).

Obtaining a SSL certificate

As mentioned above, you have two options to get a SSL certificate. You can create and self-sign your own certificate or you can purchase a CA SSL certificate.

Creating your own SSL certificate

There’s a nice blog post on how to do this.

Purchasing your own SSL certificate

There are many SSL certificate providers. Your company might even give one to you. As this application is non-commercial and I’m paying out of my own pockets, I found that SSLMate provides an excellent service. For just $16/year it provides a DV (Domain Validation) SSL certificate. Basically you subscribe to the website, provide your credit card details, install their command line tool (with brew in my case) and in under 60 seconds you can get a fully functional SSL certificate. Once the process is complete, you will obtain the following files:

  • <domain-name>.chain.crt
  • <domain-name>.chained.crt
  • <domain-name>.crt
  • <domain-name>.key

The above picture shows how they look for me. Of these, the two most important ones are the .crt (your certificate) and your .key (the private key used to sign the certificate). Save all files to a secure place as you will now use them to create the SSL store for your Spring Boot UI Microservice.

Generating a PKCS12 keystore file

After much browsing and searching, trial and error, I’ve found this brief but very useful article on how to generate a PKCS12 store file from your SSL certificates. Basically it all boils down to the following command:

openssl pkcs12 -export -out server.p12 -inkey <domain-name>.key -in <domain-name>.crt 

In my case I didn’t have to add the –certfile CACert.crt option. You will of course need to substitute the <domain-name> with your own values. In my case the command looked like the following:

You will be asked to enter a password. Make sure that you keep note of this password as we will need it later, when setting up SSL for the Spring Boot Frontend microservice.

openssl pkcs12 -export -out server.p12 -inkey feedbackpal.org.key -in feedbackpal.org.crt

The above command will generate the server.p12 file. Copy this file to the src/main/resources folder of your Spring Boot UI app but make sure that if you are committing code to GitHub, you add this file to the .gitignore file. It’s always safe not to commit any sensitive files (e.g. certificates) to a code repository.

Create a Kubernetes Cluster on GKE

Instructions can be found here.

Creating the Kubernetes secret for MySQL

Before installing the MySQL database on GKE, we need to create a Kubernetes secret to hold the MySQL database root password plus the username and password details that the application will use to connect to the database.

There are various ways to achieve this and more details can be found here. We will create a mysql-local-secrets Kubernetes secret.

To create a secret, first you need to create the base64 version of your secrets. Below you will find the commands that I’ve used to respectively create base64 (fictitious) values for:

  • rootPassword
  • db_user
  • password

Then, you want to create a YAML file (I named mine mysql-local-secrets.yml). It looks like the following:

In a production environment you’d use much stronger passwords.

MYSQL_XXX are keys of the secret file that we will use later when installing the MySQL service on GKE. These files should never be committed to a source code repository or stored in an unsafe place! 

To create the secret, open a command prompt and point to the folder where you have created the YAML file. First ensure that you are pointing to your GKE cluster. I do this by running the command:

kubectl config get-contexts

If your current context (marked by *) starts with gke_, then you’re pointing at your GKE cluster. Otherwise you can point to it by running the command:

kubectl config use-context gke_...

To create the secret for MySQL, now simply run the command:

kubectl apply -f mysql-local-secrets.yml

Obviously make sure you give your file name. After this, you can check that the secret has been created by typing the command:

kubectl get secrets

You can indeed see that mysql-local-secrets has been created.

Creating the MySQL service on GKE

To create the MySQL service on GKE, I created a Kubernetes Deployment that also defines a Service. It’s all contained in a YAML file, which I’ve named: mysql-deployment.yml. I’m going to paste parts of this file (as it’s too long otherwise).

MySQL Deployment file: the Volumes section

In order to avoid your data be lost everytime the MySQL container gets recreated, it’s normally advisable to store data in external volumes. To do this, first we ask GKE to create a Persistent Volume and a Persistent Volume Claim.

GKE creates a volume and a volume claim for you when you apply this file. To check them, you can access the “Storage” section of your GKE cluster.

And if you click on it, you can see the details:

MySQL Deployment file: The Service and Deployment

The last part of the file defines the MySQL Service and Deployment.

This file defines a Kubernetes Service of type ClusterIP, listening on port 3306 (this can only be reached from within the GKE cluster) and a Kubernetes Deployment that deploys the official MySQL image mysql:5.7 from Docker Hub.

Few things of interest here:

  • I had to pass the args with “–ignore-db-dir=lost+found” otherwise the database wouldn’t start. It might not be the case for you
  • The environment variable (env -> name) MYSQL_ROOT_PASSWORD and how to use it is documented in the official MySQL image documentation on Docker Hub. The database will be created with user root and the password you have defined here. Remember the secret we created earlier? Here we are saying to GKE that the value for this environment variable must be taken from the mysql-local-secrets secret with key MYSQL_ROOT_PASSWORD. This is an application of the 12 Factors as explained in one of my previous blogs.
  • Notice that we are using the volumes we have defined at the beginning of this file.

To create the database, just run the command:

kubectl apply -f mysql-deployment.yml

To watch the progress, you can run the command:

watch -n 3 kubectl get pods

And once the pod has been created, you can check the database instance with the following command (it assumes your pod has been named mysql-8498c4899d-w5xxc):

kubectl exec -it mysql-8498c4899d-w5xxc -- /bin/bash

This should land you on the bash shell of your MySQL container. You can then try to login to the database by executing the following command:

mysql -u root -p 

And when asked for the password, use the “clear text” one that you have created earlier as part of the secret (if you’ve followed this blog, you would enter rootPassword).

Et voila’, you are inside a fully working MySQL server.

Deploying the Backend Microservice with Spring Boot REST Data JPA

Now that we have a database in place we can spin up the Microservice that will manage the data in and out. I’ve used a Spring Boot REST Data JPA Microservice. The advantage of this kind of service is that one defines the JPA entities and the JPA repositories and Spring Boot generates all the CRUD REST APIs automatically.

Of course, as the application becomes more complex, there might be the need for some customisations, but the majority of functions is there. So this Microservice reads JSON from the UI Microservice (described later) to query/write to the database and returns JSON with database data. The UI Microservice never interacts directly with the database, an important concept in a Microservice-based architecture.

The YAML defines a Service and a Deployment.

The Kubernetes service is of type NodePort, because I don’t actually need this service to be accessible from outside the cluster. It defines the NodePort (the one you would use as part of your address in the browser) which maps to the Spring Boot port. In this case, port 30002 -> 8080. This means that whenever the UI Microservice invokes http://pulse-backend:30002/ it actually means http://pulse-backend:8080.

Two important things to notice here:

  • Look at the JDBC connection string. It contains mysql as a server, but we know that there isn’t such a thing, at least that would normally be reachable, but because we have deployed our MySQL service with name mysql, this is all that is required for the pulse-backend Microservice to be able to resolve that server name. Nobody outside the cluster would be able to resolve the same address. One gotta love the magic of Docker and Kubernetes!
  • The second part is the environment variables we pass to our Spring Boot app. Of particular importance we have three: the prod profile (as I have different configurations based on the active profile), the spring_datasource_username and the spring_datasource_password. Do you remember when we have set up the MySQL secret? We are using them here to pass to Spring Boot the username/password it should use to initialise its connection to the database. Again, this is an implementation of the 12 Factor App. My Spring Boot app doesn’t have one single sensitive configuration information (in fact no sensitive data at all) in the source code, it’s all managed through secrets. This way I can write the code once and, deploy anywhere. This also follows good DevOps principles because it can be highly automated.

Deploying the SSL Frontend Microservice with Spring Boot

For the frontend, I also used Spring Boot, this time in conjunction with Thymeleaf, Bootstrap and Spring Security. The job of the frontend Microservice is very simple: display the UI to the users and ask the pulse-backend Microservice to store/read data to/from the database in JSON format, by invoking its REST APIs.

The UI must be accessible externally. I’ve tried various configurations with trial and error.

  • First I tried with the Spring Boot app running on HTTP and fronted with a Kubernetes Ingress running on SSL. That worked except from the security bit, that is when the user tried to perform admin tasks. The /login action was somehow redirected to HTTP instead of HTTPS, so I had to abandon this avenue.
  • The second configuration was instead to deploy the frontend microservice as a Kubernetes Service of type LoadBalancer. In this configuration, I protected the Spring Boot app with SSL and the Kubernetes Service of type LoadBalancer simply mapped port 443 (HTTPS) to port 8443 (which is the port I’ve told the Spring Boot app to run on). This configuration worked and it’s the one that we’re going to see now. With this configuration, GKE automatically creates a Load Balancer for you and even a public IP address if you haven’t specified one. Very seamless.

As mentioned in the opening of this blog post, you should now have a server.p12 store file in your file system. Make sure to place this under src/main/resources of your Spring Boot app, as the following image shows:

Then make sure your application.properties file contains the following:

Let’s see what happens here:

  • First I told Spring Boot to run the app on port 8443
  • Secondly, I instruct the SSL engine that the store type is PKCS12
  • Then I give the SSL engine the path to the server.p12 file as per above. Notice the classpath: prefix to instruct Spring Boot to look into the class path (src/main/resources is in the class path).
  • When we created the server.p12 file at the beginning of this blog post, we had to enter the password. The server.ssl.key-store-password property must have that value. Now, following 12 Factors and good practices, we don’t want to enter any sensitive configuration information in the source code, therefore we use an environment variable ${SSL_PASSWORD}. Can you guess how we will be passing this value to the Microservice? Through a Kubernetes secret, exactly!

Creating the Pulse UI Kubernetes Secret

Similarly to what we have done for the MySQL database, I created another YAML file with some secrets I want to pass to my Spring Boot micro services. The file (with obviously fake values) looks like the one below:

The SSL_PASSWORD key is the one I will use when deploying this UI Microservice as Kubernetes Service, as explained below.

Deploying the UI Microservice

Similarly to the pulse-backend Microservice, I deployed this Spring Boot Microservice as a Kubernetes Service and Deployment, however with one important difference. This Kubernetes Service is of type LoadBalancer and I specify the static IP address that I created at the beginning of this post.

Here are the things to notice:

  • The Service type is LoadBalancer
  • I specified my static IP address with the loadBalancerIP property
  • The service redirects traffic from 443 (HTTPS) to 8443 (the port I asked Spring Boot to run on). Since my domain feedbackpal.org is mapped to my static IP address, this allows me to invoke: https://feedbackpal.org which will ultimately result in https://pulse-ui:8443/
  • Again, I’m passing the active profile as an environment variable and I pass a couple of password through secrets.

That’s all there is to it folks!

‘Till the next time, roger and out!