Hero Image
- Philipp Ludewig

Is my Container healthy?

bash docker Localstack

Aloha peeps how is it going? Last week was the last week of October and therefore spooky. Halloween in Covid-19 times was lame as there was no party to crash and the second lockdown is on its way to ruin everyones fun. Anyway this week i understood how to check if a service within a container is healthy and how i can react on it within a bash script and i want to share it with you. You know the drill, this is more for my forgetful future self as for everyone else but you know i still invite everyone to read it.

The Case

I needed to create a local setup, so i could understand a certain Java library, which needed a AWS SQS with a linked DLQ to run locally. As a developer you have two options here, you either use a real configured AWS SQS with a working internet connection or you use a Localstack docker container. Both options have their respected pro/contra arguments. I would like to highlight them for second. The biggest argument i heard for the first option is that you would work most of the times between office hours in a building with an working internet connection and you would also use terraform to create you infrastructure. Why would you then not just create a test-stage for your integration tests? You have the terraform code for it and the new test-stage would behave exactly like you nonlive and live stage. I agree with this statement and the difficulty of this option is up-to your setup. The reason why i always tend to use option two is because i belong to the generation "no internet available for you" and besides that i tend to be on the road while i work where here in Germany you can only laugh about the internet connection in trains. These docker container are a true live saver as you can run them locally and they dont need much of your laptops cpu/ram.

Localstack it is!

Now the thing with Localstack is that every functionality from the AWS Cloud gets simulated from Localstack. The behavior of these services is not always but mostly the same of the AWS cloud, so don't try to recreate complex infrastructure within your Localstack container. Check the issue tracker for bugs if stuff doesn't work as intended. Now that this is out of the way let's start. I will use the tool aws-cli/2.0.61 and the following docker compose file for this example:

version: '3'

services:
  localstack:
    image: localstack/localstack:0.10.5
    ports:
    - "4576:4576"
    - "${PORT_WEB_UI-9000}:${PORT_WEB_UI-8080}"
    environment:
    - SERVICES=sqs
    - DEBUG=${DEBUG- }
    - DATA_DIR=${DATA_DIR- }
    - PORT_WEB_UI=${PORT_WEB_UI- }
    - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- }
    - KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
    - DOCKER_HOST=${LOCALSTACK_DOCKER_HOST-unix:///var/run/docker.sock}
    volumes:
    - "/var/run/docker.sock:/var/run/docker.sock"

As of this day where i write this post there is already a newer version of Localstack (0.11.6) which consolidates all endpoints under one hood but i couldn't make it work and opened a issue about the problem. Anyway you start the container with this shell command:

docker-compose --file docker-compose-localstack.yaml up --build --detach

The options will:

  • build or rebuild your service
  • detach your shell from the container

If you don't understand what that means try running the command without the additional options. You will see what happens or more what doesn't happen.

Now that we have running docker container we could use certain commands with awscli to create both queues and link them up but what we most likely would get if we would automate this process with a script is this bad boy:

An error occurred (502) when calling the CreateQueue operation (reached max retries: 2): Bad Gateway

This happens as we didn't wait for the container to fully start up. Now we could just use a sleep 10 for this between the function calls but this is not precise and doesn't solve the problem instead a health check does.

Am I healthy?

Now we want to add a healthcheck to our docker compose defintion. We need to define a state when a container can be seen as healthy. In our case it's when the list-queues awscli command doesn't return an error. See the documentation for more information on healthchecks.

version: '3'

services:
  localstack:
    image: localstack/localstack:0.10.5
    ports:
    - "4576:4576"
    - "${PORT_WEB_UI-9000}:${PORT_WEB_UI-8080}"
    environment:
    - SERVICES=sqs
    - DEBUG=${DEBUG- }
    - DATA_DIR=${DATA_DIR- }
    - PORT_WEB_UI=${PORT_WEB_UI- }
    - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- }
    - KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
    - DOCKER_HOST=${LOCALSTACK_DOCKER_HOST-unix:///var/run/docker.sock}
    volumes:
    - "/var/run/docker.sock:/var/run/docker.sock"
    healthcheck:
      test: aws --endpoint-url=http://localhost:4576 --region=eu-central-1 --no-sign-request --no-paginate sqs list-queues
      interval: 1m30s
      timeout: 10s
      retries: 3

Let me quickly deconstruct the used aws command for the health check. I define the endpoint to be the local sqs endpoint of Localstack and set the region to eu-central. Any region is ok here. The option --no-sign-request tells aws to ignore the security credentials as Localstack ignores it aswell by default. The following option --no-paginate stops aws from capsulating the output in a subshell. It's ok to not do this but it gets harder to read this way.

The health status can now be seen in the status column of the output of docker ps.

CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS                             PORTS                                                                                                                                                                  NAMES
2f341613b827        localstack/localstack:0.10.5   "docker-entrypoint.sh"   23 seconds ago      Up 22 seconds (health: starting)   4567-4571/tcp, 0.0.0.0:4572->4572/tcp, 4573-4574/tcp, 0.0.0.0:4575-4576->4575-4576/tcp, 4577-4583/tcp, 0.0.0.0:4584->4584/tcp, 4585-4597/tcp, 0.0.0.0:8080->8080/tcp   blogpoststuff_localstack_1

One last piece of the puzzle remains: How can I now react on the health status within a bash script?

Don't reinvent the wheel!

You can react on the health status in an easy way by using this script from a user called Jordy. The script is a beauty and easy to read. The author uses a for-loop in which the status of the health check from the container is inspected. The container can have only three states: starting, unhealthy and healthy. 

state=$(docker inspect -f '{{ .State.Health.Status }}' ${container_name})

This how you would check for the health of a container.
Cheers

Previous Post Next Post