Project 1

What you will learn in this TP

• Install Docker
• Enable Docker SWARM mode
• Create a Dockerfile
• Modify and/or override an existing docker compose file
• Create a docker compose file from a requirement
• Deploy a docker compose file
• Create a docker stack/service with a docker compose file
• Monitor the infrastructure with Prometheus/Grafana
• Deploy a private registry with Harbor
• Deploy a Harbor registry
• Import an image into a registry
• Scan an image for vulnerabilities
• Set up a Docker Compose override file
• Advanced use of labels (implemented in Project 2)

Introduction #

What you will learn in this section

• Install Docker
• Enable Docker SWARM mode
• Create a Dockerfile
• Modify and/or override an existing docker compose file
• Create a docker compose file from a requirement
• Deploy a docker compose file
• Create a docker stack/service with a docker compose file
• Monitor the infrastructure with Prometheus/Grafana
• Deploy a private registry with Harbor

This project aims to have you manipulate different concepts around Docker and especially combine them to achieve a goal. To do this, we will install and configure a Docker cluster with SWARM enabled and deploy various sub-projects on it as a stack. The goal is to set up this architecture: A recommended hardware configuration is 16GB of RAM

Test environment #

To complete this lab, you will need an Ubuntu environment (But the lab can be done with other OS/distributions). On this machine, we will create 3 virtual machines with the following characteristics:

• Virtual machine 1:
  • Name: controlplane (Swarm manager)
  • vCPU: minimum 2
  • RAM: minimum 2 GB
  • IP: 192.168.56.10
• Virtual machine 2:
  • Name: worker1
  • vCPU: minimum 2
  • RAM: minimum 2 GB
  • IP: 192.168.56.11
• Virtual machine 3:
  • Name: worker2
  • vCPU: minimum 2
  • RAM: minimum 2 GB
  • IP: 192.168.56.12

To help you set up this environment, we have provided a repo with a ready-to-use Vagrant/Virtualbox configuration.

REMINDER: This lab is dedicated to Docker configuration, but since it is an advanced lab, you will need concepts and knowledge around DevOps that are not specific to Docker but will be used in your configurations. You will need the following skills, which you can train for free:

• Vagrant: View training
• Traefik: View training
• Prometheus: View training
• Grafana: View training
• DevOps: View training

Of course, we will provide as much information as possible to help you progress even without these prerequisites.

For this, you must install Vagrant and Virtualbox on your machine. Once done, you just have to run the following commands:

git clone https://github.com/RousselTM/docker-formation
cd docker-formation/tp
vagrant up

Pre-configuration #

The following actions are necessary to prepare your test environment.

1. Domains

   You must declare the following domains on your machine (in the hosts file). They will be used to access the different sub-projects and must point to 127.0.0.1 (In the next lab we will introduce Traefik to optimize port management):
   • tp.elearning.rousseltm.fr: To access the applications. For example, to access application 1, we will use the address http://tp.elearning.rousseltm.fr/app1
   • registry.tp.elearning.rousseltm.fr: to access Harbor
   • grafana.tp.elearning.rousseltm.fr: to access Grafana
   • prometheus.tp.elearning.rousseltm.fr: to access Prometheus
   In an enterprise environment, you must use a DNS server.

2. Docker Installation

   You can check if your machines have started correctly with the following command:

   vagrant status

   If they have started correctly (status running), you can connect to the controlplane machine (future Swarm manager) with the following command:

   vagrant ssh controlplane

   Check that Docker is installed, started and enabled at boot on all VMs, and that the vagrant user can run docker commands.

3. SWARM Mode

   You must set up the Docker cluster with SWARM mode enabled. The 'controlplane' machine must be the cluster manager, and the 'workerX' machines must be the cluster workers. Since the machines have multiple IPs, we will use the IP 192.168.56.10 as the control plane management address.

   REMINDER: On all machines created by Vagrant there is a /vagrant folder which corresponds to the mount of the folder where the Vagrantfile is located: so the content of the repository's tp folder. So you can use this folder to share files between machines. For example, the file containing the tokens to join the cluster. Of course, in Production, a more secure solution should be used

   You must provide the Docker command that will store the token to join the cluster as a worker in /vagrant/swarm-token-worker.

4. Shared network

   You must create the following networks
   • common-proxy: for communication between containers via the proxy. Will be fully used in the next lab with the use of Traefik.
   • common-observability: for communication between observability containers
   We will do a mixed deployment to manipulate communication between sub-projects deployed with SWARM mode and those without SWARM mode. It is therefore necessary to set the right parameter on the 'common-proxy' network during its creation.

Sub-project 1 #

What you will learn in this section

• Deploy a Harbor registry
• Import an image into a registry
• Scan an image for vulnerabilities
• Set up a Docker Compose override file
• Advanced use of labels (implemented in Project 2)

We will create a Docker registry with Harbor on the controlplane machine (cluster manager). We will run it on the domain registry.tp.elearning.rousseltm.fr. For this first project, we will deploy it in HTTP on port 5000 (default port for Docker registries in HTTP).

REMINDER: The official Harbor website is https://goharbor.io/ and you can review the Harbor presentation video at this address: Harbor Presentation and Deployment

1. Deployment

   You must install Harbor with the trivy service on port 5000. If everything is ok, you should be able to access Harbor from the host machine at the address

   http://registry.tp.elearning.rousseltm.fr:5000

   INFORMATION: Create a folder named 'projects' in the repository's tp folder: it is visible on all VMs in /vagrant/projects. To use the paths of this lab, create the link sudo ln -s /vagrant/projects /opt/projects on each VM. You can therefore use it to manage your files for the sub-projects. We recommend 1 folder per sub-project.

2. Customization

   The install.sh script generated a docker-compose.yml file, so if we want to modify parameters we must edit this file. But we will lose all our modifications with each execution of the script.
   
   You must propose a solution to avoid this, knowing that we want the 'proxy' container to connect to the 'common-proxy' network and that it has the dt.owner label (Explanation of this Dynatrace label) with the value 'equipe1'. At the same time, add a label to tell Filebeat not to take its logs into account.
   
   We will initially simply deploy with 'docker compose' to test, and secondly as a stack with 'docker stack'.
   
   

   ATTENTION: The override must also be taken into account by the install.sh script

Sub-project 2 #

1. Image build

   You must create a Dockerfile to build an image for the following GO application which will be named simple-app and will have the tag 1.0: This application must generate a '/app/simple-app' binary and use '/app' as the working directory.

2. Adding the image to the Harbor registry

   You must send this image to the 'library' project (Explanation of the project concept in Harbor) in Harbor. We chose 'library' because it is the default Public project. Thus, the image can be used in other projects (See next lab). As a reminder, following your installation, Harbor is available at the address

   http://registry.tp.elearning.rousseltm.fr:5000

3. Vulnerability scan

   You must scan the imported image to check its vulnerabilities against CVE flaws.

4. Docker deployment

   The configuration to build must meet these needs:
   • 2-tier application: go and redis
   • The services must only run on the workers. This configuration must be externalized to allow sharing it with other applications in the same case.
   • The services must use the latest versions (in production use fixed versions) of the official images and the GO application must use the image built previously and available on Harbor.
   • 1 instance of each service
   • The GO service must export the environment variable APP='APP1'
   • The GO service must export the TASK_SLOT environment variable in which you must put the replica number of the container (Templates).
   • The GO application service must be attached to the 'common-proxy' network

5. Load balancing

   You must verify that when you call the application's URL (http://tp.elearning.rousseltm.fr) it correctly returns this page and that the highlighted fields change properly when you refresh the page and properly switch between the two containers:

Sub-project 3 #

Observability sub-project consisting of Prometheus, Grafana, Node exporter, and cAdvisor. This sub-project is the observability one and will thus allow us to have visibility over our entire project.

1. Docker deployment

   The configuration to build must meet these needs:
   • Make Grafana accessible on port 3000
   • Make Prometheus accessible on port 9090

2. Services configuration

   The configuration to build must meet these needs:
   • Automatically import the Prometheus and Elasticsearch datasources into Grafana
   • Import the dashboards with ID: 609
   • Collect Traefik metrics
   • Collect metrics from all nodes in the cluster
   • Collect metrics from all containers

Difficulty level : ●●●○○ (3/5)