Filebeat Installation

What you will learn in this TP

• Install Filebeat
• Identify important files and directories
• Use modules
• Advanced use of labels
• Deploy a Harbor registry
• Import an image into a registry
• Scan an image for vulnerabilities
• Set up a docker compose override file
• Advanced use of labels (implemented in Project 2)

Introduction #

What you will learn in this section

• Install Filebeat
• Identify important files and directories
• Use modules
• Advanced use of labels

We will proceed with the installation of Filebeat using Docker technology.

1. Procedure

   You must provide the link to find all Filebeat installation procedures as well as the one for installing it on a Linux-type OS.

Docker #

We will install Filebeat on Docker using a docker compose file.

1. The folders

   You must provide the list of folders required for Filebeat to work properly in a Docker environment.

2. SWARM mode

   You must set up the Docker cluster with SWARM mode enabled. The 'manager' machine must be the cluster manager and the 'workerX' machines must be the cluster workers. As the machines have several IPs, we will use the IP 192.168.56.2 as the control plane management address.

   REMINDER: On every machine created by Vagrant there is a /vagrant folder which corresponds to the mount of the folder containing the Vagrantfile: in other words, the content of the 6_projet folder. You can therefore use this folder to share files between the machines, for example the file containing the tokens used to join the cluster. Of course, in production you will need to use a more secure solution

   You must provide the Docker command that stores in /vagrant/swarm-token-worker the token used to join the cluster as a worker.

3. Shared network

   You must create the following networks
   • common-proxy: for communication between containers through the proxy. It will be fully used in the next lab with Traefik.
   • common-observability: for communication between the observability containers
   We will do a mixed deployment to work with communication between sub-projects deployed with SWARM mode and those deployed without it. You therefore need to set the right parameter on the 'common-proxy' network when you create it.

Sub-project 1 #

What you will learn in this section

• Deploy a Harbor registry
• Import an image into a registry
• Scan an image for vulnerabilities
• Set up a docker compose override file
• Advanced use of labels (implemented in Project 2)

We will create a Docker registry with Harbor on the manager machine. It will run on the domain registry.tp.elearning.rousseltm.fr. For this first project, we will deploy it over HTTP on port 5000 (the default port for Docker registries over HTTP).

REMINDER: The official Harbor website https://goharbor.io/ and you can watch the Harbor presentation video again at this address: Harbor presentation and deployment

1. Deployment

   You must install Harbor with the trivy service on port 5000. If everything is OK, you should be able to access Harbor from the host machine at the address

   http://registry.tp.elearning.rousseltm.fr:5000

   INFORMATION: In the 6_projet folder, there is a folder named 'projects' whose content will automatically be mounted in /opt/projects on all the VMs. You can therefore use it to manage your files for the sub-projects. We recommend one folder per sub-project.

2. Customisation

   The install.sh script generated a docker-compose.yml file, so if you want to change settings you need to edit this file. However, you will lose all your changes every time the script runs.
   
   You must propose a solution to avoid this, knowing that we want the 'proxy' container to connect to the 'common-proxy' network and to have the dt.owner label (Explanation of this Dynatrace label) with the value 'equipe1'. While you are at it, add a label telling Filebeat to ignore its logs.
   
   We will first deploy simply with 'docker compose' to test, and then as a 'docker stack'.
   
   

   WARNING: The override must also be taken into account by the install.sh script

Sub-project 2 #

1. Build image

   You must create a Dockerfile to build an image for the following Go application, which will be named simple-app and tagged 1.0: This application must produce a '/app/simple-app' binary and use '/app' as its working directory.

2. Adding the image to the Harbor registry

   You must push this image to the 'library' project (Explanation of the project concept in Harbor) in Harbor. We chose 'library' because it is the default public project, so the image can be used in other projects (see next lab). As a reminder, after your installation Harbor is available at the address

   http://registry.tp.elearning.rousseltm.fr:5000

3. Vulnerability scan

   You must scan the imported image to check its vulnerabilities against CVEs.

4. Docker deployment

   The configuration to build must meet these requirements:
   • 2-tier application: Go and Redis
   • The services must only run on the workers. This configuration must be externalised so that it can be shared with other applications in the same situation.
   • The services must use the latest versions of the official images (in production, use pinned versions) and the Go application must use the image built earlier and available in Harbor.
   • 1 instance of each service
   • The Go service must export the environment variable APP='APP1'
   • The Go service must export the environment variable TASK_SLOT, in which you must put the replica number of the container (Templates).
   • The Go application service must be attached to the 'common-proxy' network

5. Load balancing

   You must check that when you call the application URL (http://tp.elearning.rousseltm.fr) it returns this page, and that the highlighted fields change when you refresh the page, switching between the two containers:

Sub-project 3 #

Observability sub-project made up of Prometheus, Grafana, Node exporter and cAdvisor. This is the observability sub-project, so it will give us visibility over the whole project.

1. Docker deployment

   The configuration to build must meet these requirements:
   • Make Grafana accessible on port 3000
   • Make Prometheus accessible on port 9090

2. Service configuration

   The configuration to build must meet these requirements:
   • Automatically import the Prometheus and Elasticsearch datasources into Grafana
   • Import the dashboards with ID: 609
   • Collect Traefik metrics
   • Collect metrics from all the nodes in the cluster
   • Collect metrics from all the containers

Difficulty level : ●●○○○ (2/5)