Jana Nina Friedrich
Management, Analyzer & Visualsation
Hello there.
Welcome, Moin, Привет, Bonjour, ...
This page introduces you to the diploma thesis Smart Encoding Analyzer for Logs or as we like to call it: S.E.A.L.
Short description
English
The project S.E.A.L. originated in a cooperation between the school HTL
Mössingerstraße and the company Bitmovin. The project analyzes encoding log files
from Bitmovins production system, extracts performance related metrics, stores the data and
presents the filtered information in an easily comprehensible way. In addition, the development
team is notified via the communication tool Slack in the event of an error in the log
files.
German
Das Projekt S.E.A.L. entstand in einer Kooperation zwischen der Schule HTL
Mössingerstraße und der Firma Bitmovin. Das Projekt analysiert die
Encoding-Logfiles aus dem Bitmovins-Produktionssystem, extrahiert leistungsbezogene Metriken,
speichert diese Daten und präsentiert die gefilterten Informationen auf leicht verständliche
Weise. Zusätzlich wird das Entwicklungsteam bei einem Fehler in den Logfiles über das
Kommunikationstool Slack benachrichtigt.
Details
At the beginning, we should repeat that our project is being processed and created at HTL Mössingerstraße in a cooperation with the business partner Bitmovin.
Next, we are giving you a brief overview of S.E.A.L.
report_problem Structur of a Log File
Problem
So the first question you are going to ask is: Why does Bitmovin need such a system?
There are actually many reasons. Right now, you have to look at every log file individually and each log consists of many text lines. The problem is that everyday over thousands of log files are produced and you easily lose your overview. The task to find the information you are looking for is very time consuming and quite hard. As an addition the seekers have to know the logs or they are never going to find anything.
On the left side you can see the structure of one log line. The used format is similar to JSON. This example is already prettified, normally this text would stand in one line.
library_books Project desgin
Overview
To understand S.E.A.L. you have to understand the data flow. On the
right side you can find an illustration which shows a simplifed data flow where
S.E.A.L. has been broken down in four stages:
- Stage: Get log files
- Stage: Process each log file
- Stage: Store filtered data
- Stage: Visulize the saved data
The seconde picture illustrates the detailed project desgin:
For each encoding job a log file is generated and automatically uploaded to
a
Google Cloud Storage
bucket.
The GCSBeat collects the log files and passes them via Kafka
to
Logstash, where
Logstash extracts the relevant information and stores it into
Elasticsearch.
Kibana has access to the stored data and visualizes it.
The final system is running in the Kubernetes-Cluster of
Bitmovin so that every Bitmover can access it.
Every service is running in one Docker Container. Because the stored
data is useless after half a year and to save space in Elasticsearch the extracted
data gets deleted through the service Curator.
The saved data gets monitored by the service ElastAlert. If an alert
is triggered, ElastAlert writes a message into a Slack Channel. Slack
is the primary communication tool of the company.
leaderboard Visualsation
Kibana
The video below shows the possible functions of how to use Kibana. The
description of the video can be found here:
Dashboard page
First, you see is the Dashboard page, on which you can generate
various graphs from the stored data from the Elasticsearch. In this
example video there is a bar chart (left side) and a table (right side) on
the dashboard which both show the values of the metric
"log_mtmuServiceDuration". If you want to find out more from which log lines
the displayed values come from or if you want to do more research in
general, you can go to the Discover page.
Here you can see all the saved log lines from the used database Elasticsearch. The graph shows how many log lines were saved at which time. Below is a table. One line represents a log line. If you enlarge a line you can see all available details of the stored log line. Since the page quickly becomes confusing, you can take additional actions. On the left side you can also add columns from the existing metrics to the table. So you can see at first glance which entries have the value you are looking for. But if you want to search for a specific piece of information, e.g. all values of "log_mtmuServiceDuration" that are below 10,000, you can add a filter for this.
trending_up Competitions
Positive endnote
S.E.A.L. was submitted to several competitions, including the Jugend Innovativ and innovation@school competitions. We are pleased to announce, that S.E.A.L. is among the 10 best projects of innovation@school in 2020/21.
Our Awesome Team
Our team consist of two motivated students, one dynamic teacher and
the best mentor of the greatest team at Bitmovin ;)
Ariane Krassnitzer
Design, Kubernetes & Docker
General Kenobi!
Prof. Dipl.-Ing. Dr. Daniel Wischounig-Strucl
Supervisor of HTL Mössingerstraße
Think different dynamic.
Gerald Götzenbrucker
Supervisor of Bitmovin
A bit of gin keeps you movin.