SustainKieker Hackathon: Reverse Engineering of Research Software

🗓️ Thu 27/02, 13:30
📍 Room SR 206, Building 30.70, Campus South (KIT)
ℹ️ Abstract
🔬 The Kieker Observability Framework
🌐 The SustainKieker Project Homepage

Table of Contents

1. Introduction
2. Prerequisites
   • Linux system
   • Software
3. Instructions
4. Run the Kieker Trace Analysis
5. More reads

Introduction

Kieker observability framework features monitoring and analysis capabilities. OpenTelemetry, in comparison, provides means to monitor the program but analysis. Our new Otkt DSL can define a mapping from an OpenTelemetry Span to a Kieker record. Using Otkt, we show how we can use OpenTelemetry to collect monitoring data from a Python program and send it to a Kieker Analysis endpoint.

Prerequisites

Linux system

macOS

• Multipass¹ (latest download)
  • M1, M2, or Intel based
  • macOS 10.15 Catalina or later

Windows 10/11 (two options)

• Multipass (latest download)
  • Windows 10 Pro or Enterprise version 1803 (“April 2018 Update”) or later
• WSL Install Instructions
  • Windows 10 2004 or higher
  • Old versions: Windows 10 1903 or higher

Software

1. Python3

2. Java

   # Fedora 40, 41, 42
   sudo dnf install java-21-openjdk-devel
   # Ubuntu 20.04, 22.04, 24.04, 24.10
   sudo apt install openjdk-21-jdk

3. Maven

   # Fedora 40, 41, 42
   sudo dnf install maven
   # Ubuntu 20.04, 22.04, 24.04, 24.10
   sudo apt install maven

   or install manually.

4. GNU plotutils

   # Fedora 40, 41, 42
   sudo dnf install plotutils
   # Ubuntu 20.04, 22.04, 24.04, 24.10
   sudo apt install plotutils

5. The graphviz graph visualization tools

   # Fedora 40, 41, 42
   sudo dnf install graphviz
   # Ubuntu 20.04, 22.04, 24.04, 24.10
   sudo apt install graphviz

6. The Ghostscript interpreter for PostScript language & PDF

   # Fedora 40, 41, 42
   sudo dnf install ghostscript
   # Ubuntu 20.04, 22.04, 24.04, 24.10
   sudo apt install ghostscript

Instructions

1. Build the Otkt collector.

   # Run inside the hackathon repository:
   cd collector
   mvn clean package

2. Export the PYTHONPATH variable with the parent path of the otkt directory.

   export PYTHONPATH=/path/to/hackathon-repo/python:$PYTHONPATH

3. Instrument the entry point with the following python import.

   from otkt.otelinit import tracer

4. Instrument all python files of the target software using instrument-py.sh

   # Run inside the target software repository:
   /path/to/hackathon-repo/tools/instrument-py.sh

   • It is important to adhere to the indentation of each method definitions.

5. Install all required python module dependencies. A python project usually comes with requirements.txt. Append all dependencies in res/requirements.txt to the existing (or new) requirements.txt.

   # Run inside the target software repository:
   pip install -r requirements.txt 

6. Run the Otkt collector on a separate terminal.

   java -jar /path/to/Collector-0.0.1-SNAPSHOT-jar-with-dependencies.jar -c /path/to/config.txt

7. On a new terminal, run the target program (e.g., python3 main.py).

   # Run inside the target software repository:
   python3 main.py

Run the Kieker Trace Analysis

• Prepare Trace Analysis

  # Run inside the hackathon repository
  unzip tools/trace-analysis-2.0.2.zip

• Aggregated Deployment Call Tree

  # Run inside the hackathon repository
  mkdir output-adctree
  trace-analysis-2.0.2/bin/trace-analysis  \
    --inputdirs /path/to/kieker-logs       \
    --outputdir output-adctree             \
    --plot-Aggregated-Deployment-Call-Tree
  cd output-adctree
  dot aggregatedDeploymentCallTree.dot -T pdf output.pdf

• Deployment Component Dependency Graph

  # Run inside the hackathon repository
  mkdir output-dcdgraph
  trace-analysis-2.0.2/bin/trace-analysis             \
    --inputdirs /path/to/kieker-logs                  \
    --outputdir output-dcdgraph
    --plot-Deployment-Component-Dependency-Graph none
  cd output-dcdgraph
  dot deploymentComponentDependencyGraph.dot -T pdf output.pdf

• Deployment Sequence Diagrams

  # Run inside the hackathon repository
  mkdir output-dsdiagrams
  trace-analysis-2.0.2/bin/trace-analysis             \
    --inputdirs /path/to/kieker-logs                  \
    --outputdir output-dsdiagrams
    --plot-Deployment-Sequence-Diagrams
  cd output-dsdiagrams
  pic2plot -T ps deploymentSequenceDiagram-1.pic > output.ps
  ps2pdf outout.ps output.pdf

More Reads

1. The Otkt collector stores all received records under /tmp. E.g.,

   /tmp/kieker-20250217-181132-41826294550971-UTC--KIEKER/
   /tmp/kieker-20250218-093839-12420615078112-UTC--KIEKER/
   /tmp/kieker-20250218-114626-20087904425947-UTC--KIEKER/
   

   You can change the location by changing the value for kieker.monitoring.writer.filesystem.FileWriter.customStoragePath in res/config.txt.

2. Packaging a python software

   Instrumenting a Python software may require downloading the source code and building a package. In this case, follow the instruction below and repackage it yourself. It could be as simple as running python3 -m build. Otherwise, you need to follow the provided build instruction.

   # Instrument the target software before running the below commands
   pip install build
   python3 -m build

3. Building an Otkt program

   For the Hackathon, we provide all artifacts from the Otkt build of the Otkt source code demo.otkt. For more details, refer to BUILD.md.

4. Building the Otkt compiler

   The Otkt compiler is maintained in the GitHub repository OtktDSL. The instruction will be uploaded soon.

Footnotes

1. Multipass is a lightweight VM manager for Linux, Windows and macOS from Canonical. ↩