Topotato docs anatomy overview

README.md

@@ -1,6 +1,7 @@
 TOPOTATO TESTING FRAMEWORK (WORK IN PROGRESS)
 ==========================
 
+LIVE DOCS: [https://docs.page/opensourcerouting/frr~docs](https://docs.page/opensourcerouting/frr~docs)
 
 TODOs / Known issues
 ====================

docs.json

@@ -0,0 +1,68 @@
+{
+    "name": "Topotato",
+    "theme": "#36B9B9",
+    "sidebar": [
+      ["About", "/"],
+      ["Topotato 1.0 is here!", "/"],
+      ["Todos & Known Issues", "/todos"],
+      ["Getting started", "/getting-started"],
+      [
+        "Test Anatomy",
+        [
+          ["Overview", "/anatomy/overview"],
+          ["Topology", "/anatomy/topology"],
+          ["FRR configuration", "/anatomy/frr-config"],
+          ["Writing tests", "/anatomy/writing-tests"],
+          ["Debug a test", "/anatomy/debug-test"]
+        ]
+      ],
+      [
+        "Assertions",
+        [
+          ["Overview", "/assertions/overview"],
+          ["Usage", "/assertions/usage"],
+          ["AssertKernelRoutes", "/assertions/assert-kernel-routes"],
+          ["AssertKernelRoutesV4", "/assertions/assert-kernel-routes-v4"],
+          ["AssertKernelRoutesV6", "/assertions/assert-kernel-routes-v6"],
+          ["AssertVtysh", "/assertions/assert-vtysh"],
+          ["ReconfigureFRR", "/assertions/reconfigure-frr"],
+          ["AssertPacket", "/assertions/assert-package"],
+          ["AssertLog", "/assertions/assert-log"],
+          ["Delay", "/assertions/delay"]
+        ]
+      ],
+      [
+        "Modifiers",
+        [
+          ["Overview", "/modifiers/overview"],
+          ["Usage", "/modifiers/usage"],
+          ["DaemonRestart", "/modifiers/daemon-restart"],
+          ["ModifyLinkStatus", "/modifiers/modify-link-status"],
+          ["MulticastReceiver", "/modifiers/multicast-receiver"],
+          ["ScapySend", "/modifiers/scapy-send"],
+          ["ExaBGP", "/modifiers/exabgp"]
+        ]
+      ],
+      [
+        "How-tos",
+        [
+          ["TODO", "/howtos/"]
+        ]
+      ],
+      [
+        "CLI commands",
+        [
+          ["test", "/cli-commands/test"],
+          ["bash", "/cli-commands/bash"],
+          ["vm-start", "/cli-commands/vm-start"]
+        ]
+      ],
+      [
+        "Environments and CI",
+        [
+          ["Overview", "/env-ci/overview"],
+          ["Platforms", "/env-ci/platforms"]
+        ]
+      ]
+    ]
+  }

docs/anatomy/overview.mdx

@@ -0,0 +1,108 @@
+# Anatomy of a Test
+
+Tests in the topotato framework are structured into different sections that collectively define the test's behavior, topology, and configurations. This document provides an overview of these sections and their components.
+
+<Image src="/anatomy/assets/anatomy.svg" caption="Topotato structure" />
+
+
+
+### Header
+
+Every test begins with a header that sets the initial context for the test script. The header typically includes the license information, copyright details, and a brief description of the test's purpose. The following Python code demonstrates a sample header:
+
+```python
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (C) 2070  Some Body
+"""
+Check that the top of the potato has been baked properly.
+"""
+
+from topotato import *
+```
+
+A few points to note about the header:
+
+- The `topotato` package uses SPDX license identifiers. It's recommended to use the `GPL-2.0-or-later` license unless you have a compelling reason to choose otherwise.
+- A concise docstring describing the test's objective should be included.
+- The `topotato` package is imported using a wildcard import (`*`), which is acceptable in this context.
+
+#### Compatibility Markers
+
+The test script may include compatibility markers to handle changes in the topotato framework. These markers help manage compatibility issues when the framework evolves over time. For instance:
+
+```python
+# Compatibility markers for topotato framework changes
+__topotato_version__ = 1
+__topotests_file__ = 'potato/test_top.py'
+__topotests_rev__ = 'a770da1b1c6290f53cc69218a30360accd6a0068'
+```
+
+These markers provide information about the topotato version, the test file, and the revision, aiding in managing compatibility and version control.
+
+</hr>
+
+### Topology Definition
+
+In topotato, a test topology is defined using ASCII diagrams enclosed within a function marked with the `topology_fixture` decorator. Here's an example of defining a topology named `topo1`:
+
+```python
+@topology_fixture()
+def topo1(topo):
+   """
+   [ r1 ]---[ r2 ]
+   """
+   # Optional modifier code operating on topo.* here
+```
+
+The ASCII diagram visually represents the topology. The function's name `topo1` identifies the topology. Defining multiple topologies in one file is possible.
+
+<Info>But if complexity grows, it might be better to split the tests into separate files.</Info>
+
+</hr>
+
+### FRR Configurations
+
+Topotato generates FRR configurations using [Jinja2](https://palletsprojects.com/p/jinja/) templates embedded within the test script. Configuration templates define various aspects of router behavior. Here's an example of defining FRR configurations:
+
+<Warning>This Jinja2 template used has been customzied. See [`frr configuration section`](/anatomy/frr-config)</Warning>
+
+```python
+class Configs(FRRConfigs):
+   routers = ["r1"]  # By default, all systems listed in the topology run FRR
+
+   zebra = """ 
+   #% extends "boilerplate.conf"
+   #% block main
+   #%   for iface in router.ifaces
+   interface {{ iface.ifname }}
+    description {{ iface.other.endpoint.name }}
+   !
+   #%   endfor
+   !
+   #% endblock
+   """
+
+   # Configuration templates for other daemons like staticd, ospfd, etc.
+```
+
+These configurations can reference dynamic variables, making them more maintainable and easier to understand.
+
+Each *daemon* can be configured for each routers by using their respective name and configuration. 
+
+</hr>
+
+### Test Classes
+
+All topotato tests are organized within classes that inherit from the `TestBase` class. The class definition binds together the test content, topology, and configurations. Here's an example:
+
+```python
+class TestSomething(TestBase, AutoFixture, topo=topo1, configs=Configs):
+    """
+    This docstring will be included in the HTML report, make it useful.
+    """
+```
+
+To execute tests, an instance of this class is created, and its test methods are executed in sequence. The test methods can carry data between each other using the `self` object.
+<Warning>However, due to pytest's design, the `__init__` constructor cannot be used.</Warning>
+This structured approach to test organization in topotato simplifies the creation, execution, and debugging of tests while promoting consistency and maintainability.

docs/getting-started.mdx

@@ -0,0 +1,135 @@
+---
+title: Getting Started
+description: Write your first test
+---
+
+## Install `topotato`
+
+`topotato` requires Python >= 3.8
+
+Run the following command in your command line:
+
+``` bash
+sysctl -w kernel.unprivileged_userns_clone=1
+mkdir /etc/frr
+
+# In case you don't want to follow the <<apt install steps>> install these manually:
+# unshare - run program with some namespaces unshared from parent
+# nsenter - run program with namespaces of other processes
+# tini - https://github.com/krallin/tini
+# dumpcap - Dump network traffic
+# ip - show / manipulate routing, network devices, interfaces and tunnels
+
+apt-get satisfy \
+    'graphviz' 'tshark (>=4.0)' 'wireshark-common (>=4.0)' 'tini' \
+    'python3 (>=3.8)' \
+    'python3-pytest (>=6.1)' 'python3-pytest-xdist' \
+    'python3-typing-extensions' 'python3-docutils' 'python3-pyinotify' \
+    'python3-scapy (>=2.4.5)' 'python3-exabgp (>=4.2)' \
+    'python3-jinja2 (>=3.1)' 'python3-lxml' 'python3-markupsafe' \
+
+```
+
+## Create your first test
+
+1. Create a new file called `test_sample.py`, containing a function, and a test:
+
+``` python
+# content of test_sample.py
+
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (C) 2018-2023  YOUR NAME HERE for NetDEF, Inc.
+"""
+Simple demo test for topotato.
+"""
+
+from topotato.v1 import *
+
+@topology_fixture()
+def topology(topo):
+    """
+    [ r1 ]---[ noprot ]
+    [    ]
+    [    ]---[ rip ]
+    [    ]
+    [    ]---[ ripng ]
+    [    ]
+    [    ]---[ ospfv2 ]
+    [    ]
+    [    ]---[ ospfv3 ]
+    [    ]
+    [    ]---[ isisv4 ]
+    [    ]
+    [    ]---[ isisv6 ]
+    """
+    topo.router("r1").iface_to("ripng").ip6.append("fc00:0:0:1::1/64")
+
+
+class Configs(FRRConfigs):
+    routers = ["r1"]
+
+    zebra = """
+    #% extends "boilerplate.conf"
+    #% block main
+    #%   for iface in router.ifaces
+    interface {{ iface.ifname }}
+     description {{ iface.other.endpoint.name }}
+     no link-detect
+    !
+    #%   endfor
+    !
+    ip forwarding
+    ipv6 forwarding
+    !
+    #% endblock
+    """
+
+    ripd = """
+    #% extends "boilerplate.conf"
+    #% block main
+    debug rip events
+    debug rip zebra
+    !
+    router rip
+     version 2
+     network {{ router.iface_to('rip').ip4[0].network }}
+    #% endblock
+    """
+
+    ripngd = """
+    #% extends "boilerplate.conf"
+    #% block main
+    debug ripng events
+    debug ripng zebra
+    !
+    router ripng
+     network {{ router.iface_to('ripng').ip6[0].network }}
+    #% endblock
+    """
+
+
+class AllStartupTest(TestBase, AutoFixture, topo=topology, configs=Configs):
+    """
+    docstring here
+    """
+
+    @topotatofunc
+    def test_running(self, topo, r1):
+        """
+        just check that all daemons are running
+        """
+        for daemon in Configs.daemons:
+            if not hasattr(Configs, daemon):
+                continue
+            yield from AssertVtysh.make(r1, daemon, command="show version")
+```
+
+2. Run the following command in your command line:
+
+```bash
+./run_userns.sh --frr-builddir=$PATH_TO_FRR_BUILD \
+                --log-cli-level=DEBUG \
+                -v -v -x \ 
+                sameple_test.py 
+```

docs/index.mdx

@@ -0,0 +1,30 @@
+---
+title: Topotato Documentation
+description: Learn how to get started with Topotato for FRRouting
+---
+
+# Welcome to Topotato 🔝🥔
+
+Topotato is a work-in-progress test framework designed for conducting system-level conformance tests for FRRouting.
+Its primary purpose is to execute FRR through various scenarios and validate its behavior.
+
+## Goals
+
+The creation of Topotato was motivated by addressing the limitations of the existing FRR test framework, topotests.
+ Topotato aims to achieve the following objectives:
+
+- Enhance the readability and comprehensibility of tests, particularly when failures occur.
+- Improve the reliability of tests, ensuring consistent outcomes irrespective of factors like system load, parallelization, execution order, operating system, or architecture.
+- Simplify the test-writing process.
+- Minimize the variability in expressing a specific test, ideally reducing it to a single way. This streamlines the process of identifying the correct approach to articulate a test condition, minimizing the potential for creating unstable tests (i.e., flaky tests). A standardized approach to expressing tests also reduces the learning curve and facilitates troubleshooting when a test fails.
+- Enhance the utility of test reports, primarily for failure cases but also for successful ones. Test behavior and reasons for failure should be readily understandable without the need for extensive investigation, debugging statements, or repeated test runs.
+- Enable easy execution of the test suite on developers' local development systems, ensuring accessibility and speed.
+
+## Secondary Goals
+
+In addition to the primary objectives, Topotato also aims to achieve the following secondary goals, which are influenced by the aforementioned aims:
+
+- Encapsulate tests within a single file to eliminate the need to navigate through multiple files.
+- Replace hardcoded IP addresses with semantic expressions to improve readability. For instance, while `192.168.13.57` is an opaque IPv4 address, the expression `r1.iface_to('r2').ip4[0]` could represent the same address while being more comprehensible and easier to maintain.
+- Enable the test suite to run without necessitating "root" access to the system or the installation of FRR. This approach ensures ease of execution and guarantees that the test suite cannot disrupt the developer's system due to kernel-level protection. Additionally, it mitigates issues stemming from broken or mismatched installations.
+- Support FreeBSD.

docs/todos.mdx

@@ -0,0 +1,21 @@
+### TODOs / Known issues
+
+- skipping further tests after a TopotatoModifier failure is not implemented.
+  A TopotatoModifier failure should skip everything after it since a failed
+  modifier means the testbed is in an indeterminate state.
+- some style requirements should be automatically enforced, e.g. missing
+  docstrings should cause a failure.
+- ExaBGP support is work in progress.
+- terminal-on-failure (potatool) is work in progress.
+- integrated-config mode for FRR needs to be supported.
+- FreeBSD support has not been tested & updated in ages and is probably just
+  completely broken right now.
+- `pytest-xdist` interop has not been tested & updated in ages, it probably
+  also breaks in funny and hilarious ways.
+- add more self-tests
+- protomato.js needs a bunch more work.
+    - re-add xrefs lookup to source code
+    - short-decode more protocols
+- an `index.html` file should be generated with an overview of a test run.
+- add a bunch of attributes on the JUnit XML with machine parseable exception
+  location.