H Container Overview
Edge computing is a recent computing paradigm that brings cloud services closer to the client. Among other features, edge computing offers extremely low client/server latencies. In order to consistently provide such low latencies, services need to run on edge nodes that are physically as close as possible to the clients. Thus, when a client changes physical location, a service should migrate between edge nodes to maintain proximity. Differently from data centers, edge nodes are often built with CPUs of different Instruction Set Architectures (ISA). Thus, a service that is natively compiled for one ISA cannot migrate to another, which hinders migration to the closest node.
H-Containers enable the migration of containerized application, natively compiled, across compute nodes featuring CPUs of different ISAs. H-Containers advance the state-of-the-art on heterogeneous-ISA migration systems on the following aspects:
- High compatibility. No source code or compiler toolchain modifications are needed.
- Easy deployability. H-Containers are fully implemented in user space, without any OS or hypervisor dependencies.
- Greater application support. H-Containers can migrate server applications and dynamically linked binaries.
H-Containers target Linux, adopts LLVM, extends CRIU, and integrates with Docker. Experimental studies reveal that H-Containers add very little overhead during program execution and between 10ms and 100ms during execution migration. In real-world scenarios, H-Containers increase throughput – e.g., up to 94% increase in Redis throughput on heterogenous-ISA platforms.
H-Containers consists of a compiler toolchain based on LLVM and mcsema, as well as a redesigned version of the CRIU tool. Additionally, H-Container comes with Docker support. Finally, H-Container's main technology that enables migration among heterogeneous-ISA machines comes from the Popcorn Project.
H-Containers requires the user to first prepare all the application binaries that make up a Linux container to be able to migrate across heterogeneous-ISA machines, this is provided by the compiler toolchain. The redesigned version of the CRIU tool is able to stop the binary on a machine of an ISA, and restart it on a machine of a different ISA. In this way, H-Container provides checkpoint restart of containerized applications across machines of diverse ISA. At the moment we only support ARM 64bit and x86 64bit. Notwithstanding, CRIU's live migration capabilities are also supported.
H-Containers' compiler toolchain doesn't need the source code to create binaries that can migrate between machines of diverse ISA. This is achieved by decompiling the entire application into assembly language and then lifting it back to LLVM IR. This step is called "decompiler" and depicted in the following picture. H-Containers decompiler uses mcsema. When the code is in LLVM IR H-Containers uses a modified version of Popcorn compiler framework, including a new musl libc version, to generate binaries that can migrate among machines of a different ISA. This is the "compiler" step in the figure. If the source code is available only the "compiler" step is needed.
CRIU enables checkpoint and restart among machines with the same CPU. Heterogeneous CRIU extends CRIU to enable migration across ARM and x86 and the contrary. In the following picture a container with 3 applications is depicted. Differently from CRIU, we use a new tool to freeze the applications in the container that we call Popcorn-notify ("notify" in the picture). When the applications are frozen, we take a checkpoint by using standard CRIU, which outputs a set of image files. These files are transformed by our modified CRIU from one ISA format (blue) to the other (yellow), and transferred to another machine where they will be restored with vanilla CRIU.
