NixOS as Container Host

Background

#

My previous attempts always failed miserable due to the enormous differences¹ of NixOS compared to traditional Linux distributions and the steep learning curve. It’s like if you stumbled over one thing you don’t understand, you probably come across two more things, you don’t understand, while trying to figure out the first thing 😆.

Currently, my main machine is a MacBook Pro M1 after 10 years of ThinkPads with Linux², but I still owe my ThinkPad X1 9th Generation which acts as a backup device. I was unhappy with Majaro and I have too much free time (not) so I came up with the idea to give NixOS another shoot.

This time, instead of running NixOS in a virtual machine or trying to replicate a server with NixOS, I wanted to achieve something different ³.

Goals

#

• Running a very stable and (kind of) immutable OS.
• Everything is configured as code. I am a huge advocate of the Infrastructure as Code principle utilizing tools like Terraform, Ansible, Flux2, etc. My complete system should be configurable via code, respectively more generic, via “text files”.
• Be able to reproducible configure my ThinkPad from scratch.
• My existing dotfiles managed via chezmoi must be used and should still work with my MacBook Pro.
• Be able to create dedicated environments for different customers/projects.

Constraints

#

• Focus on the very basic features of NixOS to reduce the learning curve.
• Omit Flakes to further reduce the learning curve and to be honest, I have still a hard time understanding flakes.
• Omit home-manager because not all my machines will run home-manager and I don’t want to maintain multiple dotfiles and especially duplicate my Neovim configuration.
• Use i3wm as window manager because I have experience with i3 and its configuration is straight forward within a single file.
• Single user and single machine only (for now).

Approach

#

My plan is to use NixOS as a base system / host system with very little basic programs / tools. On top of NixOS I will spawn distrobox containers including all the tools necessary for my workflows. Those containers are disposable as a) no data is stored inside and b) they are automatically configured.

NixOS aka the host system

#

Syncthing

#

In order to set up my dotfiles, I require my SSH key and my Age key to be able to run chezmoi. I am a heavy user of Syncthing to sync my files across devices, relying on my home lab as a central “data sink”. To set up Syncthing to “pull” my SSH and my Age key, I use the following snippet in my configuration.nix file:

  services = {
    syncthing = {
      enable = true;
      user = "michael";
      configDir = "/home/michael/.config/syncthing";
      overrideDevices = true;     # overrides any devices added or deleted through the WebUI
      overrideFolders = true;     # overrides any folders added or deleted through the WebUI
      extraOptions = {
        globalAnnounceEnabled = false;
      }
      devices = {
        # https://docs.syncthing.net/users/faq.html#should-i-keep-my-device-ids-secret
        "unraid" = { id = "42GWJCT-VAONXMN-UNQVPRX-MVX6VHC-CSFKYFI-7MJX7QT-7VPK7SV-XJUFHAG"; addresses = [ "tcp://192.168.178.62:22222" ]; };
      };
      folders = {
        "secrets" = {                  # Label of the folder
          id = "lkumh-nvc74";          # ID of the folder
          path = "~/.secrets/";        # Which folder to add to Syncthing
          devices = [ "unraid" ];      # Which devices to share the folder with
          type = "receiveonly";        # One of "sendreceive" "sendonly" "receiveonly" "receiveencrypted"
          ignorePerms = false;         # Whether to ignore permission changes
        };
      };
    };
  };

Packages

#

As said, NixOS should only act as a host system for my DistroBox containers and only run mostly GUI and very basic tools.

My system packages managed by Nix are as follows:

environment.systemPackages = with pkgs; [
    acpi # battery interfae
    arandr # simple app to configure displays
    brightnessctl # for manipulating screen brightness
    curl
    gcc
    git
    gnumake
    htop
    networkmanagerapplet # tray icon
    pulseaudio # for audio controls
    unzip
    wget
    xorg.xhost # give permission to access X-server from a container
    zip
];

My user packages managed by Nix are as follows:

packages = with pkgs; [
    firefox # best browser
    alacritty # terminal application
    xautolock # automatic screen locking
    flameshot # Screenshots
    keepassxc # password manager
    distrobox # Linux distribution as Podman/Docker
];

I consider this packages as common and very basic, so they are managed at the host level.

Podman

#

To enable Podman in NixOS, I added the following snippet to my configuration.nix:

  virtualisation = {
    podman = {
      enable = true;
      # Create a `docker` alias for podman, to use it as a drop-in replacement
      dockerCompat = true;
      # Required for containers under podman-compose to be able to talk to each other.
      defaultNetwork.settings = {
        dns_enabled = true;
      };
    };
  };

Now, I can utilize familiar Docker commands like docker ps.

DistroBox

#

DistroBox allows you to

Use any Linux distribution inside your terminal.

With DistroBox I can create disposable and project-/client-specific containers without altering the host system ⁴

Create the default container

#

This container acts as my default container for non project specific tasks.

distrobox-create --name default --init --image docker.io/library/archlinux:latest
distrobox-enter default

--name sets the name of the container and its hostname. --init enables the init system inside the container which also prevents from accessing the host’s processes. --image specifies which Docker image to use.

Configure the default container

#

For now, I only run a simple bash script to configure the container, but I could switch to e.g. Ansible for a more sophisticated approach.

#!/bin/bash
set -e

sudo pacman --noconfirm -S base-devel ueberzug

# Install brew
NONINTERACTIVE=1 /bin/bash -c \
	"$(curl -fsSL \
		https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

export PATH=/home/linuxbrew/.linuxbrew/bin:$PATH

# Install packages
brew install \
	age \
	chezmoi \
	fd \
	gcc \
	git \
	gitui \
	go \
	java \
	lf \
	libnotify \
	nodejs \
	ripgrep \
	tig \
	tmux \
	trash-cli \
	xclip \
	zoxide

# Install Neovim nightly
brew install neovim --head

# Download my dotfiles bootstrap script
wget https://raw.githubusercontent.com/Allaman/dots/main/bootstrap.sh
chmod +x bootstrap.sh

Host integration

#

My default terminal shortcut opens a terminal within my default container:

bindsym $mod+Return exec --no-startup-id alacritty -e distrobox-enter default

In order to allow sharing the system clipboard between containers xhost is called at startup.

exec --no-startup-id xhost +si:localuser:$USER

Nix shell

#

If you are familiar with Nix(OS) you might ask why are you not using nix shell respectively nix-shell? A legitimate question!

To be honest, the answer is because of Neovim. My Neovim config depends on Mason for downloading the required binaries for my config⁵. Due to the nature of NixOS, running binaries does not always work⁶. Regarding my setup, stylua, lua-language-server, and marksman are affected binaries with presumably hard coded paths. There are solutions described in ⁶, but they are a little bit challenging for me.

As I mentioned earlier, I am not willing to rewrite my Neovim configuration. With this approach, I can circumvent this limitation by running Neovim inside a container with a “normal” Linux file system where paths are available.

Conclusion

#

With this approach I can combine the best of two worlds. NixOS as a rock solid unbreakable and sort of immutable host system and Distrobox providing a layer where “normal Linux rules” apply.