SKATO ANSIBLE

Installation

Make sure you have git, venv, and uv installed. On GNU/Linux Debian-based distros with pre-installed apt package manager, the corresponding packages are:

• git
• python3-venv
• Follow this guide for installing uv

Ansible must also be installed on your system (see this guide). Then, in your interactive bash shell session, run the following in order:

git clone https://git.sukaato.moe/admin/skato-ansible.git
cd ./skato-ansible
uv venv .venv
uv sync
source .venv/bin/activate
ansible-galaxy collections install -r ./collections/requirements.yml

Essential Background

Software Management

The standard use-case, which we call the container routing case, of this Ansible library is to run its playbooks in such a way that one host ends up acting as a router (via VPN service) to services on another host.

In the container routing case these services are defined by a repository declared in ${ANSIBLE_PROJECT_ROOT}/vars/source_code.yml, via setting source_code.repos.compose.name and source_code.repos.compose.rpath, respectively, to the name of a version-controlled source repository and to a desired path within that repository (that path written such that it treats the repository path as root, i.e. as /).

As can then be noted, the source_code dictionary holds an attribute repos with itself attributes of fixed name representing various services or utilities, each of whom must have a name attribute defined that represents the name of the corresponding version-controlled source repository for that service or utility.

The point of this set-up is to allow the repository name at its endpoint to change without requiring manual changes in the code using or deploying that repository as consequence. One unfortunate result of this is that adding a service or utiltiy from one's git bare repositories means making manual corresponding changes in pkgs software object groups in relevant host variable files, in ${ANSIBLE_PROJECT_ROOT}/roles/init-server/handlers/{core,userspace}.yml for calling installation or post-installation actions, and in ${ANSIBLE_PROJECT_ROOT}/roles/init-server/tasks/contingent/pkg/${SOFTWARE_NAME}.yml for executing those actions. The convention is for handler notifiers, handler listeners and software names to all be equivalent, though the software name can deviate from this relational pattern.

Importantly, the following top-level attributes of the source_code variable or dictionary are necessary if your version-controlled source repository source requires authentication:

• host
• user/ssh_user
• http_password/ssh_password

If you wish to be able to use both SSH and HTTP authentication methods for the version-controlled source repository or change the default preference for HTTP credential use, you may edit ${ANSIBLE_PROJECT_ROOT}/roles/init-server/templates/user/netrc.j2. See the CURL docuemntation for more information on the NetRC file format.

The point of the source_code dictionary is that it allows plugging in and executing custom code from one's own source repositories.

All other software is simply handled through the aforementioned patttern of adding an entry to any of the following in a given host variables file:

• pkgs.mngr.core or pkgs.mngr.userspace (software installed by package manager--in this case, apt as most tasks assume a Linux Debian system at Ansible's managed node)
• pkgs.script.core or pkgs.script.userspace (since different installation shell scripts allow for different parameters, entries for installation shell scripts here are downloaded--for actual installation, the script has to be ran in a set of tasks specified in a YAML file under ${ANSIBLE_PROJECT_ROOT}/roles/init-server/tasks/contingent/pkg/)
• pkgs.archive.core or pkgs.archive.userspace (similar as before, except things like path movement or specification, building/compilation, etc., are what take place via the set of tasks specified under the YAML file under ${ANSIBLE_PROJECT_ROOT}/roles/init-server/tasks/contingent/pkg/)
• pkgs.git_repos.core or pkgs.git_repos.userspace (similar as before, fulfilling any arbitrary set of specified tasks for the pulled git source repository--you get the point)

Each entry item must have a handler attribute whose value will be used to notify a handler listener, whose listen attribute has that same value, that then includes tasks found in a YAML file under ${ANSIBLE_PROJECT_ROOT}/roles/init-server/tasks/contingent/pkg/ whose basename, conventionally, is the same as those handler and listen attribute values. All of which, again, also conventionally share the same value as the name attribute of the entry item.

The structure of entries within each pkgs.${SOFTWARE_GROUP}.core/pkgs.${SOFTWARE_GROUP}.userspace depends on $SOFTWARE_GROUP because it changes the semantics of installation. All entries are dictionary items.

Note

A table will be added at a later date, covering the attributes of the dictionary items for each $SOFTWARE_GROUP. For now, the host variable files for the staging hosts found in the YAML inventory file can be used as reference.

Configuration Management

System Configuration

The most important configuration is that needed for each host. This involves software installation, though that is already covered in the above section. Other important host variables for any given host variable file are:

variable name	type	default	semantics
password	str	auto-generated	the hashed password for the root user if the managed node had not spawned prior to playbook run
ssh_keys	list	["ed25519@staging", "ecdsa@staging"]	the basenames of the SSH key pairs for the root user
admins	list	[{"username": "senpai", "services": None, "ssh_keys": ["ed25519@staging", "ecdsa@staging"], "password": ""} for d in admins]	the list of Linux superusers, i.e. administrators, to exist
token	str	None	API key for cloud hosting account to spawn a VPS
origin	str	us-east	region or location of the VPS to determine timezones and locale; useful for Akamai Linode cloud hosting
operating_system	str	None	name or path to operating system for cloud hosted VPS or a VM / bare metal system

Software Configuration

Other configuration involves Ansible role templates or files, e.g. those found in ${ANSIBLE_PROJECT_ROOT}/roles/init-server/{templates,files}. Generally, these Ansible role templates or files are sent over to the Ansible managed node by tasks provided by that same role, and referred to by tasks in the playbooks.

The convention for templates is to set some desired though non- software native "defaults" and leave some settings or values to variables, grouped by the software for which they are relevant into variable files in ${ANSIBLE_PROJECT_ROOT}/vars/, or what we may call the playbook variable files. These YAML variable files by convention have as their basename ${SOFTWARE_NAME}_settings (with some exceptions), and must be listed (as paths relative to $ANSIBLE_PROJECT_ROOT) under vars_files on the playbook that will be running the tasks/handlers, native to itself or taken from a role, that send the respective files or (after processing) templates over to the managed Ansible node(s). These kinds of files shall be called "settings files."

Inside each such settings file, the convention is for there to be a top-level variable (i.e., a single variable that takes a dictionary value) with a name of $SOFTWARE_NAME, whose value has attributes representing configuration options for the given software.

Some common, shared attributes for settings files are:

attribute name	type	default value	required?	semantics
containerized	bool	True	only for services or software that is service-capable	determines whether the configuration is for a containerized or base system install
mode	str	"prod"	no	usefully allows for setting the mode for an individual piece of software rather than having it automatically agree with the playbook mode or having it assume production mode

Other common ones, especially for services, are port (which can be an integer or string) and--for web servers--scheme (which can be either string "http" or "https", serving to toggle SSL/TLS DNS authentication for HTTP connections). These can exist at any nesting level under the top-level dictionary, based on the feature scope of the given software.

Essential Usage

Before running any Ansible playbooks in either development or production mode, make sure to specify the username and the home directory path of the user on the system you are currently using in {ANSIBLE_PROJECT_ROOT}/vars/local_facts.yml.

Development Mode

For development purposes in the container routing case, either first manually set up two Debian virtual machines, or (recommended) change into the ${ANSIBLE_PROJECT_ROOT}/staging0.test/ directory then ${ANSIBLE_PROJECT_ROOT}/staging1.test/ directory, running vagrant up for each. Running the playbooks in development mode also requires supplying "dev" as an argument for the extra mode parameter.

Warning

Vagrantfiles for neither staging host has yet been tested. Proceed by your own discretion.

If you intend to run the playbooks in production, the argument for this extra parameter should be "prod" (skip to the Production Mode section). The additional extra parameter of chosen_host should be adjusted to reflect the name of the host enlisted in the YAML inventory file for production purposes.

Important

The names of hosts should be equivalent to their intended final domain name for services on any given host for the playbook to work correctly.

Anyway, for any mode double-check that the correct values are set for chosen_user. The first playbook you run for each host or IP in each pair of ansible-playbook commands in the upcoming shell script should always have chosen_user as "root". The extra parameter chosen_user for the second ansible-playbook command of each pair of ansible-playbook commands shown below should take an argument whose value is the same as that of the username attribute of any of the items in the admins list in the host variables file for the given host. With that in mind, execute the below for development mode for container routing case while in the $ANSIBLE_PROJECT_ROOT directory:

#!/bin/bash
set -euo pipefail

SSH_KEYS=()

sudo ansible-playbook -Kk -i hosts.yml --extra-vars "chosen_host=staging0.test" --extra-vars "chosen_user=root" --extra-vars "mode=dev" init\@vps.yml
$SSH_KEYS | xargs -I %k ssh-add %k # ADD NEEDED SSH KEY FILEPATHS
ansible-playbook -Kk -i hosts.yml --extra-vars "chosen_host=staging0.test" --extra-vars "chosen_user=senpai" --extra-vars "mode=dev" administrate\@vps.yml

sudo ansible-playbook -Kk -i hosts.yml --extra-vars "chosen_host=staging1.test" --extra-vars "chosen_user=root" --extra-vars "mode=dev" init\@homeserver.yml
# UNCOMMENT BELOW TO REASSIGN ENVIRONMENT VARIABLE FOR NEW ARRAY OF SSH KEYPAIR FILEPATHS
# SSH_KEYS=()
# UNCOMMENT BELOW TO ADD UPDATED SSH KEY PAIR FILEPATHS TO SSH_AGENT
# $SSH_KEYS | xargs -I %k ssh-add %k
ansible-playbook -Kk -i hosts.yml --extra-vars "chosen_host=staging1.test" --extra-vars "chosen_user=senpai" --extra-vars "mode=dev" administrate\@homeserver.yml

Warning

In the container routing case, development mode may not supply an accurate test of the intended networking result if the virtual machines for staging were manually created, but the playbook should run successfully. This is because apppropriate networking must be set for the virtual machine itself for VPN client containers in one virtual machine to "speak" with the VPN server in another virtual machine. The supplied Vagrantfiles are a WIP.

Production Mode

Its important to note that the above shell script example for development mode would have to be modified in production mode for the container routing case such that every ansible-playbook call instance has its extra parameter chosen_host share the same value throughout. This is due to the combination of how SSL/TLS and domain names are set up in the playbook and the requirements of the container routing case. Namely, the former treats the inventory hostname used for the playbook as the fully qualified domain name of that server host, while the container routing case requires that both server hosts share that fully qualified domain name.

This means that chosen_host's value being the same across all ansible-playbook commands does not mean that value should represent the same managed node or host IP throughout. Whether it does so is of course determined by whatever is in the YAML inventory file, so that would need to be changed when running playbooks against the second chosen host in the container routing case.

To drive all that home, in production we may instead need something like the following (again, executed while in the $ANSIBLE_PROJECT_ROOT directory):

#!/bin/bash
set -euo pipefail

SSH_KEYS=()
HOST_FQDN=web.site # a fully qualified domain name you own
CURRENT_IP= # whatever the current IP of the host $HOST_FQDN is 
UPDATED_IP= # the desired new IP for the host $HOST_FQDN

sudo ansible-playbook -Kk -i hosts.yml --extra-vars "chosen_host=${HOST_FQDN}" --extra-vars "chosen_user=root" --extra-vars "mode=prod" init\@vps.yml
$SSH_KEYS | xargs -I %k ssh-add %k # ADD NEEDED SSH KEY PAIR FILEPATHS TO SSH_AGENT
ansible-playbook -Kk -i hosts.yml --extra-vars "chosen_host=${HOST_FQDN}" --extra-vars "chosen_user=senpai" --extra-vars "mode=prod" administrate\@vps.yml

sed '/'"$HOST_FQDN"':{N;s/'"$HOST_FQDN"':\n {6}ansible_host: '"$CURRENT_IP"'/'"$HOST_FQDN"':\n      ansible_host: '"$NEW_IP"'/g}' hosts.yml

sudo ansible-playbook -Kk -i hosts.yml --extra-vars "chosen_host=${HOST_FQDN}" --extra-vars "chosen_user=root" --extra-vars "mode=prod" init\@homeserver.yml
# UNCOMMENT BELOW TO REASSIGN ENVIRONMENT VARIABLE FOR NEW ARRAY OF SSH KEYPAIR FILEPATHS
# SSH_KEYS=()
# UNCOMMENT BELOW TO ADD UPDATED SSH KEY PAIR FILEPATHS TO SSH_AGENT
# $SSH_KEYS | xargs -I %k ssh-add %k
ansible-playbook -Kk -i hosts.yml --extra-vars "chosen_host=${HOST_FQDN}" --extra-vars "chosen_user=senpai" --extra-vars "mode=prod" administrate\@homeserver.yml

There is a way to avoid reassigning the IP address of the shared inventory hostname for each host the pair of playbooks is being run against in the container routing case under production. This involves defining fqdn as a playbook variable or as an extra parameter for all ansible-playbook commands in the shell script, resulting in a shell script that looks more similar to our development mode shell script:

#!/bin/bash
set -euo pipefail

SSH_KEYS=()

sudo ansible-playbook -Kk -i hosts.yml --extra-vars "chosen_host=staging0.test" --extra-vars "chosen_user=root" --extra-vars "mode=dev" --extra-vars "fqdn=web.site" init\@vps.yml
$SSH_KEYS | xargs -I %k ssh-add %k # ADD NEEDED SSH KEY FILEPATHS
ansible-playbook -Kk -i hosts.yml --extra-vars "chosen_host=staging0.test" --extra-vars "chosen_user=senpai" --extra-vars "mode=dev" --extra-vars "fqdn=web.site" administrate\@vps.yml

sudo ansible-playbook -Kk -i hosts.yml --extra-vars "chosen_host=staging1.test" --extra-vars "chosen_user=root" --extra-vars "mode=dev" --extra-vars "fqdn=web.site" init\@homeserver.yml
# UNCOMMENT BELOW TO REASSIGN ENVIRONMENT VARIABLE FOR NEW ARRAY OF SSH KEYPAIR FILEPATHS
# SSH_KEYS=()
# UNCOMMENT BELOW TO ADD UPDATED SSH KEY PAIR FILEPATHS TO SSH_AGENT
# $SSH_KEYS | xargs -I %k ssh-add %k
ansible-playbook -Kk -i hosts.yml --extra-vars "chosen_host=staging1.test" --extra-vars "chosen_user=senpai" --extra-vars "mode=dev" --extra-vars "fqdn=web.site" administrate\@homeserver.yml

Tip

Adding the extra parameter here to the ansible-playbook commands of the development mode execution for the container routing use-case may also improve the realism of its execution and final state, virtual machine network configurations notwithstanding. Consider doing so for more realistic networking tests under the container routing case.

Custom Playbooks

As long as you are matching the appropriate set of playbook variable files or settings files with the playbook according to its need of those variable definitions/declarations, mostly determined by the intended targeted hosts' own variables together with role handlers, and as long as you have appropriately set up software installations, (see Software Management) creating new/custom playbooks should be relatively easy.

Depending on what you are trying to do with a new playbook, the best approach may be to just copy/duplicate the extant playbook files and then edit them, as this allows one to mitigate errors as well as have a reliable reference point for changes.

CLI Tool

As an alternative, you can use the skato-ansible shell script in $ANSIBLE_PROJECT_ROOT as an abstraction of the ansible-playbook commands for the container routing case. I plan to replace it with a CLI app covering more intended use-cases.

When using the skato-ansible.sh script in development mode for the container routing use case:

#!/bin/bash
set -euo pipefail

./skato-ansible.sh init vps -v 3 -m dev -n staging0.test
./skato-ansible.sh populate-agent staging
./skato-ansible.sh bootstrap vps -s 1 -v 3 -m dev -n staging0.test

./skato-ansible.sh init homeserver -v 3 -m dev -n staging1.test
# BELOW ONLY IF $SKANSIBLE_SSH_KEY ALTERED IN THE SHELL SCRIPT
# ./skato-ansible.sh populate-agent staging
./skato-ansible.sh bootstrap vps -s 1 -v 3 -m dev -n staging1.test

Otherwise, in production mode:

#!/bin/bash
set -euo pipefail

INVENTORY_HOSTNAMES=(web1 web2)
FQDN=web.site

./skato-ansible.sh init vps -v 3 -n "${INVENTORY_HOSTNAMES[0]}" -d "$FQDN"
./skato-ansible.sh populate-agent staging
./skato-ansible.sh bootstrap vps -s 1 -v 3 -n "${INVENTORY_HOSTNAMES[0]}" -d "$FQDN"

./skato-ansible.sh init homeserver -v 3 -n "${INVENTORY_HOSTNAMES[1]}" -d "$FQDN"
# BELOW ONLY IF $SKANSIBLE_SSH_KEY ALTERED IN THE SHELL SCRIPT
# ./skato-ansible.sh populate-agent staging
./skato-ansible.sh bootstrap vps -s 1 -v 3 -n "${INVENTORY_HOSTNAMES[1]}" -d "$FQDN"

Important

If you have different SSH keypairs for staging, make sure to change the value of SKANSIBLE_SSH_KEYS environment variable in the ${ANSIBLE_PROJECT_ROOT}/skato-ansible.sh shell script before running the above shell scripts from $ANSIBLE_PROJECT_ROOT.