Bobby Bruce has submitted this change. ( https://gem5-review.googlesource.com/c/public/gem5/+/71098?usp=email ) Change subject: util: Add util for GitHub runner configuration ...................................................................... util: Add util for GitHub runner configuration This adds files that can be used to configure Vagrant machines that will be used to test running gem5 alongside Github Actions. Change-Id: I52b0f39b6e6044c22481f02163d5fc01eab76788 Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/71098 Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu> Maintainer: Bobby Bruce <bbruce@ucdavis.edu> Tested-by: kokoro <noreply+kokoro@google.com> --- A util/github-runners-vagrant/README.md A util/github-runners-vagrant/Vagrantfile-builder A util/github-runners-vagrant/Vagrantfile-runner A util/github-runners-vagrant/provision_nonroot.sh A util/github-runners-vagrant/provision_root.sh A util/github-runners-vagrant/vm_manager.sh 6 files changed, 312 insertions(+), 0 deletions(-) Approvals: Bobby Bruce: Looks good to me, approved; Looks good to me, approved kokoro: Regressions pass diff --git a/util/github-runners-vagrant/README.md b/util/github-runners-vagrant/README.md new file mode 100644 index 0000000..86b9e1d --- /dev/null +++ b/util/github-runners-vagrant/README.md @@ -0,0 +1,118 @@ +# Setting up a Github Actions Runner with Vagrant + +This directory provides a way to setup a Github Actions runner using Vagrant to host the runner in a Virtual machine. + +This tutorial has been written with the assumption of running on a machine with Ubuntu 22.04. +Setting up a runner on a different OS may require some changes. + +Before anything else, copy this directory, "util/github-runners-vagrant", to the root of the location on your host system you wish to setup the VMs from. +The CWD is assumed to be this directory. + +## Install Dependencies + +```sh +sudo apt install vagrant +sudo apt-get build-dep vagrant ruby-libvirt +sudo apt-get install qemu libvirt-daemon-system libvirt-clients ebtables dnsmasq-base libxslt-dev libxml2-dev libvirt-dev zlib1g-dev ruby-dev + +# Note: The vagrant-libvirt APT package does not work as intended. We must +# remove it from the system otherwise errors will occur (we will install it +# later using the vagrant plugin command). +sudo apt purge vagrant-libvirt +``` + +## Set up the Vagrantfiles for the GitHub repository + +First, generate a Personal Access Token, which you can create [here](https://github.com/settings/tokens) +Make sure to set admin permissions on this token, then replace instances of `<PERSONAL ACCESS TOKEN>` in the Vagrantfiles ("Vagrantfile-builder" and "Vagrant-runner") with your token. + +Next, replace instances of `<GITHUB REPO>` with your GitHub account name and the repository name, separated by a forward slash. +For example, if your GitHub account name is `example` and your repository name is `example-repo`, you would replace `<GITHUB REPO>` with `example/example-repo`. + +## Install Vagrant Plugins + +Once everything is set properly, set the `VAGRANT_HOME` environment variable to the directory in which the Vagrant files and other scripts are stored (i.e., the CWD). +For example: + +```sh +export VAGRANT_HOME=`pwd` +``` + +After this, install the relevant vagrant plugins: + +``` sh +vagrant plugin install dotenv +vagrant plugin install vagrant-libvirt +vagrant plugin install vagrant-reload +``` + +## The "builder" and "runner" VMs + +The number of CPUs and the memory size differs between the "Vagrantfile-builder" and "Vagrantfile-runner". + +In our work we have two types of machines "runners" and "builders". +Runners are single core machines with 8GB of memory, and builders are 4 core machines with 16GB of memory. +The latter is used for building gem5 binaries while the former is used for running instances of gem5. +You can expect each machine to take up approximately 60GB of disk space though VMs will consume the disk space they require. + +The "Vagrantfile-builder" file is set to create a runner machine and the "Vagrantfile-builder" file is set to create a builder machine. + +Specifying which Vagrantfile to use is done by setting the `VAGRANT_VAGRANTFILE` environment variable. + +## Creating the virtual machine + +Each VM on your host system must have a unique name. +Give the VM to be created a unique name by setting the `<VM NAME>` variables in the Vagrantfile you wish to utilize. + +Then run: + +```sh +VAGRANT_VAGRANTFILE=<VAGRANTFILE> vagrant up --provider=libvirt +``` + +This should automatically create your machine, as well as configure and start up a Github Actions runner. +You can check the status of the runner here: https://github.com/<account>/<repo>/settings/actions/runners + +If the runner ever shows as offline, you can rerun the `vagrant up --provider=libvirt` command to make sure everything is working properly. + +If you wish to create more than one runner you must edit the `<VM NAME>` in the Vagrant file. + +## Helper scripts + +The "vm_manager" script can be used to set up multiple builder and runner VMs. +To use this script simply modify the `NUM_RUNNERS`, `NUM_BUILDERS`, `RUNNER_PREFIX`, and `BUILDER_PREFIX` variables to the desired values. +Then run the script with: + +```sh +./vm_manager.sh +``` + +This script will create any VMs that don't already exist and ensure those that do exists are running. + +If you wish to destroy all the VMs you can run: + +```sh +./vm_manager.sh destroy +``` + +**Note:** This script assumes "VAGRANT_HOME" is set to the CWD. + +## Troubleshooting + +### The default libvirt disk image storage pool is on the wrong drive + +By default libvirt will store disk images in "/var/lib/libvirt/images". +This is not ideal as it is on a small root partition. +A solution to this is to change the default storage location. +To do so, do the following: + +```sh +virsh pool-list --all # Confirm here a "default" pool exist. We'll modify this. +virsh pool-dumpxml default >default-pool.xml # We take a dump of the default then removed it. +virsh pool-destroy default +virsh pool-undefine default +vim default-pool.xml # Change the image path to the desired path +virsh pool-define default-pool.xml # From here we re-add the default. +virsh pool-start default +virsh pool-autostart default +``` diff --git a/util/github-runners-vagrant/Vagrantfile-builder b/util/github-runners-vagrant/Vagrantfile-builder new file mode 100644 index 0000000..242e094 --- /dev/null +++ b/util/github-runners-vagrant/Vagrantfile-builder @@ -0,0 +1,49 @@ +# -*- mode: ruby -*- +# vi: set ft=ruby : + +Vagrant.configure("2") do |config| + config.vm.box = "generic/ubuntu2204" + config.vm.box_check_update = true + config.vm.define "<VM NAME>" + config.vm.hostname = "<VM NAME>" + # allows us to ssh into the machine, addressing the problem below + # https://www.reddit.com/r/vagrant/comments/sb7hfl/new_to_vagrant_getting_efault_warning/ + config.ssh.username = "vagrant" + config.ssh.password = "vagrant" + + config.vm.provider "libvirt" do |vb| + # Customize the amount of cpus and memory on the VM: + vb.cpus = "4".to_i + vb.memory = "16384".to_i + end + + # sets up vm + config.vm.provision :shell, path: "provision_root.sh" + config.vm.provision :shell, privileged: false, path: "provision_nonroot.sh" + # The provision_root.sh adds the vagrant user to the docker group, so we need to reload the VM. + config.vm.provision :reload + config.vm.provision :shell, privileged: false, run: 'always', inline: <<-SHELL + if [ -d ~/actions-runner ]; then + # This will be run everytime the VM is run (once created). + cd actions-runner + nohup ./run.sh & + else + # This will be run the first time the VM is created. + mkdir ~/actions-runner && cd ~/actions-runner + curl -so actions-runner-linux-x64-2.304.0.tar.gz -L \ + https://github.com/actions/runner/releases/download/v2.304.0/actions-runner-linux-x64-2.304.0.tar.gz + tar xzf ./actions-runner-linux-x64-2.304.0.tar.gz + + # configure the runner + # echo automatically sets the name of the runner, and the tags + # create a personal access token with admin permission and copy it into the curl command + echo -ne '\n\nbuild\n\n' | ./config.sh --url https://github.com/<GITHUB REPO> --token $(curl -L \ + -X POST -H "Accept: application/vnd.github+json" -H "Authorization: Bearer <PERSONAL ACCESS TOKEN>" \ + -H "X-GitHub-Api-Version: 2022-11-28" \ + https://api.github.com/repos/<GITHUB REPO>/actions/runners/registration-token | jq -r '.token') + # start the runner + nohup ./run.sh & + fi + SHELL + +end diff --git a/util/github-runners-vagrant/Vagrantfile-runner b/util/github-runners-vagrant/Vagrantfile-runner new file mode 100644 index 0000000..ab601cd --- /dev/null +++ b/util/github-runners-vagrant/Vagrantfile-runner @@ -0,0 +1,48 @@ +# -*- mode: ruby -*- +# vi: set ft=ruby : + +Vagrant.configure("2") do |config| + config.vm.box = "generic/ubuntu2204" + config.vm.box_check_update = true + config.vm.define "<VM NAME>" + config.vm.hostname = "<VM NAME>" + # allows us to ssh into the machine, addressing the problem below + # https://www.reddit.com/r/vagrant/comments/sb7hfl/new_to_vagrant_getting_efault_warning/ + config.ssh.username = "vagrant" + config.ssh.password = "vagrant" + + config.vm.provider "libvirt" do |vb| + # Customize the amount of cpus and memory on the VM: + vb.cpus = "1".to_i + vb.memory = "8192".to_i + end + + # sets up vm + config.vm.provision :shell, path: "provision_root.sh" + config.vm.provision :shell, privileged: false, path: "provision_nonroot.sh" + # The provision_root.sh adds the vagrant user to the docker group, so we need to reload the VM. + config.vm.provision :reload + config.vm.provision :shell, privileged: false, run: 'always', inline: <<-SHELL + if [ -d ~/actions-runner ]; then + # This will be run everytime the VM is run (once created). + cd actions-runner + nohup ./run.sh & + else + # This will be run the first time the VM is created. + mkdir ~/actions-runner && cd ~/actions-runner + curl -so actions-runner-linux-x64-2.304.0.tar.gz -L \ + https://github.com/actions/runner/releases/download/v2.304.0/actions-runner-linux-x64-2.304.0.tar.gz + tar xzf ./actions-runner-linux-x64-2.304.0.tar.gz + + # configure the runner + # echo automatically sets the name of the runner, and the tags + # create a personal access token with admin permission and copy it into the curl command + echo -ne '\n\nrun\n\n' | ./config.sh --url https://github.com/<GITHUB REPO> --token $(curl -L \ + -X POST -H "Accept: application/vnd.github+json" -H "Authorization: Bearer <PERSONAL ACCESS TOKEN>" \ + -H "X-GitHub-Api-Version: 2022-11-28" \ + https://api.github.com/repos/<GITHUB REPO>/actions/runners/registration-token | jq -r '.token') + # start the runner + nohup ./run.sh & + fi + SHELL +end diff --git a/util/github-runners-vagrant/provision_nonroot.sh b/util/github-runners-vagrant/provision_nonroot.sh new file mode 100644 index 0000000..4465b5a --- /dev/null +++ b/util/github-runners-vagrant/provision_nonroot.sh @@ -0,0 +1,15 @@ +#!/usr/bin/env bash + +# fail on unset variables and command errors +set -eu -o pipefail # -x: is for debugging + +# Install deno +curl -fsSL https://deno.land/x/install/install.sh | sh +echo "export PATH=\"\${HOME}/.deno/bin:\${PATH}\"" >> ~/.profile +echo "export PATH=\"\${HOME}/.deno/bin:\${PATH}\"" >> ~/.bash_profile + +# Install docker compose +DOCKER_COMPOSE_VERSION=$(curl -s https://api.github.com/repos/docker/compose/releases/latest | jq -r '.tag_name') +mkdir -p "${HOME}/.docker/cli-plugins" +curl -sL "https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-$(uname -s)-$(uname -m)" -o "${HOME}/.docker/cli-plugins/docker-compose" +chmod +x "${HOME}/.docker/cli-plugins/docker-compose" diff --git a/util/github-runners-vagrant/provision_root.sh b/util/github-runners-vagrant/provision_root.sh new file mode 100644 index 0000000..660064f --- /dev/null +++ b/util/github-runners-vagrant/provision_root.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash + +# fail on unset variables and command errors +set -eu -o pipefail # -x: is for debugging + +apt-get update +apt-get upgrade -y +apt-get install -y software-properties-common +add-apt-repository --yes --update ppa:git-core/ppa +apt-get install -y \ + bash \ + build-essential \ + clang-format \ + git \ + git-lfs \ + jq \ + libffi-dev \ + libssl-dev \ + nkf \ + python3 \ + python3-dev \ + python3-pip \ + python3-venv \ + shellcheck \ + tree \ + wget \ + yamllint \ + zstd +snap install jq + +# Install docker +apt-get install -y apt-transport-https ca-certificates curl gnupg lsb-release +curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg +echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null +apt-get update -y +apt-get install -y docker-ce docker-ce-cli containerd.io +groupadd docker || true +gpasswd -a vagrant docker +newgrp docker +systemctl restart docker + +# Cleanup +apt-get autoremove -y diff --git a/util/github-runners-vagrant/vm_manager.sh b/util/github-runners-vagrant/vm_manager.sh new file mode 100755 index 0000000..1cae8d2 --- /dev/null +++ b/util/github-runners-vagrant/vm_manager.sh @@ -0,0 +1,39 @@ +#!/bin/bash + +NUM_RUNNERS=20 +NUM_BUILDERS=3 +RUNNER_PREFIX="$(hostname)-runner-" +BUILDER_PREFIX="$(hostname)-builder-" + +SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) +export VAGRANT_HOME=${SCRIPT_DIR} + +param="up" +if [[ $# -eq 1 ]]; then + param=$1 + if [[ "${param}" != "destroy" ]]; then + echo "Only valid parameter is 'destroy', to destroy all VMs" + exit 1 + fi +fi + + +for (( i=1; i<=NUM_RUNNERS; i++ )); do + sed -i "s/ config.vm.define.*/ config.vm.define \"${RUNNER_PREFIX}${i}\"/g" Vagrantfile-runner + sed -i "s/ config.vm.hostname.*/ config.vm.hostname = \"${RUNNER_PREFIX}${i}\"/g" Vagrantfile-runner + if [[ "${param}" == "destroy" ]]; then + VAGRANT_VAGRANTFILE=Vagrantfile-runner vagrant destroy -f + else + VAGRANT_VAGRANTFILE=Vagrantfile-runner vagrant up --provider=libvirt + fi +done + +for (( i=1; i<=NUM_BUILDERS; i++ )); do + sed -i "s/ config.vm.define.*/ config.vm.define \"${BUILDER_PREFIX}${i}\"/g" Vagrantfile-builder + sed -i "s/ config.vm.hostname.*/ config.vm.hostname = \"${BUILDER_PREFIX}${i}\"/g" Vagrantfile-builder + if [[ "${param}" == "destroy" ]]; then + VAGRANT_VAGRANTFILE=Vagrantfile-builder vagrant destroy -f + else + VAGRANT_VAGRANTFILE=Vagrantfile-builder vagrant up --provider=libvirt + fi +done -- To view, visit https://gem5-review.googlesource.com/c/public/gem5/+/71098?usp=email To unsubscribe, or for help writing mail filters, visit https://gem5-review.googlesource.com/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: public/gem5 Gerrit-Branch: develop Gerrit-Change-Id: I52b0f39b6e6044c22481f02163d5fc01eab76788 Gerrit-Change-Number: 71098 Gerrit-PatchSet: 6 Gerrit-Owner: Melissa Jost <melissakjost@gmail.com> Gerrit-Reviewer: Bobby Bruce <bbruce@ucdavis.edu> Gerrit-Reviewer: Gabe Black <gabe.black@gmail.com> Gerrit-Reviewer: kokoro <noreply+kokoro@google.com> Gerrit-CC: kokoro <noreply+kokoro@google.com>