Background

This document captures my "copy/paste" and other details as I try to setup a k3 cluster.

Walkthrough

Using the k3s-ansible repository

There is a k3s-ansible repository that may be easier to work with.  I'll fork it and then use it as a basis of my own version.

Repositoryhttps://github.com/tlhakhan/k3s-ansible


# get helm
https://github.com/helm/helm/releases

# get the linux amd64 version
wget https://get.helm.sh/helm-v3.8.2-linux-amd64.tar.gz
tar -zxvf helm-v3.8.2-linux-amd64.tar.gz
cd linux-amd64
mv helm /sbin
cd ..
rm -rf linux-amd64

# add helm stable repo
helm repo add rancher-stable https://releases.rancher.com/server-charts/stable

# add namespace
kubectl create namespace cattle-system

# helm needs to know location of kubeconfig file
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

# setup
helm install rancher rancher-stable/rancher \
  --namespace cattle-system \
  --set hostname=rancher.tenzin.io \
  --set replicas=3 \
  --set ingress.tls.source=secret

# DNS was created for rancher.tenzin.io and CNAMEd to kube-1, the master node.
# VMware vSwitch needs to be configured with promiscuous mode, forged transmits and MAC changes.

Playbook breakdown

Inventory breakdown

---
k3s_version: v1.22.3+k3s1
ansible_user: sysuser
systemd_dir: /etc/systemd/system
master_ip: "{{ hostvars[groups['master'][0]]['ansible_host'] | default(groups['master'][0]) }}"
extra_server_args: ""
extra_agent_args: ""

Notes


[master]
kube-1

[node]
kube-2
kube-3

[k3s_cluster:children]
master
node

Notes

Entrypoint breakdown

---
- hosts: k3s_cluster
  gather_facts: yes
  become: yes
  roles:
    - role: prereq
    - role: download

- hosts: master
  become: yes
  roles:
    - role: k3s/master

- hosts: node
  become: yes
  roles:
    - role: k3s/node

Notes

Prereq role breakdown

prereq
└── tasks
    └── main.yml

This role appears to setup some general OS specific requirements for Kubernetes to work.

# lsmod | grep netfilter
br_netfilter           32768  0
bridge                253952  1 br_netfilter
# sysctl -a | grep net.bridge.bridge-nf-call-ip
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

Download role breakdown

download
└── tasks
    └── main.yml

This role focuses only on downloading the binary and verifying its checksum.  It tries to target three platforms: x64, arm64 and armhf.

k3s/master role breakdown

k3s/master
├── defaults
│   └── main.yml
├── tasks
│   └── main.yml
└── templates
    └── k3s.service.j2

This role is run on the master node(s).  Presently it's designed for a single master and will need to be enhanced for a HA master setup.

The defaults contain only a single variable.

k3s_server_location: /var/lib/rancher/k3s

The k3s.service.j2 template.

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
After=network-online.target

[Service]
Type=notify
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s server --data-dir {{ k3s_server_location }} {{ extra_server_args | default("") }}
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

The tasks does the following:

- name: Replace https://localhost:6443 by https://master-ip:6443
  command: >-
    k3s kubectl config set-cluster default
      --server=https://{{ master_ip }}:6443
      --kubeconfig ~{{ ansible_user }}/.kube/config
  changed_when: true

k3s/node role breakdown

k3s/node
├── tasks
│   └── main.yml
└── templates
    └── k3s.service.j2

This role runs on the worker node and setups the service k3s-node.service and joins it to the control plane (master).

The template sets up the systemd service.  

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
After=network-online.target

[Service]
Type=notify
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s agent --server https://{{ master_ip }}:6443 --token {{ hostvars[groups['master'][0]]['token'] }} {{ extra_agent_args | default("") }}
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

Appendix


DocumentLink
1k3s.iohttps://k3s.io/
2Latest k3s releasehttps://github.com/rancher/k3s/releases/latest
3Installing rancher on k3shttps://rancher.com/docs/rancher/v2.5/en/installation/install-rancher-on-k8s/