Geek and I
Go back

Kubernetes VMware VSphere Infrastructure – Phase 3

Updated:
Mike Horwath5 min read

Preamble

What is Kubespray? (repo)

Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes cluster configuration management tasks.

I found Kubespray recently and thought, well, I know Ansible, and I am learning Kubernetes, so can this help me on my journey?

The answer? Yes

This is a follow-up post to phase 2, where I shared example Terraform HCL files and scripts for deployment of the infrastructure.

BTW: I am using v2.22.1 from the repository.

Table of Contents

Open Table of Contents

Requirements

Based on my example Terraform deployment, I will use the IPv4 side of things for the examples.

You will need to collect the IP addresses of your hosts to fill in the values for your Ansible Playbook inventory.

You also need the username and path to your SSH private key for passwordless access to your hosts.

You must update these items for your configuration, network setup, etc.

You need to clone the repository and check out the release.

Terminal window
git clone https://github.com/kubernetes-sigs/kubespray.git --branch=v2.22.1
cd kubespray
cp -rp inventory/sample inventory/example

WARNING: You should also read the README.md file closely, particularly the need to install some Python modules so that Ansible can do its work, but here is the quick command line that will get it done. I’d highly suggest setting up a Python venv, though below is quick and dirty.

Terminal window
cd kubespray
pip install -r requirements.txt

Within the cloned repo is a directory structure layout (see below), and I will only show examples for the three files listed as they are the bulk of the configurations used in this deployment.

# partial directory layout
inventory/
  example/
    hosts.yaml
    inventory.ini
    group_vars/
      k8s_cluster/
        addons.yml

Below I’ll make sure to have download links for my example files.

Required changes

From my example Terraform variables, I have the following information.

Control-plane nodes:

Worker nodes:

My example uses:

Here are the files relating to the above configuration. I’ll provide links in the download section.

hosts.yaml
all:
  hosts:
    worker1:
      ansible_host: 192.168.7.65
      ip: 192.168.7.65
      access_ip: 192.168.7.65
      ansible_user: linux-user
      ansible_ssh_private_key_file: ~/.ssh/id_rsa
    worker2:
      ansible_host: 192.168.7.66
      ip: 192.168.7.66
      access_ip: 192.168.7.66
      ansible_user: linux-user
      ansible_ssh_private_key_file: ~/.ssh/id_rsa
    worker3:
      ansible_host: 192.168.7.67
      ip: 192.168.7.67
      access_ip: 192.168.7.67
      ansible_user: linux-user
      ansible_ssh_private_key_file: ~/.ssh/id_rsa
    cp1:
      ansible_host: 192.168.7.71
      ip: 192.168.7.71
      access_ip: 192.168.7.71
      ansible_user: linux-user
      ansible_ssh_private_key_file: ~/.ssh/id_rsa
    cp2:
      ansible_host: 192.168.7.72
      ip: 192.168.7.72
      access_ip: 192.168.7.72
      ansible_user: linux-user
      ansible_ssh_private_key_file: ~/.ssh/id_rsa
    cp3:
      ansible_host: 192.168.7.73
      ip: 192.168.7.73
      access_ip: 192.168.7.73
      ansible_user: linux-user
      ansible_ssh_private_key_file: ~/.ssh/id_rsa
  children:
    kube_master:
      hosts:
        cp2:
        cp3:
    kube_control_plane:
      hosts:
        cp3:
        cp2:
        cp1:
    kube_node:
      hosts:
        worker1:
        worker2:
        worker3:
    etcd:
      hosts:
        cp3:
        cp2:
        cp1:
    k8s_cluster:
      children:
        kube_control_plane:
        kube_master:
        kube_node:
    calico_rr:
      hosts: {}
inventory.ini
# ## Configure the 'ip' variable to bind Kubernetes services on a
# ## different ip than the default iface
# ## We should set etcd_member_name for etcd cluster. The node that is not an etcd member does not need to set the value or can set the empty string value.
[all]
# worker1 ansible_host=95.54.0.12  # ip=10.3.0.1 etcd_member_name=etcd1
# worker2 ansible_host=95.54.0.13  # ip=10.3.0.2 etcd_member_name=etcd2
# worker3 ansible_host=95.54.0.14  # ip=10.3.0.3 etcd_member_name=etcd3
# cp1 ansible_host=95.54.0.16      # ip=10.3.0.5 etcd_member_name=etcd5
# cp2 ansible_host=95.54.0.17      # ip=10.3.0.6 etcd_member_name=etcd6
# cp3 ansible_host=95.54.0.18      # ip=10.3.0.7 etcd_member_name=etcd7


# ## configure a bastion host if your nodes are not directly reachable
# [bastion]
# bastion ansible_host=x.x.x.x ansible_user=some_user


[kube_control_plane]
cp1
cp2
cp3


[kube_control_plane:vars]
ansible_user=linux-user
ansible_ssh_private_key_file=~/.ssh/id_rsa


[etcd]
cp1
cp2
cp3


[etcd:vars]
ansible_user=linux-user
ansible_ssh_private_key_file=~/.ssh/id_rsa


[kube_node]
worker1
worker2
worker3
cp1
cp2
cp3


[kube_node:vars]
ansible_user=linux-user
ansible_ssh_private_key_file=~/.ssh/id_rsa


[calico_rr]


[k8s_cluster:children]
kube_control_plane
kube_node
calico_rr

Deployment Configuration

Lastly, there is the actual deployment configuration.

I will include my example configuration below, but I would like you to go through this file and enable the options yourself, as every deployment is different. I have also provided a download link toward the end of this post.

You enable add-ons via the addons.yml file in inventory/example/group_vars/k8s_cluster/addons.yml.

These are the three add-ons I have enabled:

# metrics-server (addons.yml ~line 15)
metrics_server_enabled: true
# ingress-nginx (addons.yml ~line 99)
ingress_nginx_enabled: true
# ingress_nginx_host_network: false
ingress_publish_status_address: ""
ingress_nginx_nodeselector:
  kubernetes.io/os: "linux"
ingress_nginx_tolerations:
  - key: "node-role.kubernetes.io/control-plane"
    operator: "Equal"
    value: ""
    effect: "NoSchedule"
ingress_nginx_namespace: "ingress-nginx"
ingress_nginx_insecure_port: 80
ingress_nginx_secure_port: 443
ingress_nginx_configmap:
  map-hash-bucket-size: "128"
  ssl-protocols: "TLSv1.2 TLSv1.3"
ingress_nginx_configmap_udp_services:
  53: "kube-system/coredns:53"
ingress_nginx_termination_grace_period_seconds: 300
ingress_nginx_class: nginx
ingress_nginx_without_class: true
ingress_nginx_default: true
# cert-manager (addons.yml ~line 139)
cert_manager_enabled: true
cert_manager_namespace: "cert-manager"
cert_manager_tolerations: []
cert_manager_dns_policy: "ClusterFirst"
cert_manager_dns_config:
  nameservers:
    - "1.0.0.1"
    - "1.1.1.1"
cert_manager_controller_extra_args:
  - --default-issuer-name=letsencrypt-prod
  - --default-issuer-kind=ClusterIssuer
  - --default-issuer-group=cert-manager.io

Deploy

Well, it’s time to see if this will all work. I wish you the best of luck!

First, let’s run the reset playbook. This will validate your base configuration (housed in hosts.yaml and inventory.ini):

Terminal window
ansible-playbook -i inventory/example/hosts.yaml --become --become-user=root reset.yml

This takes a little while to run. Within my lab environment, it is less than 15 minutes.

If you have any issues or failures, you will need to address them before you continue.

If all is clean, then it is time to take it to the next level by actually deploying via:

Terminal window
ansible-playbook -i inventory/example/hosts.yaml --become --become-user=root cluster.yml

This takes a little longer to run and was < 25 minutes in my lab. As before, look for issues or failures and address them as needed.

Now, you have a Kubernetes cluster deployed and ready to access it. You need to grab the Kubernetes config that gives you admin access.

Terminal window
ssh -i ~/.ssh/id_rsa linux-user@cp1 sudo cat /etc/kubernetes/admin.conf

You must edit or create your ~/.kube/config file and paste the above information.

Moment of truth:

Terminal window
kubectl get nodes

If successful, you should receive output that looks similar to the following:

NAME      STATUS   ROLES           AGE   VERSION
cp1       Ready    control-plane   8d    v1.27.4
cp2       Ready    control-plane   8d    v1.27.4
cp3       Ready    control-plane   8d    v1.27.4
worker1   Ready    <none>          8d    v1.27.4
worker2   Ready    <none>          8d    v1.27.4
worker3   Ready    <none>          8d    v1.27.4

You should now have a working Kubernetes cluster to start your journey.

Individual files

Each of these links will download the different files you need to have in the same directory you will be operating Terraform out of.

Summary

We have come to the end, and I hope you have been successful.

Related Posts

Previous Post
Kubernetes VMware vSphere infrastructure - Phase 2
Next Post
Kubernetes VMware vSphere infrastructure - Phase 1