9. Creating Kubernetes Clusters on the Cloud

The virtual machines provided by CloudVeneto can also be used to deploy Kubernetes clusters where users familiar with Kubernetes can run their applications. Kubernetes (K8S) is the most popular production-grade container orchestrator.

In this chapter we explain how to deploy a Kubernetes cluster using Ansible.

9.1. Intro: the idea

The provided Ansible playbooks allow you to deploy a Kubernetes cluster both on bare metal and on an OpenStack cloud. The installation is based on the kubeadm tool configured with a pre-generated admin token and flannel network. The playbooks can enrich the cluster installation with a set of services such as:

  • Dashboards: K8S legacy and Grafana;
  • Monitoring: Prometheus operator;
  • Big Data Analytics: Apache Spark and Kafka operators.

9.2. System requirements

The deployment environment requires:

  • Ansible 2.5.0+ (on the user client)
  • Ubuntu 18.04 (for master and node images)
  • 2 CPUs or more per machine, 2 GB or more of RAM per machine (any less will leave little room for your apps)

9.3. Getting Started

This section represents a quick installation and is not intended to teach you about all the components. The easiest way to get started is to clone the ‘ansible-k8s’ repository:

# git clone https://github.com/zangrand/ansible-k8s.git
# cd ansible-k8s

The directory structure should be like:

ansible-k8s/
+-- config
|   +-- config
|   +-- keystone_client.py
|   +-- tls-ca-bundle.pem
+-- examples
|   +-- spark-pi.yaml
|   +-- kcluster.yaml
|   +-- ktopic.yaml
+-- deploy_k8s.yaml
+-- deploy_master_openstack.yaml
+-- deploy_node_openstack.yaml
+-- group_vars
|   +-- all
+-- inventory
+-- k8s
|   +-- alertmanager-service.yaml
|   +-- dashboard-setup.yaml
|   +-- grafana-service.yaml
|   +-- os-k8s-node.yaml
|   +-- prometheus-service.yaml
+-- openstack_config.yaml
+-- README.md
+-- roles
    +-- auth
    |   +-- keystone
    |       +-- files
    |       |   +-- infn_ca.pem
    |       |   +-- k8s-auth-policy.yaml
    |       |   +-- k8s-keystone-auth.yaml
    |       |   +-- k8s-keystone-auth.yaml_orig
    |       |   +-- keystone_client.py
    |       |   +-- tls-ca-bundle.pem
    |       |   +-- webhookconfig.yaml
    |       +-- tasks
    |           +-- main.yml
    +-- common
    |   +-- tasks
    |       +-- main.yml
    +-- docker
    |   +-- tasks
    |       +-- main.yml
    +-- haproxy
    |   +-- tasks
    |   |   +-- main.yml
    |   +-- templates
    |       +-- haproxy.cfg.j2
    +-- keepalived
    |   +-- tasks
    |       +-- main.yml
    |   +-- templates
    |       +-- keepalived.conf.j2
    +-- kubeadm
    |   +-- tasks
    |       +-- main.yml
    +-- master
    |   +-- handlers
    |   |   +-- main.yml
    |   +-- tasks
    |       +-- main.yml
    +-- masterha
    |   +-- handlers
    |   |   +-- main.yml
    |   +-- tasks
    |       +-- main.yml
    |   +-- templates
    |       +-- kubeadm-config.yaml.j2
    +-- node
    |   +-- tasks
    |      +-- main.yml
    +-- os-node
    |  +-- tasks
    |      +-- main.yml
    +-- prometheus
    |   +-- tasks
    |       +-- main.yml
    +-- spark
        +-- tasks
            +-- main.yml

9.4. Deployment on the cloud

The provided Ansible playbook is able to create and configure properly all hosts (i.e. the VMs) on CloudVeneto and deploy Kubernetes on it. To do it:

  • edit the file openstack_config.yaml and fill up all required attributes (i.e. OS_USERNAME, OS_PASSWORD, OS_PROJECT_NAME, OS_PROJECT_ID, OS_NETWORK, etc), the same used for accessing OpenStack by its client;
  • define the VMs characteristics of the master and nodes, in terms of name, flavor, and image (default values for flavor and image are defined);
  • specify the number of nodes (i.e. OS_NODES) of your cluster.

Look at the comments inside the openstack_config.yaml file for more details.

Verify if the ‘shade’ Python module is available on your environment, otherwise install it:

# pip install shade

Execute:

# export ANSIBLE_HOST_KEY_CHECKING=False
# ansible-playbook deploy_master_openstack.yaml

Note

The deployment requires a few minutes to have the full cluster up and running.

9.5. How to access your Kubernetes cluster

There are two different ways to access the Kubernetes cluster: the kubectl command line tool or the dashboard.

9.5.1. Kubectl

The kubectl command line tool is available on the master node. If you wish to access the cluster remotely please see the following guide: Install and Set Up kubectl.

You can enable your local kubectl to access the cluster through the Keystone authentication. To do it, copy all files contained into the folder ansible-k8s/config/ to $HOME/.kube/.

The tls-ca-bundle.pem file is CA certificate required by the CloudVeneto OpenStack based cloud. Do not forget to source the openrc.sh with your OpenStack credentials and OS_CACERT variable set.

The only manual configuration required is to edit $HOME/.kube/config and set the IP address of your new K8S master.

To allow other users to access your K8S cluster and operate on a subset of its resources, edit the auth-policy file with:

# kubectl -n kube-system edit configmap k8s-auth-policy

modify in the first block the line “resources” and replace “type”: “role”, “values”: [“k8s-user”] with e.g. “type”: “user”, “values”: [“username1”, “username2”]

9.5.2. Kubernetes Dashboard

The cluster exposes the following dashboards:

To login into the K8S dashboard use the token of the kube-system:default service account. To get it, execute the following command from your environment, or from the master node:

# kubectl -n kube-system describe secret kubernetes-dashboard
[...]
Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1025 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3Vu
dC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWU
iOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi05cGc5NyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2
VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY
291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjAyYjYzZGRkLWFhYzItMTFlOC1iNTBkLWZhMTYzZTQ2OWU0ZiIsInN1
YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.XPSzu31Svk
R_xwfd3MpLQBkHc7anZlEA1FMSMrZsU6wENflLJQEPrEUJmYji24jU4vTnd2eVK1rhEB4P1iPEiVg0nCZIIhkJTtpaN
TyefV1Uq3V9JUTxEO9rMAsfSx16yqctuSi9qgUU7Ac85ZEffJqrKrQwSkQGyCnrDuAQ11Ryl5VGWbTfTfeEP-epjm0j
nAcI1akhkoS2xUESRV9Bq41rOtboJYv3hAe0pjOL7CHZ3mTsHMHXR_0IDQvCTx8tC9S_vU09-jK8c_4UAkoUDd5-_1D
Pl68AckAMtgZyPSQLKnlFW50WwQt5WCwp7VGrBL_okM-E7QeTQkrUMrGTDw

Note

To login into the Grafana dashboard as administrator use the credentials: username=admin and password=admin. The first login requires the changing of the default password for security reasons.

9.6. Testing your Kubernetes cluster

The cluster comes up by default with two K8S operators implementing the popular Big Data Analytics and Streaming platforms Apache Spark and Kafka (you can avoid this by removing the roles spark and kafka in the file deploy_k8s.yaml).

9.6.1. Launching the Spark application spark-pi

You can use the Spark application spark-pi to verify that the cluster works properly. Just take the examples/spark-pi.yaml file and execute the following kubectl commands:

# kubectl apply -f spark-pi.yaml

# kubectl get sparkapplications spark-pi
NAME       AGE
spark-pi   5m

# kubectl describe sparkapplications spark-pi
Name:         spark-pi
Namespace:    default
Labels:       none
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"sparkoperator.k8s.io/v1beta1","kind":"SparkApplication","metadata":{"annotations":{},"name":"spark-pi","namespace":"default...
API Version:  sparkoperator.k8s.io/v1beta1
Kind:         SparkApplication
[...]

# kubectl logs -f spark-pi-driver | grep "Pi is roughly"
Pi is roughly 3.1458557292786464

Note

In case of problems with the sparkoperator API Version, look at the output of

# kubectl api-versions

9.6.2. Creating a Kafka cluster with a topic

Declare the cluster structure as in the kcluster.yaml file taken from the examples directory, and execute the following kubectl command:

# kubectl apply -f kcluster.yaml

For further details on configuration see https://strimzi.io/docs/master/#assembly-deployment-configuration-str

A topic for the Kafka cluster can be declared as in the ktopic.yaml file taken from the examples directory, and created by executing the following kubectl command:

# kubectl apply -f ktopic.yaml

Kubernetes provides a port on the master for accessing the created cluster. The port number is reported by the following kubectl command:

# kubectl get service kcluster-kafka-external-bootstrap -o=jsonpath='{.spec.ports[0].nodePort}{"\n"}'

Other useful commands for monitor the status of the cluster are:

# kubectl get service
NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
kcluster-kafka-0                        NodePort    10.97.1.118      <none>        9094:31945/TCP               64s
kcluster-kafka-1                        NodePort    10.100.252.199   <none>        9094:31730/TCP               64s
kcluster-kafka-2                        NodePort    10.106.128.149   <none>        9094:31608/TCP               64s
kcluster-kafka-bootstrap                ClusterIP   10.109.113.86    <none>        9091/TCP                     65s
kcluster-kafka-brokers                  ClusterIP   None             <none>        9091/TCP                     65s
kcluster-kafka-external-bootstrap       NodePort    10.107.133.0     <none>        9094:32161/TCP               64s
kcluster-zookeeper-client               ClusterIP   10.103.223.73    <none>        2181/TCP                     93s
kcluster-zookeeper-nodes                ClusterIP   None             <none>        2181/TCP,2888/TCP,3888/TCP   93s
kubernetes                              ClusterIP   10.96.0.1        <none>        443/TCP                      3d1h

# kubectl get pod
NAME                                            READY   STATUS    RESTARTS   AGE
kcluster-entity-operator-7b8d767b5c-lh6kp       3/3     Running   0          3m55s
kcluster-kafka-0                                2/2     Running   0          4m28s
kcluster-kafka-1                                2/2     Running   0          4m28s
kcluster-kafka-2                                2/2     Running   0          4m28s
kcluster-zookeeper-0                            2/2     Running   0          4m56s
kcluster-zookeeper-1                            2/2     Running   0          4m56s
kcluster-zookeeper-2                            2/2     Running   0          4m56s
strimzi-cluster-operator-6464cfd94f-tmbqd       1/1     Running   0          3d1h

# kubectl get kafkatopics
NAME                  AGE
ktopic                12s