While I’m not an advanced Linux user, I’m comfortable navigating a Linux filesystem. But the lack of a GUI always gave me a bit of anxiety when trying to find where a binary or config file lives. To amplify that frustration, while preparing for the CKA 1.19 exam, I realized that Kubernetes doesn’t store things in one directory so to help myself visualize and remember the directories better, I made the below graphic that illustrates the directories that are necessary for troubleshooting a Kubernetes cluster. In this post, we’ll explore how Kubernetes uses each directory and it’s importance for the CKA.

Exploring Each Directory

$HOME/.kube

Contains the kubeconfig which is the brains behind the kubectl command. It specifies the Kubernetes API server address, certificates, and context (cluster, namespace, and user info) necessary for a terminal user to interact with the Kubernetes API. A sample kubeconfig is shown below:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LOTOFRANDOMNUMBERSANDLETTERSFORKEY
    server: https://192.168.1.117:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: LOTOFRANDOMNUMBERSANDLETTERSFORKEY
    client-key-data: LOTOFRANDOMNUMBERSANDLETTERSFORKEY

NOTE: While it’s called kubeconfig, Kubernetes looks for the filename config by default. Learn more about kubeconfigs in the Kubernetes documentation.

/etc/kubernetes

At first I thought, “YES! There will be a single directory that Kubernetes uses for all it’s config files!” I was willing to give Kubernetes a pass on the kubeconfig being a user-specific file…but I was wrong! But if there were a primary directory for Kubernetes, /etc/kubernetes/ would be it in my opinion. It contains two directories:

• manifest/
• pki/etcd/

/etc/kubernetes/manifest/

On master nodes, this directory contains the manifests (configurations) for the components that make up Kubernetes: etcd, kube-apiserver, kube-controller-manager, and kube-scheduler. These are referred to as Static Pods which are managed by the kubelet daemon on a node, visible to the kube-apiserver, but not scheduled by kube-scheduler. This process allows Kubernetes to run as pods without the “chicken or the egg?” problem.

This is an important directory during troubleshooting the CKA exam! If there’s anything wrong with core Kubernetes services and functionality, you can correct it here. You could have a variety of scenarios:

• kube-apiserver isn’t pointing to the correct mountPath or hostPath for certificates
• Pods are getting scheduled because kube-scheduler.yaml is pointing to the wrong image
• etcd isn’t working because there’s inconsistent TCP/IP ports
• Container command sections have typos or not pointing to the correct TCP/IP port or configuration/certificate file path

/etc/kubernetes/pki/

While pki/ itself contains certificates and keys, I haven’t found that I needed to know about them for the CKA exam. Below shows it’s content and the real reason we’re here: etcd/.

/etc/kubernetes/pki/etcd/

pki/etcd/ contains the key and certificates for etcd to run in Kubernetes. We specifically need to know about this directory and it’s contents to backup and restore etcd!

/etc/cni/net.d/

This is the default directory that contain the configuration files for the container network interface (CNI) plugin that is running on the node. It’s specified by the --cni-conf-dir switch when the CNI plugin runs.

During cluster troubleshooting, seeing a configuration file in this directory should let you know that a CNI plugin is configured and running in the cluster. You can also verify by looking at the pods running in the kube-system namespace. This particular Kubernetes cluster is running weave-net:

/opt/cni/bin/

Having a CNI configured in the cluster is critical for pod communication and cluster DNS and that’s where /opt/cni/bin/ comes into play. While /etc/cni/net.d/ contains the CNI configuration file, /opt/cni/bin/ contains the CNI binaries (and is specified in the CNI configuration using --cni-bin-dir).

On this node, the only CNI I have configured is weave-net so that binary is listed in this directory.

/var/lib/

Contains two important directories for Kubernetes:

• kubelet/
• etcd/

/var/lib/kubelet/

Primarily, this directory contains the kubelet manifest (config.yaml). The kubelet is the agent that runs on a Kubernetes node and is responsible for making sure pods are running and healthy on the node and communicating with kube-apiserver. For the CKA exam, if static pods aren’t running in the cluster, check staticPodPath and ensure that it’s pointing to /etc/kubernetes/manifests/.

/var/lib/etcd/

This is the etcd data directory and stores two important components (that I’ve found for the CKA):

• etcd database
• etcd snapshots

Notice this is etcd/member/snap/. I didn’t specify that level of detail in the graphic above since I haven’t interacted with the directory with this level of detail while preparing for the CKA. Instead, I’ve only used the --data-dir flag when restoring an etcd database which only requires specifying where the new etcd database will reside.

While preparing for the CKA, this visualization of the directories to know for the exam has been helpful to me. I hope it helps you as well as you prepare! Feel free to say hello on Twitter or LinkedIn!