Troubleshooting a misconfigured kube-controller manager

Lab The main role of a kube-controller manager is to check the health of running containers. This means that in a k8s cluster, if the kube-controller is not operational, it would be impossible to detect failed applications to trigger the sel-healing mechanism which is basically the k8s system trying to restart the pods. This means that probes would also fail.

context

• The problem here is that the kube-controller manager pod is not comming on. its is stuck on the CrashLoopBack status
• we can confirm this with the command kubectl get pods -n kube-system. This just lists all pods in the kube-system namespace
• considering that the kube-controller manager is a static pod, we can use the container runtime commands to check its logs. In context of Docker, we would use docker ps but our system uses cri-o and thus we can list all running containers using the command watch crictl ps -a. the -a flag just specifies that we want to see even the failed container
• The above commnad should list all running containers in the system even the failed and kube-controller-manager should be there
• Beaing a static pod, we can either use the crictl logs container-id, journalctl or /var/log/syslog file to check for errors

fix

• Using the commad crictl logs container-ID | grep -i error we get an output telling us that there is an error in the manifest file
• The static pod’s manifest files are located in /etc/kubernetes/manifests/
• a simple search on the patterns got from the error in the manifest file, we can get the rootcause of the error
• saving the manifest file will trigger the system to create another pod from the corrected manifest file. This might take awhile, alternatively, we can move the manifest file to another directory and put it back in the kubernetes manifests file. This will manualy trigger the creation of a new kube-controller manager pod
• running the command kubectl get pods -n kube-system will show the pod in Running state

Troubleshooting a misconfigured kublete

Lab

• A Kublete is the agent that runs in the nodes that directly communicates with the Apiserver receiving instructions from the control node as a podspec.
• The kubelet receives a podspec which is a YAML file from the apiserver that describes how the containers should be running. The kublet makes sure that the nodes are healthy and ready t

context

• In this lab, the kublete in node01 is not working and you can confirm that by getting to the node01 and checking the kubelet service status

$ ssh node01
$ systemctl status kubelet
# for older systems
$ service status kubelet

• With the kubelet down, this means that we are unable to know the state of our node. This makes it impossible to schedule any pod on it.
• Get the logs to see any issues raised

$ cat /var/log syslog | grep -i kubelet

# or

$ journalctl -xe | kubelet

This should give all logs related to the kubelet and why it failed

What is kubernetes?

This is a container ochestration tool that was initialy developed by Google for use as an internal tool. It was later on made open source and donated to the Cloud Native Foundation(CNCF). This sounds like a lot of tech words to throw around in a single sentence. Lets go to the very basics. What is a container?

A container is a package that contains an application and all the dependencies it needs to run. This makes it easy to share around applications and run them in different enviroment. This is all fun an games until you have to work with multiple containers. For multiple containerised applications, we use docker compose which is a IaaC used to define multiple container environments. This gets the job done but has limitations. We have to manualy manage and control the containers. This gets harder to scale or fix a falling container. A failing container could be a business critical service that could mean the ultimate death of the business due to frustrations faced by the customer accessing the service.

The exciting Roadtrip of a URL

Have you ever wondered what happens behind the scenes when you type “https://www.google.com" into your browser and press Enter? The seemingly instant appearance of Google’s homepage is the result of a complex and highly organised series of steps involving various technologies and systems.

In this blog post, we’ll take a deep dive into the route taken by a URL, exploring the role of DNS requests, TCP/IP, firewalls, HTTPS/SSL, load-balancers, web servers, application servers, and databases.

DevOps Zero to hero

This will be a series of articles showcasing the skills I pick up along the way.

Backstory

I’ve been a Network Support Technician for two years and recently got promoted to Network Security Engineer at the same organization. This role is more of a hybrid, as I’m responsible for the institution’s IT infrastructure.

Our current setup is quite complex, and I’ll draw an architectural diagram in future articles to illustrate the issues. Our infrastructure has many unnecessary connections and devices. For example, we host a Windows server as a VM on one PC and our Unifi controller on another.

• This is the linux packet filtering tool baked-into linux systems

• This is the userspace module for managing Netfilter

• Acts as a firewall filtering packets at the kernel level by using rules to filter packets

• Rules defined determine how network traffic is handled by the system

  Structure

• Tables- - These are collection of chains that perform a specific function. - Most commontables include filter , nat, mangle

• chains

  • These are a list of rules that packets are checked against
  • The filter table has three default chains
    • INPUT: For packets destined for local packets. packets coming into the server
    • OUTPUT: For localy generated packtes. packets leaving the server
    • FORWARD: For packets being routed through the server to another destination

• Rules

• presentatation
• Source code This was a personal budgeting application to help me counter my poor budgeting and financial habits.

Technologies

Python FastAPI MySQL Linux

• This is a budgeting app that enables you to create an account and track your regular spending, savings

Dir: API: This is the backend section of the app - Queries the DB(currently using file storage) for data - Currently only accepts Category, Date, Amount and Description. The ID is auto generated

Deploying a 3-tier application on Kubernetes

presentation source codde

UI

• This will be represeneted by the swagger-ui
• This is what the user will interact with

Django app(Twender-App)

• This is the main logic of the application
• functionality:

  User creation user validation User login trip creation payment payment validation trip update

MySQL Database

• Stores all data processed by the django app
• offers data on-request

The architecture