Fly, Penguin!

Tagged: kubernetes Toggle Comment Threads | Keyboard Shortcuts

• 

  penguin 21:28 on 2020-01-25 Permalink | Reply
  Tags: check_mk, fix ( 2 ), kubernetes, monitoring ( 2 )   

  Check MK container/k8s deployment 

  In the company everybody seems to love Check MK. Me? Not so much, but a better alternative costs time and effort, both resources we don’t really have right now. Yet there’s a positive thing about it – because there’s an official docker container. Since I already coded a helm chart for stateful single container softwares (which I personally find super useful), I just wrote a Check MK YAML and installed it on my K8S cluster.

  And then nothing worked. Turns out, Apache – which is used in that very strange “Open Monitoring Distribution” which Check MK seems to have been at one point – has a slightly sub-optimal configuration for running in a container behind a load balancer using cert-manager.

  In short, you connect to the load balancer using “cmk.my.domain”, and it redirects you to the container port, which to itself is “https://cmk.my.domain:5000/” and just wrong. Which brings me to the question if anybody has ever tried to run the Check MK container in a k8s cluster or behind a load balancer, which brings me to the question that I’d rather use software which actively embraces that, which brings me to the question WHICH ONE?!? which brings us back to “no resources, no time”.

  So, bad luck, Check MK it is. But what about the bug? Reporting it you get an email “DONT CALL US – WE CALL YOU (and we probably won’t)“, with a ticket ID but no link. So probably no help here. So I “forked” the container, fooled around with it, and found a solution. The “fixed” container is now available on docker hub (sources on GitHub) and running nicely in our internal cluster. Let’s see which hidden bugs I have introduced 😉 . The stasico-Helm-YAML file I used to deploy Check MK in K8S is also available.

  TL;DR

  • Found a bug in the Check MK docker container – you can’t run it behind a LB
  • Created a fixed version (GitHub repo here)
  • … and a YAML deployment script for my stasico helm chart
   

  Reply Cancel reply

  Required fields are marked *

  Name *

  Email *

  Website

  Save my name, email, and website in this browser for the next time I comment.

• 

  penguin 00:28 on 2020-01-11 Permalink | Reply
  Tags: helm ( 2 ), kubernetes, ops ( 2 )   

  cert-manager too old … 

  Today cert-manager stopped issuing certificates, and all requests said “insecure website”. Uncool, since this affected our Confluence and our sign-in mechanism. So let’s find out what was happening, right? Turns out cert-manager considered itself “too old” (“your ACME client is too old”, literally) and wanted to be updated.

  So far, so good. Just perform helm update cert-manager cert-manager, right?

  Wrong.

  • First, I had to upgrade to helm3. All right, I could have used helm2, but helm3 was already on here, and it seemed easy. That went (fairly) easy.
  • Then I wanted to upgrade cert-manager. Turns out for that I actually had to upgrade the running k8s version 1.12.x to 1.13.x … otherwise I’d get errors from the helm chart. That just took ages cause AKS is a bit slow.
  • Finally done I wanted to upgrade cert-manager. Until I realized a lot of stateful pods were stuck in “initialization”. Turns out that AKS had issues moving the volumes around, I still don’t know why. (Did I mention I just don’t like pretty much anything about Azure? It’s just so incredibly cumbersome to use, and nothing is where you expect it). So I had to manually mount the volumes on the host the Pod was currently on and have an open TODO now, which just sucks.
  • Finally done I wanted to upgrade cert-manager. The upgrade went just peachy, until I realized that … nothing happened. Turns out they changed pretty much all API versions and annotation keys. So I had to rewrite / upgrade all ingress annotations, update the ClusterIssuer resources and delete the now obsolete former K8S CRDs.

  And just like that I had my certificates back. Wasn’t that easy? 😀

   

   
• 

  penguin 09:50 on 2018-05-25 Permalink | Reply
  Tags: helm ( 2 ), kubernetes, rbac   

  Helm in a kops cluster with RBAC 

  I created a K8S cluster on AWS with kops.

  I ran helm init to install tiller in the cluster.

  I ran helm list  to see if it worked.

  I got this:

  Error: configmaps is forbidden: User "system:serviceaccount:kube-system:default" \ 
      cannot list configmaps in the namespace "kube-system"

  That sucked. And google proved … reluctant. What I could figure out is:

  Causes

  • kops sets up the cluster with RBAC enabled (which is good)
  • helm (well, tiller) uses a standard role for doing things (which might be ok, at least it was with my stackpoint cluster), but in that case (for whatever reason) it did not have sufficient privileges
  • so we need to prepare some cluster admin roles for helm to use

  Fixes

  Just do exactly as it says in the helm docs 🙂 :

  • apply the RBAC yaml file which creates the kube-system/tiller service account, and binds this to the cluster-admin  role.
  • install helm with: helm init –service-account tiller

  Is that secure? Not so much. With helm you can still do anything to the cluster at all. I might get to this in a later post.