CI/CD, the status quo

… is quickly summed up: Working. 🙂

So I am – naturally – a happy camper. Although, there are a couple of things I would like to change, which don’t scale or are in a state I’m not happy with for “v1.0” yet.

Container data persistency

Rancher does offer convoy, but I’m not sure it really fits my needs, and even if I’m not using it yet. And it starts to hurt not to have this. Badly. (AWS Elastic File System would be excactly what I need, but that’s not going to happen I fear). And even this is the most hurting part, not sure if this is the one I can solve quickest.

Pressure: 9


It’s embarassing, but – I don’t have host monitoring. I don’t even remember how often I needed to re-create a host cause the disk went full without me noticing (and recreation is just so much easier than fixing, actually, so at least this is a good sign). The current evaluation candiate is sysdig cloud. That might change to DataDog. And don’t get me started on log management.

Pressure: 8,5


While I like TeamCity, it’s – like all things Java – horribly, completely, fully over-engineered mess with billions of functions which all don’t quite do what you want. In our case it’s also reduced to executing basically 3 scripts. So TeamCity must go, mid term. Replacement? No idea. Gradle,, wercker, distelli or Travis seem viable candidates. (Also found an interesting article about online CI tools which is well worth a read).

Pressure: 7


It’s written by me. And it sucks, and, although it works, it must go.

Pressure: 7


Currently I am focused on “getting things to usable”. Security is not on that list. I need to address some of the issues which I ignored until now, mainly because they might have architecture impact. (Which I believe I planned pretty well, but who knows until you have to actually do it, right?)

Pressure: 6


I am still using Puppet. Might be overkill, but it’s reliable once set up. That brings me to … the set up. I am still using masterless puppet, one environment, and pulling the repo from each host each time. Simple, robust, working, but not elegant. I see different environments coming up, and then … hm. It’s gonna be complicated.

Pressure: 6


Rancher is awesome. But I’m so desperately waiting for this feature right here to arrive it’s not funny any more 🙂 . I also need the same thing for host-based services (which are not inside of a container orchestration platform). I want to spawn a service somewhere, and an Amazon Route53 entry should appear automatically. Preferably based on a consul readout. So DNS management becomes a non-issue.

Pressure: 5


If one host breaks, the whole cloud is inoperable. Which one? The NAT host. Needed for any host to get a connection to the outside in AWS. And that’s just one breaking point of a couple. Critical? Not really. Super annoying? You bet.

Pressure: 4


Currently I use cloudformation to set everything up. And although it’s an awesome product, it’s kinda limited. If a host is gone, I can’t call “cloudformation fix-it” and this very host will respawn. With tools like terraform this is possible, but terraform relies (or relied the last time I looked) on local state data, which is a big pffft. It also means to really test an updated cloudformation template I have to recreate the full thing, which is just too cumbersome. (Takes about 1 hour to completely migrate everything, if done quickly, which is awesome for a complete outage, but really bad for testing). But maybe I just don’t know enough and my template is way too tightly written.

Pressure: 3



Arch with dm-cryptt on UEFI boot

Just a collection of links, cause this is something which is not really documented in full. I am also too lazy, but next time I don’t want to search 🙂 .

  • Basic installation instructions: 3rd party article (German), Arch Linux docs (without LUKS)
    • Read the comments on the first article: gummiboot is now bootctl, you have to format the boot partition with FAT32, and you should use a better random generator on cryptsetup. It’s all in there.
    • Note: The former /boot partition is identical to the UEFI boot partition. You don’t need both.
  • Arch Linux original instructions: In German, in English
    • Careful: The setting “FONT_MAP” in /etc/vconsole.conf in the German guide should not be applied! It’s obsolete.
    • The English guide does only go into the crypt installation, but that really deep.
  • If you actually added FONT_MAP to /etc/vconsole.conf, that happens
  • The file /vmlinuz-linux comes in the “kernel” package. If it’s missing, just reinstall “pacman -S kernel”
  • You can usually choose between NetworkManager and systemd-networkd as networking management solutions. I chose NetworkManager:
    • systemctl enable NetworkManager
    • systemctl start NetworkManager
    • systemctl disable systemd-networkd
    • systemctl disable systemd-resolved
  • Installing GNOME (and the login manager gdm) is pretty simple:
    • pacman -S gnome gnome-extras
    • systemctl enable gdm

That should be it.