Max Brenner

My first steps in indie hacking

2024-03-04T00:00:00+00:00

Photo by Jake Hills / image source

Hey, thanks for stopping by. After over two years of silence on this blog I finally felt the urge to write about something that is worth sharing again. A lot has happened since my last post but that’s something for another day.

Today we are talking indie hacking, an endeavor? hobby? lifestyle? 🤷 that I started moving into at the end of last year. In case you are unfamiliar with the term indie hacking, here’s my definition:

person + computer = product (ideally with users and which turns profits)

No external money, no employees, just a single person building and selling stuff online.

The last two months to me felt a bit like back then in 2020 when I decided to go freelance full-time. A lot of uncertainty and a whole lot of new stuff to learn. Exciting times, but let’s start from the beginning.

My Background

So here’s my background in a few bullet points to set the baseline for my journey.

got a degree in computer science
started working as a software developer (Java back then 🙊)
transitioned over to SysAdmin / DevOps, managing a local data center
switched jobs to gain experience in cloud infrastructure
stepped into DevOps freelancing full-time

All the while I was working on various open source projects, which allowed me get to ~~a decent~~ some level in programming. Technologies that I was already familiar with included:

Python: main programming language for years
HTML, CSS, JS: I know the basics, no pro in any of them (can you be pro at HTML? 🤔)
Django: a few projects over the years, but nothing that made it into “production”
Angular, React, Vue: one project each, just to understand what they’re about

Why indie hacking?

Actually I really asked myself this question for the first time while writing this post and it wasn’t that easy to come up with an answer. I’m still not 100% sure but here are my thoughts.

Learning new stuff

This one is probably the biggest factor for me.

If you do something almost every day for a few years, you usually get pretty good at it. And that’s where I’m currently at when it comes to setting up, maintaining and improving cloud infrastructure. Don’t get me wrong, I’m definitely not saying that I am along the best DevOps guys out there. But for 90% of the topics the average client approaches me with, a solution is already in my head after a short time. Reason being that the challenges companies have, usually are all very similar.

Long story short, I want to dive into different topics and indie hacking allows me to do just that. Wearing the marketing, sales, technical, … hat forces me to jump into areas I have never been in and that’s what I’m looking for.

DALL-E's understanding of an indie hacker wearing multiple hats 😆

Back to simplicity

Holy f, cloud infrastructure got (was made) complicated. From single VMs that you SSH into, to hundreds of cloud resources that get created almost instantly using some Terraform module just to run an app on 0.5 vCPU cores and 1GB of for 100 dollars a month (hello AWS Fargate 😉).

Obviously, there’s a place for that kind of stuff and I still love to work on it in my freelance projects. But for my “personal” projects I’d like to take a step back and embrace simplicity again. Simple technologies, simple processes, just get things done when they need to in the shortest time possible.

This also touches the topic of perfectionism. In my professional work it’s often not about finding just any solution but the best. And that’s also what I expect from myself. Not just delivering anything, but the best possible thing.

This may be fine, but if you want to build a product, alone, from scratch, without spending months before launching it and finding out nobody cares about it, you have to focus on what matters and be fine with an 80% solution (for the moment).

Building for users

In my freelance projects, especially when not being part of the actual product development, you get fairly disconnected from the people that you are actually building for. Which makes me feel a bit sad.

If you’ve been shipping some kind of software you probably remember the first time you’ve got feedback from a random stranger on something you’ve built. Heck, even negative comments spark emotions as it’s always exciting to hear what other people think about your creation.

I hope that with my indie projects I’ll encounter these kinds of situations more often and eventually find something that helps other people in whatever they are trying to achieve.

Decoupling income from time

This one is pretty obvious and I’m probably romanticizing it too much. As a freelancer I know pretty well what it means to sell your time for money. I see these numbers every month on the invoices that I send out.

So generating some income without actively spending time working, sounds like something desirable. Who knows what I’m going to do with the freed up time. 🤷

Getting things started

Alright, that was a pretty long intro but I feel like it was necessary.

So what was my first milestone? In the beginning of January I set myself the goal to get something shipped by the end of the month. It didn’t have to be pretty, it didn’t have to get much attention/users, it just had to be functional and providing some value (at least in my eyes). Here’s what I came up with.

Blocked by CORS

If you’ve been building web applications, you’ve probably come across CORS issues. If you haven’t, just know, they are annoying, pretty hard to debug and a lot of developers encounter them each and every day (literally, just check Reddit 😄).

Recognizing this, I set out to build a “universal solution” for all kinds of CORS issues, providing a knowledge base but more importantly, tools to help developers figure out what’s going on.

As it was my first project I didn’t expect anything from it. It was just about getting something out there and that’s what I did. After just a little more than 7 days of work I started sharing blockedbycors.dev publicly.

It only included one central feature, which is still its core as of today, a CORS debugging tool. Not getting into much detail here, but it tells you whether a request from you web app would pass a server’s CORS policy.

The result a user gets from the CORS debugger

To save time, I simply used tools I already knew pretty well. Django as a fullstack framework, no fancy JS library, just a bit of HTMX for interactivity and TailwindCSS + DaisyUI to make things don’t look too bad.

A few days later I also added a configuration generator, as properly configuring CORS is not an easy task. In general, during the building process and while learning the ins and outs of CORS, so many ideas popped into my mind that one could build. But I had to resist. There’s another half (three-quarter? nine-tenth?), which is finding users for your app.

So I split up my marketing approach (it sounds so wrong to call it like that 😆) into two parts:

Short term strategy

To get some short term validation and finding out whether it’s even worth pursuing the current path I just reached out to people having CORS issues and sharing my tool with them. Finding these people was pretty easy as they are numerous 😆. Enter “blocked by CORS” into Google, get the results from the last 24 hours, done.

This way I approached two to five people every day for at least a week. The feedback was pretty good and I was able to help almost all of them to fix their issue. Very nice! Plus, the places I shared my tool (StackOverflow, various forums) still bring in some traffic up until today.

This approach is perfect to get some (almost instant) feedback but obviously it doesn’t scale well. Nobody wants to spend their time doing this for weeks or even months.

Additionally it’s the nature of problem solving tools to become useless once the problem is solved. Once the people get their CORS issue sorted, they move on and may only come back when they encounter the next one. But CORS issues are not like dirty dishes. It’s not something you have to deal with every week. At least I hope so.

What I also did was to bundle all the accumulated knowledge about CORS into an easily digestible format. The outcome is an interactive mindmap that was a nice piece to share and people really liked it. Traffic spikes from Reddit or Hacker News are motivating but they won’t bring too much results long term.

So we need a way to bring in some regular traffic.

Long term strategy

The area of CORS issues felt like a typical case for organic traffic to me. People copy paste their CORS error into Google, they find an article on blockedbycors.dev, they use the tool to find the solution to their CORS issue, user is happy.

So I started setting up a blog with Hugo (great tool!) and wrote some articles about various CORS topics. Now comes the big question, how do I get people to find my blog?

The standard answer to this question are the magical three letters S E O (search engine optimization). You remember me saying that I wanted to dive into new topics? Here we go, I’ve never had anything to do with SEO. It’s basically the language to talk to search engines and whisper them in their ears: “Hey, Google/Bing/DuckDuckGo. When someones searches for X, send them to my page!”

Here’s the simplified concept:

create content that contains keywords you want to rank for, e.g. in my example: CORS issue, blocked by CORS
get other pages to link to your page with a dofollow link

This increases your domain rating, which in turn puts your page higher up in the search result.

For me, this was probably the hardest thing to do during the last two months. It feels like a game that should not be played. Plus the whole process has a feedback loop time that a developer cannot cope with. Be happy if you can see any results within two months. I’ll keep you posted. 😅

Any Results?

OK, so what has been the outcome of all this work?

Let’s start with some traffic numbers. For the static blog that contains the articles and references to the tools, Cloudflare Analytics reported 1200 page visits since the launch on January 17th. As usual, people that are using ad blockers or privacy protection are not included in these numbers.

Top traffic sources on blockedbycors.dev of the last 30 days, numbers aren't accurate, Cloudflare Analytics just rounds everything to hundreds

I also added some server-side request tracking to the Django application that is providing the tools and this reports 2400 visits since launch.

Traffic on app.blockedbycors.dev of the last 30 days

So all together we are looking at roughly 3500 visits in total, which is a nice bit of traffic for a page that only exists for a few weeks.

I’m even more happy to see that people are regularly using the CORS debugger to, hopefully, fix their CORS issues. Currently there are 5 to 10 different sites (and some of them are pretty well known 😆) analyzed everyday using the tool.

To capture some direct user feedback I also added a small form, but I didn’t receive any submissions yet. Means that everything is working, right? 😅

During the whole process I also started sharing regular updates about anything I do on Twitter and built a bit of a following by doing so. Currently I use it as a bit of a public journal which also really helped me when writing this blog post to kind of remind me of the whole process I went through.

Was it worth it?

Well, it’s difficult to give a definitive answer here. I’m very happy that I was able to fulfil my goal of shipping an app from scratch within a very short time frame. I learned a whole lot during the process and could keep myself going even when tackling topics that I did not enjoy but were necessary.

On the other side, all of this required me to spent substantial amounts of time. For roughly three weeks I probably spent 20-30 hours per week on top of my day job, tinkering on my indie project. This left little to no time on doing other things that I enjoy. It was very exciting to go “all-in” on it but I realized that if I want to do this long-term I had to take a different approach. So what’s up next?

What’s next?

With the accumulated knowledge I’m now feeling more confident than ever on continuing my indie hacker journey. To tell you the truth, I already did 😆. The initial idea of this blog post was to cover my first two indie projects, but after writing about the first one, I realized that it’s just too much for one post. So yeah, I already worked on and even published my second project.

I’ll talk about it and what I did differently in the next blog post. In case you are curious, I already shared parts of the story in small pieces on Twitter.

By the way, if you are in a similar situation as me or have something to share or talk about, feel free to contact me. Difficult things get easier when you share them. Hope to talk to you soon. Cheers ✌️

Sharing a Kubernetes cluster between multiple tenants

2022-01-11T00:00:00+00:00

Running and maintaining Kubernetes clusters is hard. They need to be upgraded, automatically scale based on demand, be monitored and most importantly, ensured to be highly available at all time.

So the idea of sharing clusters across teams, divisions or even whole companies is definitely valid. There are various ways of achieving multi-tenancy which we will be exploring today.

Multiple clusters

This approach is probably not the one you were looking for when deciding to read this article but I don’t want it to go unnoticed. Of course it has some obvious downsides, like having to set up and maintain multiple Kubernetes control planes. On the other hand it limits the amount of damage an application can do to the scope of a single cluster keeping the remaining ones safe.

But for sure, there are good ways to safely enable multi tenancy on a single Kubernetes cluster.

Namespaces

Namespaces are a concept you are most likely already aware of. They are shipped with Kubernetes out of the box and a default installation already has few of them set up for you. Initially these namespaces are nothing more than a logical separation but in combination a few other components they introduce multi tenancy capabilities. Those are RBAC and resource quotas.

RBAC stands for role-based access control and allows you to configure fine grained permissions to certain namespaces or global resources. It may be used to realize the following use case cases:

create one admin user that is able to manage all namespaces
create several users with different levels of access to a single namespace

Resource quotas on the other hand allow you to define limits for the memory and CPU usage of all pods within a single namespace. This can be achieved by creating a ResourceQuota object.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: example-resource-quota
spec:
  hard:
    requests.cpu: "2"
    requests.memory: 2Gi
    limits.cpu: "4"
    limits.memory: 4Gi

Be aware that creating a ResourceQuota objects requires you to define CPU and memory requests and limits for each pod within this namespace. In case you want to know more about this topic check out my blog post about how to waste money using Kubernetes.

The below namespace structure may be the result when deploying a frontend and backend application into three different environments.

$ kubectl get namespaces
NAME               STATUS   AGE
kube-system        Active   1d
default            Active   1d
frontend-dev       Active   1d
frontend-staging   Active   1d
frontend-prod      Active   1d
backend-dev        Active   1d
backend-staging    Active   1d
backend-prod       Active   1d

Using RBAC we can then define a cluster admin that is able to assign the appropriate resource quota to each namespace and create further roles that provide the right level of access to other users.

What becomes clear is that this approach doesn’t scale very well. Each newly created namespace needs its own set of role bindings and resource quotas.

Additionally, depending on your policies, creating these objects may require involvement of cluster admin users. So e.g. the backend team wouldn’t be able to independently create new namespaces with resources assigned to them when necessary.

So how can we fix on these shortcomings?

Hierarchical Namespaces

Hierarchical namespaces aren’t too popular yet but are pretty self-explanatory. Instead of having a single level of namespaces, it introduces the ability to nest them, allowing for permission and object propagation. This not only simplifies the RBAC configuration but also enables teams to independently manage their resource allocation.

Assume your team’s parent namespace have been given a certain amount of CPU and memory resources, e.g. your team lead can decide how much of these to assign to the dev, staging and prod environment. This can happen without needing to bother the cluster admins.

By introducing HNs our example namespace structure from above could be converted into this:

$ kubectl hns tree
kube-system
default
frontend
└── dev
└── staging
└── prod
backend
└── dev
└── staging
└── prod

This allows to give teams complete power over their parent namespace including the assigned resources. This way each team is independently able to create and manage namespaces and distribute their available CPU and memory across them.

To make use of this concept you first need to install the hierarchical namespace controller in your cluster. Have a look at the docs to get more details on the process. Also be aware that there’s no HNC release recommended for production yet. But according to the roadmap this should happen very soon.

Virtual cluster

Maybe you remember the term DinD that came up a few years ago. It stands for Docker in Docker, meaning you can manage Docker containers from within a Docker container.

A virtual cluster is pretty much the same in the Kubernetes world. It allows you to create a semi-independent cluster within your existing cluster. At the time of this writing there are two implementations that I know of and we will have a look at.

Kubernetes VirtualCluster

Kubernetes VirtualCluster is the implementation of the official Kubernetes multi-tenancy SIG. It introduces a new CRD that can be used to create new clusters on demand. The solution consists of the following three components that share the necessary responsibilities:

vc-manager

The vc-manager manages the realization of said VirtualCluster CRD objects by creating the necessary control plane pods or even proxying an external Kubernetes cluster when providing the corresponding kubeconfig.

syncer

The syncer makes sure to reflect API object changes happening in the tenant cluster to the super cluster and to propagate events occurring in the super cluster back to the tenant cluster. During this process certain mappings may occur, e.g. to prevent naming conflicts. For example namespaces being created in the tenant cluster, will be stored with a prefix in the super cluster control plane.

vn-agent

The vn-agent is running on the worker node and proxying the API requests to the local kubelet process. It makes sure that each tenant can only access its own pods.

VirtualCluster architecture / image source

Even though VirtualCluster passes most of the Kubernetes conformance tests it still has a few limitations that you can read about here. Currently it does not support managing DaemonSets and persistent volumes from a tenant cluster just to name a few.

Similarly to the HNC, there’s no productive release of VirtualCluster at the time of this writing but is expected to be coming soon.

Loft Labs vcluster

Loft Labs implemented their own version of a virtual Kubernetes cluster and open sourced it in April 2021. It provides a similar functionality but with a slightly different user experience and architecture.

Instead of requiring you to install an operator and create an CRD object, vclusters are being deployed by applying standard Kubernetes manifests (StatefulSet, Service, Role, …) onto your cluster. This creates a pod with 2 containers.

k3s

One is running k3s, a stripped down Kubernetes distribution, to create a separate control plane including the usual API server, data store and controller manager.

syncer

The other component is called syncer yet again and acts as an replacement for the default Kubernetes scheduler. Instead of distributing pods onto worker nodes it makes sure that vcluster pod specs are being passed to the host cluster and actually being applied there.

For every pod a proxy object is being instantiated in the vcluster that reflects the status of the actual pod running in the host cluster.

vCluster architecture / image source

If you are curious about a more detailed view of a Loft Labs employee about the differences between the two virtual cluster implementations have a look here.

Conclusion

And there you have it. Four ways of providing Kubernetes to multiple tenants. Multiple clusters definitely provide the highest level of separation but at a “high” cost. Simple namespaces come with the exact opposite.

But upcoming technologies try to find a good balance between the two and broaden the arsenal of enabling multi-tenancy for Kubernetes clusters. This way, hopefully everyone is going to find the right tool of choice for their exact use case.

Moving from utterances to giscus

2021-12-27T00:00:00+00:00

Why migrate?

I added the ability to leave comments on posts of this blog around a year ago. Back then I was looking for a privacy friendly, open source, free and easy to setup solution. When I came across utterances and knew I found the right tool.

Using Github’s issue feature as a backend for comments is just a very elegant solution in my opinion. No database that you need to manage, using Github to authenticate users (although I’d like to allow for anonymous users) and an integration that only requires you to load one client-side script.

utterances served me very well and I could have used for way longer. If just there wouldn’t be a better alternative. Around March 2021 a new Github project with the name giscus has been created.

While being very similar to utterances there’s one distinct difference. Instead of using Github Issues it uses the fairly new Discussions feature to store comments. This alone would not have made me do a move as there’s no major advantage of one over the other for this use case. But there a few points and one in particular that drove my decision to migrate.

Github Dark theme

Starting with a small annoyance, utterances is missing the Github Dark theme. Although there’s a theme with this name, the colors don’t match and I dislike the looks of it.

Post reactions

utterances allows you to add reactions to comments but as an author I’m also interested in the general reception of the post itself. giscus provides this feature.

giscus displaying page reactions

Conversation view

While you are able to tag users, utterances will simply render comments as a list in the order they have been created. Following conversations this way is pretty hard. giscus groups replies to a comment instead.

This is perhaps the only point where Github Discussions has a slight advantage over using Issues as the reply feature is directly built into it.

giscus rendering replies

Missing maintenance

Coming to the most important fact that made me migrate. As with many open source projects, utterances is being taken care of by a single developer. Unfortunately it seems like he lost interest in supporting the project over the last year.

There are a lot of open issues and even PRs that aim to solve almost all of my problems with utterances. But sadly the project seems to have been abandoned completely.

Don’t get me wrong, I’m not blaming anyone here. I’m an open source maintainer myself and know how changing times and interests can highly impact the level of effort you want to put into certain projects. But of course this will (rightfully) make your users move away from your software.

So the decision has been made. But how did I migrate my blog from utterances to giscus without loosing existing comments?

Preparing your site

The setup for giscus is very similar to the one for utterances. Simply go to giscus.app, click through the configuration guide, copy the resulting script tag and paste it into your page where the comments should appear. Here’s my script tag with some explanation:

<script src="https://giscus.app/client.js" <!-- the script to load -->
        data-repo="brennerm/brennerm.github.io-comments" <!-- the name of the repo to store the comments -->
        data-repo-id="MDEwOlJlcG9zaXRvcnkzMTg1MTk0ODQ=" <!-- the id of the repo to store the comments -->
        data-category="Announcements" <!-- the name of the Discussions category to store the comments -->
        data-category-id="DIC_kwDOEvw4vM4CAcbV" <!-- the ID of the Discussions category to store the comments -->
        data-mapping="pathname" <!-- the type of page to discussions mapping -->
        data-reactions-enabled="1" <!-- flag to enable/disable post reactions -->
        data-theme="dark" <!-- the theme to use -->
        data-lang="en" <!-- the language the comment renderer should use -->
        crossorigin="anonymous"<!-- performing CORS request without credentials -->
        async> <!-- flag to load the script asynchronously -->
</script>

If your repository is public, has the Discussions feature enabled and the giscus app installed, you should see the comment renderer displaying 0 items at this point. If you don’t have any previous comments that you’d like to keep, you are basically done at this point.

Migrating comments of a single page

The most important part when migrating comments from utterances to giscus is to keep the mapping of pages to discussions working. This works by selecting to correct mapping value in your script tag.

Usually it should be the same as in your utterances script tag as both offer the same mapping techniques and I expect them to be compatible. I kept using the pathname mode and it worked flawlessly for me.

To be sure, try to create a new discussion containing at least one comment with the same title as your utterances issues. If everything works it should appear on your page. Be sure to delete this discussion to prevent any conflicts later on.

After verifying that the mapping works simply navigate to the relevant Github issue and click on Convert to discussion on the bottom right. The following dialog should appear in which you select the correct category.

Converting an issue into a discussion

If that works you are done. In my case I ended up with this error.

Error I encountered when converting an issue to a discussion

Luckily I found a workaround which should work for you as well. Navigate to the Discussions tab, edit the categories and change the, in my case, Announcements category to an Open ended discussion. Convert the issue, which should succeed now and revert the category change.

If everything worked you should see the comments pop up on your page.

Migrating comments of multiple pages

To migrate comments of multiple pages we simply need to repeat the above process for each issue. Fortunately Github provides some bulk operations to speed up the process. At first we need to label all issues that we want to migrate. Feel free to use an existing or create a new label.

Then select the issues and apply the label of choice.

Labelling multiple issues at once

Afterwards navigate to the Labels overview and select the Convert issues action of the appropriate label.

Converting multiple issues into discussions

This process happens in the background and may take some time depending on the number of issues you migrate. After it’s done all your comments should be successfully shown by the giscus comment renderer on each of your pages.

Feel free to leave a reaction and comment to make sure I did everything right. 😉 If you run into any trouble make sure to let me know or seek help in the giscus community.

Why newly created AWS Route53 records may not resolve

2021-12-23T00:00:00+00:00

So there you are with your newly created Route53 hosted zone, ready to assign some DNS names to your hosts. A simple click on the “Create Record” button and you want to check if it works immediately. But trying to resolve the DNS entry fails. What’s going on?

pending vs. insync

Creating a Route53 record is a two-step process consisting of:

pushing the record to the Route 53 database
propagating it to the AWS DNS servers

Whether you use the AWS console, the CLI tool or the API, only the first step will happen synchronously and almost immediately. Afterwards the record or rather the submitted change request will be in the PENDING state.

Using the GetChange API endpoint or the wait resource-record-sets-changed CLI command you can then make sure the state switches to INSYNC. This indicates that your changes have been propagated to all AWS DNS servers responsible for your domain. AFAIK the AWS console does not allow you to check this synchronization state. But AWS claims that propagation is finished within 60 seconds.

Alright, your newly created record is now in sync and you can finally resolve it. Just to realize that it fails again. What’s the deal now?

understanding SOA

If you have been managing a DNS domain you probably came across certain record types like A, CNAME or TXT. There’s another very important but fairly unknown one, called SOA.

SOA stands for “start of authority record” and contains several information about a given DNS zone. For shipit.dev it looks like so:

$ dig shipit.dev SOA  
...
shipit.dev. 1934 IN SOA dns1.registrar-servers.com. hostmaster.registrar-servers.com. 1638547874 43200 3600 604800 3601
...

Here’s an explanation of each part. All time value are in seconds.

shipit.dev.: the zone name
1934: the TTL of the SOA record
IN: the class of this record SOA: the type of this record
dns1.registrar-servers.com.: the primary DNS server
hostmaster.registrar-servers.com.: the administrator’s mail address
1638547874: the zone’s serial number
43200: the time secondary servers should wait until they refresh from the primary one
3600: the retry interval for failed requests of secondary servers
604800: the expiration time for secondary servers if primary is unreachable
3601: the minimum duration value for how long records may be cached

I won’t go into each fields’ detail as only two of them are relevant for our topic.

You may know the term TTL (time to live) which determines how long a response for a certain DNS record is being cached. What’s important to us is that there’s even a TTL value for records that can’t be resolved.

According to the RFC 2308, it’s being calculated by choosing the smaller value between the TTL of the SOA record, in our case 1934, and the minimum duration value of our zone, in our case 3601. The resulting value is also being referred to as the negative TTL.

This means in our scenario, a negative answer to a DNS query may be cached up to one hour at worst. And this may lead to our problem.

putting it together

In case you haven’t figured out what’s going on, let’s have a look at the timeline of events.

We create a new AWS Route53 DNS record.
We try to resolve it but the record has not yet been propagated to AWS’ DNS servers and the resolution fails.
The negative answer is being cached with a certain TTL.
We wait 60 seconds until our new DNS record has been propagated to all DNS servers.
All consecutive tries to resolve the record keep failing until the negative answer in our cache expires.

And this is why newly created Route53 records may not resolve for quite some time if you query them too quickly. This pitfall is especially important when creating records in an automated fashion and querying them immediately after.

For example in my case I was encountering this issue when using external-dns. This component automatically sets up DNS entries for applications running in Kubernetes. Deploying my application and executing tests against its external endpoints using a CI pipeline resulted in numerous failing DNS queries.

Preventing this issue from happening is fairly simple though. Waiting a minute before querying is probably the easiest solution. Actively polling the synchronization state will be a bit quicker but may be a lot harder depending on the situation.

Be aware that this is not really an issue with AWS Route53 in particular. Negative TTLs exist for every DNS domain and are there to protect the DNS server from unnecessary queries.

Hopefully you are going to remember this post when encountering similar issues. Until then, enjoy your time in the cloud.

Optimizing applications on Kubernetes using Machine Learning

2021-06-30T00:00:00+00:00

Hint: This post has been happily sponsored by StormForge, the company building the product that is being used. Despite sponsorship, they allowed me to share my honest opinion. Huge thanks for their trust and support!

intro

Manually optimizing an application on Kubernetes is hard, especially if your goal is to keep costs low. There are plenty of options, the dependency trees in modern microservice architectures are getting bigger and bigger and for some (most?) of us all of this takes place on hardware that we have no control over.

Additionally without a lot of testing and monitoring effort it’s very difficult to determine the “correct” specs for your deployments. It requires extensive trial and error to push your application into the area where it still performs without being overprovisioned.

Otherwise you’ll end up wasting resources (and thus money) or with a system that falls apart as load increases. That’s why I decided to take a look at StormForge, a tool that aims to solve this problem without occupying endless amounts of engineering time called StormForge Optimize.

Optimizing with StormForge

StormForge is a SaaS product that combines trial and error experiments with machine learning to help you determine configurations that will optimize your application for the metrics you care about. It also includes Performance Testing to place load on your system in a test environment, but for this blog I’ll focus on StormForge’s optimization capabilities. This includes three key components:

the StormForge Kubernetes controller, handling the experiment execution and the communication with the machine learning (ML) model
the ML model, responsible for selecting the parameter sets for each trial based on previous results
the StormForge dashboard, allowing you to analyze and export the resulting configurations of your experiments during and after the execution

Below you can find a visualization that explains the general workflow of an StormForge experiment.

StormForge experiment flow

You start by creating an experiment, including your parameters and metrics you want to optimize.
The baseline values of your parameters will act as a starting point.
Your Kubernetes application (defined by a Pod, Job, Deployment, StatefulSet or DaemonSet) will be patched with these parameter values.
StormForge will wait for a certain condition. Usually this will be your application getting ready to process incoming requests.
A trial job will be executed putting your application under load.
Metrics, that you can freely define, will be collected.
The gathered metrics + the parameters of this run will be fed into StormForge’s machine learning model.
Based on that, the ML model will determine new parameter values.
Steps 3 to 8 will be repeated for a configurable amount of times.

Over the course of a single experiment, StormForge will be able to predict configurations that will exceed the performance of your baseline values, based on the metrics you defined.

As Kubernetes is StormForge’s primary target platform, you interact with it using the manifests, a concept you are probably already familiar with. The main one being the Experiment manifest, which consists of the following parts:

Optimization

The optimization section allows you to set the number of iterations that the experiment will go through. Adjust this value according to your use case. Integrating StormForge experiments into your CI pipeline to prevent performance regression? 20 trials may be sufficient.

Wanting to create a baseline configuration for your newly written application? Let StormForge cycle through 200 iterations and leave it running overnight to get the most accurate result.

Example:

...
  optimization:              
  - name: "experimentBudget"
    value: "50"
...

Parameters

Parameters define the possible configuration values that StormForge can experiment with, like the amount of memory, number of replicas or different disk storage types.

As a general rule of thumb increase the number of trials along with your number of parameters. The more permutations your experiment is covering, the more data points the ML model will need to determine the optimal configurations.

Example:

...
  parameters:
  - name: memory
    min: 500
    max: 2000
    baseline: 1000
  - name: cpu
    min: 500
    max: 2000
    baseline: 500
...

Patches

Patches instruct the StormForge controller how to apply the previously declared parameters to your Kubernetes manifests. This is where StormForge’s biggest strength is in my opinion as you can patch everything that is configurable through a Kubernetes manifest.

It may start with simple things like CPU and memory resource limits. But with more and more extensions coming to Kubernetes lately the possibilities are endless. One example I can imagine is using Crossplane to cycle through VM types of different cloud providers and see on which one your application performs best.

Example:

...
  patch: |
    spec:
      template:
        spec:
          containers:
          - name: my-app
            resources:
              limits:
                memory: "{{ .Values.memory }}Mi"
                cpu: "{{ .Values.cpu }}m"
...

Trial template

The trial template specifies what your application-specific performance measurement looks like. Internally it launches a Kubernetes Job which gives you the freedom of using your tool of choice, like StormForge’s own load testing solution, or another tool like Locust or Gatling.

Also you can take care of some preparation tasks like deploying a Helm chart or a dedicated Prometheus instance. Be aware that trial jobs will be executed on the same Kubernetes cluster as your application under test. So be sure to keep room for its potential additional CPU and RAM usage.

Example for a PostgreSQL load test trial template:

...
  template:
    spec:
      initialDelaySeconds: 15
      template:
        spec:
          template:
            spec:
              containers:
              - image: crunchydata/crunchy-pgbench:centos7-11.4-2.4.1
                name: pgbench
                envFrom:
                - secretRef:
                    name: postgres-secret
...

Metrics

Metrics allow you to define the criteria that you want to gather and/or optimize, e.g. minimizing the costs while maximizing the throughput of your application. StormForge comes with certain metrics out of the box, like trial duration, but can also query popular monitoring systems like Prometheus or Datadog if you are looking to optimize more specific metrics.

Example:

...
  metrics:
  - name: duration
    minimize: true
    query: "{{duration .StartTime .CompletionTime}}"
  - name: cost
    minimize: true
    type: pods
    # calculating with $20/month/CPU core and $3/month/GB of RAM
    query: '{{resourceRequests .Pods "cpu=0.020,memory=0.000000000003"}}'
    selector:
      matchLabels:
        component: my-app
...

For more information check out StormForge’s documentation which covers all these elements in more detail.

getting started

With all the concepts clarified let’s prepare our application and cluster for the first experiment. To begin with we have to decide on an application that we want to optimize. I went with Apache Cassandra because I always see it using huge amounts of resources in Kubernetes clusters and wanted to check whether this can be improved.

So I went ahead and created a standard StatefulSet that takes care of deploying my Cassandra pods. Additionally we’ll need some way of putting load onto the database during the trial job. Luckily, Cassandra comes with a tool to do just that, called cassandra-stress.

...
  template:
    spec:
      template:
        spec:
          containers:
          - image: cassandra:4.0
            name: cassandra-stress
            # making sure that our trial job does not eat up endless amounts of resources 
            resources:
              limits:
                memory: "1024Mi"
                cpu: "2000m"
            command:
            - bash
            - -c
            - |
              CASSANDRA_URL="cassandra.default.svc.cluster.local"
              /opt/cassandra/tools/bin/cassandra-stress write n=100000 -rate threads=400 -node $CASSANDRA_URL
              /opt/cassandra/tools/bin/cassandra-stress mixed n=100000 -rate threads=400 -node $CASSANDRA_URL
              cqlsh --request-timeout=60 -e "DROP KEYSPACE keyspace1;" $CASSANDRA_URL || true

As you can see our trial job consists of the following three steps:

measuring write performance while prepopulating the cluster at the same time
measuring mixed (~50:50 read and write) performance
dropping the keyspace to make sure each test run starts in a clean environment

Because we always execute the same number of read and write operations (2 x 10000), simply measuring the time of the trial job to complete will be a good way to judge how our database performs.

With our application good to go, let’s quickly prepare our Kubernetes cluster. Assuming you have a connection to it set up, after installing the redskyctl CLI tool the following commands will deploy the StormForge controller.

$ redskyctl login # log into our StormForge account
Opening your default browser to visit:

        https://auth.carbonrelay.io/authorize?...

You are now logged in.
$ redskyctl init # deploy the StormForge controller
customresourcedefinition.apiextensions.k8s.io/experiments.redskyops.dev created
customresourcedefinition.apiextensions.k8s.io/trials.redskyops.dev created
clusterrole.rbac.authorization.k8s.io/redsky-manager-role created
clusterrolebinding.rbac.authorization.k8s.io/redsky-manager-rolebinding created
namespace/redsky-system created
deployment.apps/redsky-controller-manager created
clusterrole.rbac.authorization.k8s.io/redsky-patching-role created
clusterrolebinding.rbac.authorization.k8s.io/redsky-patching-rolebinding created
secret/redsky-manager created
$ redskyctl check controller --wait # wait for the controller to become ready
Success.

starting simple

After going through the preparation phase we can start with our first experiment. To keep it simple, we will start with two parameters (CPU and memory limits) and a single metric (duration of our trial) that we want to optimize. Below you can see the relevant parts of the experiment spec:

...
  parameters:                                                                                                   
  - name: memory                             
    min: 1000
    max: 4000
    baseline: 2000
  - name: cpu
    min: 500
    max: 3500
    baseline: 1000
  patches:
  - targetRef:
      kind: StatefulSet
      apiVersion: apps/v1
      name: cassandra
    patch: |
      spec:
        template:
          spec:
            containers:
            - name: cassandra
              resources:
                limits:
                  memory: "{{ .Values.memory }}Mi"
                  cpu: "{{ .Values.cpu }}m"
                requests:
                  memory: "{{ .Values.memory }}Mi"
                  cpu: "{{ .Values.cpu }}m"
  metrics:
  - name: duration
    minimize: true
    query: "{{duration .StartTime .CompletionTime}}"
...

After roughly 1.5 hours we end up with the following chart in the StormForge dashboard.

Results of the single metric experiment

As you may have already guessed, the results are not really surprising. The more resources that fuel your application, the better it will perform. Who would’ve thought? Anyway, I think this experiment is valuable for the following two reasons.

First, it’s a simple showcase to understand how StormForge works, and second it helped us validate our assumptions. Imagine if Cassandra performed worse with more resources or the trial duration turned out to be completely random. In this case we would know that something in our test setup is messed up. Instead we now can jump onto our first proper experiment with confidence.

adding another dimension

So let’s add another dimension of metrics. In most real world scenarios you try to achieve the best performance while keeping the resource usage as low as possible. And that’s exactly what we will be trying to do. As a general rule of thumb for all kinds of optimization, you should always try to optimize at least two metrics that contradict each other, like increasing throughput while decreasing CPU usage or reducing latency while minimizing cache size.

We will be combining the CPU and memory usage into a single cost estimation metric by using the resourceRequests function which allows us to create a weighted sum of the two. The parameters will stay the same while the metrics change to the following:

...
  metrics:
  - name: duration
    minimize: true
    query: "{{duration .StartTime .CompletionTime}}"
  - name: costs
    type: pods                                    
    # weights are equal to AWS EC2 pricing of c5.xlarge instance ($30.60/month/CPU core,$15.30/month/GB RAM)
    query: '{{resourceRequests .Pods "cpu=0.03060,memory=0.000000000001530"}}'
    selector:                                     
      matchLabels:                         
        component: cassandra
...

After applying our Experiment manifest we can have a look at the individual trials runs using kubectl like so:

$ kubectl get trials -o wide -w                                     
NAME                          STATUS      ASSIGNMENTS             VALUES
cassandra-two-metrics-1-000   Completed   memory=2000, cpu=1000   duration=69, costs=33.808642559999996
cassandra-two-metrics-1-001   Completed   cpu=2845, memory=3003   duration=36, costs=91.87477680384
cassandra-two-metrics-1-002   Completed   cpu=1782, memory=2886   duration=47, costs=59.15927121407999
cassandra-two-metrics-1-003   Completed   cpu=1529, memory=3225   duration=51, costs=51.961336128
cassandra-two-metrics-1-004   Completed   cpu=728, memory=3403    duration=87, costs=27.73630531584
cassandra-two-metrics-1-005   Completed   cpu=537, memory=2269    duration=107, costs=20.07240498432
cassandra-two-metrics-1-006   Completed   cpu=1929, memory=4000   duration=41, costs=65.44468512
...

Results of the two metrics experiment

Now that we have a chart with more useful information here’s an explanation of the different data points:

The blue square represents the result of our baseline configuration, in our case 1 CPU core and 2 GB of RAM which resulted in a trial job duration of 69 seconds.
Green circles stand for “normal” trial runs. Those can mostly be ignored as they were outperformed by others.
The beige squares show the trial runs that performed well and give you a variety of choices depending on how you prioritize lowering costs versus increasing performance.
The orange square represents the trial run that StormForge selected as the sweet spot, in our case decreasing the costs by 16% while achieving the same trial run duration.

After selecting the trial run that fits our needs, we can export the new application configuration using the redskyctl tool. I’ll go with StormForge’s recommended trial that decreased the costs by 16% while achieving the same duration run.

$ redskyctl export -t \ # export the patched resource
  -f cassandra.yaml \ # path to the Cassandra STS manifest
  -f experiment.yaml \ # path to the Experiment manifest
  cassandra-two-metrics-1-015 # the trial run with our desired configuration
...
  resources:
    limits:
      cpu: 875m
      memory: 1000Mi
    requests:
      cpu: 875m
      memory: 1000Mi
...

Comparing this configuration to our baseline values of 1000m CPU shares and 2000Mi memory we were able to save quite a bit of resources. While the CPU improvement is probably based on a bit of wriggle room of the trial run duration, the RAM savings definitely confirm that Cassandra is running perfectly fine with half of our original value.

Please be aware that the outcome of StormForge experiments is only meaningful if your load test is very similar to what your application is being exposed to when running in production. That’s why you should select a load generation tool that is capable of doing so.

conclusion

So there you have it. A basic example of optimizing Cassandra on Kubernetes by letting software take care of tedious and repetitive tasks and thereby figuring out to which position certain knobs should be set. Of course this example could have been extended endlessly by adding more and more parameters, like JVM heap size, different disk types or various compression algorithms.

But as all of those very much depend on your use case and the capabilities of your infrastructure, I’d be very happy if you are able to do that on your own using StormForge after reading this blog post. If there are any open questions, feel free to reach out to them.

Self-employment: three months update

2021-04-17T00:00:00+00:00

some background

I jumped into the world of self-employment as a DevOps freelancer at the beginning of this year. After reviewing my first month was well received by the community I decided to keep this going. Here’s my second update after three months of self-employment.

let’s start with numbers

From my previous three clients two remain. Will go into more detail on that topic in the next section.

Since the beginning of this year I worked a total of 469.5 hours which makes 156.5 a month and 7.57 per working day. That’s still a pretty average workload in Germany. Let’s have a look at my work time per day of the week.

Working time distribution across the week

No visible change here. Still a pretty standard distribution which I don’t think will change any time soon. How about my working hours per month?

Hours worked per month

Seeing this chart for the first time surprised me a little. From my feeling I worked less and less each month but the raw numbers tell that the exact opposite is the case. Apparently my brain is very good at tricking me into thinking what I want to think.

A key takeaway here, track your time no matter if you bill your clients by hour or by project and be honest to yourself. After all, there are different and more important topics in life than work. 😉

clients come and go

As already said I, finalized working with one of my clients. At the same time other opportunities came up.

After reading my first blog post about self-employment a kind fellow, currently founding a startup, approached me to show me the upcoming open source project he’s working on. While there was no option for me to support them at the moment, I’m really glad to have met him, get to know his product at such an early state and provide some feedback to hopefully make it just a little better.

Similarly after reading my last blog post about wasting money with Kubernetes I was messaged by a really nice guy from an American company. We had two calls during the last weeks, getting to know each other a little and I got a quick glance at their product. Right now we are talking about different ways to collaborate. Let’s see where this will be going. 🙂

Lastly, after attending my city’s monthly meetup of entrepreneurs and self-employed people like usual, I got in contact with a person from a local company. They were currently applying for a prototype fund backed by the German government. I was especially hooked after finding out that the resulting prototype has to be made publicly available. As I’m a huge fan of open source software I did my best to improve their application. Currently crossing fingers that we’ll get the fund and the project can start in autumn.

Why do I tell you all of this? Well mainly cause I want to showcase that you can get to know products, companies and especially human beings from all kinds of sources. Get in contact with people, shout out your more or less professional opinion and join groups that share similar interests. This allows you to open up many opportunities which helps settling your mind in case you get unexpectedly dropped by one of your clients. Additionally having the choice where to go next is always better than depending on that one job you aren’t passionate about.

taxes

Disclaimer: Be aware that I have no kind of qualification when it comes to taxes whatsoever. Everything I’m sharing about this topic is based on my experience and half-hearted research to at least somewhat understand how things work.

This quarter I gained my first experience on how taxes are being handled in Germany if you are self-employed. Two kinds of taxes are important in this case, namely sales (or VAT) and income tax.

The former only applies to sales within Germany and are paid on a monthly basis. Example: I’m selling my DevOps services with a net worth of 1000 € to a German company. They’ll pay me a total amount of 1190 € (19% is the general German sales tax) and I transfer the 119 € to my local finance authority till the beginning of the next month.

For sales within Europe the so called reverse charge mechanism applies. This transfers the responsibility for declaring sales tax to the invoice recipient according to their local tax laws. The same is true if I sell my services to companies outside of the EU.

Income tax on the other hand is based, well as the name suggests on my income. As my total earnings will only be known by the end of the year and authorities don’t want to wait for their tax payments for that long, it’s common to go with an estimation here.

Credits: xdcd.com

That means communicating something like: “I plan on earning X € during this year.” at the beginning of each year. Based on that number you’ll make income tax payments (of the same amount) per month or in my case per quarter. At the end of the year your final earnings will be determined, the income tax will get calculated, all previous tax transaction will be accumulated and either you or the authorities have to balance out the difference.

As usual, the rate of income tax increases with the amount you earn, combined from all sources (employment, self-employment, real estate income, …). It starts at 0% and goes up all the way to 45%. If you are interested in the detailed numbers check out this Wikipedia article.

payment fees

If you read my first self-employment blog post you may remember me not being sure what I’ll end up paying to be able to receive money in foreign currencies. Turned out it was a lot (>5%).

Paying a high fee just to be able to get money onto your bank account hurts. Especially if you make the calculation back to the amount of time you spent to earn it in the first place. Working 3 days a month just for a bank transfer is a very unsatisfying feeling.

Lessons learned: Clarify how you’ll get paid before joining a client. With platforms like Wise* (formerly: TransferWise) it can be very easy and cheap to pay and get paid from people all across the globe. But in certain countries their service isn’t available (yet). There are alternatives. But expect them to be (way) more expensive.

* affiliate link

conclusion after three months

So how would I describe my three month journey of self-employment? To me it was one hell of a ride with emotions ranging from excitement and curiosity to fear and uncertainty. Leaving a stable, well paying job behind “just” to be able to work under your own guidance is a tough choice.

But as usual, leaving your comfort zone always results in personal growth no matter what’s the outcome. Guess most of you remember the first time approaching that attractive girl/boy, stuttering something of coffee with oat milk foam and park. After doing that X times you get used to it. This is where I am right now after these three months, which is good and bad at the same time.

Getting used to something allows you to free up your mind which enables you to take on new challenges. On the other hand it makes everything, like dating that girl you had a crush on since ages or doubling your paycheck, normal after some time. Let’s wrap it up here before I become to philosophical. Always strive to improve but never forget to appreciate what you already have. 🙂

Self-employment isn’t easy and it’s definitely not for everyone. But I think there are more people that could benefit from it and the demand, at least on the DevOps market, is definitely there. That’s why I’d like to support people going for it. My first step is working out a getting started guide I would have loved to have a few months back.

Feel free to hit me up if you have some questions that you’d like to get answered or topics that I should cover. Until then, stay safe. ✌️

How to waste money using Kubernetes

2021-03-14T00:00:00+00:00

Kubernetes enables you to do a lot. Orchestrating your application deployments, seamlessly integrating with public cloud services, distributing workload across thousand of nodes or simply generating a huge bill by the end of each month. While the latter can be beneficial if you are on the issuing side of the bill, this blog posts showcases a few pitfalls that I’ve seen (a lot) and explains how to avoid them to shave a few bucks off of it.

I’m mainly focussing on saving up on pay-as-you-go cloud services or IaaS/PaaS offerings here. But if you are running Kubernetes on-premise you may be able to spend your freed up resources on something else instead.

the productive VM

Let’s start with the topic that is probably going to have the biggest impact on your bill, insufficient or missing autoscaling.

Are you running the same amount of pods on the same number of nodes 24/7 to be able to handle the peak usage of your application that is occurring for only a few hours a day? Or do you keep your dev setups running all day long so that your engineers can continue where they left off at the next day?

If so there’s something your are missing out on…Autoscaling

Let’s highlight the three most common autoscaling mechanisms:

vertical pod autoscaling

Pods can have hard limits for the amount of CPU and memory resources that they are allowed to consume. With vertical pod autoscaling those limits can be adjusted (within boundaries) on the fly and automatically.

horizontal pod autoscaling

Here we are talking about adjusting the number of pod replicas. Your pod reaches a certain amount of load? Let’s create another one and the let the load balancer do its job. The load decreases a lot? Let’s scale down again.

cluster autoscaling

This is by far the most important one as this is where the actual saving happens. Where’s the benefit if you scale down pods and your nodes end up sitting there idling and wasting your money anyway? This is where cluster autoscaling comes into play that takes care of destroying and spawning nodes based on their utilization.

All of those three can be applied at the same time but you should know what you do in this case.

When coming from a background (and I certainly do) where you did setup a server or VM and labeled it prod-24_7-dont-touch-super-important-app-0 there’s a mindset change necessary when moving to Kubernetes. Worker nodes are simple minions, slaves if you will with the only purpose of running your precious applications.

They can be created within seconds but also disposed at the same rate. As soon as you accept that you can start having fun by decreasing the amount of computing resources your cluster runs on. Give it a try. Having a cluster with zero worker nodes is a thing today. 😉

Sure, running a single a1.xlarge AWS EC2 instance may only cost you 2.5 dollars a day but be aware that there a lot of factors that come into play here. Multiply this by having individual setups for your ten developers and running them 24/7 for a year and you are looking at a 9000 $ bill.

fearing the spot

Cloud providers are always having a bit of extra computing capacity as requests are rising and there’s a lot of demand fluctuation during the day. That’s why most of them are offering so called spot instances. They are sold to you at a huge discount (70 - 90%) with the only premise that they can be shutdown and withdrawn with little to no announcement.

image credit: Marcelo Jaboo

As that sounds completely inappropriate for your traditional productive VM it would be great if there’d be some platform that could gracefully handle this interruption for me, right? Surprise surprise, with a little bit of (or even no) preparation Kubernetes can take care of that.

If it receives the notification of a node being shut down it transfers the pods onto other nodes or requests a new one. Even if there’s no notification, the pods that are being terminated will get shifted and your application stays up and running.

It takes some time to get confident with someone else randomly shutting down nodes that run your productive app but if you’ve seen it working a dozen times you get used to it. Honestly, depending on your bidding price for the spot instances it doesn’t really happen often anyway (at least on AWS).

So instead of paying 100% of the on-demand price try offloading (some of) your workload onto those spot instances. If you want to be on the safe side you can also run your cluster on spot as well as on-demand instances at given ratio e.g. 50:50 or 20:80. That really comes down to your needs.

lazy and hungry

What’s worse than wasting money on an application that is just idling. Exactly, it’s wasting money on an application that is just idling and consuming a lot of resources at the same time.

In my experience we are talking primarily about memory here. It has not been only once that I’ve seen a Java Spring micro service application consuming gigabytes of RAM and doing nothing. At the same time there are Go applications that you’d need hundred instances of to fill that amount of memory.

Databases and message queues is also something to watch out for. I’ve joined a client 2 months ago and Cassandra is still not letting go of that 2 gigabytes.

Of course there are a lot of other factors that come into play when choosing your tech stack but take the base capacity consumption into consideration. Another option is to extract resource hungry components and reuse them for multiple deployments of your application.

buy 3, get 2

Having a look at the following image you can see that there’s something that can have an impact on the amount of resource capacity that is allocatable for your pods.

Overview of Kubernetes' resource capacity handling

It’s called memory and CPU reservations which are being taken care of by kubelet, the application managing your Kubernetes workers. Those are very important to make sure your nodes stay healthy and have enough resources to execute their own processes besides running your pods.

In most cases your worker nodes’ sole purpose should be to take over Kubernetes workloads but mandatory OS process, the container runtime or kubelet itself introduce a certain memory and CPU overhead that need to be considered.

What’s important here is that reservations are sized correctly so that all essential functions can be executed but you don’t hold back resources that end up being unused. Most of the time you can influence these parameters by setting the –kube-reserved and –system-reserved kubelet parameters accordingly.

For certain Kubernetes offers you are bound to certain predefined values. E.g. Azure Kubernetes service comes with increased reservations for small instances (on a percentage basis) so you may decide to go with less but bigger nodes instead.

Monitor your worker nodes and adjust the parameters or research your options depending on your Kubernetes operator and minimize the amount of money wasted on resource reservations.

CPU and memory requests

As you can see in the graph above CPU and memory request values, defined for each your pods, are being used to schedule them onto worker nodes. Setting these parameters too low and your application wont be able to perform as intended or in the worst case even get killed.

Declaring them too high and you’ll waste resources that could be used for scheduling other pods. E.g. if your are setting a memory request of 1 GB of RAM and your application never consumes more than 400 Mi you are essentially wasting money for 600 Mi.

This is why it’s important to define your requests as close as possible to the actual usage with going rather a little high than too low. Very similar to the previous resource reservations.

Kubernetes’ resource requests and limits is a very complex topic and you should get used to the fact that you’ll never get it 100% right but rather aim for an “as good as possible” state. Collecting historical data of your pods’ consumption or incorporating vertical pod autoscaling will help you with that.

summary

Of course there are still plenty of other things that can be improved in order to reduce the bill. Cleaning up unused resources, preventing to collect unnecessary logs and metrics, etc. But those points pretty much apply to every other infrastructure as well and this blog post already got longer than I initially anticipated. Seems like Kubernetes is pretty good at burning money. 😉

So there you have it. Be aware that most of the topics discussed are only realizable if your applications meet certain requirements. Being able to get shutdown at any time and saving their state externally are only some. Talk with your devs, have a look at the Twelve-Factor App and make a plan.

Hope you enjoyed the read and are able to save a few bucks in the future. 🙂

AWS EC2 launch configurations vs launch templates

2021-02-11T00:00:00+00:00

At first sight AWS launch configurations and templates may seem very similar. Both allow you to define a blueprint for EC2 instances. Let’s have a look at their differences and see which one we should prefer.

They grow up so fast

Launch configuration are old. In terms of cloud technologies they are essentially ancient. During my research I found articles that date back to 2010. It’s hard to find exact details but it seems like they have been introduced together Auto Scaling Groups (ASGs) or shortly after. This also explains why there are only compatible with ASGs. Want to create a single EC2 instance based on an launch configuration? That is not going to happen.

Settings that are supported include:

the EC2 image (AMI)*
the instance type (e.g. m5.large)*
an SSH key pair to connect to the VM*
the purchase options (on-demand or spot)
an IAM profile
one or more security groups
a block device mapping to specify additional storage volumes
a few more minor things

* marks required values

Changing any of these parameters is not supported due to launch configurations’ nature of being immutable. This means instead of updating it in place you need to delete and recreate it.

All in all launch configurations have a very specific use case and a set of configuration options limited to the basic parameters. Let’s see how launch templates compare.

The hot stuff

The first big difference is the wider range of AWS services that are compatible with launch templates. Additionally to ASGs it can be used in managed EKS node groups and to create single EC2 instances.

Regarding configuration options, they support a bit more than launch configurations like network settings and a few more advanced details (interruption behavior, termination protection, CloudWatch monitoring, …).

A few of launch templates' advanced settings

The main difference here is that every setting is optional. As you can see in the image above you can set the value “Don’t include in launch template” for every parameter. You can essentially create a launch template that specifies nothing. That’s kinda pointless but you get the idea.

Combining this with the ability to source values from existing templates and you can start to imagine all the options that arise. Similar to something like Docker images you can start to create your base template(s) and inherit more specific templates from them.

As nice as this may sound I just want to advice you to be cautious with doing this. Depending on organization and your upfront template “architecture” planning this may work really well. But it hasn’t been just once that I’ve seen this ending up in dependency hell. (including rhyme in blog post ✅) So think about if you don’t wanna stick with independent templates especially when factoring in the next feature.

Launch templates support versioning. Meaning while a single version is immutable you are still able to make modifications which will result in a new one that you can refer to. In my opinion this workflow provides a much better user experience compared to the delete and recreate approach that you need to go through with launch configurations. But again it adds complexity of managing the references in your ASGs and child templates.

Which is the better choice?

So, how did both do? Is there a clear winner or can I give you at least a recommendation which one you should prefer?

I’m not sure how things really evolved so take the following with a grain of salt. To me it seems like launch configurations have been created out of necessity when introducing ASGs. Afterwards the folks from AWS noticed that having an EC2 blueprint could be useful for other services as well. Cause it was probably easier to create something new instead of making the existing solution more generic they introduced launch templates in late 2017.

Additionally as far as I know there’s nothing that you can achieve with launch configurations which is not doable using launch templates. Please let me know if there’s a use case I’m missing here. The only advantage of launch configurations is that they are just simpler. No versioning, no inheritance, immutability, just a minimal set of required and a few optional values and you are good to create your ASG.

My impression from reading through the documentation is that AWS will soon start to deprecate launch configurations. They clearly discourage from using them and even provide a guide to replace existing launch configurations with templates. That’s why I’d suggest you to use launch templates for anything new and start to migrate your existing launch configurations if you plan on continue using them long-term.

That’s all with my little comparison. Hope you got some value out of it. Enjoy your day 👍

My first month of self-employment

2021-02-05T00:00:00+00:00

my way towards self-employment

In November 2020 I decided to take a huge personal step. I quit my beloved job and started my journey in becoming an independent freelancer. During the previous five years of working at two different companies the desire to start something for myself and being able to work with people from more than one company at a time became bigger and bigger. So the day came, 1st of January 2021, the first day of me being self-employed. Here’s what happened during the first month.

some numbers

Of course I was trying to prepare as best as I could beforehand and so I was able to start off working with two companies (both through Upwork). Another one approached me after reading one my blog posts which accumulates to three total clients.

Being self-employed also means being responsible for managing when and how much to work. As it’s what I’m used to and also works best for my clients I mostly kept working during the week which results in following distribution of my working time.

Working time distribution across the week

While being surprised during the creation of this chart I guess this spread is pretty common. You start with a lot of energy at the beginning of the week and it decreases every day. So if you need something done quick, make sure to give me call at the beginning of the week. 😁

In total I was working 149 hours across January. Dividing this by 20 working days results in 7.45 hours a day or 37 hours a week. That’s pretty close to the usual 40 hour contracts that are common in Germany and I had signed at my previous employers.

I expect to stay at this level of workload at least as long as COVID is a thing. Afterwards I may spend more time on other activities I enjoy and have been missing since quite some time.

upgrading

During my first month I also bought a few things to improve the experience of my clients, my productivity (at least that’s what I’m telling myself) and to help protect my personal health.

As I’m mainly working from home and thus spending quite some time with people in video calls I got a proper microphone cause who likes having a constant noise in their ear during a two hour call, right?

Besides some small things, like a USB-C hub or wireless headphones, I finally got myself a standing desk. Did get pretty used to these things at the office of my previous employer and missed having one a lot since working from home due to COVID. I’m a fan of mechanical ones cause I’m concerned about the electronic parts breaking and I don’t move it up and down multiple times a day anyway.

Other than that I reused the simple setup I already had. A Dell XPS 13 and secondary monitor. That’s all I currently need and I don’t plan on extending my setup for the time being.

all hail the bureaucracy

I’m living in Germany and as you may know there are a lot of complex regulations here for every aspect of life. The fact that I had to create a different kind of invoice for each of my three clients should demonstrate that. Unfortunately the German law is behind its time when it comes to jobs that haven’t existed 10 years ago just like my profession as a Cloud and DevOps Engineer.

This results in having to argue with the finance authority whether you’re an entrepreneur or a freelancer (yes those are two distinct things in German law). Don’t wanna go into too much detail here cause it’s probably not relevant to you anyway but just know that I had my fights.

Tax declaration is another complex topic in Germany. You can probably take care of it on your own and spend hours on topics you don’t enjoy and prevent you from doing what you excel at.

Or you can take a decent amount of money and let it getting handled by someone else. Ideally this should be someone you trust. That’s why I’m already at my second tax accountant after my first month of self-employment. Let’s hope I can stay with this one for a while.

Last topic, bank account. Fortunately it’s not hard to get a cheap or even free bank account in Germany these days. There’s only one thing that almost no bank talks about for whatever reason and that’s receiving money from other countries or in foreign currencies.

As one of the reasons getting self-employed was to work with companies outside of Germany this point is very relevant to me. I still don’t know what I need to pay until I receive my first payment and then it’s too late anyway. Let’s see how good or bad it’ll be.

summary

So there you have it. My first month of self-employment in ~800 words. Hopefully you got some value out of my story. Feel free to reach out to me if you have some questions regarding the process of getting self-employed.

All in all I’m very happy with my new way of working and hope that I have to take care of less and less annoyances each month and to be able to focus on what I enjoy. Maybe there’ll be another update after some time. Stay safe!

update 17.04.2021

I came up with an update after three months of self-employment. Feel free to check it out.

Kubernetes operators with Python #2: Implementing Controller

2021-01-18T00:00:00+00:00

Introduction

This post is the second part of a little blog series in which we are going through the complete process of implementing our own Kubernetes operator with Python. Previously we had a look at two ways on how to register a CRD for our ExchangeRate resource. Based on this we can now start writing our controller that will query the requested currency exchange rate and make them available to our Pods and Jobs in the form of a ConfigMap.

Implementing the controller

If you decided to create your CRD(s) using the Kubernetes API you can include and package this code together with your controller. This makes sure the resources your controller acts on, are definitely available in your cluster. Otherwise you’d need to make sure that your Kubernetes manifest is being applied before starting your controller which can be cumbersome. That’s the reason I prefer the API approach and will be going forward using it.

The controller application itself is essentially an endless loop that constantly watches Kubernetes resources of a specific kind, in our case ExchangeRate objects. Upon an update (creation, modification, deletion) its business logic will react accordingly. This can be anything from:

creating DNS entries
issuing TLS certificates
or fetching exchange rates from an API and storing the result in a ConfigMap

Although this could be implemented from scratch there’s a great framework that does most of the heavy lifting called Kopf. It allows you to almost entirely focus on implementing the business logic of your controller.

Below you can see all the code that is necessary to watch for and react on a newly created ExchangeRate object.

import kopf

@kopf.on.create('operators.brennerm.github.io', 'v1', 'exchangerates')
def on_create(namespace, spec, body, **kwargs):
    print(f"An ExchangeRate object has been created: {body}")

Having this we can concentrate on:

extracting the requested currency out of the ExchangeRate object

currency = spec['currency']

querying the current exchange rate for this currency

exchange_rates_url = 'https://api.exchangeratesapi.io/latest?symbols='
rate = requests.get(f"{exchange_rates_url}{currency}").json()['rates'][currency]

creating a new ConfigMap containing the exchange rate

k8s_client.CoreV1Api().create_namespaced_config_map(namespace, 
 {
   'data': {
       'rate': str(rate)
   }
 }
)

So there you have the essential parts of the controller. All that is left is handling an update (= update the exchange rate if the currency changes) and deletion (= destroy the ConfigMap if the ExchangeRate object is deleted) of an ExchangeRate object. Updating is pretty much the same as the above code but instead of creating a ConfigMap you’ll patch the existing one using:

k8s_client.CoreV1Api().patch_namespaced_config_map(name, namespace, new_data)

Regarding handling the deletion we could choose the obvious way of implementing the kopf.on.delete handler and deleting the ConfigMap manually. The more elegant way IMO is making use of Kubernetes’ owner references. These allow to specify parent-child relationships between objects which will result in an automatic garbage collection of all children upon deleting the parent.

And as we are even too lazy to implement this ourselves we’ll let Kopf take care of it by passing our ConfigMap data to kopf.adopt. Next to a few other things this will set the owner reference of the ConfigMap to our ExchangeRate object.

And that is all to create a simple “CRUD” controller application using Kopf. Let’s package and finally deploy it to our Kubernetes cluster.

Packaging and running the controller

As an operator will, similar to any other application, run within a Pod, we need to package it as one of the Kubernetes supported image formats. I decided to go with the most popular way of building my image using a Dockerfile. Below you can find its content and a few comments that explain each step.

FROM python:3.8-alpine
apk --update add gcc build-base # required to build some of the following pip packages
RUN pip install --no-cache-dir kopf kubernetes requests # install our dependencies
ADD exchangerates-operator.py / # copy our operator into the image
CMD kopf run /exchangerates-operator.py # start our operator on container creation

The resulting image needs to be pushed to some registry your Kubernetes cluster has access to. Afterwards you can deploy the operator e.g. by using a Deployment manifest that could look like this.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: exchangerates-operator
  labels:
    app: exchangerates-operator
spec:
  replicas: 1 # make sure to not have more than one replicas
  strategy:
    type: Recreate # make sure the old pod is being killed before the new pod is being created
  selector:
    matchLabels:
      app: exchangerates-operator
  template:
    metadata:
      labels:
        app: exchangerates-operator
    spec:
      containers:
      - name: exchangerates-operator
        image: registry.brennerm.io/exchangerates-operator:latest

If you are using RBAC ensure that your operator has the sufficient permissions to register CRDs, read ExchangeRate objects, create events and ConfigMaps. The according role could look like so:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: exchangerates-operator
rules:
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["create"]
- apiGroups: ["operators.brennerm.github.io"]
  resources: ["exchangerates"]
  verbs: ["*"]
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["create, patch"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["create"]

After that’s done we are ready to try out our new operator. To do that we’re going to use the following ExchangeRate object.

# exchangerate.yml
apiVersion: operators.brennerm.github.io/v1
kind: ExchangeRate
metadata:
  name: exchange-rate-usd
spec:
  currency: USD

Apply it and a ConfigMap with the following content should appear pretty much instantly.

$ kubectl apply -f exchangerate.yml
$ kubectl get configmaps
NAME                      DATA   AGE
exchange-rate-usd-j98bc   1      2s

$ kubectl describe configmaps exchange-rate-usd-j98bc
Name:         exchange-rate-usd-j98bc
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
rate:
----
1.1901
Events:  <none>

And there we have our working operator that takes care of registering a CRD on startup and starting the controller application that watches objects of a this resource kind. Have a look at the picture below to review all the relevant parts and processes.

Overview of how our Exchange Rates operator works

Of course that’s a fairly minimal example but it should give you all the tools and knowledge to create much more complex operators.

I pushed all the code and manifests into a Git repository for you to see the operator as a whole. If you still run into issues or have open questions feel free to drop me a message. Hope you enjoyed that little guide. 👍

Update 19.01.2021

Nolar, aka Kopf’s current maintainer, pointed out that Timers would be a good fit for this use case as well. They allow you to regularly trigger your controller no matter if there were changes on your objects. For our use case this would for example allow us to automatically pull and update the exchange rate every hour like so:

@kopf.timer('operators.brennerm.github.io', 'v1', 'exchangerates', interval=3600.0)
def update_exchange_rate(namespace, name, spec, status, **kwargs):
    # update ConfigMap
    ...

Check the documentation for some further details.

Kubernetes operators with Python #1: Creating CRDs

2021-01-13T00:00:00+00:00

Introduction

A lot of the core processes happening in a Kubernetes cluster are following the so called controller pattern. This pattern describes an ongoing monitoring of resources and reacting appropriately to bring the current state closer their desired state.

A simple example is the relationship between the Deployment and Pod resources. When increasing the replica count in the Deployment object the number of Pods will be adjusted by the responsible controller.

Operators are a special kind of controllers and a popular way of extending Kubernetes clusters. They consist of a controller application and domain specific custom Kubernetes resources (CRDs). The controller is watching for changes on objects he’s responsible for and executes tasks according to his business logic.

As a huge part of the K8s ecosystem is written in Golang it’s also the de facto standard language when writing operators. Today I wanna show you how to write your own operator using a more beginner friendly programming language like Python.

To showcase the whole process we’ll create an operator that provides currency exchange rates through a ConfigMap object. These can then be referred to by our Pods and Jobs. The rates will be fetched from the Exchange Rates API and to tell the operator which rates to pull, we’ll register our own CRD. The following diagram will provide an overview of all components and their relations.

Overview of how our Exchange Rates operator works

We’ll take care of implementing the controller application in the next part of this little blog series. At first we are going to start with defining our custom ExchangeRate resource by creating a CRD.

Creating the CRD

Creating a CRD is nothing else than registering a new Kubernetes resource type with a fixed set of fields and their data types. To accomplish that we’ll have a look at two different ways.

Using a Kubernetes manifest

Every Kubernetes resource can be created using a manifest and as a CRD is a resource itself, it is no exception.

# crd.yml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: exchangerates.operators.brennerm.github.io
spec:
  group: operators.brennerm.github.io
  versions:
    - name: v1 # it's possible to provide multiple versions of a CRD
      served: true # it's possible to disable a CRD
      storage: true # there can be multiple versions but only one can be used to store the objects
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                currency:
                  type: string
                  enum: ["CAD","CHF","GBP","JPY","PLN","USD"] # we'll limit the valid currencies to these
  scope: Namespaced # resources can be namespaced or available for the whole cluster
  names:
    plural: exchangerates
    singular: exchangerate
    kind: ExchangeRate # this name is being used in manifests
    shortNames: # these short names can be used in the CLI, e.g. kubectl get er
    - er

The schema definition follows the OpenAPI v3 specification which can be used to define various data types and nested structures. After applying this file, e.g. with kubectl apply -f crd.yml we are ready to create our first ExchangeRate object using the following manifest.

# exchangerate.yml
apiVersion: operators.brennerm.github.io/v1
kind: ExchangeRate
metadata:
  name: exchange-rate-usd
spec:
  currency: USD

Using the Kubernetes API

Instead of using a manifest we can also register our CRD by using the Kubernetes API. Conveniently there’s an official Python client that we’ll use for this purpose.

Below you can find the definition of our CRD as a Python object. You’ll see a lot of similarities to the above manifest.

import kubernetes.client as k8s_client
import kubernetes.config as k8s_config

exchange_rate_crd = k8s_client.V1CustomResourceDefinition(
    api_version="apiextensions.k8s.io/v1",
    kind="CustomResourceDefinition",
    metadata=k8s_client.V1ObjectMeta(name="exchangerates.operators.brennerm.github.io"),
    spec=k8s_client.V1CustomResourceDefinitionSpec(
        group="operators.brennerm.github.io",
        versions=[k8s_client.V1CustomResourceDefinitionVersion(
            name="v1",
            served=True,
            storage=True,
            schema=k8s_client.V1CustomResourceValidation(
                open_apiv3_schema=k8s_client.V1JSONSchemaProps(
                    type="object",
                    properties={"spec": k8s_client.V1JSONSchemaProps(
                        type="object",
                        properties={"currency":  k8s_client.V1JSONSchemaProps(
                            type="string",
                            enum=["CAD","CHF","GBP","JPY","PLN","USD"]
                        )}
                    )}
                )
            )
        )],
        scope="Namespaced",
        names=k8s_client.V1CustomResourceDefinitionNames(
            plural="exchangerates",
            singular="exchangerate",
            kind="ExchangeRate",
            short_names=["er"]
        )
    )
)

Afterwards we’ll need to load our kubeconfig and call the API endpoint for finally creating the CRD.

k8s_config.load_kube_config()

with k8s_client.ApiClient() as api_client:
    api_instance = k8s_client.ApiextensionsV1Api(api_client)
    try:
        api_instance.create_custom_resource_definition(exchange_rate_crd)
    except k8s_client.rest.ApiException as e:
        if e.status == 409: # if the CRD already exists the K8s API will respond with a 409 Conflict
            print("CRD already exists")
        else:
            raise e

The function load_kube_config reads the access credentials to your K8s cluster from your local environment (most likely ~/.kube/config). If that’s not what you intend, the library also provides manually setting the configuration or loading it from within an in-cluster environment.

No matter how you end up creating your CRD, executing kubectl api-resources should list your new resource type if you’ve done everything correctly.

$ kubectl api-resources
NAME                              SHORTNAMES   APIGROUP                       NAMESPACED   KIND
...
exchangerates                     er           operators.brennerm.github.io   true         ExchangeRate
...

Personally I prefer this approach compared to using a CRD manifest. I’ll talk about the reason in the second part of this blog series in which we’ll implement the controller application.

Github Actions workflow for merged/closed PRs

2020-12-27T00:00:00+00:00

Lately I made some investigations on different events that you can use to trigger Github Actions workflows. I was especially interested in the pull request events as executing some clean up task as soon as a PR is merged was one of my goals.

Going through the list of available events, closed turned out to be what I was looking for. Additionally I added the requirement to make a distinction between the following two cases when closing a pull request.

merged - the PRs’ changes have been merged into the target branch
closed - the PR has been closed without merging its changes

Turns out the event received contains the pull request object which itself contains a lot of additional information such as whether the PR has been merged or not. Thus the following configuration is what I came up with.

name: Close Pull Request

# only trigger on pull request closed events
on:
  pull_request:
    types: [ closed ]

jobs:
  merge_job:
    # this job will only run if the PR has been merged
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    steps:
    - run: |
        echo PR #${{ github.event.number }} has been merged

  close_job:
    # this job will only run if the PR has been closed without being merged
    if: github.event.pull_request.merged == false
    runs-on: ubuntu-latest
    steps:
    - run: |
        echo PR #${{ github.event.number }} has been closed without being merged

To see the above workflow in action check out this little showcase repo I created. Feel free to use it as a starting point.

Screwing up remote access to dozens of servers within seconds

2020-12-22T00:00:00+00:00

the power of infrastructure as code

Back in the day (must have been around 2016) my team and I were using Ansible to provision our infrastructure. It’s a tool to configure servers in an automated and reproducible manner. It helped us to setup web servers, databases and all kinds of services with very little preconditions.

There’s no central server that you need to setup initially and keep running. The client machines don’t need any kind of custom agent. All that is required is a Python interpreter and remote access using SSH which both are baked into modern OS’ anyway.

Additionally Ansible allows to create separate groups of servers and apply different actions on them as well as define tasks that will be executed for every server. E.g. increasing a version of nginx should only be applied to web servers while changing the address of the DNS or NTP server is something that is relevant for all hosts.

Due to parallel execution, Ansible is able to rollout these changes to a huge amount of clients in a very short amount of time. And with this great power comes great responsibility…

the day I screwed up

As we used Ansible to make system level changes root permissions were almost always required. Hence we used the system’s root user which is debatable from my todays’ point of view. Anyway securing remote access to this user was very important.

SSH, among others, supports authentication using a password or a SSL key pair. We used the latter as it is more secure (and convenient) but logging into the root account using a password was still an option which we wanted to actively disable.

Sshd, the software realizing the remote access to the server, provides an configuration parameter called PermitRootLogin with the following options:

yes - allows root login with all authentication methods
no - disables root login using SSH completely
prohibit-password - disables root login using password authentication

Number 3 it is, right? So I sat down and wrote the Ansible “code” (it’s actually YAML). As I am an experienced engineer, so I thought, testing my changes before deploying them was obvious. I spun up a new machine running Ubuntu 16.04, deployed my change, verified the config, tried logging in using a password which failed as expected… everything was looking great.

There I was, confidently typing the Ansible command into my shell (it was actually executed on our CI server, but I’m trying to be dramatic here 😉), targeting all of our servers, hitting Enter and seeing the usual endless amounts of logs flying by.

And then boom…around 50 servers responded with an error, oddly all of them running Ubuntu 14.04. Turns out that Ubuntu 14.04. came with an older version of sshd which, instead of prohibit-password, expected the value without-password for the PermitRootLogin config.

As already said, when using Ansible 50% of what you need is SSH access but SSH access is also what you need a 100%. So take a guess, what happens to an sshd service that encounters an unknown configuration parameter during its startup? Correct, there’s no startup. All there is is an error message that never reached the display of my PC as I screwed up the SSH connection.

the learning

Luckily for me all of these servers were LXC containers and no physical machines. So fixing the sshd config was just a matter of crafting a script with the correct lxc-execute (same as docker exec if that’s more familiar to you) command. Alright fire extinguished, coming to the even more important part. What could I have done to avoid this situation in the first place?

In an ideal world replicating the whole setup in a staging environment and applying my changes there would be the way to go. You and me know that in reality this is mostly not achievable due to various reasons, e.g. limited computing capacities.

Instead of copying the whole setup I could have tested my changes against all base images of our containers, e.g. Ubuntu 14.04 additionally to 16.04. This would have revealed this particular issue but may not have helped in a different case.

Another out of the box feature of Ansible that helps preventing making erroneous configurations to all of your hosts at once is called rolling updates. By passing the serial parameter, Ansible will apply your changes to batches of the given size. If it fails for one batch the execution will stop, leaving the remaining hosts intact.

All in all I’m happy that I made this mistake for a service with no impact to the end user at all. A web or database service that is not starting would have been much worse.

And that’s my how I screwed up remote access to dozens of servers in seconds. Hopefully you had a good laugh from my postmortem story and ideally learned how to prevent something like this from happening to you. 👍

Update 22.12.2020

Another safety measure several users on Reddit recommended, is to validate the config after copying it onto the server. In case of sshd this would look like so:

- name: Upload sshd config file
  template:
    src: sshd_config
    dest: /etc/ssh/sshd_config
    validate: "/usr/sbin/sshd -T -f %s"

-T tells sshd to validate the config, -f is used to pass its path and %s will be replaced by Ansible to point to the temporary config file.

This is supported by multiple Ansible modules, like copy or template and will prevent copying the faulty config.

Kubernetes Overview Diagrams

2020-12-01T00:00:00+00:00

I started creating and sharing overviews of various Kubernetes objects on my Twitter. Some people requested to put them in a more convenient place for later use. This page is the result.

Feel free to use them for yourself. Giving credit is very much appreciated. Have fun (y)

Architecture

Overview of Kubernetes' basic architecture

Workload

Overview of Kubernetes Workload objects

Networking

Overview of Kubernetes Networking objects

Storage

Overview of Kubernetes Storage objects

RBAC

Overview of Kubernetes RBAC objects

Resource requests and limits

Overview of Kubernetes resources requests and limits

Setting up a load balancer with failover support in Azure

2020-11-19T00:00:00+00:00

the cloud solves all your problems…

Lately I was in need of a load balancer solution supporting the classic active-passive scenario. In other words, forward (ideally TCP) traffic to server A. If server A is not reachable reroute to server B. Being a fairly simple and common problem it actually took me quite some time to find a solution I was happy with.

Initially I was very confident that the Azure Load Balancer was exactly what I was looking for. Finding out that it does only support Azure internal endpoints made my anticipation decline. Next stop: Azure Application Gateway!

Being already sceptical cause its kind of an overkill service for such a simple problem I sat down and created an instance anyway. Seeing external targets being an option when configuring the backend pools was good. HTTP and HTTPS being the only supported protocols wasn’t great. Finding out that App Gateway is not capable of my desired failover scenario took it off if my option list.

There I was, sitting in front of cloud resources having terabytes of memory but not being able to realize a simple failover use case. Just before coming to the conclusion of setting up and managing a few HA proxy VMs by myself I stumbled upon a service I have never heard of… Azure Traffic Manager.

After reading about it for a few minutes I was pretty sure that this is what a was looking for. Let’s see if I was right.

…but does it?

Azure Traffic Manager’s functionality is based on DNS. It acts as an DNS alias that is being rewritten depending on the load balancing algorithm, of which it supports the following:

Priority - provide multiple endpoints, the one that is available and has the highest priority receives all of the traffic
Weighted - allows to set a weight for each endpoint and distributing the traffic based on it
Performance - routes the request to the fastest endpoint based on where the request originates from
Geographic - routes the request to the closest endpoint based on where the request originates from
Multivalue - returns all available endpoints
Subnet - allows to map the requesting source IP subnet to a fixed endpoint

For showcasing the general functionality we’ll setup a failover functionality for two DNS servers.

Overview of my DNS server failover showcase scenario

The Terraform code for setting up the corresponding Traffic Manager profile can be found below.

resource "azurerm_traffic_manager_profile" "my-traffic-manager-profile" {
  name                   = "my-traffic-manager-profile"
  resource_group_name    = "my-traffic-manager"
  traffic_routing_method = "Priority"

  dns_config {
    relative_name = "my-traffic-manager-profile" # the host name for the FQDN of this traffic manager profile
    ttl           = 60 # the TTL for the DNS alias
  }

  monitor_config { # the config for monitoring the endpoints for availability and latency
    protocol                     = "tcp" # the protocol to monitor with
    port                         = 53 # the port to monitor
    interval_in_seconds          = 30 # the time between probes
    timeout_in_seconds           = 10 # the time the probing agent waits for a response before considering a check as failed
    tolerated_number_of_failures = 3 # the number of failing health probes that will be tolerated before marking the endpoint as unhealthy
  }
}

For our failover use case the Priority method is the one we are going with. We’ll provide two endpoints, a primary and a secondary and the former will have a higher priority (=a lower value). In terms of Terraform code this configuration will look like this.

resource "azurerm_traffic_manager_endpoint" "primary" {
  name                = "cloudflare"
  resource_group_name = "my-traffic-manager"
  profile_name        = "my-traffic-manager-profile"
  target              = "1.1.1.1"
  type                = "externalEndpoints"
  priority            = 1
}

resource "azurerm_traffic_manager_endpoint" "secondary" {
  name                = "google"
  resource_group_name = "my-traffic-manager"
  profile_name        = "my-traffic-manager-profile"
  target              = "8.8.8.8"
  type                = "externalEndpoints"
  priority            = 2
}

To test our configuration we simply need to constantly send DNS requests to our Traffic Manager instance and meanwhile shutdown the Cloudflare DNS server. ;) Afterwards our DNS requests should be rerouted the Google’s DNS server.

To not interrupt DNS queries all over the world I decided to take the less interfering path of just disabling the cloudflare endpoint like so:

$ az network traffic-manager endpoint update -g my-traffic-manager --profile-name my-traffic-manager-profile -n cloudflare --endpoint-status Disabled

This simulates an outage of our primary endpoint and after a short amount of time the following happens.

$ while 1; do host shipit.dev my-traffic-manager-profile.trafficmanager.net; sleep 1; done
...
Using domain server:
Name: my-traffic-manager-profile.trafficmanager.net
Address: 1.1.1.1#53
Aliases:

shipit.dev has address 185.199.109.153
shipit.dev has address 185.199.110.153
shipit.dev has address 185.199.108.153
shipit.dev has address 185.199.111.153

Using domain server:
Name: my-traffic-manager-profile.trafficmanager.net
Address: 8.8.8.8#53 # notice that the IP changed
Aliases:

shipit.dev has address 185.199.109.153
shipit.dev has address 185.199.111.153
shipit.dev has address 185.199.110.153
shipit.dev has address 185.199.108.153
...

As you can see the failover to Google’s DNS server works just as expected. And if we re-enable the cloudflare endpoint, requests will be routed back to it again automatically.

This behavior can be accomplished for any kind of protocol running over TCP. Regarding HTTP and HTTPS, Traffic Manager even supports checking a specific path and matching given HTTP status codes to determine whether the endpoint is healthy.

And there you have it. A very easy to configure load balancer with failover support that supports targets within and outside of the Azure cloud.

Setting up an EKS cluster with IAM/IRSA integration

2020-11-15T00:00:00+00:00

Introduction

AWS’ IAM service is a powerful system to provide fine-grained control over AWS resources. Additionally it is integrated into several AWS services and EKS is no exception. Next to the cluster role, AWS introduced 2019 the concept of IRSA, which stands for IAM Roles for Service Accounts.

Together with Kubernetes’ RBAC system it allows to assign IAM role capabilities to K8s computing resources like Pods and Jobs. A simple use case you can imagine is allowing a Pod to write to a S3 bucket. After reading through this blog post you will understand how to create an EKS cluster using Terraform with IRSA support and how to make use of it.

Preparing the VPC

Before creating an EKS cluster you need to set up a VPC network that your data and control plane traffic can go through. It will also define whether your Kubernetes API endpoint will be accessible publicly or only from within a private network.

Depending on your requirements the VPC configuration can be more or less complex. If you don’t want to do anything too crazy I suggest to use aws-vpc module. It provides a nice abstraction layer for the AWS VPC Terraform resources that are being used under the hood. Below you can find an example VPC configuration that can act as a starting point for your EKS.

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name                 = "my-vpc"
  cidr                 = "10.0.0.0/16"
  azs                  = ["eu-central-1a", "eu-central-1b", "eu-central-1c"]
  public_subnets       = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  enable_dns_support   = true
  enable_dns_hostnames = true
}

Be aware that an EKS cluster needs at least two subnets in different availability zones. Enabling the DNS related flags is necessary to allow the worker nodes to find and register themselves at the API server.

Creating the EKS cluster

Similar to the VPC, I recommend you to use the aws-eks Terraform module if your EKS setup is not too far away from the ordinary. Below you can find an example Terraform code snippet that uses the previously discussed VPC.

module "cluster" {
  source          = "terraform-aws-modules/eks/aws"
  cluster_name    = "cluster"
  cluster_version = "1.18"
  vpc_id          = var.vpc_id
  subnets         = var.vpc_subnet_ids
  enable_irsa     = true

  worker_groups = [
    {
      instance_type = "t3.medium"
      asg_max_size  = 3
    }
  ]
}

The enable_irsa flag will lead to the OIDC (OpenID Connect) provider being created. Additionally we will define a Terraform output that contains its ARN (Amazon Resource Name) which will be used in the next step.

output "oidc_provider_arn" {
  value = module.cluster.oidc_provider_arn
}

And that’s all you need to continue with the next step.

Creating a service account associated with an IAM role

In this example we are going to create a service account that has full access to all of your S3 buckets. This can be easily changed by adjusting the policies that you attach to your IAM role.

To start with we need to define a few variables.

ROLE_NAME=s3-writer # the name of your IAM role
SERVICE_ACCOUNT_NAME=s3-writer # the name of your service account name
SERVICE_ACCOUNT_NAMESPACE=default # the namespace for your service account
PROVIDER_ARN=$(terraform output -json | jq -r .oidc_provider_arn.value) # the ARN of your OIDC provider
ISSUER_HOSTPATH=$(aws eks describe-cluster --name cluster --query cluster.identity.oidc.issuer --output text | cut -f 3- -d'/') # the host path of your OIDC issuer

The most important step is defining the correct assume policy for your IAM role.

$ cat > assume-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "$PROVIDER_ARN"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${ISSUER_HOSTPATH}:sub": "system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}"
        }
      }
    }
  ]
}
EOF

This will allow the service account to switch to your role using the AssumeRoleWithWebIdentity command.

Afterwards you can create the new role using the above assume policy and attach your desired policies to it.

$ aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://assume-policy.json
$ aws iam update-assume-role-policy --role-name $ROLE_NAME --policy-document file://assume-policy.json
$ aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess

The last step is to create the Kubernetes Service Account and annotate it with the role ARN.

$ kubectl create sa $SERVICE_ACCOUNT_NAME
$ S3_ROLE_ARN=$(aws iam get-role --role-name $ROLE_NAME --query Role.Arn --output text)
$ kubectl annotate sa $SERVICE_ACCOUNT_NAME eks.amazonaws.com/role-arn=$S3_ROLE_ARN

Using the service account in your application

Assigning the newly created service account to your application only requires adding a single line to your Deployment or Job manifest.

apiVersion: apps/v1
kind: Deployment
spec:
  ...
  template:
    ...
    spec:
      serviceAccountName: s3-writer
      containers:
      - image: myapp:latest
        name: myapp
        ...

This will result in the EKS Pod Identity Webhook injecting some environment variables and a volume mount into each of your pods that contain the access credentials for your IAM role. If you use a recent version of the AWS SDK these will be picked up automatically when creating a new session and you are good to go.

As a conclusion you can find an overview of all components and processes being part of the IRSA concept below.

Overview of IRSA components and processes / image source

For further details check out the official AWS blog post about IRSA.

Integrating cdk8s with Argo CD

2020-05-04T00:00:00+00:00

After having a look at how to integrate cdk8s with Flux I was asked whether I could have a look at Argo CD as well. So without further ado, let’s integrate cdk8s with Argo CD.

Argo CD

But at first, what is Argo CD? Similar to Flux it realizes Kubernetes Manifest deployments by following the GitOps paradigm. This states that the desired condition of your system should be kept in one or more Git repositories. Tools like Argo CD take care of keeping your system (Kubernetes in this case) and this state in sync. In its simplest form think of a repository containing a single Kubernetes Manifest that defines a Deployment and a Service. Argo CD will make sure to deploy them to your cluster and continue to do so for any new change.

To control Argo CD it ships with a CLI tool and a really nice web UI, that allows you to check for the status of your deployments in real time. Both allow you to create new applications or setup new cluster connections. The latter allow you to use the multi cluster functionality to orchestrate deployments over several clusters with a single instance of Argo CD. Setting one up would be next step from here.

Preparation

If you want to follow along you gonna need a running Argo CD instance. Feel free to use my snippet to set one up within 5 minutes. Otherwise go through the official Getting Started guide.

In case you are not familiar with cdk8s make sure to check out this little rundown or my dedicated blog post that goes into more detail. Otherwise let’s dive right into the action.

At first we need some cdk8s code that we can deploy later on. We are going to use the argocd-example-apps repository as a starting point. It contains the same configuration in different formats, like plain K8s manifests, a Helm Chart or kustomize files all deploying a simple guestbook application consisting of one Deployment and one Service. We are going to put our code into a new folder called cdk8s-guestbook.

As with every cdk8s application we start by executing cdk8s init python-app. I’m going to use Python but feel free to go with TypeScript. Afterwards we define the Deployment and Service objects in the main.py.

...
  label = {"app": "guestbook-ui"}

  k8s.Service(self, 'service',
              spec=k8s.ServiceSpec(
                type='LoadBalancer',
                ports=[k8s.ServicePort(port=80, target_port=k8s.IntOrString.from_number(80))],
                selector=label))

  k8s.Deployment(self, 'deployment',
                 spec=k8s.DeploymentSpec(
                   replicas=1,
                   selector=k8s.LabelSelector(match_labels=label),
                   template=k8s.PodTemplateSpec(
                     metadata=k8s.ObjectMeta(labels=label),
                     spec=k8s.PodSpec(containers=[
                       k8s.Container(
                         name='guestbook-ui',
                         image='gcr.io/heptio-images/ks-guestbook-demo:0.2',
                         ports=[k8s.ContainerPort(container_port=80)])]))))
...

If that’s done we push the code into a repo that is preferably publicly accessible. Otherwise we need to pass ArgoCD the credentials to the repo later on. Afterwards we can continue with the integration.

Integration

Currently cdk8s is not supported by Argo CD out-of-the-box. To be able to use it we need to register cdk8s as a custom config management plugin. This works by simply creating/updating the argocd-cm Kubernetes ConfigMap with something like the following content:

# config.yml
data:
  configManagementPlugins: |
    - name: cdk8s # the name of the plugin that we'll later use to reference it
      init: # some optional preprocessing commands
        command: ["bash"]
        args: ["-c", "pipenv install && cdk8s import -l python && cdk8s synth"] # making sure everything is installed and generating the K8s manifest(s)
      generate: # the output of this command will be deployed onto the target cluster
        command: ["bash"]
        args: ["-c", "cat dist/*"] # printing the generated Kubernetes manifests
kind: ConfigMap
apiVersion: v1
metadata:
  annotations:
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm
  namespace: argocd
  selfLink: /api/v1/namespaces/argocd/configmaps/argocd-cm

Apply this with kubectl apply -f config.yml and our plugin is ready to use.

So let’s try it out. I’m going to use the Argo CD CLI tool but creating the application using the web UI will work as well. After issuing this command:

$ argocd app create guestbook \ # creating an application called guestbook
      --repo https://github.com/brennerm/argocd-example-apps.git \ # the URL of our repo
      --path cdk8s-guestbook \ # the path to the folder containing our config
      --dest-server https://kubernetes.default.svc \ # the cluster we want to deploy to
      --dest-namespace default \ # the namespace we want to deploy to
      --sync-policy automated \ # enabling automatic sync of changes in the repo
      --config-management-plugin cdk8s # make sure to use our cdk8s plugin

we end up with the following error:

FATA[0006] rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: Unable to generate manifests in cdk8s-guestbook: rpc error: code = Unknown desc = ‘bash -c pipenv install && cdk8s import -l python && cdk8s synth’ failed exit status 127: bash: pipenv: command not found

Having some background knowledge of Argo CD’s internals make this issue somewhat predictable. Each config management plugin is being executed in a component called the argocd-repo-server. To make our custom plugin work we also need to make sure that the tools we use are available in this environment. In our case these are pipenv and cdk8s. The proposed solutions are the following:

using volume mounts containing the necessary binaries
providing a custom image for the argocd-repo-server

If you’ve read my last post you know that Flux has the exact same problem and that I’m pretty disappointed by these options. But that’s what we have to work with. I decided to go with the custom image as it appears easier to me. Below you can find my Dockerfile adding the missing binaries.

FROM argoproj/argocd:latest

USER root

RUN apt-get update && \
    apt-get install -y \
        curl \
        python3-pip && \
    apt-get clean && \
    pip3 install pipenv

RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN apt-get update && apt-get install -y yarn
RUN yarn global add npm cdk8s-cli

USER argocd

After building this Dockerfile and pushing the resulting image to a Docker registry of your choice (pro tip: if you are working with kind use the really nice load feature) we need to update the argocd-repo-server Kubernetes Deployment to use the new image, e.g. with kubectl edit -n argocd deployments.apps argocd-repo-server.

If we’ve made sure that the argocd-repo-server pod has been recreated with the new image we can give creating our application a second try. This time everything should work and we’ll end up with the guestbook pod being started.

$ kubectl get pods
NAME                                                   READY   STATUS              RESTARTS   AGE
cdk8s-guestbook-deployment-967cec91-65b878495d-jcczj   0/1     ContainerCreating   0          16s

To make sure Argo CD is properly syncing changes let’s set the replicaCount in the main.py to 2, push the change and et voila:

live update of the replica count change in Argo CD's Web UI

Conclusion

So there you have it. A fully functional integration of cdk8s with Argo CD. The main steps being:

registering the cdk8s configuration management plugin
making the necessary tools available in the argocd-repo-server
using the cdk8s plugin when creating the Argo CD application

I’m still not satisfied with having to customize an internal service to make everything work but AFAIK there’s currently no way around it. If you know of any better way or see some possible improvements please let me know.

Integrating cdk8s with Flux

2020-04-08T00:00:00+00:00

After I got used to cdk8s I was curious how well it integrates with some current continuous delivery tools for Kubernetes. Therefore I sat down for a quick session for integrating it with Flux. I will give you a short introduction for both tools to make sure you understand everything when we put them together. If you are already familiar with cdk8s and Flux you can probably skip the next two sections.

Flux

Flux is a tool for deploying Kubernetes manifests to your cluster. It does so by following the GitOps paradigm which essentially says that one or more Git repositories are the single source of truth for the configuration of your system (in this case being Kubernetes). So in its simplest form imagine a Git repository with a single YAML file that contains a basic Kubernetes manifest (e.g. Deployment + Service). Flux will take care of applying this to your cluster. If you push more changes into this repository Flux will take care of keeping your cluster in the desired state.

Now one could say: “Alright, so what’s the benefit over executing kubectl apply -f *.yml on every code change through my CI/CD system?” The answer is very straightforward… Flux simply doesn’t need such a system. Instead of running on any infrastructure that you or someone else has to manage Flux lives in the Kubernetes cluster itself. If the cluster is down, Flux can and will be down as well. If the cluster is up, Flux should be running.

cdk8s

AWS Labs’ cdk8s is a framework that allows you to define Kubernetes deployments with object oriented programming languages like TypeScript or Python. It provides you with predefined classes, so called structs, for each Kubernetes resource. Below you can find a part of some example code defining a single Deployment.

k8s.Deployment(self, 'deployment',
               spec=k8s.DeploymentSpec(
                 selector=k8s.LabelSelector(match_labels="my-app"),
                 template=k8s.PodTemplateSpec(
                   metadata=k8s.ObjectMeta(labels="my-app"),
                   spec=k8s.PodSpec(containers=[
                     k8s.Container(
                       name='hello-kubernetes',
                       image='paulbouwer/hello-kubernetes:1.7',
                       ports=[k8s.ContainerPort(container_port=8080)])]))))

Out of this code cdk8s is able to generate valid manifest file(s).

If you wanna get a more detailed introduction feel free to check out my last blog post.

Putting them together

After getting to know the purpose of our two tools one could ask how to put them together. How can I deploy my K8s manifests that I define and generate with cdk8s using Flux? As Flux only understands plain manifest files the main step that we have to automate is their generation. I came up with two possible solutions that you will find below. If you know of any better or different ways feel free to let me know.

the obvious way

What’s the first thing to do after celebrating that we don’t need a CI system when using Flux? Correct, introducing a CI system. ;) The idea is to put our cdk8s code into a repository and setting up a CI job that generates the Kubernetes manifests and pushes them to a separate branch. Afterwards we setup Flux to use this branch as its configuration source.

The bash script for our CI job could look like this.

cdk8s synth # `cdk8s synth` generates the K8s manifests and by default puts them in a folder called dist
git checkout --orphan manifests || git checkout manifests
git add dist
git commit -m "Update K8s manifests"
git push origin manifests

Be aware that the above script assumes that a valid cdk8s installation is already available. After putting this script into a CI pipeline that is being triggered on every code change the only thing that is left to do is correctly configuring Flux. We’ll use the following fluxctl command.

fluxctl install \
  --git-user=${USER} \
  --git-email=${USER}@mycompany.com \
  --git-url=git@git.mycompany.com:${USER}/k8s-deployments \ # the URL of the Git repository containing the cdk8s code and the generated K8s manifests
  --git-branch=manifests \ # the branch that contains the K8s manifests
  --git-path=dist \ # the folder that contains the K8s manifests
  --namespace=flux | kubectl apply -f -

fluxctl works very similar to other CLI tools in the Kubernetes ecosystem. It generates K8s manifests that you can directly pipe into kubectl apply -f - and that take care of setting up the Flux controller with the desired parameters.

After having configured everything correctly the complete flow will look like this:

some developer makes changes to the cdk8s code, e.g. in the master branch
the CI pipeline is triggered by the new commit
the bash script is being executed within the context of the master branch
the K8s manifests are being generated
the output is being pushed to the manifests branch
Flux syncs its copy of the manifests branch
Flux detects changes and applies them to the cluster

the (not yet) elegant way

If you are someone that is familiar with Flux I already hear you shouting at me: “Max, just use Flux’s generators!” and that’s exactly what my second solution is based on. Initially I thought this approach is better than the first one due to the fact that we don’t need a CI system. Though while implementing it a big issue came up. But first things first, what is a Flux generator?

Generators allow you to produce new or modify existing Kubernetes manifests on the fly before applying them on your cluster. So instead of your manifests you place a simple .flux.yaml file along with your cdk8s code into your repository that could look like this.

version: 1
commandUpdated:
  generators:
  - command: cdk8s synth > /dev/null && cat dist/*

Flux will take of care of executing the command(s), capture the output, which should be valid K8s manifests and apply them on the cluster. Be aware that you have to mute every command that does generate output not containing K8s manifests. Otherwise you’re gonna run into problems.

To enable the generators feature we have to pass the --manifest-generation=true parameters when setting up Flux.

fluxctl install \
--git-user=${USER} \
--git-email=${USER}@mycompany.com \
--git-url=git@git.mycompany.com:${GHUSER}/k8s-deployments \ # the URL of the Git repository containing the cdk8s code and the .flux.yaml file
--manifest-generation=true \ # enable generators
--namespace=flux | kubectl apply -f -

So afterwards we simply put the above .flux.yaml file next to our cdk8s code, push everything to a repo and we should be good to go, right? Of course not as here comes the big bummer. The execution of generators happen within the fluxd, the controller of Flux, container. This essentially means that we need to install the whole node/yarn- and Python/pip-stack into it to be able to execute cdk8s. For me, blowing up a container that needs to have full control over at least one K8s namespace to this extent is not acceptable. Anyway I checked if this approach is actually realizable so below you can find the working .flux.yaml.

version: 1
commandUpdated:
  generators:
  - command: (/sbin/apk add yarn python3 npm && pip3 install pipenv && pipenv run pip install constructs cdk8s && yarn global add cdk8s-cli && cdk8s import && cdk8s synth) > /dev/null && cat dist/*

According to Flux’s documentation the developers are planning to allow executing generators in a separate container but for now this is not possible.

conclusion

So there you have it. Two options of setting up a GitOps process using cdk8s and Flux. For a productive setup I’d accept the additional effort of setting up / managing a CI system (which one already has in most cases anyway) and prefer the first option for now. As soon as Flux supports the execution of generators in a separate container the second option becomes more viable in my opinion.

All in all I was surprised how well both tools work together. As cdk8s becomes more mature I’ll definitely take the two into consideration for setting up my next K8s delivery pipeline.

cdk8s, the future of Kubernetes application deployments?

2020-03-30T00:00:00+00:00

Hint: As cdk8s is fairly new at the time of this writing expect that things will change/have changed.

Intro

cdk8s (probably cloud development kit for Kubernetes) is a new framework released by AWS Labs (AWS’s open source organization)that is written in TypeScript. It allows you to define Kubernetes manifests using modern object oriented programming languages. This enables you to develop a very flexible solution that is capable of complex operations for establishing your application deployment process to Kubernetes.

To achieve this cdk8s provides so called constructs, which are abstractions of Kubernetes resources (Deployment, Service, Ingress, …). A logical collection of these is called a chart (similar to a Helm chart). Finally an app is defined by one or more charts. Later on you will find an example that’ll showcase these units and their relation in more detail.

Now you may wonder how you can use your programming language of choice if cdk8s itself is written in TypeScript. The answer is called jsii. AWS’ jsii is a tool to generate bindings for several programming languages (currently Python, Java and C#) that allow you to interact with Type-/JavaScript classes. It is already in use by the AWS CDK (cdk8s but for CloudFormation files) and will probably get support for more languages over time.

Usage

Installing cdk8s requires you to have the standard Node.js, yarn/npm stack available on your machine. To assist with bootstrapping and generating constructs for your particular Kubernetes version the kit comes with a CLI tool. Follow the Getting Started section to get a detailed introduction.

After going through the introduction we are ready to write our code. Below you find a snippet that defines a single cdk8s app consisting of a single chart containing one service and one deployment.

#!/usr/bin/env python
from constructs import Construct # importing base class for type hinting
from cdk8s import App, Chart

from imports import k8s # importing resource bindings for your particular Kubernetes version, previously generated by "cdk8s import"

class MyChart(Chart):
  def __init__(self,
               scope: Construct, # our app instance
               ns: str, **kwargs): # careful! this is not the K8s namespace but just a prefix for our resources
    super().__init__(scope, ns, **kwargs)

    # defining some common variables
    label = {"app": "hello-kubernetes"}
    container_port = 8080

    # defining a deployment with one container and two replicas
    k8s.Deployment(self, 'deployment',
                   spec=k8s.DeploymentSpec(
                     replicas=2,
                     selector=k8s.LabelSelector(match_labels=label),
                     template=k8s.PodTemplateSpec(
                       metadata=k8s.ObjectMeta(labels=label),
                       spec=k8s.PodSpec(containers=[
                         k8s.Container(
                           name='hello-kubernetes',
                           image='paulbouwer/hello-kubernetes:1.7',
                           ports=[k8s.ContainerPort(container_port=container_port)])]))))

    # defining a service for pods created by the deployment above
    k8s.Service(self, 'service',
                spec=k8s.ServiceSpec(
                  type='LoadBalancer',
                  ports=[k8s.ServicePort(port=80, target_port=k8s.IntOrString.from_number(container_port))],
                  selector=label))


app = App() # creating an App instance
MyChart(app, "hello") # installing our chart in the app under a certain namespace

app.synth() # this method call takes care of generating the K8s manifests

Executing the above code or running cdk8s synth results in the following manifest.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-deployment-c51e9e6b
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hello-kubernetes
  template:
    metadata:
      labels:
        app: hello-kubernetes
    spec:
      containers:
        - image: paulbouwer/hello-kubernetes:1.7
          name: hello-kubernetes
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: hello-service-9878228b
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: hello-kubernetes
  type: LoadBalancer

As you can see in this simple example cdk8s takes care of creating a one to one translation of your objects into a Kubernetes manifest. Take into consideration that cdk8s’ structs are very explicit. They do not provide any kind of defaults (e.g. for the replica count) or detect any relations between objects (e.g. detecting the container port and using it for the service). This is where you come into play and develop charts that implement your desired logic. On the one hand this could be a generic chart that creates a Deployment and a Service manifest for an arbitrary container image. On the other hand it could be a chart creating a ConfigMap that dynamically pulls its values out of your key value store. You can achieve whatever your chosen programming language is capable of.

Comparison against kustomize and Helm

After reading to this point you may think why you shouldn’t just keep using tools like kustomize or Helm and in most cases you are probably right in doing so.

Depending on your background kustomize and Helm may be easier to learn. They don’t require you to write actual code but use a templating language. This allows you to do simple operations, like variable replacement, conditional blocks or for-loops. But as soon as you require more complex functionality or need to communicate with external services you’re gonna run into the limits of these tools.

In my opinion cdk8s is one of the next obvious steps in the DevOps movement, bringing Ops people closer to the development and enabling developers to take over Ops tasks. As infrastructure gets more and more dynamic and complex the need for powerful application deployment orchestration rises.

Personal Opinion

Would I use cdk8s in production today? No, there’s no stable version yet and the development is very active, making it unavoidable to break your setup from time to time. Instead I’m going to keep an eye on its progress and use it for some of my side projects. After cdk8s reaches a certain level of maturity (probably the first major version) I will definitely consider using it for new projects.

Are you already using cdk8s and have some questions or experiences to share? Feel free to let me know. (y)

Minikube vs. kind vs. k3s - What should I use?

2019-12-05T00:00:00+00:00

These days there are a few tools that claim to (partially) replace a fully fledged Kubernetes cluster. Using them allows e.g. every developer to have their own local cluster instance running to play around with it, deploy their application or execute tests against applications running in K8s during CI/CD. In this post we’ll have a look at three of them, compare their pros and cons and identify use cases for each of them.

minikube

minikube is a Kubernetes SIGs project and has been started more than three years ago. It takes the approach of spawning a VM that is essentially a single node K8s cluster. Due to the support for a bunch of hypervisors it can be used on all of the major operating systems. This also allows you to create multiple instances in parallel.

From a user perspective minikube is a very beginner friendly tool. You start the cluster using minikube start, wait a few minutes and your kubectl is ready to go. To specify a Kubernetes version you can use the --kubernetes-version flag. A list of supported versions can be found here.

If you are new to Kubernetes the first class support for its dashboard that minikube offers may help you. With a simple minikube dashboard the application will open up giving you a nice overview of everything that is going on in your cluster. This is being achieved by minikube’s addon system that helps you integrating things like, Helm, Nvidia GPUs and an image registry with your cluster.

kind

Kind is another Kubernetes SIGs project but is quite different compared to minikube. As the name suggests it moves the cluster into Docker containers. This leads to a significantly faster startup speed compared to spawning VM.

Creating a cluster is very similar to minikube’s approach. Executing kind create cluster, playing the waiting game and afterwards you are good to go. By using different names (--name) kind allows you to create multiple instances in parallel.

One feature that I personally enjoy is the ability to load my local images directly into the cluster. This saves me a few extra steps of setting up a registry and pushing my image each and every time I want to try out my changes. With a simple kind load docker-image my-app:latest the image is available for use in my cluster. Very nice!

If you are looking for a way to programmatically create a Kubernetes cluster, kind kindly (you have been for waiting for this, don’t you :P) publishes its Go packages that are used under the hood. If you want to get to know more have a look at the GoDocs and check out how KUDO uses kind for their integration tests.

k3s

K3s is a minified version of Kubernetes developed by Rancher Labs. By removing dispensable features (legacy, alpha, non-default, in-tree plugins) and using lightweight components (e.g. sqlite3 instead of etcd3) they achieved a significant downsizing. This results in a single binary with a size of around 60 MB.

The application is split into the K3s server and the agent. The former acts as a manager while the latter is responsible for handling the actual workload. I discourage you from running them on your workstation as this leads to some clutter in your local filesystem. Instead put k3s in a container (e.g. by using rancher/k3s) which also allows you to easily run several independent instances.

One feature that stands out is called auto deployment. It allows you to deploy your Kubernetes manifests and Helm charts by putting them in a specific directory. K3s watches for changes and takes care of applying them without any further interaction. This is especially useful for CI pipelines and IoT devices (both target use cases of K3s). Just create/update your configuration and K3s makes sure to keep your deployments up to date.

Summary

I was a long time minikube user as there where simply no alternatives (at least I never heard of one) and to be honest…it does a pretty good job at being a local Kubernetes development environment. You create the cluster, wait a few minutes and you are good to go. However for my use cases (mostly playing around with tools that run on K8s) I could fully replace it with kind due to the quicker setup time. If you are working in an environment with a tight resource pool or need an even quicker startup time, K3s is definitely a tool you should consider.

All in all these three tools are doing the job while using different approaches and focusing on different use cases. I hope you got a better understanding on how they work and which is the best candidate for solving your upcoming issue. Feel free to share your experience and let me know about use cases you are realizing with minikube, kind or k3s at @__brennerm.

Below you can find a table that lists a few key facts of each tool.

	minikube	kind	k3s
runtime	VM	container	native
supported architectures	AMD64	AMD64	AMD64, ARMv7, ARM64
supported container runtimes	Docker,CRI-O,containerd,gvisor	Docker	Docker, containerd
startup time initial/following	5:19 / 3:15	2:48 / 1:06	0:15 / 0:15
memory requirements	2GB	8GB (Windows, MacOS)	512 MB
requires root?	no	no	yes (rootless is experimental)
multi-cluster support	yes	yes	no (can be achieved using containers)
multi-node support	no	yes	yes
project page	minikube	kind	k3s

Installing Python in OpenWRT on a USB storage

2019-09-07T00:00:00+00:00

1. The problem

Lately I was in need to periodically execute a Python script. To achieve this task my router, a TP-Link N600, running OpenWRT Chaos Calmer came to my mind as it’s running the whole day anyway. So I checked the available packages in opkg, OpenWRT’s package manager, and found what I was looking for. Confidently executing opkg install python3 (for Python 2 it’s opkg install python) resulted in the following

Collected errors:
 * wfopen: /usr/lib/python2.7/encodings/mac_latin2.py: No space left on device.
 * wfopen: /usr/lib/python2.7/encodings/iso8859_3.py: No space left on device.
 * wfopen: /usr/lib/python2.7/encodings/iso8859_1.py: No space left on device.
 * wfopen: /usr/lib/python2.7/encodings/euc_kr.py: No space left on device.
 * wfopen: /usr/lib/python2.7/encodings/cp775.py: No space left on device.
 * pkg_write_filelist: Failed to open //usr/lib/opkg/info/python-codecs.list: No space left on device.
 * opkg_install_pkg: Failed to extract data files for python-codecs. Package debris may remain!
 * opkg_install_cmd: Cannot install package python.
 * opkg_conf_write_status_files: Can't open status file //usr/lib/opkg/status: No space left on device.

Well Shit! Being used to work with devices with loads of storage I didn’t expect that. df told me that my router has an incredible amount of 4.5, in words four dot five, megabytes of storage!

root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
rootfs                    4.5M      4.4M    108.0K  98% /
/dev/root                 2.3M      2.3M         0 100% /rom
tmpfs                    61.5M      1.2M     60.4M   2% /tmp
/dev/mtdblock3            4.5M      4.4M    108.0K  98% /overlay
overlayfs:/overlay        4.5M      4.4M    108.0K  98% /
tmpfs                   512.0K         0    512.0K   0% /dev

And so the journey began.

2. Looking for solutions

After some investigation I found out that opkg supports custom installation paths for packages. The router’s RAM, mounted at /tmp, is one of the predefined targets. It’s advantage is that it provides a lot more storage (in my case 61.5Mb), but you may already have one concern. Right, as we’re working on volatile memory all data will be lost upon reboot.

As this was no option in my case I had to look further. Another solution would have been to use some kind of network storage and mount it via NFS. In my current environment I don’t have anything like that available to me so I went with the next obvious solution. Expanding my router’s storage using an USB device.

3. Accessing a USB device

The support for USB devices is documented fairly good on the two pages about basic USB support and USB storage. All I needed to do is to check the output of dmesg after plugging in my USB stick.

[5952821.750000] usb 1-1.1: new high-speed USB device number 3 using ehci-platform

In my case I had to install the driver for the ehci platform (USB 2.0).

opkg update
opkg install kmod-usb2 kmod-usb-storage
insmod ehci-hcd

Afterwards the system detected the device and made it available at /dev/sda.

Depending on the partitioning of your device you may need to install support for further filesystems like ext4.

opkg install kmod-fs-ext4

Now I was able to mount my USB stick.

mount -t ext4 /dev/sda1 /mnt/usb

As the mounting should happen automatically after every reboot installing the block-mount package, persisting the current mounts block detect > /etc/config/fstab and enabling the autostart /etc/init.d/fstab enable was necessary. All that was left was to enable the automount for the USB device by editing the file /etc/config/fstab.

4. Installing Python to a USB device

With a working USB storage I needed to tell opkg to actually use it. This was done by adding the following the file /etc/opkg.conf.

dest usb /mnt/usb

Installing packages to a non-default location may break. Some of the issues can be resolved by adjusting the path variables in /etc/profile.

export PATH=$PATH:/mnt/usb/bin:/mnt/usb/sbin:/mnt/usb/usr/bin:/mnt/usb/usr/sbin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/mnt/usb/lib:/mnt/usb/usr/lib

Afterwards I was able to install Python using opkg intall python3 -d usb. The OpenWrt docs suggest to precompile all Python modules to achieve a faster execution. python -m compileall

Et voila, we have a running Python interpreter!

[GCC 4.8.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

My top 5 positive points about Ansible in 2017

2017-09-19T00:00:00+00:00

Intro: In this post you are going to find a list of aspects where I think the provisioning engine Ansible excels. I’m a guy that lives in a Linux dominated infrastructure. Therefore my experiences may not align with yours. Make sure to let me know your opinion at @__brennerm.

Hint: The latest Ansible release as of this writing is 2.3.2.0.

1. The client requirements

Unlike other provisioning tools, like Puppet, Ansible is not in need of any additional agent software being installed on the clients. The only requirements are SSH access and Python, which are already fulfilled on most modern operating systems. The latter isn’t even required as you can use the raw module to install Python e.g. by executing apt-get install -y python2.7 before the execution of your “real” configuration. This module also gives you the ability to configure devices that do not run a fully fledged operating system, like switches or PDUs.

2. The nonexistence of the “master server”

As for the clients, the requirements to be able to start provisioning are minimal. All you need is the Ansible executable, your configuration you want to apply and a way to authenticate against the clients. This leads to the fact that theoretically any machine can become one of your provisioning servers in a matter of seconds. The clients are not fixed to one master server which results in a lot of flexibility. Additionally you don’t have to keep your provisioning server up all the time to be able to answer the client’s request for configuration updates. Also you know exactly when your clients are being updated and it’s easier to see what’s exactly happening during the deployment.

3. Python

This is the most subjective point of this list but what can I say. It’s just me liking Python and thus being able to extend Ansible in any way. I’m in contact with a lot of proprietary software where Ansible support isn’t available. For this kind of stuff it’s a real pleasure to use your beloved programming language to be able to create reusable building blocks in the form of Ansible modules. Another place where Python excels is the dynamic inventory. As there is an API for almost every IaaS implementation available for Python it’s really easy to connect your Ansible provisioning to your already existing infrastructure.

4. The documentation

Giving some credit to the maintainers, the documentation is really good. For every part of Ansible you are going to find extensive and high quality documentation that is up to date. Additionally as it’s an already established piece of software a lot of questions have already been answered. The only small issue that comes to my mind is that some of the built-in modules are missing the documentation of their return values.

5. The ecosystem

Another advantage of Ansible being widely used is the existence of a lot of community projects around it. In this case I’m particularly talking about roles. There are roles for almost every piece of software that is publicly available. For popular technologies, like PostgreSQL or Gitlab, you’re going to find roles that let you adjust basically everything. Ansible Galaxy does a good job of making all of these roles accessible over a unified interfaces. Additionally it prevents a lot of duplicate “code” as it allows you to easily handle your role dependencies.

That’s my top 5 points about Ansible in 2017. If you think I missed something feel free to let me known at @__brennerm.

Make sure you check back for my top 5 negative points about Ansible.

Cap'n Proto with Python

2015-05-19T00:00:00+00:00

If you are already familiar with Cap’n Proto and just want to see how to use it with Python click here.

Introduction to Cap’n Proto

Ever heard of Cap’n Proto? - No it’s not one of the worst named superheroes ever. Lets have a look at what the official webpage is saying:

Cap’n Proto is an insanely fast data interchange format and capability-based RPC system.

This addresses two points - data interchange format and RPC. But before we can make use of whatever this means we will have a look at Cap’n Proto’s schema language.

Schema language

This language is used to define structures for further use. It’s syntax shows similarities to the C programming language and supports common data types like Boolean, Integer, Struct, Enum, etc. Let’s have a look at it:

const blogUrl :Text = "shipit.dev/";

enum Language {
	en @0;
	de @1;
	ru @2;
}

struct Date{
	year @0 :Int16;
	month @1 :UInt8;
	day @2 :UInt8;
}

struct Post {
	availableLanguages @0 :List(Language);
	publishDate @1 :Date;
	content @2 :Text;
}

As you see containers like struct or enum are supported and can be used to easily abstract real-world objects.

You may have already noticed the numbers behind every single field. This enables Cap’n Proto to keep your evolving schema backwards compatible. As long as you follow some rules there is no need to spend time on keeping your application compatible with older versions.

There’s no point going over the whole vocabulary of the schema language. It’s just important to get a basic understanding, cause the following features use this schemas as a basis. If you want to learn more about the schema language have a look here.

data interchange format

The data interchange format is based on messages that can contain multiple instances of our self-defined objects. These messages are saved in a binary format which is controlled by the Cap’n Proto library.

Each data type is handled in a defined way which results in a well organized format. For bandwidth limited use cases Cap’n Proto provides a built-in packing that can save up a lot of unnecessary zero bytes. In some cases the size can be shrunk further by applying a suitable compression algorithm.

RPC

RPC stands for remote procedure call which is a type of inter-process communication. It allows you to call functions that are provided by another process on your machine or even over the network.

It follows the client-server-architecture with the server providing and executing the functions and the clients calling them. The interface definition is handled through the schema language we discussed earlier:

interface Calc {
	sum @0 (a :Int64, b :Int64) -> (result :Int64);
	sub @1 (a :Int64, b :Int64) -> (result :Int64);
	mul @2 (a :Int64, b :Int64) -> (result :Int64);
	div @3 (a :Int64, b :Int64) -> (result :Float64);
}

This is an example of a really basic calculator, but contains all you need to know. It’s of course possible to use your own defined structures as parameters or return value.

There are already several implementations of RPC like Java’s RMI, CORBA or DBUS, but Cap’n Proto provides a feature called promise pipelining. This lets you use a return value of one function call as an argument for another without waiting for the first function to finish. This is especially useful when your application runs in an environment with a long round trip time, cause you save up unnecessary requests. More information can be found here.

Cap’n Proto with Python

With the knowledge of what Cap’n Proto is all about we’ll have a look into pycapnp. It’s a wrapper for Cap’n Proto that makes most of the features available to use with Python. The following examples will be based on this schema:

enum Unit {
	k @0;
	f @1;
	c @2;
}

struct Temperature {
	value @0 :Float64;
	unit @1 :Unit;
}

interface TempConv {
	convert @0 (temp :Temperature, target_unit :Unit) -> (result :Temperature);
}

Message creation

Before we are able to create our messages we need to read in our schema. Pycapnp handles this pretty pythonic:

import capnp
import tempconv_capnp

This will search the current directory and your PYTHONPATH for a file called tempconv.capnp. The schema is going to be processed and made accessible like a module within your code. You can then create and define your messages like that:

temp = tempconv_capnp.Temperature.new_message()

temp.value = 100
temp.unit = 'c'
print('temperature:' + str(temp))

Pretty easy, right? Pycapnp handles all validation and errors are raised when a value is inappropriate (type mismatch, not in enum, …).

If you like dicts, this is a way you can go:

temp_dict = temp.to_dict()
temp_dict['value'] = 50

restored_temp = tempconv_capnp.Temperature.new_message(**temp_dict)
print('temperature:' + str(restored_temp))

De-/Serialization

The de- and serialization from and to the data interchange format is supported as well:

import capnp
import tempconv_capnp

temp = tempconv_capnp.Temperature.new_message()

temp.value = 100
temp.unit = 'c'
print(temp)

temp_bytes = temp.to_bytes()
print(temp_bytes)

restored_temp = tempconv_capnp.Temperature.from_bytes(temp_bytes)
print(restored_temp)

The resulting bytes can be send over the network or can be saved in a file. To use Cap’n Proto’s built-in packing just append _packed to to_bytes and from_bytes.

RPC

Let’s come to the most interesting part of the library. Our target is to build a server that offers and executes the convert function. The functionality is implemented like this:

# name of the class doesn't matter, as long as you inherit from your server class
class TempConv(tempconv_capnp.TempConv.Server):
    def convert(self, temp, target_unit, **kwargs):
        temp_dict = temp.to_dict()
		
		result = tempconv_capnp.Temperature.new_message()
        result.unit = target_unit
        result.value = CONVERTER[temp_dict['unit']][str(target_unit)](temp_dict['value'])

        return result

Be sure to name your function according to your interface definition and add **kwargs to your parameters. This will ensure your server remains compatible with newer versions that may provide more arguments.

Now it’s time to start our server:

# enables you to save capabilities and restore them later
def restore(ref):
    assert ref.as_text() == 'tempConv'
    return TempConv()

server = capnp.TwoPartyServer('127.0.0.1:12345', restore)
server.run_forever()

With the server up and running we can now connect our client:

client = capnp.TwoPartyClient('localhost:12345')

tempconv = client.ez_restore('tempConv').cast_as(tempconv_capnp.TempConv)

With the connection in hand we can finally call our convert function:

request = tempconv.convert_request()

request.temp.value = 100
request.temp.unit = 'c'
request.target_unit = 'k'

promise = request.send()

The resulting promise can be handled in two ways:

asynchronous:

promise.then(lambda ret: print(ret)).wait()

synchronous:

result = promise.wait()

And there is our result:

( result = (value = 373.15, unit = k) )

That’s all you need to know for now to make use of Cap’n Proto for your Python application. When having a look at the roadmap there will be some interesting features in the future, like shared-memory RPC or dynamic schema transmission so stay tuned for updates.
All shown code examples can be found on github.

#Update 20.05.2015 Played around using my own sockets when communicating with pycapnp’s RPC lately and had some trouble in the beginning. So I just want to let you guys know how to get it working. The first step is to connect your client and server socket:

# server
s = socket()
s.bind(('127.0.0.1', 12345))
s.listen(1)
conn, addr = s.accept()

# client
s = socket()
s.connect(('127.0.0.1', 12345))

Now instead of handing over the address + port hand over your socket to the server and client constructor.

# server
server = capnp.TwoPartyServer(conn, bootstrap=TempConv())
server.on_disconnect().wait()

# client
client = capnp.TwoPartyClient(s)
tempconv = client.bootstrap().cast_as(tempconv_capnp.TempConv)

request = tempconv.convert_request()

request.temp.value = 100
request.temp.unit = 'c'
request.target_unit = 'k'

promise = request.send()
print(promise.wait())

Be sure to use server.on_disconnect().wait() instead of server.run_forever() and it will work as expected.

Python's str() vs. repr()

2015-04-10T00:00:00+00:00

Ever wondered what happens when you call Python’s built-in str(X), with X being any object you want? The return value of this function depends on the two magic methods __str__ being the first choice and __repr__ as a fallback. But what’s the difference between them? When having a look at the docs

>>> help(str)
'Create a new string object from the given object.'
>>> help(repr)
'Return the canonical string representation of the object.'

they seem to be fairly similar. Let’s see them in action:

>>> str(123)
'123'
>>> repr(123)
'123'

Alright no difference for now.

>>> str('Python')
'Python'
>>> repr('Python')
"'Python'"

A second pair of quotes around our string. Why?
With the return value of repr() it should be possible to recreate our object using eval(). This function takes a string and evaluates it’s content as Python code. In our case passing “‘Python’“ to it works, whereas ‘Python’ leads to an error cause it’s interpreted as the variable Python which is of course undefined. Let’s move on…

>>> import datetime
>>> now = datetime.datetime.now() 
>>> str(now)
'2015-04-04 20:51:31.766862'
>>> repr(now)
'datetime.datetime(2015, 4, 4, 20, 51, 31, 766862)'

This is some significant difference. While str(now) computes a string containing the value of now, repr(now) again returns the Python code needed to rebuild our now object.</br>

The following clues might help you to decide when to use which:

str()	repr()
- make object readable	- need code that reproduces object
- generate output for end user	- generate output for developer

These points should also be considered when writing __str__ or __repr__ for your classes.