kubectl is the new, cloud-native
ssh. That is, it’s the low-level tool to break out when you need to manually inspect or manipulate the state of your running applications.
ssh makes it easy to pop open a few virtual terminal windows to perform different tasks on several systems at once. Each virtual terminal window and its PTY encapsulates its own running state. There’s no way that
cd-ing into a directory in one window affects the current working directory in another window.
kubectl breaks that paradigm. By default there is a single global
~/.kube/config file that stores the state of your authenticated session with a cluster. When you authenticate with a new
kubectl mutates this state by adding another context and switching to it.
Mutable global state, you say? Isn’t that… bad?
Yes, I think it is. And while the golden rule of global state being evil has traditionally referred to software engineering practices, I think it applies more generally.
Imagine you’re running a script that uses
kubectl to do something. Label some namespaces, kill some pods, whatever. That takes a non-zero amount of time to run. Then you pop open another virtual terminal to do some work in the meantime and switch context into another cluster. Boom, your script in the other virtual terminal is now running against the wrong cluster!
Not only that, but often OIDC providers for k8s are configured to use very long lasting, or even everlasting, tokens that are stored in
~/.kube/config. This is like configuring passwordless root access via
ssh to your production server. At any moment, any old script can just
ssh root@prod rm -rf / without any interaction.
I wanted to change the default behaviour of
kubectl to behave more like good old
- No persistent state anywhere.
- Every virtual terminal has its own
For the first part, I created an empty file in
~/.kube/config, and set permissions to
-r-------- 1 scott scott 0 Oct 11 2019 /home/scott/.kube/config
This makes it impossible for
kubectl to store global state in the default location.
For the second part, I added this snippet to my
# dynamic kubeconfig per virtual terminal to avoid mutating global state if [ -z "$KUBECONFIG" ]; then export KUBECONFIG=$(mktemp --tmpdir kubeconfig.XXXXXXXX) trap "rm -f $KUBECONFIG" EXIT fi
$KUBECONFIG to make it so that every new virtual terminal gets is own
kubectl state. All state, including authentication, is cleared when the shell exits.
This changes the usual
kubectl workflow because you have to authenticate anew each time you open a new shell. I see this as an improvment because it forces a workflow that makes you think about which cluster you’re authenticating to every time.