Updating Secrets from a Kubernetes Pod

Photo by Safar Safarov on Unsplash

Sometimes you might want to update secrets from within a Kubernetes Pod. For example, if you have the access token and refresh token for an OAuth application, you might need to update these every time you refresh them. I’m sure there’s a few different ways to go about this, and there’s always the option of just storing these in a DB. In my case, I needed to refresh my tokens for Fitbit and wanted to keep them as secrets in Kubernetes. After reading some stuff online, I found the kubectl patch command, which seemed to be what I wanted. The full command would be something like:

kubectl patch secret <secret> -p='{"data":{"<key>": "<base64_encoded_val>"}}

I wanted to do this in Python, so I wrote the following functions, which worked great from my machine:

from base64 import b64encode
from subprocess import run
def b64encodestr(string):
    return b64encode(string.encode("utf-8")).decode()
def update_secrets(secret, key, val):
    b64val = b64encodestr(val)
    cmd = f"""kubectl patch secret {secret} -p='{{"data":{{"{key}": "{b64val}"}}}}'"""
    return run(cmd, shell=True)

Then I created a simple DAG using the KubernetesPodOperator that would update my airflow secret by updating TEST_SECRET with the current timestamp every minute. To use this method, you first need to make sure you have kubectl installed in the image that you’re using. When I first ran it, I got the following error:

Error from server (Forbidden): secrets "airflow" is forbidden: User "system:serviceaccount:default:default" cannot get resource "secrets" in API group "" in the namespace "default"

After reading a bit, I found this was due to roles and permissions. In order to fix it, I first needed to create a role that would allow updating secrets. I did so with the following command (note that you’ll need the Kubernetes Engine Admin role assigned from GCP to do run these):

kubectl create role update-secrets --verb=get,patch --resource=secrets

Once I’d created the role, I had to assign it to my default service account on the default namespace (which is the default setup in pods spun up by KubernetesPodOperator). You can do so with the following:

kubectl create rolebinding --role=update-secrets default-update-secrets --serviceaccount=default:default

After this, everything is working as expected! Now I can update my access and refresh tokens for my fitbit account when pulling data each day!

NOTE: If you’re just using Python, you could also use the kubernetes python library and the patch_namespaced_secret function. For the purpose of this article I decided to use the most “portable” option.