Public-key Cryptography on Kubernetes Secret

Kubernetes secret is a common way to externalize configurations.

However, it is not doing any encryption, the data in the secret is “opaque” and it is simply a Base64 encoding.

It's noticed that the latest Kubernetes’ Secret data encryption at rest, (K8s version 1.13.0 above), makes sure that the data, when saved into the etcd store, are encrypted. When you access the secret with K8s API, the data is decrypted back for you automatically.

This may not be what you want for a high data sensitive environment. The paper explores the usage of the classic public-key cryptography algorithm for the data in the Secret of Kubernetes.

Public-key cryptography, or asymmetric cryptography, is a cryptographic system that uses pairs of keys: public keys which may be disseminated widely, and private keys which are known only to the owner…

To apply the public-key cryptography in the encryption/decryption for the data in Secret of Kubernetes, the overall idea is straightforward.

I implement this idea with Golang and tested it with MicroK8s on my Macbook.

1. Create a pair of keys

ssh-keygen -m pem -b 4096 -t rsa -f mykey -N '' -C "key for encryption"

The -m pem option makes sure the private key is created with the following format as the default ssh-keygen on Mac is creating a non rsa key.

-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----

Then we dump out the public key in PEM format,

openssl rsa -in mykey -pubout -out mykey.pub.pem

Now you should have the private key, mykey, and the public key, mykey.pub.pem.

2. Build a sample go app for encryption and decryption

Let’s build a sample golang app. I use the Cobra command line app builder. The setup and initialization are skipped. Let’s take a look of the action function for encryption command.

// lines skipped...
Run: func(cmd *cobra.Command, args []string) {
text := args[0]
data, err := ioutil.ReadFile(publicKeyFile)
if err != nil {
log.Fatalf("Failed to read pub key:%v", err)
}
block, _ := pem.Decode(data)
if block == nil {
log.Fatalf("Failed to decode the pub key")
}
pubInterface, err := x509.ParsePKIXPublicKey(block.Bytes)
if err != nil {
log.Fatalf("Failed to parse pub key:%v", err)
}
pub := pubInterface.(*rsa.PublicKey)
s, err := rsa.EncryptOAEP(sha256.New(), rand.Reader, pub, []byte(text), []byte("mylabel"))
if err != nil {
log.Fatalf("Failed to encrypt:%v", err)
}
fmt.Printf("%x\n", s)
},
// lines skipped...
var publicKeyFile stringfunc init() {
encryptCmd.Flags().StringVarP(&publicKeyFile, "pub", "p", "", "public key file to encrypt")
rootCmd.AddCommand(encryptCmd)
}

The publicKeyFile points to the public key file, which will be parsed in as a command line flag. After reading the file, the program decodes the PEM data, parse it into the public key. Then we can use this key to encrypt the text with the function of rsa.EncryptOAEP(). Here we are using a fixed label.

The same goes for the decrypt part but using the private key.

//... skipped
Run: func(cmd *cobra.Command, args []string) {
text, err := hex.DecodeString(args[0])
if err != nil {
log.Fatalf("Failed to covert hex to string:%v", err)
}
data, err := ioutil.ReadFile(privateKeyFile)
if err != nil {
log.Fatalf("Failed to read private key file:%v", err)
}
block, _ := pem.Decode(data)
if block == nil {
log.Fatalf("Failed to decode the private key")
}
priv, err := x509.ParsePKCS1PrivateKey(block.Bytes)
if err != nil {
log.Fatalf("Failed to parse pub key:%v", err)
}
s, err := rsa.DecryptOAEP(sha256.New(), rand.Reader, priv, []byte(text), []byte("mylabel"))
if err != nil {
log.Fatalf("Failed to decrypt:%v", err)
}
fmt.Printf("%s\n", s)
},
//... skipped
var privateKeyFile string
func init() {
decryptCmd.Flags().StringVarP(&privateKeyFile, "private", "p", "", "private key file to decrypt")
rootCmd.AddCommand(decryptCmd)
}

The rsa.DecryptOAEP() function call decrypt the data with the same label.

Build it. Test the encryption part,

➜ encryptor/encryptor enc -p mykey.pub.pem supersecret97b07ee4b963ce4137bd3d8f5ae4595af01bd25dd35e9f2c2dd38730bc851fc51f4b05ce7b88991b218cb00b22221dda9e3e394f8bcb628f5c5b7077a7b7e36156deff9a1192015687c4a08277339e6ce8f70d10ffc2514a7d24f68fc91b30ad1ed68802a93f0abb601bc63d95c9a27bd3d61a016d4e116841dddfb3bfd7cfe10512314fbf68e504971a7b33d1167a0d4a1d9a7b0dee13b44746a7975aa61afa3084b82248174fa6e7ec91aa8d198bcabe22e093fc724a423b37308ef4beced9449bd2d15f35d9a8e905499d46d02c78b2a7b8ada430cde1dce0c2766f350ed0866b8277b1a074fd1e05493e804892f0b308ff5cd3c2aa18f9f82fbd74da7801002b41ba820ddc3ab6a492e207868a2fb4480dc889eb822c01a61d4434dcc74355bbb3213f2d321ecc1c595ce6b2d2ab539c6b797c250a45c34ee296e40818b74be654904cc45475e8fed4c6a0cfc6c382708ae312bcc908f63ca7a1bb260d8a4b184da00938914f9abe3097b2089eafe707f339c9bc2c4b862066fc8033651a8f42eaa7aab42889445af826f66fb4c4f70e45088d8be663c9da0b45030155aadd7ac014a1e2299e7f7dd3fa1c7c762d0937f1accb3219ce7b6cb8419c02573d42ea043a6a3297c6ecd80a7749376582f310031508671568e25c2783a90a06af093b5e00c951a81867194f68dded7932198aa50198c9ac867028c04a44e7d7dc

The decryption part,

➜ encryptor/encryptor dec -p mykey $(encryptor/encryptor enc -p mykey.pub.pem supersecret)supersecret

3. Build a docker image

Cross compile the binary for Linux,

export GOARCH=amd64
export GOOS=linux
go build -o encryptor.lin main.go

Create the following Dockerfile

FROM alpine
ADD encryptor/encryptor.lin /
RUN chmod a+rx /encryptor.lin

Build the image and push it to docker hub tagged as “zhiminwen/encryptor-demo:v1”

4. Create Secrets

Let's create two secrets one for the encrypted sensitive data.

kubectl create secret generic my-credential --from-literal=cred=97b07ee4b963ce4137bd3d8f5ae4595af01bd25dd35e9f2c2dd38730bc851fc51f4b05ce7b88991b218cb00b22221dda9e3e394f8bcb628f5c5b7077a7b7e36156deff9a1192015687c4a08277339e6ce8f70d10ffc2514a7d24f68fc91b30ad1ed68802a93f0abb601bc63d95c9a27bd3d61a016d4e116841dddfb3bfd7cfe10512314fbf68e504971a7b33d1167a0d4a1d9a7b0dee13b44746a7975aa61afa3084b82248174fa6e7ec91aa8d198bcabe22e093fc724a423b37308ef4beced9449bd2d15f35d9a8e905499d46d02c78b2a7b8ada430cde1dce0c2766f350ed0866b8277b1a074fd1e05493e804892f0b308ff5cd3c2aa18f9f82fbd74da7801002b41ba820ddc3ab6a492e207868a2fb4480dc889eb822c01a61d4434dcc74355bbb3213f2d321ecc1c595ce6b2d2ab539c6b797c250a45c34ee296e40818b74be654904cc45475e8fed4c6a0cfc6c382708ae312bcc908f63ca7a1bb260d8a4b184da00938914f9abe3097b2089eafe707f339c9bc2c4b862066fc8033651a8f42eaa7aab42889445af826f66fb4c4f70e45088d8be663c9da0b45030155aadd7ac014a1e2299e7f7dd3fa1c7c762d0937f1accb3219ce7b6cb8419c02573d42ea043a6a3297c6ecd80a7749376582f310031508671568e25c2783a90a06af093b5e00c951a81867194f68dded7932198aa50198c9ac867028c04a44e7d7dc

The other one for the private key with the key named as “privateKey” and the content from the file “mykey”

kubectl create secret generic my-encryptor-key --from-file=privateKey=mykey

5. Deploy to Kubernetes

Deploy with the following deployment objects,

apiVersion: apps/v1
kind: Deployment
metadata:
name: encryptor-demo
labels:
app: encryptor-demo
spec:
replicas: 1
selector:
matchLabels:
app: encryptor-demo
template:
metadata:
labels:
app: encryptor-demo
spec:
containers:
- name: encryptor-demo
image: zhiminwen/encryptor-demo@sha256:2bc801704c79cd796a1d56b1a4fdb612a46cec33b13de078036f946d02152536
imagePullPolicy: IfNotPresent
command: ["sh"]
args: ["-c", "echo decrypted password is $(/encryptor.lin dec -p /keys/privateKey $MY_CREDENTIAL); while true; do sleep 60; done"]
env:
- name: MY_CREDENTIAL
valueFrom:
secretKeyRef:
name: my-credential
key: cred
volumeMounts:
- name: private-key-volume
mountPath: /keys
volumes:
- name: private-key-volume
secret:
secretName: my-encryptor-key

In the deployment, we mount the secret as a file in the pod, in which the private key of the encryption will be located as /keys/privateKey in the pod.

We also bring in the encrypted credential as an environment variable referenced from the Secret of my-credential

In the test program, we can call the decryptor to decrypt the data. This can be validated as below.

➜ kubectl get pods
NAME READY STATUS RESTARTS AGE
encryptor-demo-8b9955489-djw8f 1/1 Running 0 4h9m
➜ kubectl log encryptor-demo-8b9955489-djw8f
decrypted password is supersecret

Conclusion

The application running in Kubernetes has to protect its sensitive data.

The classic public-key cryptography is an effective way to encrypt and decrypt the data. Due to the maturity of the public-key cryptography, almost all major programming language has its own library/tools to utilize this technique.

Of course, the common practice of ACL still applies for the access of the private key, encrypted sensitive data, Kubernetes API, Pod and so on…

Cloud explorer