What do you use to manage Kubernetes secrets? External Secrets Operator, Sealed Secrets, Secrets Store CSI Driver, or something else?

@Luther_Luffeigh

Жыл бұрын

We moved away from sealed secrets to ESO recently, it seems easier to use with all our helm charts as compared to CSI driver. I have a dream one day ESO (or any secret solution actually) will also restart pods when a secret is rotated 🤞

@DevOpsToolkit

Жыл бұрын

@@Luther_Luffeigh A feature to reload (not restart) Pods is coming to Kubernetes :)

@pietersmit621

Жыл бұрын

Mozilla sops encrypted values in git

@DevOpsToolkit

Жыл бұрын

​@@pietersmit621 ​ True. SealedSecrets and SOPS are my personal favorites. Nevertheless, what makes SSCSID interesting is that there are no Kubernetes secrets in the cluster. Now, whether that is a good idea (no matter the implementation) is a different question.

@mtik000

Жыл бұрын

We use: sealed secrets for things that pre-date our Vault cluster and bootstrap things (i.e. the Vault cluster is down), and ESO for cluster-things (e.g. AltertManager Slack webhooks). All of our apps read directly from Vault. I would imagine that we might move to AWS Secrets Manager in the future, and maybe SSCSID is a more agnostic way to go?

I use this at work a lot. Biggest con I've found is that you HAVE to mount your SecretProviderClass to a pod for secrets to created in the cluster. This means if you have a CRD that needs a secret but no real pod that goes along with it then you need to run a small busybox pod to inject the secret.

@DevOpsToolkit

Жыл бұрын

I agree, but I would generalize it a bit and say that it can cause problems or extra work when using it with third-party resources, no matter whether those are CRDs, Helm charts designed to use only secrets, etc.

Thanks! Looking to know more about this use case.

Thank Victor for sharing this! Always good to see alternatives reagarding secret managment in Kubernetes! k8s secrets should have been like ingress, or gateway API: An Interface to whatever implementation I want to use.

@DevOpsToolkit

Жыл бұрын

I think that's where Kubernetes is going. It started with "basic" implementations of most of the things we need (Deployment, Ingress, Storage, Service, etc.). Those are only simple implementations that should be extended or reimplemented. That's how we got service meshes and are not getting a standard interface for them (Gateway API). Something similar will happen to secrets. But, until it does, we need to work with what we have.

Thanks, Victor! Exciting topic as always. Cute nails by the way 😀

nails match the background. 😄

Probably another negative thing is the fargate profile + CSI storage driver , it does not work all together, due to daemon set

JFYI the Azure Key Vault Provider for Secrets Store CSI Driver also supports setting environment variables by syncing a kubernetes secret, not only volume mounts. Despite this, a drawback is that the k8s secret needs to be attached to a pod, otherwise the secret will be deleted. So you might need to artifiacilly attach the secret to a pod (i.e: you might need to mount the secret in the ingress controller pod to be able to use it in an ingress resource).

@DevOpsToolkit

Жыл бұрын

I think that's not specific to Azure but all providers. The problem is that once you start creating secrets as well, SSCSID loses its appeal and the reason for existence. In those cases, I would go with External Secrets Operator (kzread.info/dash/bejne/ha2GvMduibmphs4.html).

🤠 Great video!!

Using both External Secrets Operator and Flux's built-in SOPS decryption (which is similar to Sealed Secrets, but uses SOPS, and is native to Flux).

...and here I am and I can't decide between secrets in ENV vars, ESO, the apps reading directly from the store, and now I have to think about SSCSID! so many choices.

@DevOpsToolkit

Жыл бұрын

That's the blessing and the curse of the ecosystem's size and speed of innovation. There are too many choices.

I think this is how OpenFaaS is doing it, it also provides secrets inside the pod where the function run as files.

@armynyus9123

Жыл бұрын

Oh wow - Mr. rpicluster himself. Tells me watching this channel is not a waste of time (ok, knew that before) and gives me the chance to say thanks for the super tutorial! Coffee due once through.

@vladoportos

Жыл бұрын

@@armynyus9123 this channel is awesome ! I would recommend it in heart beat.

Great video, thanks! I must be missing something about how the secrets are mounted for third party apps. You say that this may not work for third party apps that use Kubernetes Secrets, but it's my understanding that inside th ePod, the K8s secret (when mounted as a volume) would look the same as a SSCSID secret as they both appear as a file with the secret as the contents. Obviously the Helm chart/manifest for that app would need modifications due to the existing use of K8s secrets, so maybe that is all you meant? And of course, if the existing app is using a K8s secret mounted as an env var then I completely get the difference. I'm just talking the scenario where K8s secrets are mounted as volumes.

@DevOpsToolkit

Жыл бұрын

As you said, helm charts would need to modified and that would result in maintenance issues. The modification would effectively be a fork and you'd need to watch for future releases of the original chart and apply the changes to the fork. Alternative would be to keep using the original chart by combining it with some sort of overlaying with, let's say, ytt, but that would also increase the complexity. In other words, it's doable, but at a price.

@Tomsta17

Жыл бұрын

@@DevOpsToolkit Got it. Thanks for clarifying!

@Tomsta17

Жыл бұрын

@@DevOpsToolkit oh I forgot to ask, was I correct in my understanding that the volume mounting mechanism looks the same inside the pod? (When not using env vars for K8s secrets)

@DevOpsToolkit

Жыл бұрын

@@Tomsta17 Yes. In both cases, filesystem is mounted into containers and secrets are available as files (ignoring env. vars.).

Thanks for your video, when working with secrets one of the main use cases is, that secrets can change during lifetime (f.e. secret rotation) most applications read the secrets/config in during startup, but don't update it anymore. The way I work around this problem is stakater Reloader to auto restart on config/secret change. How can Secrets Store CSI Driver achieve this use case?

@joebowbeer

Жыл бұрын

Enable Auto Rotation of Secrets to refresh, but if you need to be notified then you'll need to implement a watcher, e.g., of the file contents of the volume mount of the CSI Driver.

@DevOpsToolkit

Жыл бұрын

From the app perspective, it's the same. Secrets are available as files and your app needs to reload itself when they change. The major difference is that SSCSID loads those files by mounting volumes instead of mounting secrets (which are files as well, but are also stored in etcd).

@nilsmuller9097

Жыл бұрын

@@DevOpsToolkit Yes exactly and so changes inside are not detected and need to be implemented as sidecar or inside the application :/

I'm a bit... confused, in case of it's integration with Hashicorp Vault. I understand it's use case with GCP/AWS/Azure however Hashicorp Vault already mounts secrets as volume (as init or sidecar container) and supports live reloading of data. So, do you see any advantage of using SSCSID with Vault instead of using plain Vault?

@DevOpsToolkit

Жыл бұрын

Vault has quite a few ways to deal with secrets, some done by HashiCorp itself, while others by third-party solutions. Are you referring to www.vaultproject.io/docs/platform/k8s/injector?

@krzysztofwiatrzyk4260

Жыл бұрын

@@DevOpsToolkit yes, I'm using it at work, except some issues with JWT validation change in Kubernetes 1.22 (or 1.21?) it works like a charm. The biggest advantage I see is that you just need to use proper annotations to injects secrets (so Helm charts that allow setting annotations from values would work). It is even possible to load secrets as environment variables with some hack but that hack should also work for this solution.

@DevOpsToolkit

Жыл бұрын

@@krzysztofwiatrzyk4260 Vault is the most advanced secrets store with decently big ecosystem so there is no big reason to use ESO or SSCSID. You can accomplish similar results with Vault Injector (similar to ESO) or Vault CSI (similar to SSCSID). The major difference is that ESO and SSCSI are more in line with the vault Kubernetes expects us to define things (through CR instead of annotations) and with more decoupling between the app (or app manifest) and secrets. There are other differences. Nevertheless, those might be significant enough to warrant a switch from Injector to ESO or Vault CSI to SSCSID. That being said, there will be more differences in the future. HashiCorp tends to ignore or go against Kubernetes so its slowly falling behind the "Kube Ecosystem". That can probably be attributed to HashiCorp not treating open source as it should, other software vendors looking to reduce the dependency on HashiCorp, their insistance on keeping Nomad alive, etc.

Hello Sir...Can you please guide me to retrieve the AWS secrets and store it to kubernetes pod/container.using helm charts? your quick reply help me a lot ..!! Thank you so much in advance

@DevOpsToolkit

9 ай бұрын

The gist with the commands i used in that video is in the description. You should be able to replicate what i did in aws with those.

How do you edit your videos, its awsm and what tool do you use to record your session, please let me know, also if possible please create a video on the same as well. it will be a good learning experience for me and for others.

@DevOpsToolkit

Жыл бұрын

I record screen, video, and sound separately and send it to an agency that does the editing, animations, banners, etc.

@devopsbucket2549

Жыл бұрын

@@DevOpsToolkit mac book screen recorder? what about lighting and all, thanks for answering

@DevOpsToolkit

Жыл бұрын

​@@devopsbucket2549 Here it goes: * Screen recording: OBS * Video recording: Sony ZV-E10 with Sigma 16 mm Lens * Audio recording: Rode NTG4 (for videos) and ElGato Wave (for streams) * Lightning: 2 ElGato Key Lights in front and LifX lightbulbs in the back (RGB) * Teleprompter: Desview T3 * Computer: iMac (will be moving to Mac Studio in the future)

@devopsbucket2549

Жыл бұрын

@@DevOpsToolkit Thanks for answering it help me a lot.

External Secrets Operator is definitely better than Secrets Store CSI Driver.

@mopsik4ever

Жыл бұрын

Not the same use case. External secret pulls from third party and creates a secret inside the cluster. Here you never create any sensitive data inside the cluster

@shulyakav

Жыл бұрын

@@mopsik4ever true. But anyway you will create volume with secrets inside a cluster (pod).

@Luther_Luffeigh

Жыл бұрын

ESO can also used to create a k8s secret store, so you can have it fetch from either source

@mopsik4ever

Жыл бұрын

@@shulyakav if security is done right you can minimize the attack surface by creating multiple service accounts for multiple purposes. If someone gets to your cluster , it’s going to be with some kind of cluster role or role attached to it. If you have secrets in a namespaces they are easier to grab than connecting to a pod since the pod/exec permission is not very common except cluster admins such as devops and product security. I am not saying it’s a 100% covering all soliton but it does minimize the attack surface

Iam using helm values to deploy all manifest, and all the secrets are stated on that (plain text) How can implement this to my values?

@DevOpsToolkit

11 ай бұрын

I'm not sure I understood the question. Are you trying to use secrets as Helm values? If that's the case, don't. If an app needs info available as a secret, that app should mount that secret.

can we unmount CSI volume as once secrets are pulled in Application ?

@DevOpsToolkit

Жыл бұрын

That would result in the destruction of the pod where the volume is mounted and the creation of a new one without the volume. Out of curiosity... Why would you remove the volume?

Can I create Env variables without creating secrets? Almost all apps I use rely on env variables.

@DevOpsToolkit

Жыл бұрын

You can add env variables to containers through secrets, config maps, or directly in pod manifests.

How to set an environmental variable from third party secrets to the container instead mount as volume?

@DevOpsToolkit

Жыл бұрын

If the secret already exists, all you have to do is mount it to the pod. Once you do that, all entries of the secret will be available as environment variables.

@ramallways6321

Жыл бұрын

How sir, for example if Mount means as file it will come and sit in the location where I've chosen. But I didn't say anything in pod spec about env...?

@DevOpsToolkit

Жыл бұрын

@@ramallways6321 If you're talking about Kuberentes secrets, than you can specify which secret entry becomes an env. variable through `secretKeyRef` (kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#define-a-container-environment-variable-with-data-from-a-single-secret).

@ramallways6321

Жыл бұрын

I'm not talking about k8s secret. For example if Mount my Azure secret into my pod via secret provider class. Then all secrets will come and sit like file in my pod. My question here is how this secret will be available in pod's env?

@DevOpsToolkit

Жыл бұрын

My bad. I misunderstood... With ESO you get only a file and the container that mounts it would need to convert it into environment variables. I think that there is also an option to create kubernetes secrets as well but I haven't used it much since that would defy the purpose of using ESO.

There is a caveat i want to add with csi. If ur syncing certs as k8s tls secrets. It can only create tls.crt and tls.key but it cant create ca.crt as secret data.

@DevOpsToolkit

Жыл бұрын

Use service mesh for internal certs and cert manager for external facing ones. Do not mount certs yourself into apps.

The main problem I have with this is that I can’t inject the secret as an environment variable…

@mgna20

Жыл бұрын

You can, you need to set it to create a k8s secret in your SecretProviderClass and then mount it as a volume in your pod. It will create the k8s secret and you pull from there with envFrom. Not ideal but it does work

@joebowbeer

Жыл бұрын

Fix: SSCSID volume mounts can be projected to k8s secrets and any k8s secret can be injected as an environment variable

@DevOpsToolkit

Жыл бұрын

That's true. However, that goes against the main reason for existence of SSCSID (no secrets).

million dollar question, how do I get the secrets from the file to be set as environmental variables in cases where the application just requires it

@DevOpsToolkit

Жыл бұрын

I think you'd need to read that file and convert into env. vars. As far as I know, SSCSID does not do that and you'd either need to create your own process (in a sidecar) or look for some tool that does that. P.S. I'm ignoring the fact that SSCSID can create secrets as well (and thus provider envs. vars.) since that defies its purpose (to not have secrets).

@travissobeck4939

Жыл бұрын

@@DevOpsToolkit as common as a need as this is I'm surprised I've not found an established and popular method that works with any of pod setup. All I ever see is people just say well you can do it and no actual useful information about how

@DevOpsToolkit

Жыл бұрын

@@travissobeck4939 I guess that the need is not that big. Most people use Kubernetes secrets or configmaps which do provide env. vars.

@DevOpsToolkit

Жыл бұрын

@@travissobeck4939 Assuming that the file is in key/value format, a simple `source my-file` should do. You can put it into the command of the container (e.g., `source my-file && my-app`).

It seems the video has a huge echo.

@DevOpsToolkit

Жыл бұрын

That's true. I recoded it in a different location that has terrible echo. It's a temporary place where I go during summer and that I did not yet sound proof.

I was told that, the secret store needs a `hostPath` mount to work!! Is that true? Production clusters don't allow to use hostPath

@DevOpsToolkit

Жыл бұрын

That is indeed true. `hostPath` is required. Now, I would not go as far as deny anything to use `hostPath`. Instead, like with almost anything else, I'd make sure that only certain processes can use it in a controlled manner. Otherwise, you'd have trouble running quite a few other tools effectively (e.g., Tekton, Argo Workflows, etc.).

@JaydeepDave12

Жыл бұрын

@@DevOpsToolkit Security guys are adopting (learning) Kubernetes and the security tools around Kubernetes. They want to tick as many boxes as they can :D

@DevOpsToolkit

Жыл бұрын

@@JaydeepDave12 That's true and, given everything else being equal, I'd pick a tool that does not mount `hostPath`. I still recommend External Secrets Operator over SSCSID for the cases where secrets are in external secrets manager.

@joebowbeer

Жыл бұрын

Covered in the SSCSID best practices guide.

You should have asked your daughter to paint them grey!

@DevOpsToolkit

Жыл бұрын

I'm afraid that it'll be pink or green the next time.

@teebu

Жыл бұрын

@@DevOpsToolkit I just thought you went emo.