Secret Management

[epinzur]

The kaskada project as it exists today doesn't have any requirement for secret management. Currently there are no inputs or outputs that require secrets for configuration.

But this is about to change... the pulsar branch contains code for reading from and writing to pulsar streams. The API for configuring access to pulsar includes a parameter named auth_params for passing authentication secrets. In the current implementation this secret will be stored un-encrypted in the manager's database.

This will not be the only secret that the system will need access to. In the future there will probably be many input and output connectors that require secrets for authentication and/or authorization.

We need to figure out methods for safely storing secrets in the system and/or provide ways to access secrets stored in other systems.

[epinzur]

For the initial implementation, I propose that we focus on enabling ways to access secrets stored in other systems, and don't worry about safely encrypting secrets in our database.

For example, if a value is passed with a env: prefix, we could check for a matching environment variable to pull the secret from:

   secret_auth_token="env:MY_SECRET_AUTH_TOKEN"

Similarly we could add functionality in the future to pull secrets from Hashicorp Vault when the is prefixed with vault: and from AWS Secrets Manager when prefixed by aws:secretsmanager:.

The functionality to pull secrets from other systems could be entirely contained within either the manager or engine codebases. There is no need to do it in both locations. To start we can implement just the environment variable support.

[bjchambers]

That makes sense. My inclination would be to put this in the engine if possible.

I'd also wonder if we colud do better than string prefixes. If this is being used in a proto, could we have a protobuf that we use for these? Something that had a oneof for a string coming from different places (eg., a literal string, an environment variable name, etc.). We could always allow parsing a string into the protobuf. On the other hand, maybe it's easier for the user to pass the prefixed string? It just always makes me worry about things like -- what if I want the value to be the string that starts with env:, etc.

[bjchambers]

Also, we should look to see if there are any crates that provide this kind of functionality we could lean on here.