CyberArk Secrets Provider for Kubernetes

This topic describes how to populate Kubernetes secrets with secrets stored in Conjur.

How it works

The CyberArk Secrets Provider for Kubernetes enables you to use secrets stored and managed in the CyberArk Vault using Conjur and consume them as Kubernetes secrets in your application containers. To do so, the CyberArk Secrets Provider for Kubernetes image (cyberark/secrets-provider-for-k8s) runs as an init container (cyberark-secrets-provider-for-k8s) and populates the Kubernetes secrets required by the pod, from Conjur.

  1. The cyberark-secrets-provider-for-k8s init container starts and authenticates with the Conjur server using the Kubernetes Authenticator (authn-k8s).​

  2. The cyberark-secrets-provider-for-k8s init container reads all Kubernetes secrets required by the pod.

  3. For each mapped Kubernetes secret, the cyberark-secrets-provider-for-k8s init container retrieves Conjur secrets and updates the Kubernetes secret with the Conjur secret value.

  4. The cyberark-secrets-provider-for-k8s init container runs to completion.

  5. The application container consumes the Kubernetes secrets and starts.

Step 1: Configure Conjur resources

To enable the CyberArk Secrets Provider for Kubernetes to retrieve Conjur secrets, it needs to authenticate to Conjur.

To do so, you need to assign the CyberArk Secrets Provider for Kubernetes a Conjur host identity that authenticates against the Kubernetes Authenticator:

  1. Prerequisite: Verify that a Kubernetes Authenticator has been enabled

    For details, see Enable Authenticators.

  2. Enable Conjur hosts to authenticate using the Kubernetes Authenticator

    In this step, define a Conjur host used to authenticate the cyberark-secrets-provider-for-k8s container with the Kubernetes Authenticator:

    1. Use the following policy to define a Conjur host. Remember to specify the Kubernetes namespace (APP_NAMESPACE_NAME) where you are deploying your app.

      The service provider host should follow Machine Identity guidelines.

      For more details, see Policy for authentication identities.

      For example, to control authentication and secret access by namespace < TEST_APP_NAMESPACE_NAME >, create the following policy:

       
      - !policy
        id: apps
        body:
          - !layer
       
          - &hosts
            - !host
              id: < TEST_APP_NAMESPACE_NAME >/*/*
              annotations:
                kubernetes/authentication-container-name: cyberark-secrets-provider-for-k8s
       
          - !grant
            role: !layer
            members: *hosts
       

      The value of the kubernetes/authentication-container-name annotation of the host states the container name from which it authenticates with Conjur. When you create the application deployment manifest below, verify that the CyberArk Secrets Provider for Kubernetes init container has the same name.

    2. Save the policy as a yaml file using the following file naming convention: authn-k8s-<authenticator-id>-apps.yml

    3. Load the policy file into authn-k8s-<authenticator-id>:

       
      conjur policy load conjur/authn-k8s/<authenticator-id> authn-k8s-<authenticator-id>-apps.yml
     

    For the rest of this topic, all the examples will refer to the host, conjur/authn-k8s/<authenticator-id>/apps/< TEST_APP_NAMESPACE_NAME >/*/*.

  3. Define variables to hold the secrets for your application

    This section describes how to define the secrets that the cyberark-secrets-provider-for-k8s container needs access to.

    1. Use the following sample policy to define the database credentials as Conjur variables and grant access to these credentials to the host defined in the previous step.

       
      - !policy
        id: db_credentials
        body:
        - &variables
            - !variable password
            - !variable url
            - !variable username
       
      - !permit
        resources: *variables
        role: !layer conjur/authn-k8s/<authenticator-id>/apps
        privileges: [ read, execute ]
    2. Save the policy as a yaml file using the following file naming convention: db-credentials.yml

    3. Load the policy file into root:

       
      conjur policy load root db-credentials.yml

Step 2: Map Kubernetes secrets to Conjur secrets

To populate secrets from Conjur to Kubernetes secrets, create or modify the yaml file that defines the Kubernetes secrets.

Create the mapping

Add a conjur-map data entry to specify the mapping between the Kubernetes secret data key and the secret in Conjur.

To create the mapping:

  1. Edit each Kubernetes secret to include a conjur-map entry for each key stored in Conjur. For example:

     
    # authn-k8s-<authenticator_id>-secrets.yml
    ---
    apiVersion: v1
    kind: Secret
    metadata:
      name: db-credentials
    type: Opaque
    data:
      DBName:   bXlhcHBEQg==
    stringData:
      conjur-map: |-   
        username: db_credentials/username   
        password: db_credentials/password
  2. Update the secret in Kubernetes:

     
    # Kubernetes
    kubectl apply -f authn-k8s-<authenticator_id>-secrets.yml
     
    # OpenShift
    oc apply -f authn-k8s-<authenticator_id>-secrets.yml

Example: Update Kubernetes secrets

The following example describes how to update Kubernetes secrets with secrets from Conjur:

 

The Kubernetes secret db-credentials is defined in a yaml file as follows:

 
# authn-k8s-<authenticator_id>-secrets.yml
---
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  DBName:   bXlhcHBEQg==
  username: dGhlLXVzZXItbmFtZQ==
  password: dGhlLXBhc3N3b3Jk

You determine that the username and password values are sensitive and therefore should be stored in Conjur. However, the value of "DBName" can remain in Kubernetes.

The Conjur admin defines the username and password values as Conjur variables in the following policy:

 
- !policy
  id: db_credentials
  body:
  - !variable username
  - !variable password

and loads the values to Conjur:

 
conjur variable values add db_credentials/username the-user-name
conjur variable values add db_credentials/password the-password

To consume these secrets as Kubernetes secrets, update the original yaml file as follows:

 
# authn-k8s-<authenticator_id>-secrets.yml
---
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  DBName:   bXlhcHBEQg==
stringData:
  conjur-map: |-   
    username: db_credentials/username   
    password: db_credentials/password
 

When loaded, "stringData" values are encoded.

Finally, update the secret in Kubernetes:

 
# Kubernetes
kubectl apply -f authn-k8s-<authenticator_id>-secrets.yml
 
# OpenShift
oc apply -f authn-k8s-<authenticator_id>-secrets.yml

Step 3: Configure RBAC permissions

Create a ServiceAccount, and bind a Role or a ClusterRole to that ServiceAccount to grant privileges to it. These permissions enable the CyberArk Secrets Provider for Kubernetes to update Kubernetes secrets with Conjur secrets.

The following manifest defines a ServiceAccount. The ServiceAccount is bound to the role secrets-access that defines the RBAC permissions on the Kubernetes secrets resource:

To have the Conjur ServiceAccount authenticate your application, you also need to add permissions. For details, see Kubernetes RBAC Permissions.

Load this manifest by running the following:

 
# Kubernetes
kubectl apply -f conjur-role.yml
  
  
# OpenShift
oc apply -f conjur-role.yml

Step 4: Add the CyberArk Secrets Provider for Kubernetes image to the application deployment manifest

The CyberArk Secrets Provider for Kubernetes is a public image provided to you.

  1. Add the CyberArk Secrets Provider for Kubernetes image as an init container to the application deployment manifest. The manifest defines a deployment which includes an application container, test-app, and an init container of the cyberark/secrets-provider-for-k8s image.

    Configure the CyberArk Secrets Provider for Kubernetes with the following variables:

     

    The value of serviceAccountName should be identical to the name of the ServiceAccount you defined in the Step 3: Configure RBAC permissions section.

    Parameter

    Description

    CONTAINER_MODE

    Specify how the container should run.

    Supported value: init

    MY_POD_NAME

    This variable identifies the pod where your application runs. Use the Downward API to insert this information into the manifest. In the example above, this value is auto-populated from the app container info.

    MY_POD_NAMESPACE

    This variable identifies the namespace where your application runs. Use the Downward API to insert this information into the manifest. In the example above, this value is auto-populated from the app container info.

    MY_POD_IP

    This variable identifies the IP of the pod where your application runs. Use the Downward API to insert this information into the manifest. In the example above, this value is auto-populated from the app container info.

    CONJUR_VERSION

    The Conjur release number.

    Supported value: 5

    CONJUR_AUTHN_URL

    Specify the credential service used to log into Conjur.

    If you deployed Conjur through the Helm chart or GKE marketplace app, change conjur-server in these examples to the name of the service.

    Syntax:

     
    https://conjur-server.<CONJUR_NAMESPACE_NAME>.svc.cluster.local/authn-k8s/<AUTHENTICATOR_ID>

    CONJUR_APPLIANCE_URL

    The URL of the service defined for the Conjur server.

    Syntax:

     
    https://< CONJUR_SERVICE_NAME >.< CONJUR_NAMESPACE >.svc/api

    CONJUR_ACCOUNT

    The account name designated to the Conjur server during initial configuration.

    CONJUR_AUTHN_LOGIN

    Specify the Conjur host that authenticates to Conjur. Set this value to a host id that you defined in the policy above.

    See Machine Identity for the host id syntax and the list of Kubernetes resources that can be declared as hosts.

    CONJUR_SSL_CERTIFICATE

    The public SSL certificate value required for connecting to the Conjur server.

    The SSL certificate is generated during Conjur server configuration and stored in a .pem file located in the root folder where the Conjur server was created. The file name is conjur-account.pem, where account is the account name provided for the Conjur server.

    We recommend using a ConfigMap to store this value.

    Example

    • Kubernetes:

       
      kubectl create configmap conjur-cert --from-file=ssl-certificate="/path/to/ssl/cert"
    • OpenShift:

       
      oc create configmap conjur-cert --from-file=ssl-certificate="/path/to/ssl/cert"
      

    The above commands create a ConfigMap that loads the certificate value into it. Once the certificate value is populated, you can add the following configuration to the manifest:

     
    configMapKeyRef:
        name: conjur-cert        
        key: ssl_certificate

    DEBUG

    Enable logs.

    Supported value: true

     

    Do not include this parameter in a production environment.

    SECRETS_DESTINATION

    Required destination for the Conjur secrets, to be retrieved by the application container.

    Supported values: k8s_secrets

    K8S_SECRETS

    A comma-separated list of Kubernetes secret names that the applications in the pod consume and their value is sourced from Conjur.

    For example:

     
    name: K8S_SECRETS
    value: db-credentials,other-secret
  2. Load the manifest by running the following:

    • Kubernetes:

       
      kubectl apply -f deployment-with-conjur.yml
    • OpenShift:

       
      oc apply -f deployment-with-conjur.yml

Step 5: Start the application

To start the application:

  • Kubernetes:

     
    kubectl create -f deployment-with-conjur.yml
  • OpenShift:

     
    oc create -f deployment-with-conjur.yml

Troubleshooting the CyberArk Secrets Provider for Kubernetes

This section describes how to troubleshoot common CyberArk Secrets Provider for Kubernetes issues.

Enable logs

To enable logs, add the debug parameter to the application deployment manifest. For details, see Enable debug logs.

Display logs

To display the CyberArk Secrets Provider for Kubernetes init container logs:

Troubleshooting

The table below describes common issues and their resolution:

Issue

Error code

Resolution

SECRETS_DESTINATION=k8s_secrets is not configured as an init container

CSPFK007E

Validate that the SECRETS_DESTINATION variable is defined as k8s_secrets in the application manifest.

The Service Account doesn’t have permissions to fetch Kubernetes secrets data

CSPFK020E

Validate that the Service Account assigned to the pod has get permissions on the secrets resource.

For details, see Step 3: Configure RBAC permissions.

The Service Account doesn’t have permissions to patch Kubernetes secrets data

CSPFK022E

Validate that the Service Account assigned to the pod has patch permissions on the secret's resource.

For details, see Step 3: Configure RBAC permissions.

CyberArk Secrets Provider for Kubernetes is unable to authenticate

CSPFK010E

First, validate that you provided the correct host name in the manifest for the init container that authenticates to Conjur. For details, see CONJUR_AUTHN_LOGIN.

Second, validate that you assigned the correct permissions to the authn-k8s service.

Kubernetes secrets resources that are defined in the K8S_SECRETS environment variable are not defined in Kubernetes

CSPFK020E

Validate that all Kubernetes secrets resources defined for the K8S_SECRETS environment variable exist in Kubernetes

For details, see Step 2: Map Kubernetes secrets to Conjur secrets

Conjur secrets are not accessible

CSPFK034E

Validate all Conjur variable paths defined in the conjur-map data entry can be retrieved by the Conjur host identity.

For details, see Step 1: Configure Conjur resources.

Client application failed because of invalid or non existing Kubernetes secrets data

No error is raised by the cyberark-secrets-provider-for-k8s init container

Validate that the secrets-provider-for-k8s image is configured as an init container.

For details, see Step 3: Configure RBAC permissions.

Kubernetes secrets resources defined in the K8S_SECRETS environment variable are not properly configured.

CSPFK028E

Validate all Kubernetes secrets resources are defined in the K8S_SECRETS environment variable are configured with a conjur-map data entry.

For details, see Step 2: Map Kubernetes secrets to Conjur secrets.

 
True