Kubernetes, OpenShift, and GKE

Overview

This integration securely passes secrets stored in Conjur to applications running in RedHat OpenShift, Google Kubernetes Engine (GKE), or other Kubernetes implementations. By configuring your environment to integrate with Conjur, secrets are never exposed to third parties.

The Conjur integration provides these features to your Kubernetes environment:

  • End-to-end encryption of secrets through mutual TLS.
  • Robust authentication and authorization incorporating Conjur policy, signed certificates, and an internal Kubernetes authenticator.
  • Conjur policy provides separation of duties, letting your security teams control container access while development teams define application requirements.
  • Easy deployment of applications across environments and pods.
  • Secret rotation, centralized auditing, and all other advantages of Conjur.
  • Scalability and performance advantages of the Enterprise Conjur master-follower architecture. Followers provide read-only activity for clients. Scale-out is easy by simply adding more followers.
 
  • This documentation describes both the Conjur Enterprise and Conjur Open Source integrations.
  • Unless otherwise noted, all mention of Kubernetes applies to native Kubernetes as well as the OpenShift and GKE implementations of Kubernetes.
  • All mentions of Kubernetes namespaces intentionally includes the OpenShift concept of Project.

Requirements

  • Access to one of the following:

    • OpenShift environment (3.3 or higher) with an internal Docker registry.

    • GKE cluster:

      • version 1.9 or higher is required for easy installation with Google Cloud Marketplace; version 1.10 is recommended.
      • version 1.5 and higher are supported with Helm installation.
    • Other Kubernetes environment (1.5 or higher)

  • For Conjur Enterprise, you also need: 
    • A license for a Conjur cluster with a master, two standbys, and at least two followers.

    • The latest conjur-appliance Docker image from your Conjur support representative

How it Works

Machine Identity for Kubernetes Resources

By assigning machine identities to Kubernetes resources, you can use policy to control:

  • The granularity and identity of the Kubernetes resources that authenticate to Conjur.

  • The granularity and identity of the Kubernetes resources that can access secrets. The access usually occurs in the context of an application-specific policy, where specific host roles are granted access to specific secrets.

To assign machine identity, you use policy to declare a desired Kubernetes resource as a Conjur host. See About Host Ids for host id syntax for each of the supported Kubernetes resources.

Here are the Kubernetes resources that you can define as Conjur hosts, with guidelines for choosing them.

Deploy Conjur

Deploy Applications

Per Cluster Preparations

Per Application Requirements

Step 1 - Prepare and Load Required Conjur Policy

Define Policy for Human Users

At least one user needs write permission to load policy and variables into Conjur. This is standard Conjur policy that creates an administrative group of users for Conjur.

Use the following policy as a template:

# initializes users
  # ted - kubernetes admin
  # bob - devops admin
  # alice - db admin
  # carol - developer

    - !group kube_admin
    - !group devops
    - !group ops
    - !group db_admin

# kube_admin and devops groups are members of the ops admin group
    - !grant
      role: !group ops
      members:
        - !group kube_admin
        - !group devops

    - !user ted
    - !grant
        role: !group kube_admin
        member: !user ted

    - !user bob
    - !grant
        role: !group devops
        member: !user bob

    - !user alice
    - !grant
        role: !group db_admin
        member: !user alice

    - !user carol

Define Policy for the Kubernetes Authenticator Service

You need one Kubernetes Authenticator policy per Kubernetes cluster. It services multiple namespaces and applications. See Machine Identity for Kubernetes Resources for context.

This policy defines:

  • The authenticator's service-id.
  • The Conjur webservice that generates the CSRs.
  • The CA certificate and key.
  • A group to represent the hosts and applications that can use this authenticator.
  • Permissions for the above group to use this authentication service. (read and authenticate permissions).
  • A layer of hosts (namespaces, pods, and service accounts) that can use this authentication service.
  • Optional annotations that turn on platform-specific icon use and enable platform-specific searches in the UI.
  • A layer of applications that can use this authentication service.
About the apps Sub-Policy Id

A sub-policy with the apps id is required.

- !policy  
  id: apps

The apps sub-policy defines:

  • Hosts that can authenticate. A host is a machine entity that can login to Conjur and authenticate.
  • Applications that can authenticate. Policy for user applications is typically defined in separate policy files. This section links your application-specific policies to this Kubernetes authentication policy.
About Application Layer References

In the apps sub-policy, each application is referenced.

- !grant
    role: !layer /application-policy-id

This reference links the application to this Kubernetes authenticator policy. It is a reference to a layer that exists in a different policy.

About Host Ids

The host ids represent Kubernetes resources. The format of the host id determines the granularity of authentication and secret access that you want to enforce.

  • [namespace]/*/* = authentication and secret access is by namespace. The two asterisks are wildcards for all Kubernetes resource types and all resource names within the namespace. With this format, all resources in a namespace are controlled by the same grants and permissions.
  • [namespace]/deployment/[name] = authentication and secret access is by deployment name within a namespace. Grants and permissions are specific to a deployment within a namespace.
  • [namespace]/deployment_config/[name] = authentication and secret access is by OpenShift deployment config name within a namespace. Grants and permissions are specific to a deployment config within a namespace.
  • [namespace]/stateful_set/[name] = authentication and secret access is by stateful set name within a namespace. Grants and permissions are specific to a stateful set within a namespace.
  • [namespace]/pod/[name] = authentication and secret access is by pod name within a named namespace. Grants and permissions are specific to a pod within a namespace.
  • [namespace]/service_account/[name] = authentication and secret access is by service account name within a namespace. Grants and permissions are specific to a service account within a namespace.

The authentication service prepends the namespace in your host id with additional known information, transforming the entire host id value to host/conjur/authn-k8s/<webservice>/apps/<namespace>/...

When the two asterisks are used to represent all resources in a namespace, the host id becomes host/conjur/authn-k8s/<webservice>/apps/<namespace>/<controller_type>/<controller_id>

Example host ids

body:
    - !layer
      - &hosts                                    
	  # list hosts here
        - !host
          id: namespace-1/*/*                     
	  # namespace-1 authenticates. The wildcards are required.

        - !host
          id: namespace-1/deployment/web03        
	  # deployment 'web03' in namespace 'namespace-1' authenticates.

        - !host
          id: namespace-1/deployment_config/web04 
	  # OpenShift deployment config 'web04' in namespace 'namespace-1' authenticates.

        - !host
          id: namespace-2/stateful_set/db         
	  # stateful set 'db' in namespace 'namespace-2' authenticates.

        - !host
          id: namespace-2/pod/pod-10              
	  # pod-10 in namespace-2 authenticates.

        - !host
          id: namespace-3/service_account/sa-20   
	  # sa-20 in namespace-3 authenticates.
About Host Annotations

Hosts can have these annotations:

  • kubernetes/authentication-container-name: is required to identify the authenticator
  • openshift: true is optional but useful to identify OpenShift hosts in the UI
  • kubernetes: true is optional but useful to identify Kubernetes hosts in the UI

The platform-specific annotations let you see the platform type in the UI and filter hosts by platform.

- &hosts
    - !host
      id: some-namespace-1/*/*
      annotations:
        kubernetes/authentication-container-name: authenticator
        openshift: true
Kubernetes Authenticator Service Policy Example

Use the following policy as a template. Typically, only the first policy id, the list of hosts, and the list of applications would need to be edited.

  - !policy
    id: conjur/authn-k8s/subcluster-1  
	# In the id, conjur/authn-k8s is required. 
	# Subsequent components in the id identify the service id 
	#  for the Kubernetes authenticator service.
    # It is the SERVICE_ID variable set during deployment.
	
    body:
    - !webservice
      annotations:
        description: Authentication service for the "subcluster-1" cluster.

    - !policy
      id: ca                 # ca policy - do not change
      body:
      - !variable
        id: cert
        annotations:
          description: CA cert for Kubernetes Pods.

      - !variable
        id: key
        annotations:
          description: CA key for Kubernetes Pods.

    - !group
      id: clients
      annotations:
        description:
          Members of this group can use the subcluster-1 
		  authentication service. This group typically has one 
		  member, which is a layer containing the enrolled
          applications.

    - !permit
      resource: !webservice
      privilege: [ read, authenticate ]
      role: !group clients

    - !policy
      id: apps                    #apps policy - the id must be apps
      annotations:
        description: Apps and services in the "subcluster-1" Kubernetes cluster.
      body:
      - !layer
	  
      #list hosts here
      - &hosts                    
        - !host
          id: some-namespace-1/*/*  
	      #host id can represent an entire namespace, controller, pod, or service account. 	  
          annotations:
            kubernetes/authentication-container-name: authenticator
            openshift: true            
			#The openshift: annotation enables platform-specific UI features
            #replace with kubernetes: true if appropriate
 
      # the role references a layer named test-app that exists in a different policy.
      - !grant
        role: !layer /test-app         
        members:
        - !host some-namespace-1/*/*

      #add all hosts to the apps layer
      - !grant                         
        role: !layer
        members: *hosts
		
      #add the apps layer to the clients group, 
	  #which has permission to authenticate to Conjur
      - !grant                  
        role: !group clients
        member: !layer apps

Load Policy into Conjur

  1. Save policy as .yml files in a location accessible to the Conjur master.

  2. Log into Conjur.

  3. Load each policy file:

    $ conjur policy load root policy-file-name.yml

    $ conjur policy load root k8s_policy.yml

Step 2 - Initialize the CA

The Kubernetes policy (described above) declares variables to hold a CA certificate and key. After loading the policy, run the following commands to initialize those resources.

  • The value of SERVICE_ID must match the service ID in the name of the Kubernetes Authenticator policy defined the section above. For example, if the policy ID is "conjur/authn-k8s/subcluster-1", the value of SERVICE_ID is "subcluster-1".
  • The value of CONJUR_ACCOUNT must match the Conjur account used in the Deploy Conjur
SERVICE_ID='**SERVICE_ID**'
CONJUR_ACCOUNT='**CONJUR_ACCOUNT**'

# Generate OpenSSL private key
openssl genrsa -out ca.key 2048

CONFIG="
[ req ]
distinguished_name = dn
x509_extensions = v3_ca
[ dn ]
[ v3_ca ]
basicConstraints = critical,CA:TRUE
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always,issuer:always
"

# Generate root CA certificate
openssl req -x509 -new -nodes -key ca.key -sha1 -days 3650 -set_serial 0x0 -out ca.cert \
  -subj "/CN=conjur.authn-k8s.$SERVICE_ID/OU=Conjur Kubernetes CA/O=$CONJUR_ACCOUNT" \
  -config <(echo "$CONFIG")

# Verify cert
openssl x509 -in ca.cert -text -noout

# Load variable values
conjur variable values add conjur/authn-k8s/$SERVICE_ID/ca/key "$(cat ca.key)"
conjur variable values add conjur/authn-k8s/$SERVICE_ID/ca/cert "$(cat ca.cert)"

These commands create a private key and root certificate and store contents of those files in the variables conjur/authn-k8s/$SERVICE_ID/ca/key and conjur/authn-k8s/$SERVICE_ID/ca/cert.

Login or auth calls to the webservice will fail if these resources are not properly defined in policy and initialized.

Step 3 - Configure Conjur Authenticators

NOTE: The deployment scripts for Conjur Enterprise have already performed this step using the value you set in the SERVICE_ID environment variable. You need to be aware of this step to add additional application clusters or additional authenticator types.

The CONJUR_AUTHENTICATORS environment variable in the Conjur deployment YAML file defines the authentication types used to authenticate with the Conjur cluster. To enable Kubernetes authentication, use:

CONJUR_AUTHENTICATORS=authn-k8s/**service_id**

where service_id is the id assigned to the authn-k8s webservice in Conjur policy. It is important that the service_id used here match the webservice id declared in the Kubernetes policy.

For example, in this snippet from the Conjur webservice policy, a policy branch named conjur declares the authn-k8s service with the service_id of prod/gke:

- !policy
  id: conjur/authn-k8s/prod/gke

The authentication value would be:

CONJUR_AUTHENTICATORS=authn-k8s/prod/gke

One authn-k8s service can serve multiple application service ids. Additional Conjur policy for hosts and applications will control which namespaces get access to Conjur and which applications get access to specific secrets. There should be a separate authn-k8s policy (and corresponding service id) for each or Kubernetes cluster.

CONJUR_AUTHENTICATORS can include more than one authenticator and more than one authentication type as a comma-separated list. For example, the following shows two authn-k8s services and another unrelated authenticator:

CONJUR_AUTHENTICATORS=authn-k8s/prod/gke,authn-k8s/dev/gke,authn-jwt/something

To disable an authenticator, remove it from the list.

Step 4 - Deploy the K8S Authenticator Client (Sidecar or Init)

Each application pod requires the K8S Authenticator Client running as either a sidecar or an init container. See Sidecar versus Init Container for Authenticator Client for a comparison.

  1. Reference the authenticator client image in the appropriate section of the application manifest. Because the image is available on Docker Hub, you can reference cyberark/conjur-authn-k8s-client in the manifest to deploy the image.

     

    Get the K8S Authenticator Client from either of these locations:

  2. For the sidecar, the reference appears as an additional - image in the spec section of the application manifest.

  3. For an init container, the reference appears under an added initContainers field in the spec section of the application manifest.

    The next section shows and explains example manifests.

  4. Set the CONTAINER_MODE variable in the application manifest. The default mode is sidecar if this variable is not provided.

    - name: CONTAINER_MODE
        value: <init | sidecar>

Step 5 - Add Resources to the Application Manifest

Edit your application manifest, adding the following required resources to the spec section:

  • a sidecar or init container
  • the shared volume
  • required Conjur variables for the application container

Step 6 - Prepare Application to Retrieve Secrets

There are two options for applications to retrieve secrets from Conjur.

Use the Conjur API to retrieve secrets from Conjur

  • Using API calls, the application gets the access token from the shared volume (/run/conjur/access-token), authenticates to Conjur, and then requests secrets.

  • The access token may not be available immediately on application startup. The application may need to wait for the volume to mount, the authentication to occur, and the access token to be written into the shared file.

    NOTE: The conjur-api-go API handles the wait gracefully and seamlessly. For other APIs, the application containers should check for the file on startup and retry until the file exists.

  • See the sidecar README for more information about the API and the access token. The same sidecar is used for both Kubernetes and OpenShift integrations.

  • See our demo repository for examples of complete Conjur deployments and test applications that use the API to get secrets from Conjur.

    https://github.com/conjurdemos/kubernetes-conjur-demo

Use Summon to retrieve secrets from Conjur on behalf of the application

  • Summon is a CyberArk Conjur tool that reads a file in secrets.yml format, obtains secrets from a source, and injects the secret values as environment variables into any process. Once the process exits, the secrets are gone.
  • Summon uses source-specific providers to fetch secrets. For the Conjur Kubernetes integration, applications use the summon-conjur provider.
  • Summon gracefully and seamlessly handles the wait for the access token. If your application is using Summon with the summon-conjur provider to get secrets, and has the environment variable CONJUR/_AUTHN/_TOKEN/_FILE=/run/conjur/access-token set, then Summon will keep retrying until that file exists.
  • See Summon documentation for information about installation, configuration, the summon-conjur provider, and the secrets.yml file format.

Example: secrets.yml for Summon

The following example of a secrets.yml file shows several types of allowed entries.

  DB_USERNAME: !var my-app/db/username
  DB_PASSWORD: !var my-app/db/password
  REGION: us-east-1
  SSL_CERT: !var:file ssl/certs/private
  • Lines 1 and 2 specify variables with pathnames containing a secret. In this case, assume that a policy with an id of my-app was loaded into Conjur.
  • Line 3 specifies a literal string for the secret.
  • Line 4 specifies a variable that is a file containing the secret. The contents of the file would also be retrieved from Conjur.

Step 7 - Define Policy for Applications

Application policy defines:

  • Secrets (as Conjur variables) used by the application
  • Permission for at least one human user or group for write access to the secrets (to load the initial value)
  • Permission for application instances to read the secrets

Each application has its own Conjur policy, usually each in a separate policy file.

  1. Use the following policy as a template.

    - !policy
      id: test-app
      owner: !group developers
      body:
      - !layer
    
    - !policy
      id: test-app-db
      owner: !group operations
      body:
      - &variables
        - !variable password
    
      - !permit
        resources: *variables
        privilege: [ read, execute ]
        role: !layer /test-app          
    #reference this layer in Kubernetes policy when adding relevant hosts
  2. Load the application policy:

    $ conjur policy load root policy-file-name.yml

Step 8 - Reference the Application in the Authenticator Policy

Remember to add your application layer (/webapp in the above example policy) in the apps sub-policy of the authenticator policy. See About the apps Sub-Policy Id.

Step 9 - Load Initial Secret Values into Conjur

For each secret defined in an application policy, load the initial secret value. You can use the Conjur API, the UI, or the CLI for this step.

Here is an example using the CLI:

$ conjur variable value cluster-1/db/password abc$xyz

Step 10 - Start the Application

To start the application, use this Kubernetes command:

kubectl create -f your-manifest.yaml

The equivalent OpenShift command is:

oc create -f your-manifest.yaml

Administration