In our previous post, we discussed the issues administrators face when it comes to the security of outside container content in both development and runtime environments.
Open source components are a significant weak link in container environments, and the layered aspect of container images means careful review is required before deployment into a runtime environment. Accomplishing this without diminishing agility or speed requires tools created specifically for integration and implementation.
Red Hat Quay is the container and application registry which allows developers to securely build, analyze, and distribute container images. Quay also provides a Docker registry service, and then seamlessly integrates with Clair to implement scanning of container images for security vulnerabilities.
The Clair project is an open source engine that powers Red Hat Quay Security Scanner to detect vulnerabilities in all images within Red Hat Quay, then notify developers as those issues are discovered.
This article assumes you have a functioning Quay environment. Our article is based on having Red Hat Quay 3 installed with Red Hat OpenShift 3.11.98 and Ansible 2.6. This also assumes that you have a supported Object Oriented Storage location established and is already being used by Quay.
Configure Quay
Quay needs to be configured to communicate with Clair. Configuration must be done using a Quay user account with Superuser permissions.
1. Login to the Quay web UI using a Superuser type user account2. Navigate to:Super User Admin Panel --> Registry Settings --> Security Scanner3. Select checkbox next to "Enable Security Scanning"4. Set Security Scanner Endpoint: http://clair-app-api.quay-enterprise.svc.cluster.local:60605. Create Authentication Key (record Key ID and private key):Generate Key -> Generate Shared Key -> Continue -> Generate Key6. Write private key to openshift_certs/security_scanner.pem7. Click “Save Configuration Changes” button at the bottom of the page.Stage container images
Depending on the organization and its security policies, Direct internet access may be allowed and the OCP infrastructure servers may be configured to download directly from Red Hat or Docker. If this is the case, the next section can be skipped.
However; if an on-Prem registry is already in place and is going to be the preferred location for image storage and retrieval. This will need to be done if the OCP servers will NOT be allowed to download directly from Red Hat or Docker.
$ docker pull quay.io/coreos/clair-jwt:v2.0.8$ docker tag quay.io/coreos/clair-jwt:v2.0.8 quay.io/coreos/clair-jwt:latest$ docker tag quay.io/coreos/clair-jwt:latest \ registry.apps.ocp-dev01-bna.ssc.tsc/quay-enterprise/clair-jwt$ docker push registry.apps.ocp-dev01-bna.ssc.tsc/quay-enterprise/clair-jwt
Create databases
The Clair tool requires that it installs a small version of the Postgres database. This database is used for CVE (Common Vulnerabilities and Exposures)files that are downloaded from Red Hat and Docker. These files are needed for the scanning of images that will be used in the OCP environment.
$ oc rsh <postgresql pod name>sh-4.2$ createdb clair -O quayusersh-4.2$ createdb quayuser -O quayuser
Configure authentication
Configure credentials to enable Clair to communicate with the Postgresql database and Quay.
This communication is vital for the in depth scanning process that Clair will perform and compare an imported image from the developers against the CVE information housed in the Postgres database. Since this information contains sensitive information and may expose certain vulnerabilities, allowing a “service” account to perform the tasks without human intervention is necessary for the automation process. Once that process is complete, this information is then taken by Quay to show results and reports of what it has found.
# Set Postgresql password in config/config.yml, then create secret.# Important - URL encode any special characters in password (see https://developer.mozilla.org/en-US/docs/Glossary/percent-encoding)$ vi config/config.ymlclair: database: type: pgsql options: # A PostgreSQL Connection string pointing to the Clair Postgres database. # Documentation on the format can be found at: http://www.postgresql.org/docs/9.4/static/libpq-connect.html source: postgres://quayuser:<POSTGRESQL_PASSWORD>@postgresql.quay-enterprise.svc.cluster.local:5432/clair?sslmode=disable cachesize: 16384<snip>$ oc create configmap clair-app --from-file=config/ -n quay-enterprise# ca.crt and security_scanner.pem should be located in openshift_certs directory$ cd ../openshift_certs$ oc create secret generic ca-crt --from-file=ca.crt -n quay-enterprise$ oc create secret generic securityscanner \ --from-file=security_scanner.pem -n quay-enterprise$ cd ../quaydeploy/ Deploy Clair
Deploying Clair is similar to when we deployed Quay, OCP treats this just like an internal service and therefore should be deployed as projects that will in turn create containers to run the service and start up all of the necessary processes , opening of the published ports needed for network communication and all the required internal certificate needed for secure communication. This also allows for the service to be monitored for health checks and self-healing by the OCP platform.
$ oc create -f clair-service.yml -n quay-enterprise$ oc create -f clair-deployment.yml -n quay-enterprise Using Clair
Quay will automatically start using Clair to scan images as they are pushed to Quay once this implementation process has been completed. Clair scanning results will be displayed in the “SECURITY SCAN” column on the “Repository Tags” page for each Repository in the Quay web UI. Click the scan result for additional details such as the number of vulnerabilities, vulnerability criticality, links to advisories, etc.
Application deployment
This section shows how to deploy an application container using Quay as the container image registry. The source image is pulled from the Quay registry. Then the image resulting from the application build process is pushed in the Quay registry. Next, Quay scans the new image using Clair. Finally, the application container is pulled from the Quay registry so that it can be deployed by OpenShift. In this example, we’ll call this application “HelloWorld”. Our application is going to deploy and Apache Web Server with a greeting page.
*NOTE* Our example is based on the assumption that you have been able to successfully import an image into your own environment or you have access to a registry that already has test images that you would like to use. Some information may need to be filled in by you the reader for functionality and to be relative to your environment.
Create application deployment files
Create a directory to hold the application deployment template and subdirectory for application content. Then create the template and application content.
$ mkdir -p helloworld-app-demo/build_content$ cd helloworld-app-demo/
httpd-helloworld-demo.yml
Create httpd-helloworld-demo.yml in the helloworld-app-demo/ directory.
apiVersion: template.openshift.io/v1kind: Templatelabels: app: httpd-helloworld-demo template: httpd-helloworld-demomessage: |- The following service(s) have been created in your project: $. For more information about using this template, including OpenShift considerations, see https://github.com/openshift/httpd-ex/blob/master/README.md.metadata: annotations: description: An example Apache HTTP Server (httpd) application that serves static content. For more information about using this template, including OpenShift considerations, see https://github.com/openshift/httpd-ex/blob/master/README.md. iconClass: icon-apache openshift.io/display-name: Apache HTTP Server openshift.io/documentation-url: https://github.com/openshift/httpd-ex openshift.io/long-description: This template defines resources needed to develop a static application served by Apache HTTP Server (httpd), including a build configuration and application deployment configuration. openshift.io/provider-display-name: Red Hat, Inc. openshift.io/support-url: https://access.redhat.com tags: quickstart,httpd template.openshift.io/bindable: "false" creationTimestamp: 2019-04-17T03:35:08Z name: httpd-helloworld-demo namespace: openshift resourceVersion: "1726" selfLink: /apis/template.openshift.io/v1/namespaces/openshift/templates/httpd-helloworld-demo uid: <<find the UID of the image template from the above directory.copy and paste it here>>objects:- apiVersion: v1 kind: Service metadata: annotations: description: Exposes and load balances the application pods name: $ spec: ports: - name: web port: 8080 targetPort: 8080 selector: name: $- apiVersion: v1 kind: Route metadata: name: $ spec: host: $ tls: termination: edge to: kind: Service name: $- apiVersion: v1 kind: ImageStream metadata: annotations: description: Keeps track of changes in the application image name: $ spec: lookupPolicy: local: false tags: - annotations: null from: kind: DockerImage name:</Provide/an/image/to/be/used/here/$:latest generation: 1 importPolicy: {} name: latest referencePolicy: type: Source- apiVersion: v1 kind: BuildConfig metadata: annotations: description: Defines how to build the application template.alpha.openshift.io/wait-for-ready: "true" name: $ spec: output: to: kind: DockerImage name: </Provide/an/image/to/be/used/here/$:latest pushSecret: name: "quay-helloworld-pull-secret"(This needs to be created) source: contextDir: $ git: ref: $ uri: $ type: Git strategy: sourceStrategy: from: kind: DockerImage name: </Provide/an/image/to/be/used/here/$/httpd-24-rhel7:latest forcePull: true type: Source triggers: - type: ImageChange - type: ConfigChange - gitlab: secret: $ type: GitLab - generic: secret: $ type: Generic- apiVersion: v1 kind: DeploymentConfig metadata: annotations: description: Defines how to deploy the application server template.alpha.openshift.io/wait-for-ready: "true" name: $ spec: replicas: 1 selector: name: $ strategy: type: Rolling template: metadata: labels: name: $ name: $ spec: containers: - env: [] image: ' ' livenessProbe: httpGet: path: / port: 8080 initialDelaySeconds: 30 timeoutSeconds: 3 name: httpd-quay-demo ports: - containerPort: 8080 readinessProbe: httpGet: path: / port: 8080 initialDelaySeconds: 3 timeoutSeconds: 3 resources: limits: memory: $ triggers: - imageChangeParams: automatic: true containerNames: - httpd-quay-demo from: kind: ImageStreamTag name: $:latest type: ImageChange - type: ConfigChangeparameters:- description: The name assigned to all of the frontend objects defined in this template. displayName: Name name: NAME required: true value: httpd-helloworld-demo- description: The OpenShift Namespace where the ImageStream resides. displayName: Namespace name: NAMESPACE required: true value: quay-helloworld-test- description: Maximum amount of memory the container can use. displayName: Memory Limit name: MEMORY_LIMIT required: true value: 512Mi- description: The URL of the repository with your application source code. displayName: Git Repository URL name: SOURCE_REPOSITORY_URL required: true value: https://github.com/openshift/httpd-ex.git- description: Set this to a branch name, tag or other ref of your repository if you are not using the default branch. displayName: Git Reference name: SOURCE_REPOSITORY_REF- description: Set this to the relative path to your project if it is not in the root of your repository. displayName: Context Directory name: CONTEXT_DIR- description: The exposed hostname that will route to the httpd service, if left blank a value will be defaulted. displayName: Application Hostname name: APPLICATION_DOMAIN- description: GitLab trigger secret. A difficult to guess string encoded as part of the webhook URL. Not encrypted. displayName: GitLab Webhook Secret from: '[a-zA-Z0-9]' generate: expression name: GITLAB_WEBHOOK_SECRET- description: A secret string used to configure the Generic webhook. displayName: Generic Webhook Secret from: '[a-zA-Z0-9]' generate: expression name: GENERIC_WEBHOOK_SECRET build_content/index.html
Create index.html in the build_content/ directory. We want a custom index.html page to show that we were successful in our deployment. This Page is what will be displayed when we access our Apache Web server after its been deployed
<html><header><title>Quay App Demo</title></header><body>Hello <Your name>!Let's test Quay.</body></html>
Create Git repo
Create a new repo in Github using your own Git instance. Open a web browser and login to your Git repo using your credentials. Open the “New” menu at the top of the page, then click “New Project”. Set Git repo access to private.
Commit files to Git
Commit application template and content to Git. This is an optional step but one that is used to demonstrate having and using a Git repo as part of the CI/CD chain.
Note - In this example, the new Git repo is named “quay-helloworld-demo”.
$ git init$ git add .$ git commit -a$ git remote add origin git@gitprd.<your.git.repo>:<username>/quay-helloworld-demo.git$ git push --set-upstream origin master
Create Quay Repository
Create a new Repository and robot account in Quay for this application. The robot account’s credentials will be used by OpenShift to pull/push images to the application’s private repository in Quay. This is for an automated CI/CD pipeline update and publishing using Github. Treat this as a type of Service account.
1. Quay web UI → Create New Repository → Enter “quay-app-demo” as the repository name → Leave Repository Visibility set to Private → Create Private Repository2. Settings → User and Robot Permissions → Expand drop down menu → Create New Robot Account → Enter name for robot account → Create Robot Account3. Toggle permissions for robot account to “Write” → Add Permission4. Click Robot account name to view Docker login info → Docker Configuration Download ***-auth.json file and rename to config.json (place outside application template/content directory)
Pull secrets
Create secret using robot account’s Docker credentials and link to service accounts.
Note - The “--for=pull” option should not be used when linking the secret to the builder service account.
$ oc create secret generic quay-pull-secret \ --from-file=".dockerconfigjson=config.json" \ --type='kubernetes.io/dockerconfigjson' -n quay-app-test $ oc secrets link default quay-pull-secret --for=pull -n quay-app-test$ oc secrets link builder quay-pull-secret -n quay-app-test
Git secret
Create secret for accessing GitLab with SSH private key. Add annotation to secret so that the secret will be used automatically by OpenShift when a build in this project references ssh://gitprd.ssc.tsc/*
Important - Modify path to ssh-privatekey as required so that it references private key used to access Git repo.
$ oc create secret generic gitlab-auth \ --from-file=ssh-privatekey=~/.ssh/id_rsa \ --type=kubernetes.io/ssh-auth \ -n quay-app-test$ oc annotate secret gitlab-auth \ 'build.openshift.io/source-secret-match-uri-1=\ssh://gitprd.ssc.tsc/*' --overwrite=true -n quay-app-test
Deploy application
# Generate 2 different long random string of characters using the openssl command.$ openssl rand -base64 40$ openssl rand -base64 40$ oc process -f httpd-quay-demo.yml \ -p=SOURCE_REPOSITORY_URL=ssh://git@gitprd.ssc.tsc/<repo/project>/quay-app-demo.git \ -p=GITLAB_WEBHOOK_SECRET=<1st long random string of characters> \ -p=GENERIC_WEBHOOK_SECRET=<2nd long random string of characters> \ -p=CONTEXT_DIR="build_content" \ | oc create -n quay-app-test -f -
Configure webhook in Gitlab
Configure webhook integration for the application repo in the Gitlab web UI. Value of GITLAB_WEBHOOK_SECRET must match value used when deploying the application in the previous step.
GitLab Repo --> Settings --> IntegrationsSet URL = https://master.ocp-dev01-bna.ssc.tsc:443/apis/build.openshift.io/v1/namespaces/\quay-app-test/buildconfigs/httpd-quay-demo/webhooks/<GITLAB_WEBHOOK_SECRET>/gitlabSecret Token = (leave blank)Trigger: Push eventsUncheck "Enable SSL verification"Click "Add webhook"
Application builds
Manually start build
A build can be manually initiated using the c start-build command.
$ oc start-build httpd-quay-demo -n quay-app-test
Rebuild application image
OpenShift will automatically rebuild the application container image anytime a new commit and push is made to the application’s Git repo.
# Update content$ vi quay-app-demo/build_content/index.html(update and save file)
# Note: If files are added, renamed, or moved, run “git add .”, then commit.
$ git commit -a$ git push
Deploy new application image
Assuming that a new application container image has been built, it can be deployed by using oc import-image
$ oc import-image httpd-quay-demo -n quay-app-testConclusion
Congratulations on your deployment of the Red Hat Quay servers using Clair for image scanning. This software was designed to be integrated into the OpenShift platform. By following some of our examples and suggestions, you have experienced what resources are required to deploy Quay. You have learned how we deploy Quay and Clair as projects in the OCP environment. You have also been given the challenge of deploying a “HelloWorld” application using your Quay environment. While there are other ways of deploying Quay, hopefully this exercise gave you some insight for planning your supported Enterprise solution.
As containers pave the way forward for ever faster and more productive DevOps, more refined and comprehensive toolkits are required to keep environments secure. Lack of container image security leaves vulnerabilities which can be exploited, but failing to implement containers can significantly cripple an organization’s ability to move forward with high speed development and rollouts.
According to the Tripwire's 2019 The State of Container Security Report, 42% of respondents said they were delaying container adoption due to security concerns, and 82% said they were considering restructuring security responsibilities and controls to prepare for container adoption or address existing issues with container security.
However, with a combination of Red Hat Quay and Clair, you can effectively secure outside container content and prevent security vulnerabilities while driving development forward in an agile environment.
About the Author
Mike McDonough is a Senior Red Hat Architect with Stone Door Group and is responsible for designing and blueprinting OpenShift and Ansible architectures for Fortune 500 customers spanning a broad array of Red Hat technologies. With a 25+ year career in enterprise IT, Mike has deep experience in cloud architecture, automation, performance tuning and information security. Stone Door Group offers rapid adoption of Red Hat cloud technologies with their OpenShift Container Platform Accelerator. To speak to Mike, drop us a line at letsdothis@stonedoorgroup.com
References:
https://containerjournal.com/2019/03/22/the-4-most-vulnerable-areas-of-container-security-in-2019/
https://www.gartner.com/smarterwithgartner/gartner-top-10-security-projects-for-2019/
https://access.redhat.com/solutions/3533201
https://github.com/VeerMuchandi/QuayOnOpenShift
https://access.redhat.com/RegistryAuthentication
https://www.tripwire.com/state-of-security/devops/organizations-container-security-incident/