Explore Knative Build on On-Premise Kubernetes Cluster— IBM Cloud Private

This is the second part of my exploration of Knative on the on-premise Kubernetes cluster, ICP (IBM Cloud Private), Version 2.1.0.3. This time I focused on the Knative Build.

Knative Build uses some of the Google container tools to avoid the traditional way of build and upload a docker image by “docker build”. Here I will explore using Knative to build and push the docker images into the private registry of ICP.

At first part of this story, I explored the Knative build with my HTTPTestHandler go source code. I build the image and push into ICP private registry. In the second half part, I created a spring boot hello world app to show case the Knative’s source code to URL feature.

One of the use case for ICP is for those airgap environments where there is no internet connection. For the source code repository to Knative Build, I will be using the on-premise Gitlab setup, though I also publish the code in Github.

Prepare Service Account

First lets create the K8s service account that is used by the Knative Build. I associate some K8S secret to this account such as GitLab SSH Keys, ICP Private Registry Keys.

See the following yaml files. It was extracted from some Ruby rake code, where the #{} denotes some Ruby method or variables.

I encode the SSH private key, where its public key pair is loaded into Gitlab. The host key of the Gitlab can be obtained by running ssh-keyscan -p #{node_port} #{node_ip} in which the node_port and node_ip can be replaced with the K8s’ NodePort IP and port.

Notice the stringData in the second Secret, if the ‘stringData’ is specified as the key, the data are not required to be Base64 encoded.

Update Root CA

As the ICP’s private registry is using a self-signed cert, the default Knative build template is not able to run successfully. We have to update the root CA of the Knative Build image with the CA of the ICP’s private registry.

As what has been done in Part 1 of these exploration series, I update the root CA with ICP’s CA and create a configmap to hold the updated root CA.

cat ca-certificates.crt.old mycluster.icp.ca.crt > new.ca.crt
kubectl create cm my-root-ca --from-file=my-root-ca=new.ca.crt

This time, I create the configmap in the default namespace as I am going to run the Knative Build in the default namespace.

Knative Build

Based on the Kaniko template, we create the following Knative build CRD,

Here we create a volume from the configmap of “my-root-ca” and mount it over inside the step images to /kaniko/ssl/certs where kaniko read the root CA certs. This resolve the self-signed cert can not be verified issue when using private registry of ICP.

I am having issues with the GitLab ssh key. Seems like it could not populate my public key to the right location and therefore the SSH git retrieval is not successful. I switch the project to public to bypass this issue.

Notice for the URL of the gitlab, I am using the K8s service name of the gitlab. It is possible due to both my gitlab and knative build are in the same default namespace.

The debug step is a technique I used for debugging. Since all the steps images are under a same pod, the file system can be seen through. Putting the debug step in between allows me to hold the execution, go inside the container by running a kubectl exec, and release the executing by touch a file flag, touch /go-ahead-flagafter the debugging.

The hostname of the private registry mycluster.icpis resolvable, because the introduce of the Ubound DNS. (Please see more details in first part of the exploration Issue Fix 1: Use of Ubound to Allow Private Name Resolving)

Apply this yaml file, I see my image succesfully pushed into ICP’s private registry.

Knative Source to URL with Spring Boot Sample

Next I examined the source to URL feature of Knative. I am using a hello world spring boot application. See the source code in Github.

Lets use the JIB tool to build the docker image. Add the following plugin into the mvn pom.xml file,

<plugin>
    <groupId>com.google.cloud.tools</groupId>
    <artifactId>jib-maven-plugin</artifactId>
    <version>0.9.7</version>
    <configuration>
        <to>
            <image>mycluster.icp:8500/default/knative-springboot:v1</image>
        </to>
    </configuration>
</plugin>

The image built will be pushed into the private registry, tagged as mycluster.icp:8500/default/knative-springboot:v1.

Again to resolve the self-signed CA issue, I create the following Knative Build Template, named as “icp-jib-maven”

apiVersion: build.knative.dev/v1alpha1
kind: BuildTemplate
metadata:
  name: icp-jib-maven
spec:
  # parameters:
  #   - name: CMJAVAROOTCACRT
  #     description: The configmap for the ROOT java ca keystore
  #     default: java-root-ca-crt
  
  steps:
    - name: build-and-push
      image: gcr.io/cloud-builders/mvn
      args: ['compile', 'jib:build']
      volumeMounts:
      - name: ca-cert
        mountPath: /etc/ssl/certs/java
  volumes:
    - name: ca-cert
      configMap:
        # currently knative/build doesn't replace parameters under ".volumes". so hardcoded here
        name: my-java-root-ca
        items:
        - key: my-java-root-ca
          path: cacerts

We use the same technique here by mounting the updated CA root cert file through a configmap, overwriting the default java CA cert file.

Following shows how I update the Java CA root keystore.

1. Copy the default JAVA CA keystore from docker image

cd #{target_dir}
sudo docker run --rm -i --entrypoint=/bin/sh -v $(pwd):/data gcr.io/cloud-builders/mvn -c "cp /etc/ssl/certs/java/cacerts /data"

Download and save it as “java_cacerts” on my Windows 7 laptop.

2. Add the ICP CA cert filemycluster.icp.ca.crt into the keystore using Java keytool utility

set PATH=%path%;C:\\Program Files\\Java\\jre1.8.0_144\\bin
keytool -noprompt -keystore java_cacerts -storepass changeit -importcert -alias mycluster.icp -file mycluster.icp.ca.crt

3. Upload the java_cacerts file and create the configmap,

kubectl create cm my-java-root-ca --from-file=my-java-root-ca=#{target_dir}/java_cacerts

Now with this new build template, we can perform the source to URL with Knative. Apply the following yaml file.

apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
  name: my-knative-springboot
  namespace: default
spec:
  runLatest:
    configuration:
      build:
        serviceAccountName: my-build-service-account
        source:
          git:
            url: http://turbulent-penguin-gitlab-ce/wenzm/knative-springboot.git
            revision: master
        template:
          name: icp-jib-maven
      revisionTemplate:
        spec:
          container:
            image: mycluster.icp:8500/default/knative-springboot:v1
            imagePullPolicy: Always

Once the revision pods are running, I can identify the domain and browser to the springboot hello page.

Enhancement Request on the Template Parameters for Volumes

I have filed an issue on the Knative Build to request takeing in parameters on Volumes. If this can be achieved then we don’t need to hard code the configmap in the template for Volume.