Airgap Installation for OpenShift Operators

Airgap installation is always a challenging thing for OpenShift. By setting up a mirror registry and applying ImageContentSourcePolicy CRD to the cluster, we can instruct the OCI container engine to retrieve the source image from its mirrored image hosted in the mirror registry. This solves the airgap images for the cluster and the apps.

There is still a 3rd type of image for an airgap environment to tackle, that is the Operator related images. This paper documents the Operator based installation in an air-gapped environment, the steps, and the hiccups, and how it is being resolved.

OpenShift manages operators through Operator Catalog.

An Operator catalog is a repository of metadata that Operator Lifecycle Manager (OLM) can query to discover and install Operators and their dependencies on a cluster.

To install an operator in an air-gapped environment, we first need to sync the operator catalog. Starting from OCP4.6, the operator catalog is released as a container image, called an index image.

Create a custom pruned index image

First, we must know what are the available operators from the RedHat operator provider.

In an internet-connected box, run the following index image,

podman run -p50051:50051 --name operator-index -d registry.redhat.io/redhat/redhat-operator-index:v4.7

Then download the grpcurl tool, run

grpcurl -plaintext localhost:50051 api.Registry/ListPackages > packages.out

From the output, we know the operators we are interested in are named as

  • servicemeshoperator
  • serverless-operator
  • openshift-pipelines-operator-rh

We then use the opm tool to create a new pruned index image. The opm tool can be downloaded from this URL with OCP 4.7.2 release.

opm index prune -f registry.redhat.io/redhat/redhat-operator-index:v4.7 -p servicemeshoperator,serverless-operator,openshift-pipelines-operator-rh -t my-redhat-operator-index:v4.7

The -f tells where is the source image, -p is the list of operators to keep. Opm will prune the operators, and tag the image created. Notice the image will be saved in the local container storage.

As we need to get this image from a registry in the next step, we need to set up a local registry and push the image into it. Assuming this registry is ocp47-mirror-registry:5000 The step of setting it up can be referred to in my past paper. To push it into this registry,

podman tag my-redhat-operator-index:v4.7 ocp47-mirror-registry:5000/olm-mirror/my-redhat-operator-index:v4.7podman login ocp47-mirror-registry:5000podman push ocp47-mirror-registry:5000/olm-mirror/my-redhat-operator-index:v4.7

Mirror the catalog into the disk

mkdir -p olmmirror
cd olmmirror
oc adm catalog mirror ocp47-mirror-registry:5000/olm-mirror/my-redhat-operator-index:v4.7 file://mirror -a /run/user/1000/containers/auth.json

Use our pruned image from a registry, mirror the catalog (meta data and images) into the file system. Notice the file://mirror, will be mapped to “v2/mirror” in the current directory.

In my setup, the mirrored data is about 30GB and took 30 minutes.

Now tar v2 directory, and copy it to the target offline mirror registry server. As some of the files/directories are symbolic links, we have to use tar to keep the structure correct. It was found the loading will validate the type of the file.

It is noticed that the custom pruned index image is included in the mirror directory, located at

v2/mirror/olm-mirror/my-redhat-operator-index

olm-mirror is the namespace we used for the image.

Therefore we don’t need to copy and load this image separately.

Load the catalog into the mirrored registry

cd olmmirroroc adm catalog mirror file://mirror/olm-mirror/my-redhat-operator-index:v4.7 dev-airgap47-lb.dev-airgap47.ibmcloud.io.cpak:5000/olm-mirror -a /run/user/1000/containers/auth.json

However, the following error message pop up,

error: unable to retrieve source image file://mirror/olm-mirror/my-redhat-operator-index/openshift-serverless-1-tech-preview/eventing-channel-broker-rhel8 manifest sha256:72d7c6d34f6b83dd65f583b50620936358150f503658480344363f3acefa95fc: open v2/mirror/olm-mirror/my-redhat-operator-index/openshift-serverless-1-tech-preview/eventing-channel-broker-rhel8/manifests/sha256:72d7c6d34f6b83dd65f583b50620936358150f503658480344363f3acefa95fc: no such file or directory

Not exactly sure if this is a bug or expected behavior. To resolve it move all namespace level directory such as openshift-serverless-1-tech-preview into the olm-mirror/my-redhat-operator-index subdirectory to satisfy it.

Rerun it. Now we have all the images loaded into the mirror registry.

Update ImageContentSourcePolicy

The oc adm catalog mirror command creates an ImageContentSourcePolicy YAML file in a manifest subdirectory after running.

However check it out,

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
name: olm-mirror-my-redhat-operator-index
spec:
repositoryDigestMirrors:
- mirrors:
- dev-airgap47-lb.dev-airgap47.ibmcloud.io.cpak:5000/olm-mirror/openshift-serverless-1-tech-preview-serving-autoscaler-rhel8
source: mirror/olm-mirror/my-redhat-operator-index/openshift-serverless-1-tech-preview/serving-autoscaler-rhel8
- mirrors:
...

The source is not correct. Change all these prefix mirror/olm-mirror/my-redhat-operator-index to the actual source of the image, which is registry.redhat.io

Apply this YAML to the cluster. Monitor the nodes are updated for the mirror registry changes.

Apply the catalog source

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: mirror-operator-catalog
namespace: openshift-marketplace
spec:
sourceType: grpc
image: dev-airgap47-lb.dev-airgap47.ibmcloud.io.cpak:5000/olm-mirror/mirror-olm-mirror-my-redhat-operator-index:v4.7
displayName: Mirrored Operator Catalog
publisher: redhat

Apply this YAML.

Watch the OperatorHub is populated with the 3 operators we selected. We can then install the operators.