OpenShift Operator Hub on Restricted Networks

OpenShift’s Operator Lifecycle Manager (AKA Operator Hub) is designed to work with internet access. But, sometimes we have to deal with an environment that does not have internet access (disconnected environment), thus preventing us from installing Operators.

To make things harder, some operators need related images (declared on its CSV) to work properly.

Creating a local disconnected registry and upload operator images to it is not enough to get things up and running.

So, what should I do to solve this?

Custom Repository Catalog

Before you proceed, it is important to understand a few things. Some Red Hat Operators are self contained, and don’t need related images. But some of them do. The process of pruning an index catalog is suitable for Operators that don’t need extra images. You must know how the operator was built to be able to consider which method is right for you.

Use a command like the one below to get some information about the operator.

# oc get packagemanifests quay-operator -o yaml
(...)
relatedImages:
- registry.redhat.io/quay/clair-rhel8@sha256:bc7eab15a3047951f559fa9e5eb1c4f8993796307a1404198e2391efd7759b27
- registry.redhat.io/quay/quay-builder-rhel8@sha256:7175525fc0db5ee2f35230d029d39d8fe6f2b733d7a91b89d8144b6216bb61e6
- registry.redhat.io/quay/quay-builder-qemu-rhcos-rhel8@sha256:ea4fca6fa4a1dbf32ed0d300e930c46fa0b53cdeeaf1cb1b055abda1e161a43f
- registry.redhat.io/rhel8/postgresql-10@sha256:a52ad402458ec8ef3f275972c6ebed05ad64398f884404b9bb8e3010c5c95291
- registry.redhat.io/rhel8/redis-6@sha256:b15335d88b240bd6bbedeb7fe26f0a1de2f6c330af9758b0fb2edab7ca69f6df
- registry.redhat.io/quay/quay-operator-rhel8@sha256:2b17143553e30e62aa15f9c63d0be55b89dd02c5d032aa58c279dbb8bb5100b1
- registry.redhat.io/quay/quay-rhel8@sha256:0bfb107a07c6b275d8064e3fba32836cf94d20383edb115da6ee25d1fa52ec9d
(...)

What options are available?

There is an very easy approach, which is to mirror the entire Operator Lifecycle Manager (OLM) catalog, but this will take up to 1.5TB of disk space as the time of this writing (November 2022), making this approach nearly impossible to adopt on the vast majority of disconnected OpenShift clusters.

Maybe that’s not what you’re looking for.

Mirror Operators With Related Images

Some operators require related Images that aren’t downloaded in the index prune stage. Look at your Operator CSV to search if it requires additional images. For an example, OCS requires the following related images:

image:
registry.redhat.io/ocs4/ocs-operator-bundle@sha256:64cdf16dbf585e2829b1c145bc2f4649716b0e8487bd863b4f2578d64a14ee9d"
relatedImages:
registry.redhat.io/rhceph/rhceph-4-rhel8@sha256:4b16d6f54a9ae1e43ab0f9b76f1b0860cc4feebfc7ee0e797937fc9445c5bb0a
registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:e90420f29e9cbf9e7c23ad910c721bea793811fc76c852aef2fa826aa6c3b2c6
registry.redhat.io/ocs4/mcg-core-rhel8@sha256:2e9993a365a1032d84610563817453aade2de343e1e1abfc09b4e8d7f24f91c0
registry.redhat.io/ocs4/ocs-rhel8-
(...)
  • If that’s the case, you should upload the related images to your private registry in the same catalog as the operator bundle, otherwise your Operator installation will fail.

Now that you understand that some operators need additional content, let’s understand how to solve this issue.

Update Catalog With Related Images

You can do the whole process of creating a new catalog containing all the necessary content 100% manually, but everything becomes a little more time-consuming and complex.

To make things easier, you can use a python script developed by the Red Hat CoP to download the operator images from a predefined list, alongside with it’s related images.

  • You should run it from your bastion host.

Assumptions

  • OpenShift Version: 4.8
  • Registry: You already have a local private registry up and running.
  • Local Registry URL: registry.rhbrlabs.com
  • Bastion has access to the internet.

You don’t have a local private registry? I wrote about building one. Click here to check it out.

External Registry Access

Install the python script requisites according to the README.md file. Then do the logins to the required registries:

[root@bastion ocp]# podman login registry.rhbrlabs.com:5000
[root@bastion ocp]# podman login registry.redhat.io
[root@bastion ocp]# podman login quay.io

This will generate the auth.json file (/run/user/0/containers/auth.json). Save it to be able to use it later.

Mirror List

Edit the files offline_operators_list and offline_operators_list.yaml to include the operators you wish to mirror.

In my scenario, I needed Quay and OCS with their related images. These operators also need local-storage (not declared in the CSV). It is also important to install cluster-logging.

  • I’d recommend always enabling both.
[root@bastion openshift-disconnected-operators-master]# cat offline_operators_list
local-storage-operator
ocs-operator
quay-operator
openshift-marketplace
ocs-must-gather-rhel8
odf-operator
cluster-logging

[root@bastion openshift-disconnected-operators-master]# cat offline_operators_list.yaml
operators:
- name: local-storage-operator
- name: ocs-operator
- name: quay-operator
- name: openshift-marketplace
- name: ocs-must-gather-rhel8
- name: odf-operator
- name: cluster-logging

Now, run the mirror process:

[root@bastion openshift-disconnected-operators-master]# ./mirror-operator-catalogue.py \
--catalog-version v4.8 \
--authfile ~/ocp/localregistry/openshift-disconnected-operators-master/auth.json \
--registry-olm registry.rhbrlabs.com:5000/repository \
--registry-catalog registry.rhbrlabs.com:5000/repository \
--operator-file ~/ocp/localregistry/openshift-disconnected-operators-master/offline_operators_list \
--icsp-scope=namespace \
--ocp-version 4.8 \
--operator-channel 4.8

At the end of the process, you should see messages like this:

Finished mirroring related images.
Creating Image Content Source Policy YAML...
Creating Mapping File...
Creating Image manifest file...
Creating Catalog Source YAML...
Catalogue creation and image mirroring complete
See Publish folder for the image content source policy and catalog source yaml files to apply to your cluster

The mirror process will generate some YAML files in the publish directory.

[root@bastion openshift-disconnected-operators-master]# ls publish/
image_manifest.txt mapping.txt mirror_log.txt olm-icsp.yaml rh-catalog-source.yaml

Custom Mirrored Catalog

Disable the default OLM catalog sources:

[root@bastion publish]# oc patch OperatorHub cluster --type json -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'

Apply the yaml files in the publish directory. The image content source policy will create a new MCO (Machine Config Operator) render which will start a rolling reboot of your cluster nodes. Keep this in mind before performing this step.

  • You have to wait until that is complete before attempting to install operators from the catalog.
[root@bastion publish]# oc create -f olm-icsp.yaml
imagecontentsourcepolicy.operator.openshift.io/olm-image-content-source created

[root@bastion publish]# oc create -f rh-catalog-source.yaml
catalogsource.operators.coreos.com/custom-redhat-operators created

After Operator Lifecycle Manager (OLM) automatically polls the index image referenced in the catalog source at its regular interval (30 minutes), verify that the new packages are successfully added:

[root@bastion localregistry]# oc get packagemanifests -n openshift-marketplace
NAME CATALOG AGE
quay-operator Red Hat Operators 42m
ocs-operator Red Hat Operators 42m
cluster-logging Red Hat Operators 42m
local-storage-operator Red Hat Operators 42m

Now go to your Openshift Operator Hub and check the available Operators.

Disconnected Custom OLM
Disconnected Custom OLM

The Operators added has its related images, making them complete in terms of requisites. Install them to check if everything is in place.

You can have as many custom catalogs as you need. Go to OpenShift Global Configuration and search for Operator Hub. The list of available catalogs can be found there.

Disconnected Operators Installed
Disconnected Operators Installed

That’s it. This one was a little tricky to sort out. Hope this post was useful to you. Drop a message if you have questions. See ya!

It worked!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Andre Rocha

I'm just a SysAdmin with some experience in OpenSource, DevOps and Datacenter Services, who likes to share knowledge.