Back to index

4.12.0-rc.2

Jump to: Complete Features | Incomplete Features | Complete Epics | Incomplete Epics | Other Complete | Other Incomplete |

Changes from 4.11.49

Note: this page shows the Feature-Based Change Log for a release

Complete Features

These features were completed when this image was assembled

Feature OBSDA-143: Add runbook_url to alerts in the OCP UI

1. Proposed title of this feature request
Add runbook_url to alerts in the OCP UI

2. What is the nature and description of the request?
If an alert includes a runbook_url label, then it should appear in the UI for the alert as a link.

3. Why does the customer need this? (List the business requirements here)
Customer can easily reach the alert runbook and be able to address their issues.

4. List any affected packages or components.

Feature OCPPLAN-8108: The details of this Jira Card are restricted (Only Red Hat employees and contractors)
The details of this Jira Card are restricted (Only Red Hat employees and contractors)
Epic OCPBUILD-30: Build Rebases OCP 4.11

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

  • Rebase OpenShift components to k8s v1.24

Why is this important?

  • Rebasing ensures components work with the upcoming release of Kubernetes
  • Address tech debt related to upstream deprecations and removals.

Scenarios

  1. ...

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.
  • ...

Dependencies (internal and external)

  1. k8s 1.24 release

Previous Work (Optional):

  1. …

Open questions::

  1. …

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>

Incomplete Features

When this image was assembled, these features were not yet completed. Therefore, only the Jira Cards included here are part of this release

Feature OCPPLAN-4577: The details of this Jira Card are restricted (Only Red Hat employees and contractors)
The details of this Jira Card are restricted (Only Red Hat employees and contractors)
Epic MGMT-8890: [kube-api] Integration of multiple architectures support / heterogeneous clusters

Epic Goal

  • Enabling integration of single hub cluster to install both ARM and x86 spoke clusters
  • Enabling support for heterogeneous OCP clusters
  • document requirements deployment flows
  • support in disconnected environment

Why is this important?

  • clients request

Scenarios

  1. Users manage both ARM and x86 machines, we should not require to have two different hub clusters
  2. Users manage a mixed architecture clusters without requirement of all the nodes to be of the same architecture

Acceptance Criteria

  • Process is well documented
  • we are able to install in a disconnected environment
Story MGMT-11348: Assisted images should be multiarch

We have a set of images

  • quay.io/edge-infrastructure/assisted-installer-agent:latest
  • quay.io/edge-infrastructure/assisted-installer-controller:latest
  • quay.io/edge-infrastructure/assisted-installer:latest

that should become multiarch images. This should be done both in upstream and downstream.

As a reference, we have built internally those images as multiarch and made them available as

  • registry.redhat.io/rhai-tech-preview/assisted-installer-agent-rhel8:latest
  • registry.redhat.io/rhai-tech-preview/assisted-installer-reporter-rhel8:latest
  • registry.redhat.io/rhai-tech-preview/assisted-installer-rhel8:latest

They can be consumed by the Assisted Serivce pod via the following env

    - name: AGENT_DOCKER_IMAGE
      value: registry.redhat.io/rhai-tech-preview/assisted-installer-agent-rhel8:latest
    - name: CONTROLLER_IMAGE
      value: registry.redhat.io/rhai-tech-preview/assisted-installer-reporter-rhel8:latest
    - name: INSTALLER_IMAGE
      value: registry.redhat.io/rhai-tech-preview/assisted-installer-rhel8:latest
Epic MIXEDARCH-26: OLM support for node selection

OLM would have to support a mechanism like podAffinity which allows multiple architecture values to be specified which enables it to pin operators to the matching architecture worker nodes

Ref: https://github.com/openshift/enhancements/pull/1014

 

Task OPRUN-2690: [downstream] bring Subscription API and OLM controller changes to the downstream

Cut a new release of the OLM API and update OLM API dependency version (go.mod) in OLM package; then
Bring the upstream changes from OLM-2674 to the downstream olm repo.

A/C:

 - New OLM API version release
 - OLM API dependency updated in OLM Project
 - OLM Subscription API changes  downstreamed
 - OLM Controller changes  downstreamed
 - Changes manually tested on Cluster Bot

Feature OCPPLAN-4927: The details of this Jira Card are restricted (Only Red Hat employees and contractors)
The details of this Jira Card are restricted (Only Red Hat employees and contractors)
Feature OCPPLAN-6007: OpenShift Core Networking Improvements

Feature Overview

We drive OpenShift cross-market customer success and new customer adoption with constant improvements and feature additions to the existing capabilities of our OpenShift Core Networking (SDN and Network Edge). This feature captures that natural progression of the product.

Goals

  • Feature enhancements (performance, scale, configuration, UX, ...)
  • Modernization (incorporation and productization of new technologies)

Requirements

  • Core Networking Stability
  • Core Networking Performance and Scale
  • Core Neworking Extensibility (Multus CNIs)
  • Core Networking UX (Observability)
  • Core Networking Security and Compliance

In Scope

  • Network Edge (ingress, DNS, LB)
  • SDN (CNI plugins, openshift-sdn, OVN, network policy, egressIP, egress Router, ...)
  • Networking Observability

Out of Scope

There are definitely grey areas, but in general:

  • CNV
  • Service Mesh
  • CNF

Documentation Considerations

Questions to be addressed:

  • What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
  • Does this feature have doc impact?
  • New Content, Updates to existing content, Release Note, or No Doc Impact
  • If unsure and no Technical Writer is available, please contact Content Strategy.
  • What concepts do customers need to understand to be successful in [action]?
  • How do we expect customers will use the feature? For what purpose(s)?
  • What reference material might a customer want/need to complete [action]?
  • Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
  • What is the doc impact (New Content, Updates to existing content, or Release Note)?
Epic NE-682: Capture Metrics and Telemetry Associated with Use of Routes and Shards

Goal: Provide queryable metrics and telemetry for cluster routes and sharding in an OpenShift cluster.

Problem: Today we test OpenShift performance and scale with best-guess or anecdotal evidence for the number of routes that our customers use. Best practices for a large number of routes in a cluster is to shard, however we have no visibility with regard to if and how customers are using sharding.

Why is this important? These metrics will inform our performance and scale testing, documented cluster limits, and how customers are using sharding for best practice deployments.

Dependencies (internal and external):

Prioritized epics + deliverables (in scope / not in scope):

Not in scope:

Estimate (XS, S, M, L, XL, XXL):

Previous Work:

Open questions:

Acceptance criteria:

Epic Done Checklist:

  • CI - CI Job & Automated tests: <link to CI Job & automated tests>
  • Release Enablement: <link to Feature Enablement Presentation> 
  • DEV - Upstream code and tests merged: <link to meaningful PR orf GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
  • Notes for Done Checklist: 
    • Adding links to the above checklist with multiple teams contributing; select a meaningful reference for this Epic.
    • Checklist added to each Epic in the description, to be filled out as phases are completed - tracking progress towards “Done” for the Epic.
Story CFE-551: As a developer I want to support sending of metrics from OpenShift clusters to Red Hat premises for Telemeter by allowlisting metrics on the cluster side

Description:

As described in the Metrics to be sent via telemetry section of the Design Doc, the following metrics is needed to be sent from OpenShift cluster to Red Hat premises:

  • Minimum Routes per Shard
    • Recording Rule – cluster:route_metrics_controller_routes_per_shard:min  : min(route_metrics_controller_routes_per_shard)
    • Gives the minimum value of Routes per Shard.
  • Maximum Routes per Shard
    • Recording Rule – cluster:route_metrics_controller_routes_per_shard:max  : max(route_metrics_controller_routes_per_shard)
    • Gives the maximum value of Routes per Shard.
  • Average Routes per Shard
    • Recording Rule – cluster:route_metrics_controller_routes_per_shard:avg  : avg(route_metrics_controller_routes_per_shard)
    • Gives the average value of Routes per Shard.
  • Median Routes per Shard
    • Recording Rule – cluster:route_metrics_controller_routes_per_shard:median  : quantile(0.5, route_metrics_controller_routes_per_shard)
    • Gives the median value of Routes per Shard.
  • Number of Routes summed by TLS Termination type
    • Recording Rule – cluster:openshift_route_info:tls_termination:sum : sum (openshift_route_info) by (tls_termination)
    • Gives the number of Routes for each tls_termination value. The possible values for tls_termination are edge, passthrough and reencrypt. 

The metrics should be allowlisted on the cluster side.

The steps described in Sending metrics via telemetry are needed to be followed. Specifically step 5.

Depends on CFE-478.

Acceptance Criteria:

  • Support for sending the above mentioned metrics from OpenShift clusters to the Red Hat premises by allowlisting metrics on the cluster side
Story CFE-478: As a developer I want to support exporting of additional metrics from Cluster Ingress Operator

Description:

As described in the Design Doc, the following information is needed to be exported from Cluster Ingress Operator:

  • Number of routes/shard

Design 2 will be implemented as part of this story.

 

Acceptance Criteria:

  • Support for exporting the above mentioned metrics by Cluster Ingress Operator
Feature OCPPLAN-7589: The details of this Jira Card are restricted (Only Red Hat employees and contractors)
The details of this Jira Card are restricted (Only Red Hat employees and contractors)
Epic CONSOLE-3160: Make console operator optional

Epic Goal

  • Make it possible to disable the console operator at install time, while still having a supported+upgradeable cluster.

Why is this important?

  • It's possible to disable console itself using spec.managementState in the console operator config. There is no way to remove the console operator, though. For clusters where an admin wants to completely remove console, we should give the option to disable the console operator as well.

Scenarios

  1. I'm an administrator who wants to minimize my OpenShift cluster footprint and who does not want the console installed on my cluster

Acceptance Criteria

  • It is possible at install time to opt-out of having the console operator installed. Once the cluster comes up, the console operator is not running.

Dependencies (internal and external)

  1. Composable cluster installation

Previous Work (Optional):

  1. https://docs.google.com/document/d/1srswUYYHIbKT5PAC5ZuVos9T2rBnf7k0F1WV2zKUTrA/edit#heading=h.mduog8qznwz
  2. https://docs.google.com/presentation/d/1U2zYAyrNGBooGBuyQME8Xn905RvOPbVv3XFw3stddZw/edit#slide=id.g10555cc0639_0_7

Open questions::

  1. The console operator manages the downloads deployment as well. Do we disable the downloads deployment? Long term we want to move to CLI manager: https://github.com/openshift/enhancements/blob/6ae78842d4a87593c63274e02ac7a33cc7f296c3/enhancements/oc/cli-manager.md

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story CONSOLE-3220: Update console-operator's manifests with capability annotation

In the console-operator repo we need to add `capability.openshift.io/console` annotation to all the manifests that the operator either contains creates on the fly.

 

Manifests are currently present in /bindata and /manifest directories.

 

Here is example of the insights-operator change.

Here is the overall enhancement doc.

 

Feature OCPPLAN-7744: Declarative Index Config

This is a epic bucket for all activities surrounding the creation of declarative approach to release and maintain OLM catalogs.

Epic OPRUN-2211: Operator Authors have a clean veneer for releasing bundles (File based config/declarative config)

Epic Goal

  • Allow Operator Authors to easily change the layout of the update graph in a single location so they can version/maintain/release it via git and have more approachable controls about graph vertices than today's replaces, skips and/or skipRange taxonomy
  • Allow Operators authors to have control over channel and bundle channel membership

Why is this important?

  • The imperative catalog maintenance approach so far with opm is being moved to a declarative format (OLM-2127 and OLM-1780) moving away from bundle-level controls but the update graph properties are still attached to a bundle
  • We've received feedback from the RHT internal developer community that maintaining and reasoning about the graph in the context of a single channel is still too hard, even with visualization tools
  • making the update graph easily changeable is important to deliver on some of the promises of declarative index configuration
  • The current interface for declarative index configuration still relies on skips, skipRange and replaces to shape the graph on a per-bundle level - this is too complex at a certain point with a lot of bundles in channels, we need to something at the package level

Scenarios

  1. An Operator author wants to release a new version replacing the latest version published previously
  2. After additional post-GA testing an Operator author wants to establish a new update path to an existing released version from an older, released version
  3. After finding a bug post-GA an Operator author wants to temporarily remove a known to be problematic update path
  4. An automated system wants to push a bundle inbetween an existing update path as a result of an Operator (base) image rebuild (Freshmaker use case)
  5. A user wants to take a declarative graph definition and turn it into a graphical image for visually ensuring the graph looks like they want
  6. An Operator author wants to promote a certain bundle to an additional / different channel to indicate progress in maturity of the operator.

Acceptance Criteria

  • The declarative format has to be user readable and terse enough to make quick modifications
  • The declarative format should be machine writeable (Freshmaker)
  • The update graph is declared and modified in a text based format aligned with the declarative config
  • it has to be possible to add / removes edges at the leave of the graph (releasing/unpublishing a new version)
  • it has to be possible to add/remove new vertices between existing edges (releasing/retracting a new update path)
  • it has to be possible to add/remove new edges in between existing vertices (releasing/unpublishing a version inbetween, freshmaker user case)
  • it has to be possible to change the channel member ship of a bundle after it's published (channel promotion)
  • CI - MUST be running successfully with tests automated
  • it has to be possible to add additional metadata later to implement OLM-2087 and OLM-259 if required

Dependencies (internal and external)

  1. Declarative Index Config (OLM-2127)

Previous Work:

  1. Declarative Index Config (OLM-1780)

Related work

Open questions:

  1. What other manipulation scenarios are required?
    1. Answer: deprecation of content in the spirit of OLM-2087
    2. Answer: cross-channel update hints as described in OLM-2059 if that implementation requires it

 

When working on this Epic, it's important to keep in mind this other potentially related Epic: https://issues.redhat.com/browse/OLM-2276

 

Story OPRUN-2726: downstream opm server change to use pre-existent cache

Jira Description

As an OPM maintainer, I want to downstream the PR for (OCP 4.12 ) and backport it to OCP 4.11 so that IIB will NOT be impacted by the changes when it upgrades the OPM version to use the next/future opm upstream release (v1.25.0).

Summary / Background

IIB(the downstream service that manages the indexes) uses the upstream version and if they bump the OPM version to the next/future (v1.25.0) release with this change before having the downstream images updated then: the process to manage the indexes downstream will face issues and it will impact the distributions. 

Acceptance Criteria

  • The changes in the PR are available for the releases which uses FBC -> OCP 4.11, 4.12

Definition of Ready

  • PRs merged into downstream OCP repos branches 4.11/4.12

Definition of Done

  • We checked that the downstream images are with the changes applied (i.e.: we can try to verify in the same way that we checked if the changes were in the downstream for the fix OLM-2639 )
Story OPRUN-2753: (downstream) Feature: opm alpha render-veneer support for pipe stdin

enhance the veneer rendering to be able to read the input veneer data from stdin, via a pipe, in a manner similar to https://dev.to/napicella/linux-pipes-in-golang-2e8j

then the command could be used in a manner similar to many k8s examples like

```shell
opm alpha render-veneer semver -o yaml < infile > outfile
```

Upstream issue link: https://github.com/operator-framework/operator-registry/issues/1011

Feature OCPPLAN-7783: CSI Cloud Providers

Feature Overview
Provide CSI drivers to replace all the intree cloud provider drivers we currently have. These drivers will probably be released as tech preview versions first before being promoted to GA.

Goals

  • Framework for rapid creation of CSI drivers for our cloud providers
  • CSI driver for AWS EBS
  • CSI driver for AWS EFS
  • CSI driver for GCP
  • CSI driver for Azure
  • CSI driver for VMware vSphere
  • CSI Driver for Azure Stack
  • CSI Driver for Alicloud
  • CSI Driver for IBM Cloud

Requirements

Requirement Notes isMvp?
Framework for CSI driver  TBD Yes
Drivers should be available to install both in disconnected and connected mode   Yes
Drivers should upgrade from release to release without any impact   Yes
Drivers should be installable via CVO (when in-tree plugin exists)    

Out of Scope

This work will only cover the drivers themselves, it will not include

  • enhancements to the CSI API framework
  • the migration to said drivers from the the intree drivers
  • work for non-cloud provider storage drivers (FC-SAN, iSCSI) being converted to CSI drivers

Background, and strategic fit
In a future Kubernetes release (currently 1.21) intree cloud provider drivers will be deprecated and replaced with CSI equivalents, we need the drivers created so that we continue to support the ecosystems in an appropriate way.

Assumptions

  • Storage SIG won't move out the changeover to a later Kubernetes release

Customer Considerations
Customers will need to be able to use the storage they want.

Documentation Considerations

  • Target audience: cluster admins
  • Updated content: update storage docs to show how to use these drivers (also better expose the capabilities)
Epic STOR-880: CSI operator for Google Cloud Filestore (Tech Preview)

This Epic is to track the GA of this feature

Goal

  • Make available the Google Cloud File Service via a CSI driver, it is desirable that this implementation has dynamic provisioning
  • Without GCP filestore support, we are limited to block / RWO only (GCP PD 4.8 GA)
  • Align with what we support on other major public cloud providers.

Why is this important?

  • There is a know storage gap with google cloud where only block is supported
  • More customers deploying on GCE and asking for file / RWX storage.

Scenarios

  1. Install the CSI driver
  2. Remove the CSI Driver
  3. Dynamically provision a CSI Google File PV*
  4. Utilise a Google File PV
  5. Assess optional features such as resize & snapshot

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • ...

Dependencies (internal and external)

  1. ...

Previous Work (Optional):

  1. …

Open questions::

  1. …

Customers::

  • Telefonica Spain
  • Deutsche Bank

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story STOR-956: Build images for GCP Filestore CSI Driver and Operator

As an OCP user, I want images for GCP Filestore CSI Driver and Operator, so that I can install them on my cluster and utilize GCP Filestore shares.

Feature OCPPLAN-7784: Storage Maintainability

We need to continue to maintain specific areas within storage, this is to capture that effort and track it across releases.

Goals

  • To allow OCP users and cluster admins to detect problems early and with as little interaction with Red Hat as possible.
  • When Red Hat is involved, make sure we have all the information we need from the customer, i.e. in metrics / telemetry / must-gather.
  • Reduce storage test flakiness so we can spot real bugs in our CI.

Requirements

Requirement Notes isMvp?
Telemetry   No
Certification   No
API metrics   No
     

Out of Scope

n/a

Background, and strategic fit
With the expected scale of our customer base, we want to keep load of customer tickets / BZs low

Assumptions

Customer Considerations

Documentation Considerations

  • Target audience: internal
  • Updated content: none at this time.

Notes

In progress:

  • CI flakes:
    • Configurable timeouts for e2e tests
      • Azure is slow and times out often
      • Cinder times out formatting volumes
      • AWS resize test times out

 

High prio:

  • Env. check tool for VMware - users often mis-configure permissions there and blame OpenShift. If we had a tool they could run, it might report better errors.
    • Should it be part of the installer?
    • Spike exists
  • Add / use cloud API call metrics
    • Helps customers to understand why things are slow
    • Helps build cop to understand a flake
      • With a post-install step that filters data from Prometheus that’s still running in the CI job.
    • Ideas:
      • Cloud is throttling X% of API calls longer than Y seconds
      • Attach / detach / provisioning / deletion / mount / unmount / resize takes longer than X seconds?
    • Capture metrics of operations that are stuck and won’t finish.
      • Sweep operation map from executioner???
      • Report operation metric into the highest bucket after the bucket threshold (i.e. if 10minutes is the last bucket, report an operation into this bucket after 10 minutes and don’t wait for its completion)?
      • Ask the monitoring team?
    • Include in CSI drivers too.
      • With alerts too

Unsorted

  • As the number of storage operators grows, it would be grafana board for storage operators
    • CSI driver metrics (from CSI sidecars + the driver itself  + its operator?)
    • CSI migration?
  • Get aggregated logs in cluster
    • They're rotated too soon
    • No logs from dead / restarted pods
    • No tools to combine logs from multiple pods (e.g. 3 controller managers)
  • What storage issues customers have? it was 22% of all issues.
    • Insufficient docs?
    • Probably garbage
  • Document basic storage troubleshooting for our supports
    • What logs are useful when, what log level to use
    • This has been discussed during the GSS weekly team meeting; however, it would be beneficial to have this documented.
  • Common vSphere errors, their debugging and fixing. 
  • Document sig-storage flake handling - not all failed [sig-storage] tests are ours
Epic STOR-500: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Story STOR-794: Block 4.12 to 4.13 upgrade if Hardware version < 15 or vSphere < 7.0

The End of General support for vSphere 6.7 will be on October 15, 2022. So, vSphere 6.7 will be deprecated for 4.11.

We want to encourage vSphere customers to upgrade to vSphere 7 in OCP 4.11 since VMware is EOLing (general support) for vSphere 6.7 in Oct 2022.

We want the cluster Upgradeable=false + have a strong alert pointing to our docs / requirements.

related slack: https://coreos.slack.com/archives/CH06KMDRV/p1647541493096729

Epic STOR-857: OCP 4.12 release chores

Epic Goal

  • Update all images that we ship with OpenShift to the latest upstream releases and libraries.
  • Exact content of what needs to be updated will be determined as new images are released upstream, which is not known at the beginning of OCP development work. We don't know what new features will be included and should be tested and documented. Especially new CSI drivers releases may bring new, currently unknown features. We expect that the amount of work will be roughly the same as in the previous releases. Of course, QE or docs can reject an update if it's too close to deadline and/or looks too big.

Traditionally we did these updates as bugfixes, because we did them after the feature freeze (FF). Trying no-feature-freeze in 4.12. We will try to do as much as we can before FF, but we're quite sure something will slip past FF as usual.

Why is this important?

  • We want to ship the latest software that contains new features and bugfixes.

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.
Story STOR-863: Chore: Update vmware-vsphere-csi-driver to the latest release

Update the driver to the latest upstream release. Notify QE and docs with any new features and important bugfixes that need testing or documentation.

(Using separate cards for each driver because these updates can be more complicated)

Story STOR-864: Chore: Update azure-disk-csi-driver to the latest release

Update the driver to the latest upstream release. Notify QE and docs with any new features and important bugfixes that need testing or documentation.

(Using separate cards for each driver because these updates can be more complicated)

Story STOR-867: Chore: Update ibm-vpc-block-csi-driver to the latest release

Update the driver to the latest upstream release. Notify QE and docs with any new features and important bugfixes that need testing or documentation.

This includes ibm-vpc-node-label-updater!

(Using separate cards for each driver because these updates can be more complicated)

Story STOR-1060: Chore: Update ibm-vpc-block-csi-driver to v5.0.0

There is a new driver release 5.0.0 since the last rebase that includes snapshot support:

https://github.com/kubernetes-sigs/ibm-vpc-block-csi-driver/releases/tag/v5.0.0

Rebase the driver on v5.0.0 and update the deployments in ibm-vpc-block-csi-driver-operator.
There are no corresponding changes in ibm-vpc-node-label-updater since the last rebase.

Story STOR-860: Chore: Update aws-ebs-csi-driver to the latest release

Update the driver to the latest upstream release. Notify QE and docs with any new features and important bugfixes that need testing or documentation.

(Using separate cards for each driver because these updates can be more complicated)

Story STOR-858: Chore: update libraries in all operators

Update all OCP and kubernetes libraries in storage operators to the appropriate version for OCP release.

This includes (but is not limited to):

  • Kubernetes:
    • client-go
    • controller-runtime
  • OCP:
    • library-go
    • openshift/api
    • openshift/client-go
    • operator-sdk

Operators:

  • aws-ebs-csi-driver-operator 
  • aws-efs-csi-driver-operator
  • azure-disk-csi-driver-operator
  • azure-file-csi-driver-operator
  • openstack-cinder-csi-driver-operator
  • gcp-pd-csi-driver-operator
  • gcp-filestore-csi-driver-operator
  • manila-csi-driver-operator
  • ovirt-csi-driver-operator
  • vmware-vsphere-csi-driver-operator
  • alibaba-disk-csi-driver-operator
  • ibm-vpc-block-csi-driver-operator
  • csi-driver-shared-resource-operator

 

  • cluster-storage-operator
  • csi-snapshot-controller-operator
  • local-storage-operator
  • vsphere-problem-detector
Story STOR-862: Chore: Update gcp-pd-csi-driver to the latest release

Update the driver to the latest upstream release. Notify QE and docs with any new features and important bugfixes that need testing or documentation.

(Using separate cards for each driver because these updates can be more complicated)

Story STOR-865: Chore: Update azure-file-csi-driver to the latest release

Update the driver to the latest upstream release. Notify QE and docs with any new features and important bugfixes that need testing or documentation.

(Using separate cards for each driver because these updates can be more complicated)

Feature OCPPLAN-7786: CSI Migration

Epic Goal

  • Enable the migration from a storage intree driver to a CSI based driver with minimal impact to the end user, applications and cluster
  • These migrations would include, but are not limited to:
    • CSI driver for AWS EBS
    • CSI driver for GCP
    • CSI driver for Azure (file and disk)
    • CSI driver for VMware vSphere

Why is this important?

  • OpenShift needs to maintain it's ability to enable PVCs and PVs of the main storage types
  • CSI Migration is getting close to GA, we need to have the feature fully tested and enabled in OpenShift
  • Upstream intree drivers are being deprecated to make way for the CSI drivers prior to intree driver removal

Scenarios

  1. User initiated move to from intree to CSI driver
  2. Upgrade initiated move from intree to CSI driver
  3. Upgrade from EUS to EUS

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.
  • ...

Dependencies (internal and external)

  1. ...

Previous Work (Optional):

  1. …

Open questions::

  1. …

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Epic STOR-690: CSI Migration for GCE (GA)

This Epic tracks the GA of this feature

Epic Goal

Why is this important?

  • OpenShift needs to maintain it's ability to enable PVCs and PVs of the main storage types
  • CSI Migration is getting close to GA, we need to have the feature fully tested and enabled in OpenShift
  • Upstream intree drivers are being deprecated to make way for the CSI drivers prior to intree driver removal

Scenarios

  1. User initiated move to from intree to CSI driver
  2. Upgrade initiated move from intree to CSI driver
  3. Upgrade from EUS to EUS

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.
  • ...

Dependencies (internal and external)

  1. ...

Previous Work (Optional):

  1. …

Open questions::

  1. …

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story STOR-752: Change the default StorageClass to the CSI one

On new installations, we should make the StorageClass created by the CSI operator the default one. 

However, we shouldn't do that on an upgrade scenario. The main reason is that users might have set  a different quota on the CSI driver Storage Class.

Exit criteria:

  • New clusters get the CSI Storage Class as the default one.
  • Existing clusters don't get their default Storage Classes changed.
Epic STOR-575: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Story STOR-764: Change the default StorageClass to the CSI one

On new installations, we should make the StorageClass created by the CSI operator the default one. 

However, we shouldn't do that on an upgrade scenario. The main reason is that users might have set  a different quota on the CSI driver Storage Class.

Exit criteria:

  • New clusters get the CSI Storage Class as the default one.
  • Existing clusters don't get their default Storage Classes changed.
Feature OCPPLAN-7878: NetEdge - Maintainability and Debugability & Tech Backlog

tldr: three basic claims, the rest is explanation and one example

  1. We cannot improve long term maintainability solely by fixing bugs.
  2. Teams should be asked to produce designs for improving maintainability/debugability.
  3. Specific maintenance items (or investigation of maintenance items), should be placed into planning as peer to PM requests and explicitly prioritized against them.

While bugs are an important metric, fixing bugs is different than investing in maintainability and debugability. Investing in fixing bugs will help alleviate immediate problems, but doesn't improve the ability to address future problems. You (may) get a code base with fewer bugs, but when you add a new feature, it will still be hard to debug problems and interactions. This pushes a code base towards stagnation where it gets harder and harder to add features.

One alternative is to ask teams to produce ideas for how they would improve future maintainability and debugability instead of focusing on immediate bugs. This would produce designs that make problem determination, bug resolution, and future feature additions faster over time.

I have a concrete example of one such outcome of focusing on bugs vs quality. We have resolved many bugs about communication failures with ingress by finding problems with point-to-point network communication. We have fixed the individual bugs, but have not improved the code for future debugging. In so doing, we chase many hard to diagnose problem across the stack. The alternative is to create a point-to-point network connectivity capability. this would immediately improve bug resolution and stability (detection) for kuryr, ovs, legacy sdn, network-edge, kube-apiserver, openshift-apiserver, authentication, and console. Bug fixing does not produce the same impact.

We need more investment in our future selves. Saying, "teams should reserve this" doesn't seem to be universally effective. Perhaps an approach that directly asks for designs and impacts and then follows up by placing the items directly in planning and prioritizing against PM feature requests would give teams the confidence to invest in these areas and give broad exposure to systemic problems.

Relevant links:

Epic NE-1067: [Tech Debt] Create E2E testing for DNS local endpoint preference

OCP/Telco Definition of Done

Epic Template descriptions and documentation.

Epic Goal

Why is this important?

  • This regression is a major performance and stability issue and it has happened once before.

Drawbacks

  • The E2E test may be complex due to trying to determine what DNS pods are responding to DNS requests. This is straightforward using the chaos plugin.

Scenarios

  • CI Testing

Acceptance Criteria

  • CI - MUST be running successfully with tests automated

Dependencies (internal and external)

  1. SDN Team

Previous Work (Optional):

  1. N/A

Open questions::

  1. Where do these E2E test go? SDN Repo? DNS Repo?

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub
    Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub
    Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story NE-1066: Add chaos plugin to DNS to support DNS Endpoint identification.

Enable the chaos plugin https://coredns.io/plugins/chaos/ in our CoreDNS configuration so that we can use a DNS query to easily identify what DNS pods are responding to our requests.

Epic NE-1072: [Tech Debt] Default HAProxy maxconn value to 50000 for OCP 4.12

Epic Goal

  • Change the default value for the spec.tuningOptions.maxConnections field in the IngressController API, which configures the HAProxy maxconn setting, to 50000 (fifty thousand).

Why is this important?

  • The maxconn setting constrains the number of simultaneous connections that HAProxy accepts. Beyond this limit, the kernel queues incoming connections. 
  • Increasing maxconn enables HAProxy to queue incoming connections intelligently.  In particular, this enables HAProxy to respond to health probes promptly while queueing other connections as needed.
  • The default setting of 20000 has been in place since OpenShift 3.5 was released in April 2017 (see BZ#1405440, commit, RHBA-2017:0884). 
  • Hardware capabilities have increased over time, and the current default is too low for typical modern machine sizes. 
  • Increasing the default setting improves HAProxy's performance at an acceptable cost in the common case. 

Scenarios

  1. As a cluster administrator who is installing OpenShift on typical hardware, I want OpenShift router to be tuned appropriately to take advantage of my hardware's capabilities.

Acceptance Criteria

  • CI is passing. 
  • The new default setting is clearly documented. 
  • A release note informs cluster administrators of the change to the default setting. 

Dependencies (internal and external)

  1. None.

Previous Work (Optional):

  1. The  haproxy-max-connections-tuning enhancement made maxconn configurable without changing the default.  The enhancement document details the tradeoffs in terms of memory for various settings of nbthreads and maxconn with various numbers of routes. 

Open questions::

  1. ...

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>

 

Feature OCPPLAN-8030: Console: Customer Happiness (RFEs) for 4.8-4.12

Feature Overview

  • This Section:* High-Level description of the feature ie: Executive Summary
  • Note: A Feature is a capability or a well defined set of functionality that delivers business value. Features can include additions or changes to existing functionality. Features can easily span multiple teams, and multiple releases.

 

Goals

  • This Section:* Provide high-level goal statement, providing user context and expected user outcome(s) for this feature
  • …

 

Requirements

  • This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.

 

Requirement Notes isMvp?
CI - MUST be running successfully with test automation This is a requirement for ALL features. YES
Release Technical Enablement Provide necessary release enablement details and documents. YES

 

(Optional) Use Cases

This Section: 

  • Main success scenarios - high-level user stories
  • Alternate flow/scenarios - high-level user stories
  • ...

 

Questions to answer…

  • ...

 

Out of Scope

  • …

 

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

 

Assumptions

  • ...

 

Customer Considerations

  • ...

 

Documentation Considerations

Questions to be addressed:

  • What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
  • Does this feature have doc impact?  
  • New Content, Updates to existing content,  Release Note, or No Doc Impact
  • If unsure and no Technical Writer is available, please contact Content Strategy.
  • What concepts do customers need to understand to be successful in [action]?
  • How do we expect customers will use the feature? For what purpose(s)?
  • What reference material might a customer want/need to complete [action]?
  • Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
  • What is the doc impact (New Content, Updates to existing content, or Release Note)?
Epic CONSOLE-3193: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Story CONSOLE-3244: Make status.HostIP for Pods visible in the OCP Web Console

As a developer, I want to make status.HostIP for Pods visible in the Pod details page of the OCP Web Console. Currently there is no way to view the node IP for a Pod in the OpenShift Web Console.  When viewing a Pod in the console, the field status.HostIP is not visible.

 

Acceptance criteria:

  • Make pod's HostIP field visible in the pod details page, similarly to PodIP field
Story CONSOLE-3252: Inform OCP users that the platform is performing an upgrade

When OCP is performing cluster upgrade user should be notified about this fact.

There are two possibilities how to surface the cluster upgrade to the users:

  • Display a console notification throughout OCP web UI saying that the cluster is currently under upgrade.
  • Global notification throughout OCP web UI saying that the cluster is currently under upgrade.
  • Have an alert firing for all the users of OCP stating the cluster is undergoing an upgrade. 

 

AC:

  • Console-operator will create a ConsoleNotification CR when the cluster is being upgraded. Once the upgrade is done console-operator will remote that CR. These are the three statuses based on which we are determining if the cluster is being upgraded.
  • Add unit tests

 

Note: We need to decide if we want to distinguish this particular notification by a different color? ccing Ali Mobrem 

 

Created from: https://issues.redhat.com/browse/RFE-3024

Story CONSOLE-3067: Adding Rollout Restart function to the OpenShift Console

As a console user I want to have option to:

  • Restart Deployment
  • Retry latest DeploymentConfig if it failed

 

For Deployments we will add the 'Restart rollout' action button. This action will PATCH the Deployment object's 'spec.template.metadata.annotations' block, by adding 'openshift.io/restartedAt: <actual-timestamp>' annotation. This will restart the deployment, by creating a new ReplicaSet.

  • action is disabled if:
    • Deployment is paused

 

For DeploymentConfig we will add 'Retry rollout' action button.  This action will PATCH the latest revision of ReplicationController object's 'metadata.annotations' block by setting 'openshift.io/deployment/phase: "New"' and removing openshift.io/deployment.cancelled and openshift.io/deployment.status-reason.

  • action is enabled if:
    • latest revision of the ReplicationController resource is in Failed phase
  • action is disabled if:
    • latest revision of the ReplicationController resource is in Complete phase
    • DeploymentConfig does not have any rollouts
    • DeploymentConfigs is paused

 

Acceptance Criteria:

  • Add the 'Restart rollout' action button for the Deployment resource to both action menu and kebab menu
  • Add the 'Retry rollout' action button for the DeploymentConfig resource to both action menu and kebab menu

 

BACKGROUND:

OpenShift console will be updated to allow rollout restart deployment from the console itself.

Currently, from the OpenShift console, for the resource “deploymentconfigs” we can only start and pause the rollout, and for the resource “deployment” we can only resume the rollout. None of the resources (deployment & deployment config) has this option to restart the rollout. So, that is the reason why the customer wants this functionality to perform the same action from the CLI as well as the OpenShift console.

The customer wants developers who are not fluent with the oc tool and terminal utilities, can use the console instead of the terminal to restart deployment, just like we use to do it through CLI using the command “oc rollout restart deploy/<deployment-name>“.
Usually when developers change the config map that deployment uses they have to restart pods. Currently, the developers have to use the oc rollout restart deployment command. The customer wants the functionality to get this button/menu to perform the same action from the console as well.

Design
Doc: https://docs.google.com/document/d/1i-jGtQGaA0OI4CYh8DH5BBIVbocIu_dxNt3vwWmPZdw/edit

Feature OCPPLAN-8150: Agent-Based Installer GA

Feature Overview

  • As an infrastructure owner, I want a repeatable method to quickly deploy the initial OpenShift cluster.
  • As an infrastructure owner, I want to install the first (management, hub, “cluster 0”) cluster to manage other (standalone, hub, spoke, hub of hubs) clusters.

Goals

  • Enable customers and partners to successfully deploy a single “first” cluster in disconnected, on-premises settings

Requirements

4.11 MVP Requirements

  • Customers and partners needs to be able to download the installer
  • Enable customers and partners to deploy a single “first” cluster (cluster 0) using single node, compact, or highly available topologies in disconnected, on-premises settings
  • Installer must support advanced network settings such as static IP assignments, VLANs and NIC bonding for on-premises metal use cases, as well as DHCP and PXE provisioning environments.
  • Installer needs to support automation, including integration with third-party deployment tools, as well as user-driven deployments.
  • In the MVP automation has higher priority than interactive, user-driven deployments.
  • For bare metal deployments, we cannot assume that users will provide us the credentials to manage hosts via their BMCs.
  • Installer should prioritize support for platforms None, baremetal, and VMware.
  • The installer will focus on a single version of OpenShift, and a different build artifact will be produced for each different version.
  • The installer must not depend on a connected registry; however, the installer can optionally use a previously mirrored registry within the disconnected environment.

Use Cases

  • As a Telco partner engineer (Site Engineer, Specialist, Field Engineer), I want to deploy an OpenShift cluster in production with limited or no additional hardware and don’t intend to deploy more OpenShift clusters [Isolated edge experience].
  • As a Enterprise infrastructure owner, I want to manage the lifecycle of multiple clusters in 1 or more sites by first installing the first  (management, hub, “cluster 0”) cluster to manage other (standalone, hub, spoke, hub of hubs) clusters [Cluster before your cluster].
  • As a Partner, I want to package OpenShift for large scale and/or distributed topology with my own software and/or hardware solution.
  • As a large enterprise customer or Service Provider, I want to install a “HyperShift Tugboat” OpenShift cluster in order to offer a hosted OpenShift control plane at scale to my consumers (DevOps Engineers, tenants) that allows for fleet-level provisioning for low CAPEX and OPEX, much like AKS or GKE [Hypershift].
  • As a new, novice to intermediate user (Enterprise Admin/Consumer, Telco Partner integrator, RH Solution Architect), I want to quickly deploy a small OpenShift cluster for Poc/Demo/Research purposes.

Questions to answer…

  •  

Out of Scope

Out of scope use cases (that are part of the Kubeframe/factory project):

  • As a Partner (OEMs, ISVs), I want to install and pre-configure OpenShift with my hardware/software in my disconnected factory, while allowing further (minimal) reconfiguration of a subset of capabilities later at a different site by different set of users (end customer) [Embedded OpenShift].
  • As an Infrastructure Admin at an Enterprise customer with multiple remote sites, I want to pre-provision OpenShift centrally prior to shipping and activating the clusters in remote sites.

Background, and strategic fit

  • This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

  1. The user has only access to the target nodes that will form the cluster and will boot them with the image presented locally via a USB stick. This scenario is common in sites with restricted access such as government infra where only users with security clearance can interact with the installation, where software is allowed to enter in the premises (in a USB, DVD, SD card, etc.) but never allowed to come back out. Users can't enter supporting devices such as laptops or phones.
  2. The user has access to the target nodes remotely to their BMCs (e.g. iDrac, iLo) and can map an image as virtual media from their computer. This scenario is common in data centers where the customer provides network access to the BMCs of the target nodes.
  3. We cannot assume that we will have access to a computer to run an installer or installer helper software.

Customer Considerations

  • ...

Documentation Considerations

Questions to be addressed:

  • What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
  • Does this feature have doc impact?
  • New Content, Updates to existing content, Release Note, or No Doc Impact
  • If unsure and no Technical Writer is available, please contact Content Strategy.
  • What concepts do customers need to understand to be successful in [action]?
  • How do we expect customers will use the feature? For what purpose(s)?
  • What reference material might a customer want/need to complete [action]?
  • Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
  • What is the doc impact (New Content, Updates to existing content, or Release Note)?

 

References

 

 

Epic AGENT-84: Support requesting components such as MCE at installation time

Epic Goal

As an OpenShift infrastructure owner, I want to deploy a cluster zero with RHACM or MCE and have the required components installed when the installation is completed

Why is this important?

BILLI makes it easier to deploy a cluster zero. BILLI users know at installation time what the purpose of their cluster is when they plan the installation. Day-2 steps are necessary to install operators and users, especially when automating installations, want to finish the installation flow when their required components are installed.

Acceptance Criteria

  • A user can provide MCE manifests and have it installed without additional manual steps after the installation is completed
  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story AGENT-328: Install MCE using extra manifests at install time (connected)

User Story:

As a customer, I want to be able to:

  • Install MCE with the agent-installer

so that I can achieve

  • create an MCE hub with my openshift install

Acceptance Criteria:

Description of criteria:

  • Upstream documentation including examples of the extra manifests needed
  • Unit tests that include MCE extra manifests
  • Ability to install MCE using agent-installer is tested
  • Point 3

(optional) Out of Scope:

We are only allowing the user to provide extra manifests to install MCE at this time. We are not adding an option to "install mce" on the command line (or UI)

Engineering Details:

This requires/does not require a design proposal.
This requires/does not require a feature gate.

Story AGENT-330: Install MCE using extra manifests at install time (disconnected)

User Story:

As a customer, I want to be able to:

  • Install MCE with the agent-installer

so that I can achieve

  • create an MCE hub with my openshift install

Acceptance Criteria:

Description of criteria:

  • Upstream documentation including examples of the extra manifests needed
  • Unit tests that include MCE extra manifests
  • Ability to install MCE using agent-installer is tested
  • Point 3

(optional) Out of Scope:

We are only allowing the user to provide extra manifests to install MCE at this time. We are not adding an option to "install mce" on the command line (or UI)

Engineering Details:

This requires/does not require a design proposal.
This requires/does not require a feature gate.

Epic AGENT-189: Provision IPv4/IPv6 dual-stack and single-stack IPv6 clusters

Epic Goal

As a OpenShift infrastructure owner, I want to deploy OpenShift clusters with dual-stack IPv4/IPv6

As a OpenShift infrastructure owner, I want to deploy OpenShift clusters with single-stack IPv6

Why is this important?

IPv6 and dual-stack clusters are requested often by customers, especially from Telco customers. Working with dual-stack clusters is a requirement for many but also a transition into a single-stack IPv6 clusters, which for some of our users is the final destination.

Acceptance Criteria

  • Agent-based installer can deploy IPv6 clusters
  • Agent-based installer can deploy dual-stack clusters
  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.

Previous Work

Karim's work proving how agent-based can deploy IPv6: IPv6 deploy with agent based installer]

Done Checklist * CI - CI is running, tests are automated and merged.

  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>|
Task AGENT-351: Add MachineNetwork to agent-cluster-install generate for dual-stack

For dual-stack installations the agent-cluster-install.yaml must have both an IPv4 and IPv6 subnet in the networkking.MachineNetwork or assisted-service will throw an error. This field is in InstallConfig but it must be added to agent-cluster-install in its Generate().

For IPv4 and IPv6 installs, setting up the MachineNetwork is not needed but it also does not cause problems if its set, so it should be fine to set it all times.

Epic AGENT-61: Ensure FIPS compliant deployment

Set the ClusterDeployment CRD to deploy OpenShift in FIPS mode and make sure that after deployment the cluster is set in that mode

Task AGENT-375: AgentClusterInstall asset generation include installconfigoverrides annotation when FIPS is set in installconfig

In order to install FIPS compliant clusters, we need to make sure that installconfig + agentoconfig based deployments take into account the FIPS config in installconfig.

This task is about passing the config to agentclusterinstall so it makes it into the iso. Once there, AGENT-374 will give it to assisted service

Feature OCPPLAN-9015: The details of this Jira Card are restricted (Only Red Hat employees and contractors)
The details of this Jira Card are restricted (Only Red Hat employees and contractors)
Epic OCPCLOUD-1663: Rebase cluster autoscaler on Kubernetes 1.25

Epic Goal

  • Rebase cluster autoscaler on top of Kubernetes 1.25

Why is this important?

  • Need to pick up latest upstream changes

Scenarios

  1. ...

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.
  • ...

Dependencies (internal and external)

  1. ...

Previous Work (Optional):

  1. …

Open questions::

  1. …

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story OCPCLOUD-1677: Update cluster autoscaler operator to use --record-duplicated-events flag

User Story

As a user I would like to see all the events that the autoscaler creates, even duplicates. Having the CAO set this flag will allow me to continue to see these events.

Background

We have carried a patch for the autoscaler that would enable the duplication of events. This patch can now be dropped because the upstream added a flag for this behavior in https://github.com/kubernetes/autoscaler/pull/4921

Steps

  • add the --record-duplicated-events flag to all autoscaler deployments from the CAO

Stakeholders

  • openshift eng

Definition of Done

  • autoscaler continues to work as expected and produces events for everything
  • Docs
  • this does not require documentation as it preserves existing behavior and provides no interface for user interaction
  • Testing
  • current tests should continue to pass
Feature OCPPLAN-9052: IBM Public Cloud support (GA)

Feature Overview

Add GA support for deploying OpenShift to IBM Public Cloud

Goals

Complete the existing gaps to make OpenShift on IBM Cloud VPC (Next Gen2) General Available

Requirements

Optional requirements

  • OpenShift can be deployed using Mint mode and STS for cloud provider credentials (future release, tbd)
  • OpenShift can be deployed in disconnected mode https://issues.redhat.com/browse/SPLAT-737)
  • OpenShift on IBM Cloud supports User Provisioned Infrastructure (UPI) deployment method (future release, 4.14?)
Epic SPLAT-731: Enable installation of private clusters on IBM Cloud

Epic Goal

  • Enable installation of private clusters on IBM Cloud. This epic will track associated work.

Why is this important?

  • This is required MVP functionality to achieve GA.

Scenarios

  1. Install a private cluster on IBM Cloud.

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.

Dependencies (internal and external)

  1. ...

Previous Work (Optional):

  1. …

Open questions::

  1. …

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Feature OCPPLAN-9332: Ability to apply hotfix packages to RHCOS via Layering

Background and Goal

Currently in OpenShift we do not support distributing hotfix packages to cluster nodes. In time-sensitive situations, a RHEL hotfix package can be the quickest route to resolving an issue. 

Acceptance Criteria

  1. Under guidance from Red Hat CEE, customers can deploy RHEL hotfix packages to MachineConfigPools.
  2. Customers can easily remove the hotfix when the underlying RHCOS image incorporates the fix.
Epic MCO-293: Replace machine-os-content with new format base image

Before we ship OCP CoreOS layering in https://issues.redhat.com/browse/MCO-165 we need to switch the format of what is currently `machine-os-content` to be the new base image.

The overall plan is:

  • Publish the new base image as `rhel-coreos-8` in the release image
  • Also publish the new extensions container (https://github.com/openshift/os/pull/763) as `rhel-coreos-8-extensions`
  • Teach the MCO to use this without also involving layering/build controller
  • Delete old `machine-os-content`
Story MCO-289: Teach the MCO to use new extensions container

After https://github.com/openshift/os/pull/763 is in the release image, teach the MCO how to use it. This is basically:

  • Schedule the extensions container as a kubernetes service (just serves a yum repo via http)
  • Change the MCD to write a file into `/etc/yum.repos.d/machine-config-extensions.repo` that consumes it instead of what it does now in pulling RPMs from the mounted container filesystem
Story MCO-371: Add telemetry to collect data on how many cluster is using custom osImageURL

As a OCP CoreOS layering developer, having telemetry data about number of cluster using osImageURL will help understand how broadly this feature is getting used and improve accordingly.

Acceptance Criteria:

  • Cluster using Custom osImageURL is available via telemetry
Feature OCPSTRAT-103: Ensuring the Control Plane is Fully Decoupled for Hosted Control Planes

 

Why?

  • Decouple control and data plane. 
    • Customers do not pay Red Hat more to run HyperShift control planes and supporting infrastructure than Standalone control planes and supporting infrastructure.
  • Improve security
    • Shift credentials out of cluster that support the operation of core platform vs workload
  • Improve cost
    • Allow a user to toggle what they don’t need.
    • Ensure a smooth path to scale to 0 workers and upgrade with 0 workers.

 

Assumption

  • A customer will be able to associate a cluster as “Infrastructure only”
  • E.g. one option: management cluster has role=master, and role=infra nodes only, control planes are packed on role=infra nodes
  • OR the entire cluster is labeled infrastructure , and node roles are ignored.
  • Anything that runs on a master node by default in Standalone that is present in HyperShift MUST be hosted and not run on a customer worker node.

 

 

Doc: https://docs.google.com/document/d/1sXCaRt3PE0iFmq7ei0Yb1svqzY9bygR5IprjgioRkjc/edit 

Epic STOR-960: Ensure openshift cluster-storage-operator & aws-ebs-csi-driver-operator are running on hosted control planes

Overview 

Customers do not pay Red Hat more to run HyperShift control planes and supporting infrastructure than Standalone control planes and supporting infrastructure.

Assumption

  • A customer will be able to associate a cluster as “Infrastructure only”
  • E.g. one option: management cluster has role=master, and role=infra nodes only, control planes are packed on role=infra nodes
  • OR the entire cluster is labeled infrastructure, and node roles are ignored.
  • Anything that runs on a master node by default in Standalone that is present in HyperShift MUST be hosted and not run on a customer worker node.

DoD 

Run cluster-storage-operator (CSO) + AWS EBS CSI driver operator + AWS EBS CSI driver control-plane Pods in the management cluster, run the driver DaemonSet in the hosted cluster.

More information here: https://docs.google.com/document/d/1sXCaRt3PE0iFmq7ei0Yb1svqzY9bygR5IprjgioRkjc/edit 

 

Story STOR-1053: Sync hypershift dump-guest-cluster with must-gather

As OCP support engineer I want the same guest cluster storage-related objects in output of "hypershift dump cluster --dump-guest-cluster" as in "oc adm must-gather ", so I can debug storage issues easily.

 

must-gather collects: storageclasses persistentvolumes volumeattachments csidrivers csinodes volumesnapshotclasses volumesnapshotcontents

hypershift collects none of this, the relevant code is here: https://github.com/openshift/hypershift/blob/bcfade6676f3c344b48144de9e7a36f9b40d3330/cmd/cluster/core/dump.go#L276

 

Exit criteria:

  • verify that hypershift dump cluster --dump-guest-cluster has storage objects from the guest cluster.
Story STOR-1040: Update AWS EBS CSI driver operator for HyperShift

As HyperShift Cluster Instance Admin, I want to run AWS EBS CSI driver operator + control plane of the CSI driver in the management cluster, so the guest cluster runs just my applications.

  • Add a new cmdline option for the guest cluster kubeconfig file location
  • Parse both kubeconfigs:
    • One from projected service account, which leads to the management cluster.
    • Second from the new cmdline option introduced above. This one leads to the guest cluster.
  • Only on HyperShift:
    • When interacting with Kubernetes API, carefully choose the right kubeconfig to watch / create / update objects in the right cluster.
    • Replace namespaces in all Deployments and other objects that are created in the management cluster. They must be created in the same namespace as the operator.
  •  
  •  
    • Pass only the guest kubeconfig to the operand (control-plane Deployment of the CSI driver).

Exit criteria:

  • Control plane Deployment of AWS EBS CSI driver runs in the management cluster in HyperShift.
  • Storage works in the guest cluster.
  • No regressions in standalone OCP.
Story STOR-1039: Update cluster-storage-operator for HyperShift

As HyperShift Cluster Instance Admin, I want to run cluster-storage-operator (CSO) in the management cluster, so the guest cluster runs just my applications.

  • Add a new cmdline option for the guest cluster kubeconfig file location
  • Parse both kubeconfigs:
    • One from projected service account, which leads to the management cluster.
    • Second from the new cmdline option introduced above. This one leads to the guest cluster.
  • Tag manifests of objects that should not be deployed by CVO in HyperShift
  • Only on HyperShift:
    • When interacting with Kubernetes API, carefully choose the right kubeconfig to watch / create / update objects in the right cluster.
    • Replace namespaces in all Deployments and other objects that are created in the management cluster. They must be created in the same namespace as the operator.
    • Pass only the guest kubeconfig to the operands (AWS EBS CSI driver operator).

Exit criteria:

  • CSO and AWS EBS CSI driver operator runs in the management cluster in HyperShift
  • Storage works in the guest cluster.
  • No regressions in standalone OCP.
Epic SDN-3198: [4.12] Hypershift: Changes to move networking operators to hosted control plane

Epic Goal

  • To improve debug-ability of ovn-k in hypershift
  • To verify the stability of of ovn-k in hypershift
  • To introduce a EgressIP reach-ability check that will work in hypershift

Why is this important?

  • ovn-k is supposed to be GA in 4.12. We need to make sure it is stable, we know the limitations and we are able to debug it similar to the self hosted cluster.

Acceptance Criteria

  • CI - MUST be running successfully with tests automated

Dependencies (internal and external)

  1. This will need consultation with the people working on HyperShift

Previous Work (Optional):

  1. https://issues.redhat.com/browse/SDN-2589

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Epic STOR-961: Ensure cluster-snapshot-controller-operator is running on hosted control planes

Overview 

Customers do not pay Red Hat more to run HyperShift control planes and supporting infrastructure than Standalone control planes and supporting infrastructure.

Assumption

  • A customer will be able to associate a cluster as “Infrastructure only”
  • E.g. one option: management cluster has role=master, and role=infra nodes only, control planes are packed on role=infra nodes
  • OR the entire cluster is labeled infrastructure, and node roles are ignored.
  • Anything that runs on a master node by default in Standalone that is present in HyperShift MUST be hosted and not run on a customer worker node.

DoD 

cluster-snapshot-controller-operator is running on the CP. 

More information here: https://docs.google.com/document/d/1sXCaRt3PE0iFmq7ei0Yb1svqzY9bygR5IprjgioRkjc/edit 

Story STOR-970: Refactor snapshot operator

As OpenShift developer I want cluster-csi-snapshot-controller-operator to use existing controllers in library-go, so I don’t need to maintain yet another code that does the same thing as library-go.

  • Check and remove manifests/03_configmap.yaml, it does not seem to be useful.
  • Check and remove manifests/03_service.yaml, it does not seem to be useful (at least now).
  • Use DeploymentController from library-go to sync Deployments.
  • Get rid of common/ package? It does not seem to be useful.
  • Use StaticResourceController for static content, including the snapshot CRDs.

Note: if this refactoring introduces any new conditions, we must make sure that 4.11 snapshot controller clears them to support downgrade! This will need 4.11 BZ + z-stream update!

Similarly, if some conditions become obsolete / not managed by any controller, they must be cleared by 4.12 operator.

Exit criteria:

  • The operator code is smaller.
  • No regressions in standalone OCP.
  • Upgrade/downgrade from/to standalone OCP 4.11 works.
Story STOR-971: Update snapshot operator for HyperShift

As HyperShift Cluster Instance Admin, I want to run cluster-csi-snapshot-controller-operator in the management cluster, so the guest cluster runs just my applications.

  • Add a new cmdline option for the guest cluster kubeconfig file location
  • Parse both kubeconfigs:
    • One from projected service account, which leads to the management cluster.
    • Second from the new cmdline option introduced above. This one leads to the guest cluster.
  • Move creation of manifests/08_webhook_service.yaml from CVO to the operator - it needs to be created in the management cluster.
  • Tag manifests of objects that should not be deployed by CVO in HyperShift by
  • Only on HyperShift:
    • When interacting with Kubernetes API, carefully choose the right kubeconfig to watch / create / update objects in the right cluster.
    • Replace namespaces in all Deployments and other objects that are created in the management cluster. They must be created in the same namespace as the operator.
    • Don’t create operand’s PodDisruptionBudget?
    • Update ValidationWebhookConfiguration to point directly to URL exposed by manifests/08_webhook_service.yaml instead of a Service. The Service is not available in the guest cluster.
    • Pass only the guest kubeconfig to the operands (both the webhook and csi-snapshot-controller).
    • Update unit tests to handle two kube clients.

Exit criteria:

  • cluster-csi-snapshot-controller-operator runs in the management cluster in HyperShift
  • csi-snapshot-controller runs in the management cluster in HyperShift
  • It is possible to take & restore volume snapshot in the guest cluster.
  • No regressions in standalone OCP.
Feature OCPSTRAT-173: OC mirror enhancements

OC mirror is GA product as of Openshift 4.11 .

The goal of this feature is to solve any future customer request for new features or capabilities in OC mirror 

Epic CFE-538: oc mirror IBM CloudPak support

Epic Goal

  • Mirror to mirror operations and custom mirroring flows required by IBM CloudPak catalog management

Why is this important?

  • IBM needs additional customization around the actual mirroring of images to enable CloudPaks to fully adopt OLM-style operator packaging and catalog management
  • IBM CloudPaks introduce additional compute architectures, increasing the download volume by 2/3rds to day, we need the ability to effectively filter non-required image versions of OLM operator catalogs during filtering for other customers that only require a single or a subset of the available image architectures
  • IBM CloudPaks regularly run on older OCP versions like 4.8 which require additional work to be able to read the mirrored catalog produced by oc mirror

Scenarios

  1. Customers can use the oc utility and delegate the actual image mirror step to another tool
  2. Customers can mirror between disconnected registries using the oc utility
  3. The oc utility supports filtering manifest lists in the context of multi-arch images according to the sparse manifest list proposal in the distribution spec

Acceptance Criteria

  • Customers can use the oc utility to mirror between two different air-gapped environments
  • Customers can specify the desired computer architectures and oc mirror will create sparse manifest lists in the target registry as a result

Dependencies (internal and external)

Previous Work:

  1. WRKLDS-369
  2. Disconnected Mirroring Improvement Proposal

Related Work:

  1. https://github.com/opencontainers/distribution-spec/pull/310
  2. https://github.com/distribution/distribution/pull/3536
  3. https://docs.google.com/document/d/10ozLoV7sVPLB8msLx4LYamooQDSW-CAnLiNiJ9SER2k/edit?usp=sharing
Feature OCPSTRAT-402: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Epic CONSOLE-3234: Phase 3 - Pre-Work Unite Consoles (ACM, OCP)

Pre-Work Objectives

Since some of our requirements from the ACM team will not be available for the 4.12 timeframe, the team should work on anything we can get done in the scope of the console repo so that when the required items are available in 4.13, we can be more nimble in delivering GA content for the Unified Console Epic.

Overall GA Key Objective
Providing our customers with a single simplified User Experience(Hybrid Cloud Console)that is extensible, can run locally or in the cloud, and is capable of managing the fleet to deep diving into a single cluster. 
Why customers want this?

  1. Single interface to accomplish their tasks
  2. Consistent UX and patterns
  3. Easily accessible: One URL, one set of credentials

Why we want this?

  • Shared code -  improve the velocity of both teams and most importantly ensure consistency of the experience at the code level
  • Pre-built PF4 components
  • Accessibility & i18n
  • Remove barriers for enabling ACM

Phase 2 Goal: Productization of the united Console 

  1. Enable user to quickly change context from fleet view to single cluster view
    1. Add Cluster selector with “All Cluster” Option. “All Cluster” = ACM
    2. Shared SSO across the fleet
    3. Hub OCP Console can connect to remote clusters API
    4. When ACM Installed the user starts from the fleet overview aka “All Clusters”
  2. Share UX between views
    1. ACM Search —> resource list across fleet -> resource details that are consistent with single cluster details view
    2. Add Cluster List to OCP —> Create Cluster
Story CONSOLE-3125: Disable unsupported clusters in cluster dropdown

As a developer I would like to disable clusters like *KS that we can't support for multi-cluster (for instance because we can't authenticate). The ManagedCluster resource has a vendor label that we can use to know if the cluster is supported.

cc Ali Mobrem Sho Weimer Jakub Hadvig 

UPDATE: 9/20/22 : we want an allow-list with OpenShift, ROSA, ARO, ROKS, and  OpenShiftDedicated

Acceptance criteria:

  • Investigate if console-operator should pass info about which cluster are supported and unsupported to the frontend
  • Unsupported clusters should not appear in the cluster dropdown
  • Unsupported clusters based off
    • defined vendor label
    • non 4.x ocp clusters
Feature OCPSTRAT-507: RHCOS based on RHEL 9.2

Feature Overview

RHEL CoreOS should be updated to RHEL 9.2 sources to take advantage of newer features, hardware support, and performance improvements.

 

Requirements

  • RHEL 9.x sources for RHCOS builds starting with OCP 4.13 and RHEL 9.2.

 

Requirement Notes isMvp?
CI - MUST be running successfully with test automation This is a requirement for ALL features. YES
Release Technical Enablement Provide necessary release enablement details and documents. YES

(Optional) Use Cases

  • 9.2 Preview via Layering No longer necessary assuming we stay the course of going all in on 9.2

Assumptions

  • ...

Customer Considerations

  • ...

Documentation Considerations

Questions to be addressed:

  • What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
  • Does this feature have doc impact?
  • New Content, Updates to existing content, Release Note, or No Doc Impact
  • If unsure and no Technical Writer is available, please contact Content Strategy.
  • What concepts do customers need to understand to be successful in [action]?
  • How do we expect customers will use the feature? For what purpose(s)?
  • What reference material might a customer want/need to complete [action]?
  • Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
  • What is the doc impact (New Content, Updates to existing content, or Release Note)?
Epic OKD-4: MVP: Get a sprintly release of OKD running on SCOS

PROBLEM

We would like to improve our signal for RHEL9 readiness by increasing internal engineering engagement and external partner engagement on our community OpehShift offering, OKD.

PROPOSAL

Adding OKD to run on SCOS (a CentOS stream for CoreOS) brings the community offering closer to what a partner or an internal engineering team might expect on OCP.

ACCEPTANCE CRITERIA

Image has been switched/included: 

OKD-23

DEPENDENCIES

The SCOS build payload.

RELATED RESOURCES

OKD+SCOS proposal: https://docs.google.com/presentation/d/1_Xa9Z4tSqB7U2No7WA0KXb3lDIngNaQpS504ZLrCmg8/edit#slide=id.p

OKD+SCOS work draft: https://docs.google.com/document/d/1cuWOXhATexNLWGKLjaOcVF4V95JJjP1E3UmQ2kDVzsA/edit

 

Acceptance Criteria

A stable OKD on SCOS is built and available to the community sprintly.

 

Story OKD-49: Update multus script to recognize scos

This comes up when installing ipi-on-aws on arm64 with the custom payload build at quay.io/aleskandrox/okd-release:4.12.0-0.okd-centos9-full-rebuild-arm64 that is using scos as machine-content-os image

 

```

[root@ip-10-0-135-176 core]# crictl logs c483c92e118d8
2022-08-11T12:19:39+00:00 [cnibincopy] FATAL ERROR: Unsupported OS ID=scos
```

 

The probable fix has to land on https://github.com/openshift/cluster-network-operator/blob/master/bindata/network/multus/multus.yaml#L41-L53

Feature OCPSTRAT-598: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Epic USHIFT-228: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Story USHIFT-219: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Feature OCPSTRAT-845: [Tech Preview] Proper MCO State Reporting

Feature Overview (aka. Goal Summary)  

The MCO should properly report its state in a way that's consistent and able to be understood by customers, troubleshooters, and maintainers alike. 

Some customer cases have revealed scenarios where the MCO state reporting is misleading and therefore could be unreliable to base decisions and automation on.

In addition to correcting some incorrect states, the MCO will be enhanced for a more granular view of update rollouts across machines.

Epic MCO-452: [tech-preview] Proper state reporting when the MCO changes state

The MCO should properly report its state in a way that's consistent and able to be understood by customers, troubleshooters, and maintainers alike. 

For this epic, "state" means "what is the MCO doing?" – so the goal here is to try to make sure that it's always known what the MCO is doing. 

This includes: 

  • Conditions
  • Some Logging 
  • Possibly Some Events 

While this probably crosses a little bit into the "status" portion of certain MCO objects, as some state is definitely recorded there, this probably shouldn't turn into a "better status reporting" epic.  I'm interpreting "status" to mean "how is it going" so status is maybe a "detail attached to a state". 

 

Exploration here: https://docs.google.com/document/d/1j6Qea98aVP12kzmPbR_3Y-3-meJQBf0_K6HxZOkzbNk/edit?usp=sharing

 

https://docs.google.com/document/d/17qYml7CETIaDmcEO-6OGQGNO0d7HtfyU7W4OMA6kTeM/edit?usp=sharing

 

Story MCO-400: RFE: clarify "current" for MachineConfigPool status.configuration

The current property description is:

configuration represents the current MachineConfig object for the machine config pool.

But in a 4.12.0-ec.4 cluster, the actual semantics seem to be something closer to "the most recent rendered config that we completely leveled on". We should at least update the godocs to be more specific about the intended semantics. And perhaps consider adjusting the semantics?

Complete Epics

This section includes Jira cards that are linked to an Epic, but the Epic itself is not linked to any Feature. These epics were completed when this image was assembled

Epic CONSOLE-3059: OCP 4.11 - Dynamic Plugins Epic

This epic contains all the Dynamic Plugins related stories for OCP release-4.11 

Epic Goal

  • Track all the stories under a single epic

Acceptance Criteria

  •  
Story CONSOLE-3083: Add missing ts-doc for dynamic plugin SDK components

This story only covers API components. We will create a separate story for other utility functions.

Today we are generating documentation for Console's Dynamic Plugin SDK in
frontend/packages/dynamic-plugin-sdk. We are missing ts-doc for a set of hooks and components.

We are generating the markdown from the dynamic-plugin-sdk using

yarn generate-doc

Here is the list of the API that the dynamic-plugin-sdk is exposing:

https://gist.github.com/spadgett/0ddefd7ab575940334429200f4f7219a

Acceptance Criteria:

  • Add missing jsdocs for the API that dynamic-plugin-sdk exposes

Out of Scope:

  • This does not include work for integrating the API docs into the OpenShift docs
  • This does not cover other public utilities, only components.
Epic CONSOLE-3198: 4.12 Console Dependencies & Tech Debt

An epic we can duplicate for each release to ensure we have a place to catch things we ought to be doing regularly but can tend to fall by the wayside.

Story CONSOLE-3091: Remove unused markup after integration of Patterfly's dark mode

As a developer, I want to be able to clean up the css markup after making the css / scss changes required for dark mode and remove any old unused css / scss content. 

 

Acceptance criteria:

  • Remove any unused scss / css content after revamping for dark mode
Epic HELM-219: Basic authentication for ProjectHelm Chart repository in helmchartrepositories.helm.openshift.io CRD.

1. Proposed title of this feature request
Basic authentication for Helm Chart repository in helmchartrepositories.helm.openshift.io CRD.

2. What is the nature and description of the request?
As of v4.6.9, the HelmChartRepository CRD only supports client TLS authentication through spec.connectionConfig.tlsClientConfig.

3. Why do you need this? (List the business requirements here)
Basic authentication is widely used by many chart repositories managers (Nexus OSS, Artifactory, etc.)
Helm CLI also supports them with the helm repo add command.
https://helm.sh/docs/helm/helm_repo_add/

4. How would you like to achieve this? (List the functional requirements here)
Probably by extending the CRD:

spec:
connectionConfig:
username: username
password:
secretName: secret-name

The secret namespace should be openshift-config to align with the tlsClientConfig behavior.

5. For each functional requirement listed in question 4, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented.
Trying to pull helm charts from remote private chart repositories that has disabled anonymous access and offers basic authentication.
E.g.: https://github.com/sonatype/docker-nexus

Story HELM-342: Implement basic authentication for ODC Helm backend

Owner: Architect:

Story (Required)

As an OCP user I will like to be able to install helm charts from repos added to ODC with basic authentication fields populated

Background (Required)

We need to support helm installs for Repos that have the basic authentication secret name and namespace.

Glossary

Out of scope

Updating the ProjectHelmChartRepository CRD, already done in diff story
Supporting the HelmChartRepository CR, this feature will be scoped first to project/namespace scope repos.

In Scope

<Defines what is included in this story>

Approach(Required)

If the new fields for basic auth are set in the repo CR then use those credentials when making API calls to helm to install/upgrade charts. We will error out if user logged in does not have access to the secret referenced by Repo CR. If basic auth fields are not present we assume is not an authenticated repo.

Dependencies

Nonet

Edge Case

NA

Acceptance Criteria

I can list, install and update charts on authenticated repos from ODC
Needs Documentation both upstream and downstream
Needs new unit test covering repo auth

INVEST Checklist

Dependencies identified
Blockers noted and expected delivery timelines set
Design is implementable
Acceptance criteria agreed upon
Story estimated

Legend

Unknown
Verified
Unsatisfied

Epic NE-942: Add field in installer to get LB type for AWS

Epic Goal

  • We need the installer to accept a LB type from user and then we could set type of LB in the following object.
    oc get ingress.config.openshift.io/cluster -o yaml
    Then we can fetch info from this object and reconcile the operator to have the NLB changes reflected.

 

This is an API change and we will consider this as a feature request.

Why is this important?

https://issues.redhat.com/browse/NE-799 Please check this for more details

 

Scenarios

https://issues.redhat.com/browse/NE-799 Please check this for more details

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.
  • ...

Dependencies (internal and external)

  1. installer
  2. ingress operator

Previous Work (Optional):

 No

Open questions::

N/A

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Epic OCPRHV-812: Tests for oVirt Cluster API Provider and CSI Driver

We need tests for the ovirt-csi-driver and the cluster-api-provider-ovirt. These tests help us to

  • minimize bugs,
  • reproduce and fix them faster and
  • pin down current behavior of the driver

Also, having dedicated tests on lower levels with a smaller scope (unit, integration, ...) has the following benefits:

  • fast feedback cycle (local test execution)
  • developer in-code documentation
  • easier onboarding for new contributers
  • lower resource consumption
Epic ODC-4707: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Story ODC-6771: Update user in topology and add flow if Quotas has been reached in a namespace

Description

As a user, I would like to be informed in an intuitive way,  when quotas have been reached in a namespace

Acceptance Criteria

  1. Show an alert banner on the Topology and add page for this project/namespace when there is a RQ (Resource Quota) / ACRQ (Applied Cluster Resource Quota) issue
    PF guideline: https://www.patternfly.org/v4/components/alert/design-guidelines#using-alerts 
  2. The above alert should have a CTA link to the search page with all RQ, ACRQ and if there is just one show the details page for the same
  3. For RQ, ACRQ list view show one more column called status with details as shown in the project view.

Additional Details:

 

Refer below for more details 

Story ODC-6772: In topology view, intuitively update user if any of deployment have reached quota limits

Description

As a user, In the topology view, I would like to be updated intuitively if any of the deployments have reached quota limits

Acceptance Criteria

  1. Show a yellow border around deployments if any of the deployments have reached the quota limit
  2. For deployments, if there are any errors associated with resource limits or quotas, include a warning alert in the side panel.
    1. If we know resource limits are the cause, include link to Edit resource limits
    2. If we know pod count is the cause, include a link to Edit pod count

Additional Details:

 

Refer below for more details 

Epic ODC-5387: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Epic ODC-5897: Provide ability for admins to hide perspective

Goal

Provide a form driven experience to allow cluster admins to manage the perspectives to meet the ACs below.

Problem:

We have heard the following requests from customers and developer advocates:

  • Some admins do not want to provide access to the Developer Perspective from the console
  • Some admins do not want to provide non-priv users access to the Admin Perspective from the console

Acceptance criteria:

  1. Cluster administrator is able to "hide" the admin perspective for non-priv users
  2. Cluster administrator is able to "hide" the developer perspective for all users
  3. Be user that User Preferences for individual users behaves appropriately. If only one perspective is available, the perspective switcher is not needed.

Dependencies (External/Internal):

Design Artifacts:

Exploration:

Note:

Story ODC-6785: Hide perspective(s) based on customization

Description

As an admin, I want to hide user perspective(s) based on the customization.

Acceptance Criteria

  1. Hide perspective(s) based on the customization
    1. When the admin perspective is disabled -> we hide the admin perspective for all unprivileged users
    2. When the dev perspective is disabled -> we hide the dev perspective for all users
  2. When all the perspectives are hidden from a user or for all users, show the Admin perspective by default

Additional Details:

Story ODC-6784: Provide a code snippet for the console CRD

Description

As an admin, I should be able to see a code snippet that shows how to add user perspectives

Based on the https://issues.redhat.com/browse/ODC-6732 enhancement proposal, the cluster admin can add user perspectives

To support the cluster-admin to configure the perspectives correctly, the developer console should provide a code snippet for the customization of yaml resource (Console CRD).

Customize Perspective Enhancement PR: https://github.com/openshift/enhancements/pull/1205

Acceptance Criteria

  1. When the admin opens the Console CRD there is a snippet in the sidebar which provides a default YAML which supports the admin to add user perspectives

Additional Details:

Previous work:

  1. https://issues.redhat.com/browse/ODC-5080
  2. https://issues.redhat.com/browse/ODC-5449
Story ODC-6783: Extend the console-operator CRD and API for hiding user perspectives

Description

As an admin, I want to hide the admin perspective for non-privileged users or hide the developer perspective for all users

Based on the https://issues.redhat.com/browse/ODC-6730 enhancement proposal, it is required to extend the console configuration CRD to enable the cluster admins to configure this data in the console resource

Acceptance Criteria

  1. Extend the "customization" spec type definition for the CRD in the openshift/api project

Additional Details:

Previous customization work:

  1. https://issues.redhat.com/browse/ODC-5416
  2. https://issues.redhat.com/browse/ODC-5020
  3. https://issues.redhat.com/browse/ODC-5447
Story ODC-6786: Allow the user to hide perspective(s) using a form driven experience

Description

As an admin, I want to be able to use a form driven experience  to hide user perspective(s)

Acceptance Criteria

  1. Add checkboxes with the options
    1. Hide "Administrator" perspective for non-privileged users
    2.  Hide "Developer" perspective for all users
  2. The console configuration CR should be updated as per the selected option

Additional Details:

Epic SDN-3156: Egress IP reachability refactor

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

  • Come up with a consistent way to detect node down on OCP and hypershift. Current mechanism for OCP (probe port 9) does not work for hypershift, meaning, hypershift node down detection will be longer (~40 secs). We should aim to have a common mechanism for both. As well, we should consider alternatives to the probing port 9. Perhaps BFD, or other detection.
  • Get clarification on node down detection times. Some customers have (apparently) asked for detection on the order of 100ms, recommendation is to use multiple Egress IPs, so this may not be a hard requirement. Need clarification from PM/Customers.

Why is this important?

  • …

Scenarios

  1. ...

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.
  • ...

Dependencies (internal and external)

  1. ...

Previous Work (Optional):

  1. …

Open questions::

  1. …

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story SDN-3283: Add sock proxy to cluster-network-operator so egressip can use grpc to reach worker nodes

Add sock proxy to cluster-network-operator so egressip can use grpc to reach worker nodes.
 
With the introduction of grpc as means for determining the state of a given egress node, hypershift should
be able to leverage socks proxy and become able to know the state of each egress node.
 
References relevant to this work:
1281-network-proxy
[+https://coreos.slack.com/archives/C01C8502FMM/p1658427627751939+]
[+https://github.com/openshift/hypershift/pull/1131/commits/28546dc587dc028dc8bded715847346ff99d65ea+]

Incomplete Epics

This section includes Jira cards that are linked to an Epic, but the Epic itself is not linked to any Feature. These epics were not completed when this image was assembled

Epic BUILD-529: Kubernetes 1.25 Rebases

Epic Goal

  • Update OpenShift components that are owned by the Builds + Jenkins Team to use Kubernetes 1.25

Why is this important?

  • Our components need to be updated to ensure that they are using the latest bug/CVE fixes, features, and that they are API compatible with other OpenShift components.

Acceptance Criteria

  • Existing CI/CD tests must be passing
Epic CCXDEV-9285: IO maintenance OCP 4.12

This is epic tracks "business as usual" requirements / enhancements / bug fixing of Insights Operator.

Bug OCPBUGS-2915: InsightsRecommendationActive should link cluster-specific page

Today the links point at a rule-scoped page, but that page lacks information about recommended resolution.  You can click through by cluster ID to your specific cluster and get that recommendation advice, but it would be more convenient and less confusing for customers if we linked directly to the cluster-scoped recommendation page.

We can implement by updating the template here to be:

fmt.Sprintf("https://console.redhat.com/openshift/insights/advisor/clusters/%s?first=%s%%7C%s", clusterID, ruleIDStr, rec.ErrorKey)

or something like that.

 

unknowns

request is clear, solution/implementation to be further clarified

Epic CONSOLE-3174: OCP 4.12 - Dynamic Plugins Epic -GA

This epic contains all the Dynamic Plugins related stories for OCP release-4.12

Epic Goal

  • Track all the stories under a single epic

Acceptance Criteria

Story CONSOLE-3183: Improve/Fix 'console.dashboards/overview/detail/item' Extension

The extension `console.dashboards/overview/detail/item` doesn't constrain the content to fit the card.

The details-card has an expectation that a <dd> item will be the last item (for spacing between items). Our static details-card items use a component called 'OverviewDetailItem'. This isn't enforced in the extension and can cause undesired padding issues if they just do whatever they want.

I feel our approach here should be making the extension take the props of 'OverviewDetailItem' where 'children' is the new 'component'.

Acceptance Criteria:

  • Deprecate the old extension (in docs, with date/stamp)
  • Make a new extension that applies a stricter type
  • Include this new extension next to the old one (with the error boundary around it)
Bug CONSOLE-3337: When defining more then one proxy in dynamic plugin, only the last one works

when defining two proxy endpoints, 
apiVersion: console.openshift.io/v1alpha1
kind: ConsolePlugin
metadata:
...
name: forklift-console-plugin
spec:
displayName: Console Plugin Template
proxy:

  • alias: forklift-inventory
    authorize: true
    service:
    name: forklift-inventory
    namespace: konveyor-forklift
    port: 8443
    type: Service
  • alias: forklift-must-gather-api
    authorize: true
    service:
    name: forklift-must-gather-api
    namespace: konveyor-forklift
    port: 8443
    type: Service

service:
basePath: /
I get two proxy endpoints
/api/proxy/plugin/forklift-console-plugin/forklift-inventory
and
/api/proxy/plugin/forklift-console-plugin/forklift-must-gather-api

but both proxy to the `forklift-must-gather-api` service

e.g.
curl to:
[server url]/api/proxy/plugin/forklift-console-plugin/forklift-inventory
will point to the `forklift-must-gather-api` service, instead of the `forklift-inventory` service

Story CONSOLE-3164: Move main nav components to console-app package

Move `frontend/public/components/nav` to `packages/console-app/src/components/nav` and address any issues resulting from the move.

There will be some expected lint errors relating to cyclical imports. These will require some refactoring to address.

Story CONSOLE-3276: Expose ErrorBoundary capabilities in the SDK

The console has good error boundary components that are useful for dynamic plugin.
Exposing them will enable the plugins to get the same look and feel of handling react errors as console
The minimum requirement right now is to expose the ErrorBoundaryFallbackPage component from
https://github.com/openshift/console/blob/master/frontend/packages/console-shared/src/components/error/fallbacks/ErrorBoundaryFallbackPage.tsx

Story CONSOLE-3140: Expose ResourceIcon to dynamic-plugin-sdk

Following https://coreos.slack.com/archives/C011BL0FEKZ/p1650640804532309, it would be useful for us (network observability team) to have access to ResourceIcon in dynamic-plugin-sdk.

Currently ResourceLink is exported but not ResourceIcon

 

AC:

  • Require the ResourceIcon  from public to dynamic-plugin-sdk
  • Add the component to the dynamic-demo-plugin
  • Add a CI test to check for the ResourceIcon component

 

Story CONSOLE-3188: Alert users on Console plugins page when console operator `spec.managementState` is `Unmanaged`

We should have a global notification or the `Console plugins` page (e.g., k8s/cluster/operator.openshift.io~v1~Console/cluster/console-plugins) should alert users when console operator `spec.managementState` is `Unmanaged` as changes to `enabled` for plugins will have no effect.

Story CONSOLE-3224: Provide an explanation why a dynamic plugin has a status of `Failed`

During the development of https://issues.redhat.com/browse/CONSOLE-3062, it was determined additional information is needed in order to assist a user when troubleshooting a Failed plugin (see https://github.com/openshift/console/pull/11664#issuecomment-1159024959). As it stands today, there is no data available to the console to relay to the user regarding why the plugin Failed. Presumably, a message should be added to NotLoadedDynamicPlugin to address this gap.

 

AC: Add `message` property to NotLoadedDynamicPluginInfo type.

Story CONSOLE-3181: Deprecate useAccessReviewAllowed and useSafetyFirst

Based on API review CONSOLE-3145, we have decided to deprecate the following APIs:

  • useAccessReviewAllowed (use useAccessReview instead)
  • useSafetyFirst

cc Andrew Ballantyne Bryan Florkiewicz 

Currently our `api.md` does not generate docs with "tags" (aka `@deprecated`) – we'll need to add that functionality to the `generate-doc.ts` script. See the code that works for `console-extensions.md`

Story CONSOLE-3336: Make dynamic plugin dependencies optional

To align with https://github.com/openshift/dynamic-plugin-sdk, plugin metadata field dependencies as well as the @console/pluginAPI entry contained within should be made optional.

If a plugin doesn't declare the @console/pluginAPI dependency, the Console release version check should be skipped for that plugin.

Story CONSOLE-3069: Promote ConsolePlugins API version to v1

Currently the ConsolePlugins API version is v1alpha1. Since we are going GA with dynamic plugins we should be creating a v1 version.

This would require updates in following repositories:

  1. openshift/api (add the v1 version and generate a new CRD)
  2. openshift/client-go (picku the changes in the openshift/api repo and generate clients & informers for the new v1 version)
  3. openshift/console-operator repository will using both the new v1 version and v1alpha1 in code and manifests folder.

AC:

  • both v1 and v1alpha1 ConsolePlugins should be passed to the console-config.yaml when the plugins are enabled and present on the cluster.

 

NOTE: This story does not include the conversion webhook change which will be created as a follow on story

Story CONSOLE-3186: Add shared-plugin to the SDK Packages tables docs

`@openshift-console/plugin-shared` (NPM) is a package that will contain shared components that can be upversioned separately by the Plugins so they can keep core compatibility low but upversion and support more shared components as we need them.

This isn't documented today. We need to do that.

Acceptance Criteria

  • Add a note in the "SDK packages" section of the README about the existence of this package and it's purpose
    • The purpose of being a static utility delivery library intended not to be tied to OpenShift Console versions and compatible with multiple version of OpenShift Console
Story CONSOLE-3167: Remove static plugin nav extensions

We neither use nor support static plugin nav extensions anymore so we should remove the API in the static plugin SDK and get rid of related cruft in our current nav components.

 

AC: Remove static plugin nav extensions code. Check the navigation code for any references to the old API.

Epic CONSOLE-3189: OCP 4.12 - OLM Epic

This epic contains all the OLM related stories for OCP release-4.12

Epic Goal

  • Track all the stories under a single epic
Story CONSOLE-3190: Heterogeneous architecture clusters - Operator Hub filtering

This enhancement Introduces support for provisioning and upgrading heterogenous architecture clusters in phases.

 

We need to scan through the compute nodes and build a set of supported architectures from those. Each node on the cluster has a label for architecture: e.g. `kuberneties.io/arch:arm64`, `kubernetes.io/arch:amd64` etc. Based on the set of supported architectures console will need to surface only those operators in the Operator Hub, which are supported on our Nodes. Each operator's PackageManifest contains a labels that indicates whats the operator's supported architecture, e.g.  `operatorframework.io/arch.s390x: supported`. An operator can be supported on multiple architectures

AC:

  1. Implement logic in the console's backend to read the set of architecture types from console-config.yaml and set it as a SERVER_FLAG.nodeArchitectures (Change similar to https://github.com/openshift/console/commit/39aabe171a2e89ed3757ac2146d252d087fdfd33)
  2. In Operator hub render only operators that are support on any given node, based on the SERVER_FLAG.nodeArchitectures field implemented in CONSOLE-3242.

 

OS and arch filtering: https://github.com/openshift/console/blob/2ad4e17d76acbe72171407fc1c66ca4596c8aac4/frontend/packages/operator-lifecycle-manager/src/components/operator-hub/operator-hub-items.tsx#L49-L86

 

@jpoulin is good to ask about heterogeneous clusters.

Story CONSOLE-3242: Heterogeneous architecture clusters - Console backend changes

This enhancement Introduces support for provisioning and upgrading heterogenous architecture clusters in phases.

 

We need to scan through the compute nodes and build a set of supported architectures from those. Each node on the cluster has a label for architecture: e.g. kubernetes.io/arch=arm64, kubernetes.io/arch=amd64 etc. Based on the set of supported architectures console will need to surface only those operators in the Operator Hub, which are supported on our Nodes.

 

AC: 

  1. Implement logic in the console-operator that will scan though all the nodes and build a set of all the architecture types that the cluster nodes run on and pass it to the console-config.yaml
  2. Add unit and e2e test cases in the console-operator repository.

 

@jpoulin is good to ask about heterogeneous clusters.

Epic CORS-1774: Enable OpenShift IPI Installer to deploy OCP to a shared VPC in GCP [Tech Preview]

Epic Goal

  • Enable OpenShift IPI Installer to deploy OCP to a shared VPC in GCP.
  • The host project is where the VPC and subnets are defined. Those networks are shared to one or more service projects.
  • Objects created by the installer are created in the service project where possible. Firewall rules may be the only exception.
  • Documentation outlines the needed minimal IAM for both the host and service project.

Why is this important?

  • Shared VPC's are a feature of GCP to enable granular separation of duties for organizations that centrally manage networking but delegate other functions and separation of billing. This is used more often in larger organizations where separate teams manage subsets of the cloud infrastructure. Enterprises that use this model would also like to create IPI clusters so that they can leverage the features of IPI. Currently organizations that use Shared VPC's must use UPI and implement the features of IPI themselves. This is repetative engineering of little value to the customer and an increased risk of drift from upstream IPI over time. As new features are built into IPI, organizations must become aware of those changes and implement them themselves instead of getting them "for free" during upgrades.

Scenarios

  1. Deploy cluster(s) into service project(s) on network(s) shared from a host project.

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.
  • ...

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story CORS-2043: [GCP] Passsthrough installer service account to Terraform resources.

User Story:

As a user, I want to be able to:

  • skip creating service accounts in Terraform when using passthrough credentialsMode.
  • pass the installer service account to Terraform to be used as the service account for instances when using passthrough credentialsMode.

so that I can achieve

  • creating an IPI cluster using Shared VPC networks using a pre-created service account with the necessary permissions in the Host Project.

Acceptance Criteria:

Description of criteria:

  • Upstream documentation
  • Point 1
  • Point 2
  • Point 3

(optional) Out of Scope:

Detail about what is specifically not being delivered in the story

Engineering Details:

Epic IR-192: Manifest lists support

Epic Goal

  • Support manifest lists by image streams and the integrated registry. Clients should be able to pull/push manifests lists from/into the integrated registry. They also should be able to import images via `oc import-image` and them pull them from the internal registry.

Why is this important?

  • Manifest lists are becoming more and more popular. Customers want to mirror manifest lists into the registry and be able to pull them by digest.

Scenarios

  1. Manifest lists can be pushed into the integrated registry
  2. Imported manifests list can be pulled from the integrated registry
  3. Image triggers work with manifest lists

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.
  • Existing functionality shouldn't change its behavior

Dependencies (internal and external)

  1. ...

Previous Work (Optional)

  1. https://github.com/openshift/enhancements/blob/master/enhancements/manifestlist/manifestlist-support.md

Open questions

  1. Can we merge creation of images without having the pruner?

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story IR-259: As an openshift/client-go user I want to be able to specify that an ImageStream tag refers to a manifest list so that I can import all manifests from it

ACCEPTANCE CRITERIA

  • The ImageStream object should contain a new flag indicating that it refers to a manifest list
  • openshift-controller-manager uses new openshift/api code to import image streams
  • changing `importMode` of an image stream tag triggers a new import (i.e. updates generation in the tag spec)

NOTES

Epic MCO-297: Finish upgrade in place and spike layering-in-hypershift

This is a follow up Epic to https://issues.redhat.com/browse/MCO-144, which aimed to get in-place upgrades for Hypershift. This epic aims to capture additional work to focus on using CoreOS/OCP layering into Hypershift, which has benefits such as:

 

 - removing or reducing the need for ignition

 - maintaining feature parity between self-driving and managed OCP models

 - adding additional functionality such as hotfixes

Story MCO-258: Harden nodepool logic to handle multiple upgrades

Right now in https://github.com/openshift/hypershift/pull/1258 you can only perform one upgrade at a time. Multiple upgrades will break due to controller logic

 

Properly create logic to handle manifest creation/updates and deletion, so the logic is more bulletproof

Story MCO-256: MCD: add handling for disruptionless updates

Currently not implemented, and will require the MCD hypershift mode to be adjusted to handle disruptionless upgrades like regular MCD

Epic METAL-158: Streamline Machine OS image management

Changes made in METAL-1 open up opportunities to improve our handling of images by cleaning up redundant code that generates extra work for the user and extra load for the cluster.

Story METAL-157: Don't run image cache unless required

We only need to run the image cache DaemonSet if there is a QCOW URL to be mirrored (effectively this means a cluster installed with 4.9 or earlier). We can stop deploying it for new clusters installed with 4.10 or later.

Currently, the image-customization-controller relies on the image cache running on every master to provide the shared hostpath volume containing the ISO and initramfs. The first step is to replace this with a regular volume and an init container in the i-c-c pod that extracts the images from machine-os-images. We can use the copy-metal -image-build flag (instead of -all used in the shared volume) to provide only the required images.

Once i-c-c has its own volume, we can switch the image extraction in the metal3 Pod's init container to use the -pxe flag instead of -all.

The machine-os-images init container for the image cache (not the metal3 Pod) can be removed. The whole image cache deployment is now optional and need only be started if provisioningOSDownloadURL is set (and in fact should be deleted if it is not).

Epic METAL-159: Build Ironic Container Images on top of RHEL9

We plan to build Ironic Container Images using RHEL9 as base image in OCP 4.12

This is required because the ironic components have abandoned support for CentOS Stream 8 and Python 3.6/3.7 upstream during the most recent development cycle that will produce the stable Zed release, in favor of CentOS Stream 9 and Python 3.8/3.9

More info on RHEL8 to RHEL9 transition in OCP can be found at https://docs.google.com/document/d/1N8KyDY7KmgUYA9EOtDDQolebz0qi3nhT20IOn4D-xS4

Epic MGMT-10400: Disk cleaning improvements

Epic Goal

  • To improve the reliability of disk cleaning before installation and to provide the user with sufficient warning regarding the consequences of the cleaning

Why is this important?

  • Insufficient cleaning can lead to installation failure
  • Insufficient warning can lead to complaints of unexpected data loss

Scenarios

  1.  

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.
  • ...

Dependencies (internal and external)

  1. ...

Previous Work (Optional):

  1. …

Open questions::

  1. …

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Bug MGMT-11695: Disk cleanup does not clean all volume groups in a physical volume

Description of the problem:
When running assisted-installer on a machine where is more than one volume group per physical volume. Only the first volume group will be cleaned up. This leads to problems later and will lead to errors such as 

Failed - failed executing nsenter [--target 1 --cgroup --mount --ipc --pid -- pvremove /dev/sda -y -ff], Error exit status 5, LastOutput "Can't open /dev/sda exclusively. Mounted filesystem?

How reproducible:

Set up a VM with more than one volume group per physical volume. As an example, look at the following sample from a customer cluster.

List block devices
/usr/bin/lsblk -o NAME,MAJ:MIN,SIZE,TYPE,FSTYPE,KNAME,MODEL,UUID,WWN,HCTL,VENDOR,STATE,TRAN,PKNAME
NAME              MAJ:MIN   SIZE TYPE FSTYPE      KNAME MODEL            UUID                                   WWN                HCTL       VENDOR   STATE   TRAN PKNAME
loop0               7:0   125.9G loop xfs         loop0                  c080b47b-2291-495c-8cc0-2009ebc39839                                                       
loop1               7:1   885.5M loop squashfs    loop1                                                                                                             
sda                 8:0   894.3G disk             sda   INTEL SSDSC2KG96                                        0x55cd2e415235b2db 1:0:0:0    ATA      running sas  
|-sda1              8:1     250M part             sda1                                                          0x55cd2e415235b2db                                  sda
|-sda2              8:2     750M part ext2        sda2                   3aa73c72-e342-4a07-908c-a8a49767469d   0x55cd2e415235b2db                                  sda
|-sda3              8:3      49G part xfs         sda3                   ffc3ccfe-f150-4361-8ae5-f87b17c13ac2   0x55cd2e415235b2db                                  sda
|-sda4              8:4   394.2G part LVM2_member sda4                   Ua3HOc-Olm4-1rma-q0Ug-PtzI-ZOWg-RJ63uY 0x55cd2e415235b2db                                  sda
`-sda5              8:5     450G part LVM2_member sda5                   W8JqrD-ZvaC-uNK9-Y03D-uarc-Tl4O-wkDdhS 0x55cd2e415235b2db                                  sda
  `-nova-instance 253:0     3.1T lvm  ext4        dm-0                   d15e2de6-2b97-4241-9451-639f7b14594e                                          running      sda5
sdb                 8:16  894.3G disk             sdb   INTEL SSDSC2KG96                                        0x55cd2e415235b31b 1:0:1:0    ATA      running sas  
`-sdb1              8:17  894.3G part LVM2_member sdb1                   6ETObl-EzTd-jLGw-zVNc-lJ5O-QxgH-5wLAqD 0x55cd2e415235b31b                                  sdb
  `-nova-instance 253:0     3.1T lvm  ext4        dm-0                   d15e2de6-2b97-4241-9451-639f7b14594e                                          running      sdb1
sdc                 8:32  894.3G disk             sdc   INTEL SSDSC2KG96                                        0x55cd2e415235b652 1:0:2:0    ATA      running sas  
`-sdc1              8:33  894.3G part LVM2_member sdc1                   pBuktx-XlCg-6Mxs-lddC-qogB-ahXa-Nd9y2p 0x55cd2e415235b652                                  sdc
  `-nova-instance 253:0     3.1T lvm  ext4        dm-0                   d15e2de6-2b97-4241-9451-639f7b14594e                                          running      sdc1
sdd                 8:48  894.3G disk             sdd   INTEL SSDSC2KG96                                        0x55cd2e41521679b7 1:0:3:0    ATA      running sas  
`-sdd1              8:49  894.3G part LVM2_member sdd1                   exVSwU-Pe07-XJ6r-Sfxe-CQcK-tu28-Hxdnqo 0x55cd2e41521679b7                                  sdd
  `-nova-instance 253:0     3.1T lvm  ext4        dm-0                   d15e2de6-2b97-4241-9451-639f7b14594e                                          running      sdd1
sr0                11:0     989M rom  iso9660     sr0   Virtual CDROM0   2022-06-17-18-18-33-00                                    0:0:0:0    AMI      running usb

Now run the assisted installer and try to install an SNO node on this machine, you will find that the installation will fail with a message that indicates that it could not exclusively access /dev/sda

Actual results:

 The installation will fail with a message that indicates that it could not exclusively access /dev/sda

Expected results:

The installation should proceed and the cluster should start to install.

Suspected Cases
https://issues.redhat.com/browse/AITRIAGE-3809
https://issues.redhat.com/browse/AITRIAGE-3802
https://issues.redhat.com/browse/AITRIAGE-3810

Bug MGMT-10085: [Assisted-4.10][Staging] Disk cleanup fails the installation in case disk setup is lvm on raid

Description of the problem:

Cluster Installation fail if installation disk has lvm on raid:

Host: test-infra-cluster-3cc862c9-master-0, reached installation stage Failed: failed executing nsenter [--target 1 --cgroup --mount --ipc --pid -- mdadm --stop /dev/md0], Error exit status 1, LastOutput "mdadm: Cannot get exclusive access to /dev/md0:Perhaps a running process, mounted filesystem or active volume group?"

How reproducible:

100%

Steps to reproduce:

1. Install a cluster while master nodes has disk with LVM on RAID (reproduces using test: https://gitlab.cee.redhat.com/ocp-edge-qe/kni-assisted-installer-auto/-/blob/master/api_tests/test_disk_cleanup.py#L97)

Actual results:

Installation failed

Expected results:

Installation success

Epic MGMT-8804: Fix flaky/broken jobs

Epic Goal

  • Increase success-rate of of our CI jobs
  • Improve debugability / visibility or tests 

Why is this important?

  • Failed presubmit jobs (required or optional) can make an already tested+approved PR to not get in
  • Failed periodic jobs interfere our visibility around stability of features
Epic NHE-6: Track cross-project development needs for HWOL
Bug OCPBUGS-3468: Disable check_pkt_length in OVN-K for OvS Hardware Offload Cases

Description of problem:

check_pkt_length cannot be offloaded without
1) sFlow offload patches in Openvswitch
2) Hardware driver support.

Since 1) will not be done anytime soon. We need a work around for the check_pkt_length issue.

Version-Release number of selected component (if applicable):

4.11/4.12

How reproducible:

Always

Steps to Reproduce:

1. Any flow that has check_pkt_len()
  5-b: Pod -> NodePort Service traffic (Pod Backend - Different Node)
  6-b: Pod -> NodePort Service traffic (Host Backend - Different Node)
  4-b: Pod -> Cluster IP Service traffic (Host Backend - Different Node)
  10-b: Host Pod -> Cluster IP Service traffic (Host Backend - Different Node)
  11-b: Host Pod -> NodePort Service traffic (Pod Backend - Different Node)
  12-b: Host Pod -> NodePort Service traffic (Host Backend - Different Node)  

Actual results:

Poor performance due to upcalls when check_pkt_len() is not supported.

Expected results:

Good performance.

Additional info:

https://docs.google.com/spreadsheets/d/1LHY-Af-2kQHVwtW4aVdHnmwZLTiatiyf-ySffC8O5NM/edit#gid=670206692
Epic OCPBUILD-44: Dev Preview - User Namespace Builds

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

  • Run OpenShift builds that do not execute as the "root" user on the host node.

Why is this important?

  • OpenShift builds require an elevated set of capabilities to build a container image
  • Builds currently run as root to maintain adequate performance
  • Container workloads should run as non-root from the host's perspective. Containers running as root are a known security risk.
  • Builds currently run as root and require a privileged container. See BUILD-225 for removing the privileged container requirement.

Scenarios

  1. Run BuildConfigs in a multi-tenant environment
  2. Run BuildConfigs in a heightened security environment/deployment

Acceptance Criteria

  • Developers can opt into running builds in a cri-o user namespace by providing an environment variable with a specific value.
  • When the correct environment variable is provided, builds run in a cri-o user namespace, and the build pod does not require the "privileged: true" security context.
  • User namespace builds can pass basic test scenarios for the Docker and Source strategy build.
  • Steps to run unprivileged builds are documented.

Dependencies (internal and external)

  1. Buildah supports running inside a non-privileged container
  2. CRI-O allows workloads to opt into running containers in user namespaces.

Previous Work (Optional):

  1. BUILD-225 - remove privileged requirement for builds.

Open questions::

  1. …

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story BUILD-433: Run Unprivileged Builds with Environment Variable

User Story

As a developer building container images on OpenShift
I want to specify that my build should run without elevated privileges
So that builds do not run as root from the host's perspective with elevated privileges

Acceptance Criteria

  • Developers can provide an environment variable to indicate the build should not use privileged containers
  • When the correct env var + value is specified, builds run in a user namespace (non-root on the host)

QE Impact

No QE required for Dev Preview. OpenShift regression testing will verify that existing behavior is not impacted.

Docs Impact

We will need to document how to enable this feature, with sufficient warnings regarding Dev Preview.

PX Impact

This likely warrants an OpenShift blog post, potentially?

Notes

Epic OCPRHV-684: CSI driver should refresh credentials automatically

Make sure that the CSI driver automatically updates oVirt credentials when they are updated in OpenShift.

Epic ODC-6695: Allow admins to disable the Dev Catalog, or one or more of its sub-catalogs

Problem:

Customers don't want their users to have access to some/all of the items which are available in the Developer Catalog.  The request is to change access for the cluster, not per user or persona.

Goal:

Provide a form driven experience to allow cluster admins easily disable the Developer Catalog, or one or more of the sub catalogs in the Developer Catalog.

Why is it important?

Multiple customer requests.

Acceptance criteria:

  1. As a cluster admin, I can hide/disable access to the developer catalog for all users across all namespaces.
  2. As a cluster admin, I can hide/disable access to a specific sub-catalog in the developer catalog for all users across all namespaces.
    1. Builder Images
    2. Templates
    3. Helm Charts
    4. Devfiles
    5. Operator Backed

Notes

We need to consider how this will work with subcatalogs which are installed by operators: VMs, Event Sources, Event Catalogs, Managed Services, Cloud based services

Dependencies (External/Internal):

Design Artifacts:

Exploration:

Note:

Story ODC-6779: Hide sub-catalog(s) in the developer catalog or the entire dev catalog based on customization

Description

As an admin, I want to hide sub-catalogs in the developer catalog or hide the developer catalog completely based on the customization.

Acceptance Criteria

  1. Hide all links to the sub-catalog(s) from the add page, topology actions, empty states, quick search, and the catalog itself
  2. The sub-catalog should show Not found if the user opens the sub-catalog directly
  3. The feature should not be hidden if a sub-catalog option is disabled

Additional Details:

Story ODC-6778: Extend the console-operator CRD and API for disabling the access to specific sub-catalog(s) in the dev catalog or the complete dev catalog

Description

As an admin, I want to hide/disable access to specific sub-catalogs in the developer catalog or the complete dev catalog for all users across all namespaces.

Based on the https://issues.redhat.com/browse/ODC-6732 enhancement proposal, it is required to extend the console configuration CRD to enable the cluster admins to configure this data in the console resource

Acceptance Criteria

Extend the "customization" spec type definition for the CRD in the openshift/api project

Additional Details:

Previous customization work:

  1. https://issues.redhat.com/browse/ODC-5416
  2. https://issues.redhat.com/browse/ODC-5020
  3. https://issues.redhat.com/browse/ODC-5447
Story ODC-6780: Provide a code snippet for the console CRD

Description

As a cluster-admin, I should be able to see a code snippet that shows how to enable sub-catalogs or the entire dev catalog.

Based on the https://issues.redhat.com/browse/ODC-6732 enhancement proposal, the cluster admin can add sub-catalog(s)  from the Developer Catalog or the Dev catalog as a whole.

To support the cluster-admin to configure the sub-catalog list correctly, the developer console should provide a code snippet for the customization yaml resource (Console CRD).

Acceptance Criteria

  1. When the admin opens the Console CRD there is a snippet in the sidebar which provides a default YAML, which supports the admin to add sub-catalogs/the whole dev catalog

Additional Details:

Previous work:

  1. https://issues.redhat.com/browse/ODC-5080
  2. https://issues.redhat.com/browse/ODC-5449
Epic OPNET-19: OPNET Backlog

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

  • ...

Why is this important?

  • …

Scenarios

  1. ...

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.
  • ...

Dependencies (internal and external)

  1. ...

Previous Work (Optional):

  1. …

Open questions::

  1. …

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Feature Request RFE-2962: Swap order of configure-ovs and nodeip-configuration

We have been running into a number of problems with configure-ovs and nodeip-configuration selecting different interfaces in OVNK deployments. This causes connectivity issues, so we need some way to ensure that everything uses the same interface/IP.

Currently configure-ovs runs before nodeip-configuration, but since nodeip-configuration is the source of truth for IP selection regardless of CNI plugin, I think we need to look at swapping that order. That way configure-ovs could look at what nodeip-configuration chose and not have to implement its own interface selection logic.

I'm targeting this at 4.12 because even though there's probably still time to get it in for 4.11, changing the order of boot services is always a little risky and I'd prefer to do it earlier in the cycle so we have time to tease out any issues that arise. We may need to consider backporting the change though since this has been an issue at least back to 4.10.

Epic OPRUN-2600: Update OLM to address the security changes coming from PSA in 4.12

Epic Goal

  • Facilitate the transition to for OLM and content to PSA enforcing the `restricted` security profile
  • Use the label synch'er to enforce the required security profile
  • Current content should work out-of-the-box as is
  • Upgrades should not be blocked

Why is this important?

  • PSA helps secure the cluster by enforcing certain security restrictions that the pod must meet to be scheduled
  • 4.12 will enforce the `restricted` profile, which will affect the deployment of operators in `openshift-*` namespaces 

Scenarios

  1. Admin installs operator in an `openshift-*`namespace that is not managed by the label syncher -> label should be applied
  2. Admin installs operator in an `openshift-*` namespace that has a label asking the label syncher to not reconcile it -> nothing changes

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.
  • Done only downstream
  • Transition documentation written and reviewed

Dependencies (internal and external)

  1. label syncher (still searching for the link)

Open questions::

  1. Is this only for openshift-* namespaces?

Resources

Stakeholders

  • Daniel S...?

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story OPRUN-2695: [downstream] OLM controller plug-in for openshift-* namespace labelling

As an admin, I would like openshift-* namespaces with an operator to be labeled with security.openshift.io/scc.podSecurityLabelSync=true to ensure the continual functioning of operators without manual intervention. The label should only be applied to openshift-* namespaces with an operator (the presence of a ClusterServiceVersion resource) IF the label is not already present. This automation will help smooth functioning of the cluster and avoid frivolous operational events.

Context: As part of the PSA migration period, Openshift will ship with the "label sync'er" - a controller that will automatically adjust PSA security profiles in response to the workloads present in the namespace. We can assume that not all operators (produced by Red Hat, the community or ISVs) will have successfully migrated their deployments in response to upstream PSA changes. The label sync'er will sync, by default, any namespace not prefixed with "openshift-", of which an explicit label (security.openshift.io/scc.podSecurityLabelSync=true) is required for sync.

A/C:
 - OLM operator has been modified (downstream only) to label any unlabelled "openshift-" namespace in which a CSV has been created
 - If a labeled namespace containing at least one non-copied csv becomes unlabelled, it should be relabelled 
 - The implementation should be done in a way to eliminate or minimize subsequent downstream sync work (it is ok to make slight architectural changes to the OLM operator in the upstream to enable this)

Epic OU-34: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Epic RHSTOR-3173: Provide an indication of advanced features usage

Goal
Provide an indication that advanced features are used

Problem

Today, customers and RH don't have the information on the actual usage of advanced features.

Why is this important?

  1. Better focus upsell efforts
  2. Compliance information for customers that are not aware they are not using the right subscription

 

Prioritized Scenarios

In Scope
1. Add a boolean variable in our telemetry to mark if the customer is using advanced features (PV encryption, encryption with KMS, external mode). 

Not in Scope

Integrate with subscription watch - will be done by the subscription watch team with our help.

Customers

All

Customer Facing Story
As a compliance manager, I should be able to easily see if all my clusters are using the right amount of subscriptions

What does success look like?

A clear indication in subscription watch for ODF usage (either essential or advanced). 

Task MON-2654: Add a boolean value to indicate advanced feature usage into telemetry

1. Proposed title of this feature request

  • Request to add a bool variable into telemetry which indicates the usage of any of the advanced feature, like PV encryption or KMS encryption or external mode etc.

2. What is the nature and description of the request?

  • Today, customers and RH don't have the information on the actual usage of advanced features. This feature will help RH to have a better indication on the statistics of customers using the advanced features and focus better on upsell efforts.

3. Why does the customer need this? (List the business requirements here)

  • As a compliance manager, I should be able to easily see if all my clusters are using the right amount of subscriptions.

4. List any affected packages or components.

  • Telemetry

_____________________

Link to main epic: https://issues.redhat.com/browse/RHSTOR-3173

 

Epic RHSTOR-3379: Migrate to odf-console from OCP Console (remaining bits)

We migrated most component as part of https://issues.redhat.com/browse/RHSTOR-2165
We now have a few components remaining roughly 15 to 20%. This epic tragets

Epic SDN-3127: [4.12] Network Tooling

This epic tracks network tooling improvements for 4.12

New framework and process should be developed to make sharing network tools with devs, support and customers convenient. We are going to add some tools for ovn troubleshooting before ovn-k goes default, also some tools that we got from customer cases, and some more to help analyze and debug collected logs based on stable must-gather/sosreport format we get now thanks to 4.11 Epic.

Our estimation for this Epic is 1 engineer * 2 Sprints

WHY:
This epic is important to help improve the time it takes our customers and our team to understand an issue within the cluster.
A focus of this epic is to develop tools to quickly allow debugging of a problematic cluster. This is crucial for the engineering team to help us scale. We want to provide a tool to our customers to help lower the cognitive burden to get at a root cause of an issue.

 

Story SDN-3432: OVN-K alerts: add OVN controller disconnection alert

Alert if any of the ovn controllers disconnected for a period of time from the southbound database using metric ovn_controller_southbound_database_connected.

The metric updates every 2 minutes so please be mindful of this when creating the alert.

If the controller is disconnected for 10 minutes, fire an alert.

DoD: Merged to CNO and tested by QE

Epic SDN-3199: [4.12] k8s 1.25 rebase

This Epic is here to track the rebase we need to do when kube 1.25 is GA https://www.kubernetes.dev/resources/release/

Keeping this in mind can help us plan our time better. ATTOW GA is planned for August 23

https://docs.google.com/document/d/1h1XsEt1Iug-W9JRheQas7YRsUJ_NQ8ghEMVmOZ4X-0s/edit --> this is the link for rebase help

Other Complete

This section includes Jira cards that are not linked to either an Epic or a Feature. These tickets were completed when this image was assembled

Bug OCPBUGS-1950: Devfile samples (in Developer Catalog) link doesn't include the current selected namespace

Description of problem:
When opening the Devfile sample developer catalog, switch the project in another browser tab, and then open devfile samples link in a new tab, the current project context is getting lost.

Version-Release number of selected component (if applicable):
4.12, expecting that this happen also in older versions

How reproducible:
Always

Steps to Reproduce:
1. Switch to the developer perspective, navigate to Add > Samples
2. Open a new browser tab and create a new project
3. Ctrl+click a sample in the first tab.

Actual results:
The project has also changed in the "Import sample" page

Expected results:
The project should be used also for the new "Import sample" page

Additional info:
We had this issue earlier for other catalog entries. Other samples works already fine, just the Devfile sample links doesn't contain the current namespace.

Bug OCPBUGS-2493: TestUnmanagedDNSToManagedDNSInternal E2E failing to verify connectivity

Description of problem:

TestUnmanagedDNSToManagedDNSInternalIngressController E2E test is failing on the error:
{
unmanaged_dns_test.go:272: failed to verify connectivity with workload with reqURL http://10.0.128.7 using external client: timed out waiting for the condition

How reproducible:

About 75% of the time.

Version-Release number of selected component (if applicable):

4.12

How reproducible:

75%

Steps to Reproduce:

1. Run CI E2E tests on cluster-ingress-operator or 
make test-e2e TEST=TestUnmanagedDNSToManagedDNSInternalIngressController

Actual results:

E2E test fails about 75% of the time

Expected results:

E2E should always pass

Additional info:

 
Bug OCPBUGS-3346: [perf/scale] libovsdb builds transaction logs but throws them away

libovsdb builds transaction log messages for every transaction and then throws them away if the log level is not 4 or above. This wastes a bunch of CPU at scale and increases pod ready latency.

Bug OCPBUGS-435: Dropdown items on storageclass creation page need i18n support

Description of problem:

On storageclass creation page, the dropdown items for "Reclaim policy" and "Volume binding tyep" are not marked for i18n.

Version-Release number of selected component (if applicable):

4.12.0-0.nightly-2022-08-22-143022

How reproducible:

always

Steps to Reproduce:
1.Go to storageclass creation page, check if dropdown items for "Reclaim policy" and "Volume binding type" support i18n.
2.
3.

Actual results:

1. They are not marked for i18n.

Expected results:

1. Should support i18n.

Additional info:

Bug OCPBUGS-825: Available=False with no reason

Searching recent 4.12 CI, there are a number of failures in the clusteroperator/machine-config should not change condition/Available test case:

$ w3m -dump -cols 200 'https://search.ci.openshift.org/?search=clusteroperator%2Fmachine-config+should+not+change+condition%2FAvailable&maxAge=48h&type=junit' | grep '4[.]12.*failures match' | sort
periodic-ci-openshift-release-master-ci-4.12-e2e-aws-ovn-upgrade (all) - 129 runs, 53% failed, 6% of failures match = 3% impact
periodic-ci-openshift-release-master-ci-4.12-e2e-aws-sdn-techpreview-serial (all) - 6 runs, 50% failed, 67% of failures match = 33% impact
periodic-ci-openshift-release-master-ci-4.12-e2e-azure-ovn-upgrade (all) - 60 runs, 50% failed, 3% of failures match = 2% impact
periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade (all) - 129 runs, 56% failed, 8% of failures match = 5% impact
periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-azure-sdn-upgrade (all) - 129 runs, 69% failed, 12% of failures match = 9% impact
periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-gcp-ovn-rt-upgrade (all) - 8 runs, 38% failed, 67% of failures match = 25% impact
periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-gcp-ovn-upgrade (all) - 60 runs, 57% failed, 6% of failures match = 3% impact
periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-gcp-sdn-upgrade (all) - 12 runs, 42% failed, 20% of failures match = 8% impact
periodic-ci-openshift-release-master-nightly-4.12-e2e-aws-sdn-upgrade (all) - 60 runs, 40% failed, 4% of failures match = 2% impact
periodic-ci-openshift-release-master-nightly-4.12-e2e-metal-ipi-sdn-serial-virtualmedia (all) - 6 runs, 100% failed, 17% of failures match = 17% impact
periodic-ci-openshift-release-master-nightly-4.12-e2e-metal-ipi-sdn-upgrade (all) - 6 runs, 67% failed, 25% of failures match = 17% impact
periodic-ci-openshift-release-master-nightly-4.12-e2e-metal-ipi-serial-ovn-dualstack (all) - 6 runs, 67% failed, 25% of failures match = 17% impact
periodic-ci-openshift-release-master-nightly-4.12-e2e-vsphere-ovn-techpreview-serial (all) - 9 runs, 56% failed, 20% of failures match = 11% impact
periodic-ci-openshift-release-master-nightly-4.12-upgrade-from-stable-4.11-e2e-metal-ipi-upgrade (all) - 6 runs, 100% failed, 17% of failures match = 17% impact
periodic-ci-openshift-release-master-nightly-4.12-upgrade-from-stable-4.11-e2e-metal-ipi-upgrade-ovn-ipv6 (all) - 6 runs, 83% failed, 20% of failures match = 17% impact
periodic-ci-openshift-release-master-okd-4.12-e2e-vsphere (all) - 25 runs, 100% failed, 4% of failures match = 4% impact
release-openshift-ocp-installer-e2e-gcp-serial-4.12 (all) - 6 runs, 83% failed, 20% of failures match = 17% impact

Doesn't seem like reason is getting set?

$ curl -s 'https://search.ci.openshift.org/search?name=periodic-ci-openshift-release-master-ci-4.12-e2e-aws-ovn-upgrade&search=clusteroperator%2Fmachine-config+should+not+change+condition%2FAvailable&maxAge=48h&type=junit&context=15' | jq -r 'to_entries[].value | to_entries[].value[].context[]' | grep 'clusteroperator/machine-config condition/Available status/False reason'
Aug 31 01:13:56.724 - 698s  E clusteroperator/machine-config condition/Available status/False reason/Cluster not available for [{operator 4.12.0-0.ci-2022-08-30-194744}]
Aug 31 09:09:15.460 - 1078s E clusteroperator/machine-config condition/Available status/False reason/Cluster not available for [{operator 4.12.0-0.ci-2022-08-30-194744}]
Sep 01 03:31:24.808 - 1131s E clusteroperator/machine-config condition/Available status/False reason/Cluster not available for [{operator 4.12.0-0.ci-2022-08-31-111359}]
Sep 01 07:15:58.029 - 1085s E clusteroperator/machine-config condition/Available status/False reason/Cluster not available for [{operator 4.12.0-0.ci-2022-08-31-111359}]

Example runs in the job I've randomly selected to drill into:

$ curl -s 'https://search.ci.openshift.org/search?name=periodic-ci-openshift-release-master-ci-4.12-e2e-aws-ovn-upgrade&search=clusteroperator%2Fmachine-config+should+not+change+condition%2FAvailable&maxAge=48h&type=junit' | jq -r 'keys[]'
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.12-e2e-aws-ovn-upgrade/1564757706458271744
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.12-e2e-aws-ovn-upgrade/1564879945233076224
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.12-e2e-aws-ovn-upgrade/1565158084484009984
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.12-e2e-aws-ovn-upgrade/1565212566194491392

Drilling into that last run, the Available=False was the whole pool-update phase:

And details from the origin's monitor:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.12-e2e-aws-ovn-upgrade/1565212566194491392/artifacts/e2e-aws-ovn-upgrade/openshift-e2e-test/build-log.txt | grep clusteroperator/machine-config
Sep 01 07:15:57.629 E clusteroperator/machine-config condition/Degraded status/True reason/RenderConfigFailed changed: Failed to resync 4.12.0-0.ci-2022-08-31-111359 because: refusing to read osImageURL version "4.12.0-0.ci-2022-09-01-053740", operator version "4.12.0-0.ci-2022-08-31-111359"
Sep 01 07:15:57.629 - 49s   E clusteroperator/machine-config condition/Degraded status/True reason/Failed to resync 4.12.0-0.ci-2022-08-31-111359 because: refusing to read osImageURL version "4.12.0-0.ci-2022-09-01-053740", operator version "4.12.0-0.ci-2022-08-31-111359"
Sep 01 07:15:58.029 E clusteroperator/machine-config condition/Available status/False changed: Cluster not available for [{operator 4.12.0-0.ci-2022-08-31-111359}]
Sep 01 07:15:58.029 - 1085s E clusteroperator/machine-config condition/Available status/False reason/Cluster not available for [{operator 4.12.0-0.ci-2022-08-31-111359}]
Sep 01 07:16:47.000 I /machine-config reason/OperatorVersionChanged clusteroperator/machine-config-operator started a version change from [{operator 4.12.0-0.ci-2022-08-31-111359}] to [{operator 4.12.0-0.ci-2022-09-01-053740}]
Sep 01 07:16:47.377 W clusteroperator/machine-config condition/Progressing status/True changed: Working towards 4.12.0-0.ci-2022-09-01-053740
Sep 01 07:16:47.377 - 1037s W clusteroperator/machine-config condition/Progressing status/True reason/Working towards 4.12.0-0.ci-2022-09-01-053740
Sep 01 07:16:47.405 W clusteroperator/machine-config condition/Degraded status/False changed: 
Sep 01 07:18:02.614 W clusteroperator/machine-config condition/Upgradeable status/False reason/PoolUpdating changed: One or more machine config pools are updating, please see `oc get mcp` for further details
Sep 01 07:34:03.000 I /machine-config reason/OperatorVersionChanged clusteroperator/machine-config-operator version changed from [{operator 4.12.0-0.ci-2022-08-31-111359}] to [{operator 4.12.0-0.ci-2022-09-01-053740}]
Sep 01 07:34:03.699 W clusteroperator/machine-config condition/Available status/True changed: Cluster has deployed [{operator 4.12.0-0.ci-2022-08-31-111359}]
Sep 01 07:34:03.715 W clusteroperator/machine-config condition/Upgradeable status/True changed: 
Sep 01 07:34:04.065 I clusteroperator/machine-config versions: operator 4.12.0-0.ci-2022-08-31-111359 -> 4.12.0-0.ci-2022-09-01-053740
Sep 01 07:34:04.663 W clusteroperator/machine-config condition/Progressing status/False changed: Cluster version is 4.12.0-0.ci-2022-09-01-053740
[bz-Machine Config Operator] clusteroperator/machine-config should not change condition/Available
[bz-Machine Config Operator] clusteroperator/machine-config should not change condition/Degraded

No idea if whatever was happening there is the same thing that was happening in other runs, and I haven't checked 4.11 and earlier either. The test-case is non-fatal, so it doesn't break CI, but it can cause noise like ClusterOperatorDown if it continues for 10 or more minutes. Whic PromeCIeus says actually fired in this run, although apparently the origin monitors didn't notice to complain:

So parallel asks (and I'm happy to shard into separate bugs, if that's helpful):

  • Set a reason when you go Available=False, so Telemetry can collect information to aggregate and hunt for frequent reasons to prioritize improvements.
  • Figure out at least one reason why we're going Available=False in apparently healthy CI runs. If we find and fix one reason, we can circle back later to see if there are more that remain unfixed.
Bug OCPBUGS-1: Test Bug

Test description

Bug OCPBUGS-1718: prometheus-k8s-0 ends in CrashLoopBackOff with evel=error err="opening storage failed: /prometheus/chunks_head/000002: invalid magic number 0" on SNO after hard reboot tests

Description of problem:

prometheus-k8s-0 ends in CrashLoopBackOff with evel=error err="opening storage failed: /prometheus/chunks_head/000002: invalid magic number 0" on SNO after hard reboot tests

Version-Release number of selected component (if applicable):

4.11.6

How reproducible:

Not always, after ~10 attempts

Steps to Reproduce:

1. Deploy SNO with Telco DU profile applied
2. Hard reboot node via out of band interface
3. oc -n openshift-monitoring get pods prometheus-k8s-0

Actual results:

NAME               READY   STATUS             RESTARTS          AGE
prometheus-k8s-0   5/6     CrashLoopBackOff   125 (4m57s ago)   5h28m

Expected results:

Running

Additional info:

Attaching must-gather.

The pod recovers successfully after deleting/re-creating.


[kni@registry.kni-qe-0 ~]$ oc -n openshift-monitoring logs prometheus-k8s-0
ts=2022-09-26T14:54:01.919Z caller=main.go:552 level=info msg="Starting Prometheus Server" mode=server version="(version=2.36.2, branch=rhaos-4.11-rhel-8, revision=0d81ba04ce410df37ca2c0b1ec619e1bc02e19ef)"
ts=2022-09-26T14:54:01.919Z caller=main.go:557 level=info build_context="(go=go1.18.4, user=root@371541f17026, date=20220916-14:15:37)"
ts=2022-09-26T14:54:01.919Z caller=main.go:558 level=info host_details="(Linux 4.18.0-372.26.1.rt7.183.el8_6.x86_64 #1 SMP PREEMPT_RT Sat Aug 27 22:04:33 EDT 2022 x86_64 prometheus-k8s-0 (none))"
ts=2022-09-26T14:54:01.919Z caller=main.go:559 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2022-09-26T14:54:01.919Z caller=main.go:560 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-09-26T14:54:01.921Z caller=web.go:553 level=info component=web msg="Start listening for connections" address=127.0.0.1:9090
ts=2022-09-26T14:54:01.922Z caller=main.go:989 level=info msg="Starting TSDB ..."
ts=2022-09-26T14:54:01.924Z caller=tls_config.go:231 level=info component=web msg="TLS is disabled." http2=false
ts=2022-09-26T14:54:01.926Z caller=main.go:848 level=info msg="Stopping scrape discovery manager..."
ts=2022-09-26T14:54:01.926Z caller=main.go:862 level=info msg="Stopping notify discovery manager..."
ts=2022-09-26T14:54:01.926Z caller=manager.go:951 level=info component="rule manager" msg="Stopping rule manager..."
ts=2022-09-26T14:54:01.926Z caller=manager.go:961 level=info component="rule manager" msg="Rule manager stopped"
ts=2022-09-26T14:54:01.926Z caller=main.go:899 level=info msg="Stopping scrape manager..."
ts=2022-09-26T14:54:01.926Z caller=main.go:858 level=info msg="Notify discovery manager stopped"
ts=2022-09-26T14:54:01.926Z caller=main.go:891 level=info msg="Scrape manager stopped"
ts=2022-09-26T14:54:01.926Z caller=notifier.go:599 level=info component=notifier msg="Stopping notification manager..."
ts=2022-09-26T14:54:01.926Z caller=main.go:844 level=info msg="Scrape discovery manager stopped"
ts=2022-09-26T14:54:01.926Z caller=manager.go:937 level=info component="rule manager" msg="Starting rule manager..."
ts=2022-09-26T14:54:01.926Z caller=main.go:1120 level=info msg="Notifier manager stopped"
ts=2022-09-26T14:54:01.926Z caller=main.go:1129 level=error err="opening storage failed: /prometheus/chunks_head/000002: invalid magic number 0"
Bug OCPBUGS-2155: Etcd scaling test was mistakenly added to the parallel suite

https://github.com/openshift/origin/pull/27444 was intended to move the scaling test out of serial to it's own test suite, but it added it to parallel – meaning it's running in all our normal upgrade jobs, causing them to frequently fail with repeating pathological events as well as greatly increasing their run time.

See https://github.com/openshift/origin/pull/27444#discussion_r991296925 for more info

Bug OCPBUGS-1806: OCP cluster install on baremetal fails when hostname of master nodes does not include the text "master" (take 2)

Description of problem:

Disconnected IPI OCP 4.11.5 cluster install on baremetal fails when hostname of master nodes does not include "master"    

Version-Release number of selected component (if applicable): 4.11.5

How reproducible:  Perform disconnected IPI install of OCP 4.11.5 on bare metal with master nodes that do not contain the text "master"

Steps to Reproduce:

Perform disconnected IPI install of OCP 4.11.5 on bare metal with master nodes that do not contain the text "master"

Actual results: master nodes do come up.

Expected results: master nodes should come up despite that the text "master" is not in their hostname.

Additional info:

Disconnected IPI OCP 4.11.5 cluster install on baremetal fails when hostname of master nodes does not include "master"    

My cust reinstall new cluster using the fix here . But they have the exact same issue. The metal3 pod have  PROVISIONING_MACS value  empty.  Can we work together with them to understand why the new code fix https://github.com/openshift/cluster-baremetal-operator/commit/76bd6bc461b30a6a450f85a42e492a0933178aee is not working.

cat metal3-static-ip-set/metal3-static-ip-set/logs/current.log
2022-09-27T14:19:38.140662564Z + '[' -z 10.17.199.3/27 ']'
2022-09-27T14:19:38.140662564Z + '[' -z '' ']'
2022-09-27T14:19:38.140662564Z + '[' -n '' ']'
2022-09-27T14:19:38.140722345Z ERROR: Could not find suitable interface for "10.17.199.3/27"
2022-09-27T14:19:38.140726312Z + '[' -n '' ']'
2022-09-27T14:19:38.140726312Z + echo 'ERROR: Could not find suitable interface for "10.17.199.3/27"'
2022-09-27T14:19:38.140726312Z + exit 1

 

cat metal3-b9bf8d595-gv94k.yaml
...
initContainers:

command: /set-static-ip
env: name: PROVISIONING_IP
value: 10.17.199.3/27 name: PROVISIONING_INTERFACE name: PROVISIONING_MACS <------------------------- missing MACS
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f04793bd109ecba2dfe43be93dc990ac5299272482c150bd5f2eee0f80c983b
imagePullPolicy: IfNotPresent
name: metal3-static-ip-set
.... 
  • omc logs machine-api-controllers-6b9ffd96cd-grh6l -c nodelink-controller  -n openshift-machine-api
2022-09-21T16:13:43.600517485Z I0921 16:13:43.600513       1 nodelink_controller.go:408] Finding machine from node "blocp-1-106-m-0.c106-1.sc.evolhse.hydro.qc.ca"
2022-09-21T16:13:43.600521381Z I0921 16:13:43.600517       1 nodelink_controller.go:425] Finding machine from node "blocp-1-106-m-0.c106-1.sc.evolhse.hydro.qc.ca" by ProviderID
2022-09-21T16:13:43.600525225Z W0921 16:13:43.600521       1 nodelink_controller.go:427] Node "blocp-1-106-m-0.c106-1.sc.evolhse.hydro.qc.ca" has no providerID
2022-09-21T16:13:43.600528917Z I0921 16:13:43.600524       1 nodelink_controller.go:448] Finding machine from node "blocp-1-106-m-0.c106-1.sc.evolhse.hydro.qc.ca" by IP
2022-09-21T16:13:43.600532711Z I0921 16:13:43.600529       1 nodelink_controller.go:453] Found internal IP for node "blocp-1-106-m-0.c106-1.sc.evolhse.hydro.qc.ca": "10.17.192.33"
2022-09-21T16:13:43.600551289Z I0921 16:13:43.600544       1 nodelink_controller.go:477] Matching machine not found for node "blocp-1-106-m-0.c106-1.sc.evolhse.hydro.qc.ca" with internal IP "10.17.192.33"

From @dtantsur WIP PR: https://github.com/openshift/cluster-baremetal-operator/pull/299

Customer is waiting for this fix. The previous code change don't fix customer situation.

Please refer to this slack thread :https://coreos.slack.com/archives/CFP6ST0A3/p1664215102459219

Bug OCPBUGS-1810: must gather for gather_ingress_node_firewall breaks with permission issues

Description of problem:

$ oc adm must-gather -- gather_ingress_node_firewall
[must-gather      ] OUT Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dec5a08681e11eedcd31f075941b74f777b9187f0e711a498a212f9d96adb2f
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
ClusterID: 0ef60b50-4378-431d-8ca2-faa5af098274
ClusterVersion: Stable at "4.12.0-0.nightly-2022-09-26-111919"
ClusterOperators:
    clusteroperator/insights is not available (Reporting was not allowed: your Red Hat account is not enabled for remote support or your token has expired: UHC services authentication failed
) because Reporting was not allowed: your Red Hat account is not enabled for remote support or your token has expired: UHC services authentication failed[must-gather      ] OUT namespace/openshift-must-gather-fr7kc created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-xx2fh created
[must-gather      ] OUT pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dec5a08681e11eedcd31f075941b74f777b9187f0e711a498a212f9d96adb2f created
[must-gather-xvfj4] POD 2022-09-28T16:57:00.887445531Z /bin/bash: /usr/bin/gather_ingress_node_firewall: Permission denied
[must-gather-xvfj4] OUT waiting for gather to complete
[must-gather-xvfj4] OUT downloading gather output
[must-gather-xvfj4] OUT receiving incremental file list
[must-gather-xvfj4] OUT ./
[must-gather-xvfj4] OUT 
[must-gather-xvfj4] OUT sent 27 bytes  received 40 bytes  26.80 bytes/sec
[must-gather-xvfj4] OUT total size is 0  speedup is 0.00
[must-gather      ] OUT namespace/openshift-must-gather-fr7kc deleted
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-xx2fh deleted
Reprinting Cluster State:
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
ClusterID: 0ef60b50-4378-431d-8ca2-faa5af098274
ClusterVersion: Stable at "4.12.0-0.nightly-2022-09-26-111919"
ClusterOperators:
    clusteroperator/insights is not available (Reporting was not allowed: your Red Hat account is not enabled for remote support or your token has expired: UHC services authentication failed
) because Reporting was not allowed: your Red Hat account is not enabled for remote support or your token has expired: UHC services authentication failed

Version-Release number of selected component (if applicable):

4.12

How reproducible:

Always

Steps to Reproduce:

1.
2.
3.

Actual results:

 

Expected results:

 

Additional info:

 
Bug OCPBUGS-2102: Resource quota e2e tests fails after latest changes to master

Description of problem:

 

Version-Release number of selected component (if applicable):

 

How reproducible:

 

Steps to Reproduce:

1.
2.
3.

Actual results:

 

Expected results:

 

Additional info:

 
Bug OCPBUGS-2249: Conditional gatherer cluster_version_matches issues

Description of problem:

When solving flakiness of a test in IO tests, we found that there are some issues in the cluster_version_matches condition for the conditional gatherer. 

Firstly the character limit should be increased as 32 characters does not cover every possible release version as some exceed that limit. 
Furthermore, there is an error in the schema

https://github.com/openshift/insights-operator/blob/master/pkg/gatherers/conditional/gathering_rule.schema.json#L101

There is no name, it should be version

How reproducible:

Sometimes

Steps to Reproduce:

1. Spin a cluster from a PR
2. If version exceeds 32 characters, we get in the pod logs: 'Could not get version from string: "<"'

Actual results:

'Could not get version from string: "<"'

Expected results:

Metadata should contain "Metadata should contain invalid range error"

Additional info:

However, since there's the possibility for versions to exceed 32 characters, we shouldn't expect an error in this situation. Therefore, there might be more than one issue.
Bug OCPBUGS-3552: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Bug OCPBUGS-3750: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Bug OCPBUGS-2757: rebase should handle idempotency

Derrick got an "old and new refs are equal" on rebase error; this is similar to OCPBUGS-1899 but I think has a different root cause. In this case, when a manual rollback is performed via the bootloader, we've computed that there's an osimageurl diff between the expected and desired state, but actually the desired state is already set.

We just need to skip doing the rebase if we're already in the target state.

(A real root of this problem again is that the whole "current/desired config" thing is trying to track state independently of the bootloader...if we made node state == container image, all of that goes away. The MCO would understand that it got booted into a previous state)

Bug OCPBUGS-3028: panic in WaitForBootstrapComplete

This is a clone of issue OCPBUGS-3018. The following is the description of the original issue:
—
Description of problem:

When running an overnight run in dev-scripts (COMPACT_IPV4) with repeated installs I saw this panic in WaitForBootstrapComplete occur once.

level=debug msg=Agent Rest API Initialized
E1101 05:19:09.733309 1802865 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 1 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x4086520?, 0x1d875810})
    /home/stack/go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00056fb00?})
    /home/stack/go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
panic({0x4086520, 0x1d875810})
    /usr/local/go/src/runtime/panic.go:838 +0x207
github.com/openshift/installer/pkg/agent.(*NodeZeroRestClient).getClusterID(0xc0001341e0)
    /home/stack/go/src/github.com/openshift/installer/pkg/agent/rest.go:121 +0x53
github.com/openshift/installer/pkg/agent.(*Cluster).IsBootstrapComplete(0xc000134190)
    /home/stack/go/src/github.com/openshift/installer/pkg/agent/cluster.go:183 +0x4fc
github.com/openshift/installer/pkg/agent.WaitForBootstrapComplete.func1()
    /home/stack/go/src/github.com/openshift/installer/pkg/agent/waitfor.go:31 +0x77
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x1d8fa901?)
    /home/stack/go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0001958c0?, {0x1a53c7a0, 0xc0011d4a50}, 0x1, 0xc0001958c0)
    /home/stack/go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0009ab860?, 0x77359400, 0x0, 0xa?, 0x8?)
    /home/stack/go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /home/stack/go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:92
github.com/openshift/installer/pkg/agent.WaitForBootstrapComplete({0x7ffd7fccb4e3?, 0x40d7e7?})
    /home/stack/go/src/github.com/openshift/installer/pkg/agent/waitfor.go:30 +0x1bc
github.com/openshift/installer/pkg/agent.WaitForInstallComplete({0x7ffd7fccb4e3?, 0x5?})
    /home/stack/go/src/github.com/openshift/installer/pkg/agent/waitfor.go:73 +0x56
github.com/openshift/installer/cmd/openshift-install/agent.newWaitForInstallCompleteCmd.func1(0xc0003b6c80?, {0xc0004d67c0?, 0x2?, 0x2?})
    /home/stack/go/src/github.com/openshift/installer/cmd/openshift-install/agent/waitfor.go:73 +0x126
github.com/spf13/cobra.(*Command).execute(0xc0003b6c80, {0xc0004d6780, 0x2, 0x2})
    /home/stack/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:876 +0x67b
github.com/spf13/cobra.(*Command).ExecuteC(0xc0013b0a00)
    /home/stack/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
    /home/stack/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:918
main.installerMain()
    /home/stack/go/src/github.com/openshift/installer/cmd/openshift-install/main.go:61 +0x2b0
main.main()
    /home/stack/go/src/github.com/openshift/installer/cmd/openshift-install/main.go:38 +0xff
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x33d3cd3]

Version-Release number of selected component (if applicable):

4.12.0-0.nightly-2022-10-25-210451

How reproducible:

Occurred on the 12th run, all previous installs were successfule

Steps to Reproduce:

1.Set up dev-scripts for  AGENT_E2E_TEST_SCENARIO=COMPACT_IPV4, no mirroring
2. Run 'make clean; make agent' in a loop
3. After repeated installs got the failure

Actual results:

Panic in WaitForBootstrapComplete

Expected results:

No failure

Additional info:

It looks like clusterResult is used here even on failure, which causes the dereference - https://github.com/openshift/installer/blob/master/pkg/agent/rest.go#L121

 

Bug OCPBUGS-268: vsphere: installer for vsphere does not have steal clock accounting enabled

The linux kernel was updated:
https://lkml.org/lkml/2020/3/20/1030
to include steal

{time,clock}

accounting

This would greatly assist in troubleshooting vSphere performance issues
caused by over-provisioned ESXi hosts.

Bug OCPBUGS-3523: Operator attempts to render both GA and Tech Preview API Extensions

This is a clone of issue OCPBUGS-3123. The following is the description of the original issue:
—
Description of problem:

Support for tech preview API extensions was introduced in https://github.com/openshift/installer/pull/6336 and https://github.com/openshift/api/pull/1274 .  In the case of https://github.com/openshift/api/pull/1278 , config/v1/0000_10_config-operator_01_infrastructure-TechPreviewNoUpgrade.crd.yaml was introduced which seems to result in both 0000_10_config-operator_01_infrastructure-TechPreviewNoUpgrade.crd.yaml and 0000_10_config-operator_01_infrastructure-Default.crd.yaml being rendered by the bootstrap.  As a result, both CRDs are created during bootstrap.  However, one of them(in this case the tech preview CRD) fails to be created.  

We may need to modify the render command to be aware of feature gates when rendering manifests during bootstrap.  Also, I'm open hearing other views on how this might work.

Version-Release number of selected component (if applicable):

https://github.com/openshift/cluster-config-operator/pull/269 built and running on 4.12-ec5

How reproducible:

consistently

Steps to Reproduce:

1. bump the version of OpenShift API to one including a tech preview version of the infrastructure CRD
2. install openshift with the infrastructure manifest modified to incorporate tech preview fields
3. those fields will not be populated upon installation

Also, checking the logs from bootkube will show both being installed, but one of them fails.

Actual results:

 

Expected results:

 

Additional info:

Excerpts from bootkube log
Nov 02 20:40:01 localhost.localdomain bootkube.sh[4216]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_infrastructure-TechPreviewNoUpgrade.crd.yaml
Nov 02 20:40:01 localhost.localdomain bootkube.sh[4216]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_infrastructure-Default.crd.yaml


Nov 02 20:41:23 localhost.localdomain bootkube.sh[5710]: Created "0000_10_config-operator_01_infrastructure-Default.crd.yaml" customresourcedefinitions.v1.apiextensions.k8s.io/infrastructures.config.openshift.io -n
Nov 02 20:41:23 localhost.localdomain bootkube.sh[5710]: Skipped "0000_10_config-operator_01_infrastructure-TechPreviewNoUpgrade.crd.yaml" customresourcedefinitions.v1.apiextensions.k8s.io/infrastructures.config.openshift.io -n  as it already exists

 

 

 

Bug OCPBUGS-3795: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Bug OCPBUGS-3055: 4.12 backport: Wait-for install-complete did not exit upon completion.

Description of problem:

When the cluster install finished, wait-for install-complete command didn't exit as expected.

Version-Release number of selected component (if applicable):

 

How reproducible:

 

Steps to Reproduce:

1. Get the latest agent-installer and build image
git clone https://github.com/openshift/installer.git
cd installer/
hack/build.sh
Edit agent-config and install-config yaml file
Create the agent.iso image:
OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/openshift-release-dev/ocp-release:4.12.0-ec.3-x86_64 bin/openshift-install agent create image --log-level debug

2. Install SNO cluster
virt-install --connect qemu:///system -n control-0 -r 33000 --vcpus 8 --cdrom ./agent.iso --disk pool=installer,size=120 --boot uefi,hd,cdrom --os-variant=rhel8.5 --network network=default,mac=52:54:00:aa:aa:aa --wait=-1 

3. Run 'bin/openshift agent wait-for bootstrap-complete --log-level debug' and the command finished as expected.

4. After 'bootstrap' completion, run 'bin/openshift agent wait-for install-complete --log-level debug', the command didn't finish as expected.

Actual results:

 

Expected results:

 

Additional info:

 
Bug OCPBUGS-1361: Expect more detail info when report vSphere privilege alert

Description of problem:

When alert raised for vSphere privilege check which is reported by vsphere-problem-detector, we could only get the very simple info as below:

 

=======================================

Description

The vsphere-problem-detector monitors the health and configuration of OpenShift on VSphere. If problems are found which may prevent machine scaling, storage provisioning, and safe upgrades, the vsphere-problem-detector will raise alerts.

 

Summary

VSphere cluster health checks are failing

 

Message

VSphere cluster health checks are failing with CheckAccountPermissions

=======================================

 

  1. Please add vSphere privilege check in the Description, currently only mention "prevent machine scaling, storage provisioning, and safe upgrades" 
  2. Could we at least add something like "Check vsphere-problem-detector pod log in openshift-cluster-storage-operator namespace to see the detail info" if we could not list which privilege is missing.

(We could get the namespace/pod info from metric, but I think adding it in alert Description or Message should be more clear)

Version-Release number of selected component (if applicable):

4.12.0-0.nightly-2022-09-12-152748

 

How reproducible:

Always

 

Steps to Reproduce:

See description

Actual results:

Alert info is not so clear

 

Expected results:

Add more Alert info

Bug OCPBUGS-1677: CI: Backend unit tests fails because devfile registry was updated (fix assertion)

Description of problem:
pkg/devfile/sample_test.go fails after devfile registry was updated (https://github.com/devfile/registry/pull/126)

This issue is about updating our assertion so that the CI job runs successfully again. We might want to backport this as well.

OCPBUGS-1678 is about updating the code that the test should use a mock response instead of the latest registry content OR check some specific attributes instead of comparing the full JSON response.

Version-Release number of selected component (if applicable):
4.12

How reproducible:
Always

Steps to Reproduce:
1. Clone openshift/console
2. Run ./test-backend.sh

Actual results:
Unit tests fail

Expected results:
Unit tests should pass again

Additional info:

Bug OCPBUGS-2654: Console OLM Integration Tests Reference Operator Not Present in 4.12 Certified Operators CatalogSource

Description of problem:

The Console Operator has a suite of tests responsible for assuring that Console can successfully interact with Operators managed by OLM. The operator-hub.spec test references an operator no longer present in the 4.12 certified operators catalog source: https://github.com/openshift/console/blob/master/frontend/packages/operator-lifecycle-manager/integration-tests-cypress/tests/operator-hub.spec.ts#L64

OLM is unable to set the default catalog sources to the 4.12 image tag until the test is update to reference an operator in both the 4.11 and 4.12 images of the certified operators catalog source.

Version-Release number of selected component (if applicable):4.12

How reproducible: always

Steps to Reproduce:

1. Update the certified operators catalogSource images to the 4.12 tag
2. Attempt to run the operatorhub.spec test suite.

Actual results:

The test fails

Expected results:

The test passes

Additional info:
Bug OCPBUGS-3095: IPI for Power VS: Deploy fails when there is are certain preexisting resources

This is a clone of issue OCPBUGS-2290. The following is the description of the original issue:
—
Description of problem:

If you try to deploy with Internal publishing strategy, and you have either already have a pubilc gateway or already permitted the VPC subnet to the DNS service, deploy will always fail.

Version-Release number of selected component (if applicable):

 

How reproducible:

Easily

Steps to Reproduce:

1. Add a public gateway to VPC network and/or add VPC subnet to permitted DNS networks
2. Set publish strategy to Internal
3. Deploy

Actual results:

Deploy fails

Expected results:

If the resources exist simply skip trying to create them.

Additional info:

Fix here https://github.com/openshift/installer/pull/6481
Bug OCPBUGS-889: 4.12 installer is pointing at stable-4.11 channel

Description of problem:

a freshly installed 4.12 cluster should have stable-4.12 channel by default

Version-Release number of selected component (if applicable):

4.12.0-0.nightly-2022-09-02-154321

How reproducible:

100%

Steps to Reproduce:

install 4.12 cluster

Actual results:

oc get clusterversion/version -ojson | jq .spec.channel
"stable-4.11"

Expected results:

oc get clusterversion/version -ojson | jq .spec.channel
"stable-4.12"

Additional info:

 
Bug OCPBUGS-3503: CRD-based and openshift-apiserver-based Route validation/defaulting must use the shared implementation

Description of problem:

OCPBUGS-3499 and OCPBUGS-3501 both require a more recent version of openshift/library-go containing the shared validation and host-assignment logic.
Bug OCPBUGS-872: provisioning interface on master node not getting ipv4 dhcp ip address from bootstrap dhcp server on OCP IPI BareMetal install

Description of problem:

Provisioning interface on master node not getting ipv4 dhcp ip address from bootstrap dhcp server on OCP 4.10.16 IPI BareMetal install.

Customer is performing an OCP 4.10.16 IPI BareMetal install and bootstrap node provisions just fine, but when master nodes are booted for provisioning, they are not getting an ipv4 address via dhcp. As such, the install is not moving forward at this point.

Version-Release number of selected component (if applicable):

OCP 4.10.16

How reproducible:

Perform OCP 4.10.16 IPI BareMetal install.

Actual results:

provisioning interface comes up (as evidenced by ipv6 address) but is not getting an ipv4 address via dhcp. OCP install / provisioning fails at this point.

Expected results:

provisioning interface successfully received an ipv4 ip address and successfully provisioned master nodes (and subsequently worker nodes as well.)

Additional info:

As a troubleshooting measure, manually adding an ipv4 ip address did allow the coreos image on the bootstrap node to be reached via curl.

Further, the kernel boot line for the first master node was updated for a static ip addresss assignment for further confirmation that the master node would successfully image this way which further confirming that the issue is the provisioning interface not receiving an ipv4 ip address from the dhcp server.

Bug OCPBUGS-102: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Bug OCPBUGS-2512: apiserver pods cannot reach etcd on single node IPv6 cluster: transport: authentication handshake failed: x509: certificate is valid for ::1, 127.0.0.1, ::1, fd69::2, not 2620:52:0:198::10"

Description of problem:

openshift-apiserver, openshift-oauth-apiserver and kube-apiserver pods cannot validate the certificate when trying to reach etcd reporting certificate validation errors:

}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for ::1, 127.0.0.1, ::1, fd69::2, not 2620:52:0:198::10"
W1018 11:36:43.523673      15 logging.go:59] [core] [Channel #186 SubChannel #187] grpc: addrConn.createTransport failed to connect to {
  "Addr": "[2620:52:0:198::10]:2379",
  "ServerName": "2620:52:0:198::10",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for ::1, 127.0.0.1, ::1, fd69::2, not 2620:52:0:198::10"

Version-Release number of selected component (if applicable):

4.12.0-0.nightly-2022-10-18-041406

How reproducible:

100%

Steps to Reproduce:

1. Deploy SNO with single stack IPv6 via ZTP procedure

Actual results:

Deployment times out and some of the operators aren't deployed successfully.

NAME                                       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.12.0-0.nightly-2022-10-18-041406   False       False         True       124m    APIServerDeploymentAvailable: no apiserver.openshift-oauth-apiserver pods available on any node....
baremetal                                  4.12.0-0.nightly-2022-10-18-041406   True        False         False      112m    
cloud-controller-manager                   4.12.0-0.nightly-2022-10-18-041406   True        False         False      111m    
cloud-credential                           4.12.0-0.nightly-2022-10-18-041406   True        False         False      115m    
cluster-autoscaler                         4.12.0-0.nightly-2022-10-18-041406   True        False         False      111m    
config-operator                            4.12.0-0.nightly-2022-10-18-041406   True        False         False      124m    
console                                                                                                                      
control-plane-machine-set                  4.12.0-0.nightly-2022-10-18-041406   True        False         False      111m    
csi-snapshot-controller                    4.12.0-0.nightly-2022-10-18-041406   True        False         False      111m    
dns                                        4.12.0-0.nightly-2022-10-18-041406   True        False         False      111m    
etcd                                       4.12.0-0.nightly-2022-10-18-041406   True        False         True       121m    ClusterMemberControllerDegraded: could not get list of unhealthy members: giving up getting a cached client after 3 tries
image-registry                             4.12.0-0.nightly-2022-10-18-041406   False       True          True       104m    Available: The registry is removed...
ingress                                    4.12.0-0.nightly-2022-10-18-041406   True        True          True       111m    The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: DeploymentReplicasAllAvailable=False (DeploymentReplicasNotAvailable: 0/1 of replicas are available)
insights                                   4.12.0-0.nightly-2022-10-18-041406   True        False         False      118s    
kube-apiserver                             4.12.0-0.nightly-2022-10-18-041406   True        False         False      102m    
kube-controller-manager                    4.12.0-0.nightly-2022-10-18-041406   True        False         True       107m    GarbageCollectorDegraded: error fetching rules: Get "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules": dial tcp [fd02::3c5f]:9091: connect: connection refused
kube-scheduler                             4.12.0-0.nightly-2022-10-18-041406   True        False         False      107m    
kube-storage-version-migrator              4.12.0-0.nightly-2022-10-18-041406   True        False         False      117m    
machine-api                                4.12.0-0.nightly-2022-10-18-041406   True        False         False      111m    
machine-approver                           4.12.0-0.nightly-2022-10-18-041406   True        False         False      111m    
machine-config                             4.12.0-0.nightly-2022-10-18-041406   True        False         False      115m    
marketplace                                4.12.0-0.nightly-2022-10-18-041406   True        False         False      116m    
monitoring                                                                      False       True          True       98m     deleting Thanos Ruler Route failed: Timeout: request did not complete within requested timeout - context deadline exceeded, deleting UserWorkload federate Route failed: Timeout: request did not complete within requested timeout - context deadline exceeded, reconciling Alertmanager Route failed: retrieving Route object failed: the server was unable to return a response in the time allotted, but may still be processing the request (get routes.route.openshift.io alertmanager-main), reconciling Thanos Querier Route failed: retrieving Route object failed: the server was unable to return a response in the time allotted, but may still be processing the request (get routes.route.openshift.io thanos-querier), reconciling Prometheus API Route failed: retrieving Route object failed: the server was unable to return a response in the time allotted, but may still be processing the request (get routes.route.openshift.io prometheus-k8s), prometheuses.monitoring.coreos.com "k8s" not found
network                                    4.12.0-0.nightly-2022-10-18-041406   True        False         False      124m    
node-tuning                                4.12.0-0.nightly-2022-10-18-041406   True        False         False      111m    
openshift-apiserver                        4.12.0-0.nightly-2022-10-18-041406   True        False         False      104m    
openshift-controller-manager               4.12.0-0.nightly-2022-10-18-041406   True        False         False      107m    
openshift-samples                                                               False       True          False      103m    The error the server was unable to return a response in the time allotted, but may still be processing the request (get imagestreams.image.openshift.io) during openshift namespace cleanup has left the samples in an unknown state
operator-lifecycle-manager                 4.12.0-0.nightly-2022-10-18-041406   True        False         False      111m    
operator-lifecycle-manager-catalog         4.12.0-0.nightly-2022-10-18-041406   True        False         False      111m    
operator-lifecycle-manager-packageserver   4.12.0-0.nightly-2022-10-18-041406   True        False         False      106m    
service-ca                                 4.12.0-0.nightly-2022-10-18-041406   True        False         False      124m    
storage                                    4.12.0-0.nightly-2022-10-18-041406   True        False         False      111m  

Expected results:

Deployment succeeds without issues.

Additional info:

I was unable to run must-gather so attaching the pods logs copied from the host file system.
Bug OCPBUGS-69: No event log was emitted when egressIP exceeds capacity limit for cloud providers with SDN plugin

There is capacity limit on egressIP for different cloud provider, for example, GCP, the limit is 10.

If the number of egressIP added to hostsubnet exceeds the capability limit, it is expected some logging message is emitted to event log, that can be seen through "oc get event"

 

On a GCP with SDN plugin, configure egressCIDRs on one worker node, configured 12 netnamespaces, each has 1 egressIP configured, the total number of egressIP for the hostsubnet has exceeded its capacity limit of 10.   No event log was seen to indicate that the number of egressIP for the hostsubnet has exceeded the limit.

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-08-02-014045   True        False         160m    Cluster version is 4.11.0-0.nightly-2022-08-02-014045

 

See attachment for more details.

 

Bug OCPBUGS-1231: base image can't be fetched in a disconnected environment

Description of problem:

The user mirrored the 4.11.0 release and attempted to use it to generate the the installation ISO in a completely disconnected environment.

When it was the turn for extracting the os image from machine-os-images, the agent based installer ran : oc adm release info --image-for=machine-os-images --insecure=true quay.io/openshift-release-dev/ocp-release@sha256:300bce8246cf880e792e106607925de0a404484637627edf5f517375517d54a4 --registry-config=/tmp/registry-config1141450352

This does not include the --icsp-file, and thus the image reference can be retrieved to perform the extraction.

 

Version-Release number of selected component (if applicable):

https://github.com/openshift/installer/releases/tag/agent-installer-v4.11.0-dev-preview-2

How reproducible:

100%

Steps to Reproduce:

1. Mirroring the images of 4.11.0 using oc adm mirror command to the local registry.
2. Created install-config.yaml with mirror config
3. Created agent-config.yaml 
4. openshift-install-sep1 agent create image --dir kni-22

 

Actual results:

INFO[0001] Start configuring static network for 3 hosts  pkg=manifests
INFO[0002] Adding NMConnection file <bond0.nmconnection>  pkg=manifests
INFO[0002] Adding NMConnection file <eno49.nmconnection>  pkg=manifests
INFO[0002] Adding NMConnection file <eno50.nmconnection>  pkg=manifests
INFO[0003] Adding NMConnection file <bond0.nmconnection>  pkg=manifests
INFO[0003] Adding NMConnection file <eno49.nmconnection>  pkg=manifests
INFO[0003] Adding NMConnection file <eno50.nmconnection>  pkg=manifests
INFO[0004] Adding NMConnection file <bond0.nmconnection>  pkg=manifests
INFO[0004] Adding NMConnection file <eno49.nmconnection>  pkg=manifests
INFO[0004] Adding NMConnection file <eno50.nmconnection>  pkg=manifests
DEBUG   Fetching BaseIso Image...
DEBUG     Fetching Agent Manifests...
DEBUG     Reusing previously-fetched Agent Manifests
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG     Fetching Mirror Registries Config...
DEBUG     Reusing previously-fetched Mirror Registries Config
DEBUG   Generating BaseIso Image...
INFO[0004] Extracting base ISO from release payload
ERRO[0014] command 'oc adm release info --image-for=machine-os-images --insecure=true quay.io/openshift-release-dev/ocp-release@sha256:300bce8246cf880e792e106607925de0a404484637627edf5f517375517d54a4 --registry-config=/tmp/registry-config1141450352' exited with non-zero exit code 1:
error: unable to read image quay.io/openshift-release-dev/ocp-release@sha256:300bce8246cf880e792e106607925de0a404484637627edf5f517375517d54a4: Get "http://quay.io/v2/": dial tcp: lookup quay.io on 10.92.86.56:53: server misbehaving
WARN[0014] Failed to extract base ISO from release payload - check registry configuration
INFO[0014] Downloading base ISO
DEBUG Obtaining RHCOS image file from 'https://rhcos.mirror.openshift.com/art/storage/releases/rhcos-4.11/411.86.202207150124-0/x86_64/rhcos-411.86.202207150124-0-live.x86_64.iso'
ERROR failed to write asset (Agent Installer ISO) to disk: image reader not available
FATAL failed to fetch Agent Installer ISO: failed to fetch dependency of "Agent Installer ISO": failed to generate asset "BaseIso Image": failed to get base ISO image: command 'oc adm release info --image-for=machine-os-images --insecure=true quay.io/openshift-release-dev/ocp-release@sha256:300bce8246cf880e792e106607925de0a404484637627edf5f517375517d54a4 --registry-config=/tmp/registry-config1141450352' exited with non-zero exit code 1:
FATAL error: unable to read image quay.io/openshift-release-dev/ocp-release@sha256:300bce8246cf880e792e106607925de0a404484637627edf5f517375517d54a4: Get "http://quay.io/v2/": dial tcp: lookup quay.io on 10.92.86.56:53: server misbehaving
FATAL

Expected results:

Image correctly generated

Additional info:

Host OS: RHEL 8.4
NMstate version: nmstate-1.0.2-5.el8.noarch
Bug OCPBUGS-3184: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Bug OCPBUGS-3520: Install ends in preparing-failed due to container-images-available validation

This is a clone of issue OCPBUGS-3280. The following is the description of the original issue:
—
I have a script that does continuous installs using AGENT_E2E_TEST_SCENARIO=COMPACT_IPV4, just starting a new install after the previous one completes. What I'm seeing is that eventually I end up getting installation failures due to the container-images-available validation failure. What gets logged in wait-for bootstrap-complete is:

level=debug msg=Host master-0: New image status quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0f6ddae72f6d730ca07a265691401571a8d8f7e62546f1bcda26c9a01628f4d6. result: failure. 

level=debug msg=Host master-0: validation 'container-images-available' that used to succeed is now failing
level=debug msg=Host master-0: updated status from preparing-for-installation to preparing-failed (Host failed to prepare for installation due to following failing validation(s): Failed to fetch container images needed for installation from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0f6ddae72f6d730ca07a265691401571a8d8f7e62546f1bcda26c9a01628f4d6. This may be due to a network hiccup. Retry to install again. If this problem persists, check your network settings to make sure you’re not blocked. ; Host couldn't synchronize with any NTP server)

Sometimes the image gets loaded onto the other masters OK and sometimes there are failures with more than one host. In either case the install stalls at this point.

When using a disconnected environment (MIRROR_IMAGES=true) I don't see this occurring.

Containers on host0
[core@master-0 ~]$ sudo podman ps
CONTAINER ID  IMAGE                                                                                                                   COMMAND               CREATED       STATUS           PORTS       NAMES
00a0eebb989c  localhost/podman-pause:4.2.0-1661537366                                                                                                       11 hours ago  Up 11 hours ago              cef65dd7f170-infra
5d0eced94979  quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:caa73897dcb9ff6bc00a4165f4170701f4bd41e36bfaf695c00461ec65a8d589  /bin/bash start_d...  11 hours ago  Up 11 hours ago              assisted-db
813bef526094  quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:caa73897dcb9ff6bc00a4165f4170701f4bd41e36bfaf695c00461ec65a8d589  /assisted-service     11 hours ago  Up 11 hours ago              service
edde1028a542  quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e43558e28be8fbf6fe4529cf9f9beadbacbbba8c570ecf6cb81ae732ec01807f  next_step_runner ...  11 hours ago  Up 11 hours ago              next-step-runner

Some relevant logs from assisted-service for this container image:
time="2022-11-03T01:48:44Z" level=info msg="Submitting step <container-image-availability> id <container-image-availability-b72665b1> to infra_env <17c8b837-0130-4b8c-ad06-19bcd2a61dbf> host <df170326-772b-43b5-87ef-3dfff91ba1a9>  Arguments: <[{\"images\":[\"registry.ci.openshift.org/ocp/release:4.12.0-0.nightly-2022-10-25-210451\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca122ab3a82dfa15d72a05f448c48a7758a2c7b0ecbb39011235bcf0666fbc15\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0f6ddae72f6d730ca07a265691401571a8d8f7e62546f1bcda26c9a01628f4d6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9e52a45b47cd9d70e7378811f4ba763fd43ec2580378822286c7115fbee6ef3a\"],\"timeout\":960}]>" func=github.com/openshift/assisted-service/internal/host/hostcommands.logSteps file="/src/internal/host/hostcommands/instruction_manager.go:285" go-id=841 host_id=df170326-772b-43b5-87ef-3dfff91ba1a9 infra_env_id=17c8b837-0130-4b8c-ad06-19bcd2a61dbf pkg=instructions request_id=47cc221f-4f47-4d0d-8278-c0f5af933567

time="2022-11-03T01:49:35Z" level=error msg="Received step reply <container-image-availability-9788cfa7> from infra-env <17c8b837-0130-4b8c-ad06-19bcd2a61dbf> host <845f1e3c-c286-4d2f-ba92-4c5cab953641> exit-code <2> stderr <> stdout <{\"images\":[

{\"name\":\"registry.ci.openshift.org/ocp/release:4.12.0-0.nightly-2022-10-25-210451\",\"result\":\"success\"}

,{\"download_rate\":159.65409925994226,\"name\":\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca122ab3a82dfa15d72a05f448c48a7758a2c7b0ecbb39011235bcf0666fbc15\",\"result\":\"success\",\"size_bytes\":523130669,\"time\":3.276650405},{\"name\":\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0f6ddae72f6d730ca07a265691401571a8d8f7e62546f1bcda26c9a01628f4d6\",\"result\":\"failure\"},{\"download_rate\":278.8962416008878,\"name\":\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9e52a45b47cd9d70e7378811f4ba763fd43ec2580378822286c7115fbee6ef3a\",\"result\":\"success\",\"size_bytes\":402688178,\"time\":1.443863767}]}>" func=github.com/openshift/assisted-service/internal/bminventory.logReplyReceived file="/src/internal/bminventory/inventory.go:3287" go-id=845 host_id=845f1e3c-c286-4d2f-ba92-4c5cab953641 infra_env_id=17c8b837-0130-4b8c-ad06-19bcd2a61dbf pkg=Inventory request_id=3a571ba6-5175-4bbe-b89a-20cdde30b884                         

time="2022-11-03T01:49:35Z" level=info msg="Adding new image status for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0f6ddae72f6d730ca07a265691401571a8d8f7e62546f1bcda26c9a01628f4d6 with status failure to host 845f1e3c-c286-4d2f-ba92-4c5cab953641" func="github.com/openshift/assisted-service/internal/host.(*Manager).UpdateImageStatus" file="/src/internal/host/host.go:805" pkg=host-state

 

Story CONSOLE-3179: Plugins: Deprecate react-helmet shared dependency

We should deprecate and eventually remove react-helmet as a shared plugin dependency. This dependency is small, and plugins can bring their own version if needed.

This requires updated our webpack plugin to allow dependency fallbacks when a shared dependency is not present.

cc Vojtech Szocs 

 

AC:

  • Update docs in the GitHub pages to state that we are deprecating the react-helmet as a shared plugin dependency
Bug MGMT-8855: [Assisted-4-9][Staging] repeated event messages "install strategy completed with no errors"

Description of the problem:

I installed a cluster with OCS and CNV.

The issue is that cluster event contain repeated messages:

1/9/2022, 6:17:31 PM    Operator ocs status: available message: install strategy completed with no errors
1/9/2022, 6:17:30 PM    Operator lso status: available message: install strategy completed with no errors
1/9/2022, 6:17:30 PM    Operator cnv status: available message: install strategy completed with no errors
1/9/2022, 6:17:06 PM    Successfully completed installing cluster
1/9/2022, 6:17:06 PM    Updated status of the cluster to installed
1/9/2022, 6:17:01 PM    Operator ocs status: available message: install strategy completed with no errors
1/9/2022, 6:17:00 PM    Operator lso status: available message: install strategy completed with no errors
1/9/2022, 6:17:00 PM    Operator cnv status: available message: install strategy completed with no errors
1/9/2022, 6:16:31 PM    Operator ocs status: progressing message: installing: waiting for deployment ocs-operator to become ready: deployment "ocs-operator" not available: Deployment does not have minimum availability.
1/9/2022, 6:16:30 PM    Operator lso status: available message: install strategy completed with no errors
1/9/2022, 6:16:30 PM    Operator cnv status: available message: install strategy completed with no errors
1/9/2022, 6:16:01 PM    Operator ocs status: progressing message: installing: waiting for deployment ocs-operator to become ready: deployment "ocs-operator" not available: Deployment does not have minimum availability.
1/9/2022, 6:16:00 PM    Operator lso status: available message: install strategy completed with no errors
1/9/2022, 6:16:00 PM    Operator cnv status: available message: install strategy completed with no errors
1/9/2022, 6:15:31 PM    Operator ocs status: progressing message: installing: waiting for deployment ocs-operator to become ready: deployment "ocs-operator" not available: Deployment does not have minimum availability.
1/9/2022, 6:15:31 PM    Operator lso status: available message: install strategy completed with no errors
1/9/2022, 6:15:30 PM    Operator cnv status: available message: install strategy completed with no errors

 

How reproducible:

100%

Steps to reproduce:

1. Install cluster with OCS and CNV

2. Watch cluster events

Actual results:

repeated message when olm operator completed installation

Expected results:

1 event record for olm operator finished successfully 

Bug OCPBUGS-1416: ODC add-page e2e tests doesn't pass (outdated checks)

Description of problem:
Tests failure when running dev-console tests locally.

Version-Release number of selected component (if applicable):
At least on 4.11 and 4.12

How reproducible:
Always

Steps to Reproduce:
1. Start cypress: yarn run test-cypress-dev-console
2. Run add-page

Actual results:
Fails

Expected results:
Should pass

Additional info:

Bug OCPBUGS-505: Input box aria-label and name wrong for editing PDB inside Deployments

Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Go to the detail page of some Deployments with PDB connected to it
2. Click Edit PDB from the kebab menu
3. Inspect the second input box under the `Availability requirement `

Actual results: The name and aria-label attributes always show minAvailable

Expected results: They should be consistent with the first input box

Additional info:

Bug OCPBUGS-650: "opm alpha render-veneer semver" raise error when no "Candidate" in config yaml

Description of problem:

"opm alpha render-veneer semver" raise error when no "Candidate" in config yaml

Version-Release number of selected component (if applicable):

zhaoxia@xzha-mac semver % opm version
Version: version.Version{OpmVersion:"11644a543", GitCommit:"11644a5433442c33698d2eee8d3f865b0d9386c0", BuildDate:"2022-08-29T08:16:54Z", GoOs:"darwin", GoArch:"amd64"}

How reproducible:

always

Steps to Reproduce:

1. prepare catalog-semver-veneer-wrong.yaml 
zhaoxia@xzha-mac semver % cat catalog-semver-veneer-wrong.yaml 
Schema: olm.semver
GenerateMajorChannels: false
GenerateMinorChannels: true
Stable:
  Bundles:
  - Image: quay.io/olmqe/nginxolm-operator-bundle:v1.0.2
  - Image: quay.io/olmqe/nginxolm-operator-bundle:v2.1.0
Fast:
  Bundles:
  - Image: quay.io/olmqe/nginxolm-operator-bundle:v0.0.1
  - Image: quay.io/olmqe/nginxolm-operator-bundle:v2.0.1
  - Image: quay.io/olmqe/nginxolm-operator-bundle:v2.1.0 

2. run "opm alpha render-veneer semver"
zhaoxia@xzha-mac semver % opm alpha render-veneer semver catalog-semver-veneer-wrong.yaml
2022/08/29 16:48:56 semver "catalog-semver-veneer-wrong.yaml": semver-render: no bundles specified or no bundles could be rendered

3.

Actual results:

error "no bundles specified or no bundles could be rendered" is raised.

Expected results:

no error

Additional info:

 
Bug OCPBUGS-766: Missing the instance-type/region/zone labels in Machine CRs

Description of problem:

Currently in 4.11, MAPI nutanix machine-controller does not provide the machine (VM)’s instance-type, region, zone, etc. labels to the Machine CR. And these columns are empty when viewing the Machine CRs, via cli “oc get Machine” or from the OCP cluster web console. 
$ oc -n openshift-machine-api get machine 
NAME                                  PHASE      TYPE REGION ZONE   AGE 
demo-ocp-cluster-g1-77nws-master-0   Running                        133m 
demo-ocp-cluster-g1-77nws-master-1   Running                        133m 
demo-ocp-cluster-g1-77nws-master-2   Running                        133m 
demo-ocp-cluster-g1-77nws-worker-2bsxn Running                      129m 
demo-ocp-cluster-g1-77nws-worker-75hr5 Running                      129m 
demo-ocp-cluster-g1-77nws-worker-rg7b9 Running                      129m

We can add something like the below labels to the Machine CR in the mapi-nutanix when reconciling for the Machine CRs: 
machine.openshift.io/instance-type: AHV 
machine.openshift.io/region: <prism-central-address> 
machine.openshift.io/zone: <prism-element-name/uuid>

Version-Release number of selected component (if applicable):

 

How reproducible:

run cli “oc get Machine” or from the OCP cluster web console to view the Machines resource

Steps to Reproduce:

1.
2.
3.

Actual results:

The "Type", "Region", "Zone" columns are empty for each Machine CR.

Expected results:

The "Type", "Region", "Zone" columns showing data for each Machine CR.

Additional info:

 
Story TRT-540: Track down lingering Pod Security issues

We do not have a well defined method to find these all just yet, identifying that would be a good first step.

Bug MGMT-12077: Assisted Installer doesn't remove a PV if it's not attached to a volume group

Description of the problem:

During install, we assume all PVs on a host have been added to a volume group and only remove them if they are. This could let other PVs that are not attached to volume groups persist and prevent coreos from installing properly. 

Relevant assisted installer links:

https://github.com/openshift/assisted-installer/blob/9bec593930995220a2a4550b067f5a186de3b042/src/installer/installer.go#L809 

https://github.com/openshift/assisted-installer/blob/9bec593930995220a2a4550b067f5a186de3b042/src/ops/ops.go#L414

 

Found while investigating triage issue https://issues.redhat.com/browse/AITRIAGE-4017 

See slack thread for more details https://coreos.slack.com/archives/C02CP89N4VC/p1663263128420489 

How reproducible:

100%

Steps to reproduce:

1. Create a host with a PV w/o a volume group

2. Add host to cluster and install 

3. Observe the install fail

Actual results:

Installation fails with 

"Error: checking for exclusive access to /dev/sda 
Caused by:
| 0: couldn't reread partition table: device is in use |
| 1: EBUSY: Device or resource busy"

Expected results:

All PVs and VGs are removed so that the installation will succeed

Bug OCPBUGS-1265: "Prometheus when installed on the cluster should report telemetry" should consider telemeterClient.enabled

I'd disabled Telemetry for the bulk of the CI fleet in OTA-740. But that lead to many
failures for:

[sig-instrumentation] Prometheus when installed on the cluster should report telemetry if a cloud.openshift.com token is present [Late] [Skipped:Disconnected] [Suite:openshift/conformance/parallel]

We should extend the checks for Telemetry enablement to include telemeterClient.enabled in the monitoring-specific ConfigMap, as well as the previously-checked pull-secret token.

Bug OCPBUGS-861: Rebase openshift/etcd 4.12 onto v3.5.5
Bug OCPBUGS-3878: RouteTargetReference missing default for "weight" in Route CRD v1 schema

The dependency on openshift/api needs to be bumped in openshift/kubernetes in order to pull in the fix from https://issues.redhat.com/browse/OCPBUGS-3635.

Bug OCPBUGS-439: DVO gatherer relies on the namespace name

The DVO metrics gatherer in the Insights operator relies on the "deployment-validation-operator" namespace name, but this is not very good, because the DVO can be installed in other namespaces (e.g it's installed in the "openshift-operators" namespace when installing through OperatorHub)

Bug OCPBUGS-2219: ConsolePlugin CRs cannot be garbage collected due to missing spec.i18n.loadType value

Description of problem:

https://github.com/openshift/api/pull/1186 - https://issues.redhat.com/browse/CONSOLE-3069 promoted ConsolePlugin CRD to v1.

The PR introduces also a conversion webhook from v1alpha1 to v1.

In new CRD version I18n ConsolePluginI18n is marked as optional.
The conversion webhook will not set a default valid ("Lazy"/"Preload") value writing the v1 object and a v1 object completely omitting spec.i18n will be accepted we no valid default value as well.

On the other side, at garbage collection time the object will be stuck forever due to the lack of a valid value for spec.i18n.loadType

Example,
create a v1 ConsolePlugin object:

cat <<EOF | oc apply -f -
apiVersion: console.openshift.io/v1
kind: ConsolePlugin
metadata:
  name: test472
spec:
  backend:
    service:
      basePath: /
      name: test472-service
      namespace: kubevirt-hyperconverged
      port: 9443
    type: Service
  displayName: Test 472 Plugin
EOF

Delete it in foreground mode:
stirabos@t14s:~$ oc delete consoleplugin test472 --timeout=30s --cascade='foreground' -v 7
I1011 18:20:03.255605   31610 loader.go:372] Config loaded from file:  /home/stirabos/.kube/config
I1011 18:20:03.266567   31610 round_trippers.go:463] DELETE https://api.ci-ln-krdzphb-72292.gcp-2.ci.openshift.org:6443/apis/console.openshift.io/v1/consoleplugins/test472
I1011 18:20:03.266581   31610 round_trippers.go:469] Request Headers:
I1011 18:20:03.266588   31610 round_trippers.go:473]     Accept: application/json
I1011 18:20:03.266594   31610 round_trippers.go:473]     Content-Type: application/json
I1011 18:20:03.266600   31610 round_trippers.go:473]     User-Agent: oc/4.11.0 (linux/amd64) kubernetes/fcf512e
I1011 18:20:03.266606   31610 round_trippers.go:473]     Authorization: Bearer <masked>
I1011 18:20:03.688569   31610 round_trippers.go:574] Response Status: 200 OK in 421 milliseconds
consoleplugin.console.openshift.io "test472" deleted
I1011 18:20:03.688911   31610 round_trippers.go:463] GET https://api.ci-ln-krdzphb-72292.gcp-2.ci.openshift.org:6443/apis/console.openshift.io/v1/consoleplugins?fieldSelector=metadata.name%3Dtest472
I1011 18:20:03.688919   31610 round_trippers.go:469] Request Headers:
I1011 18:20:03.688928   31610 round_trippers.go:473]     Authorization: Bearer <masked>
I1011 18:20:03.688935   31610 round_trippers.go:473]     Accept: application/json
I1011 18:20:03.688941   31610 round_trippers.go:473]     User-Agent: oc/4.11.0 (linux/amd64) kubernetes/fcf512e
I1011 18:20:03.840103   31610 round_trippers.go:574] Response Status: 200 OK in 151 milliseconds
I1011 18:20:03.840825   31610 round_trippers.go:463] GET https://api.ci-ln-krdzphb-72292.gcp-2.ci.openshift.org:6443/apis/console.openshift.io/v1/consoleplugins?fieldSelector=metadata.name%3Dtest472&resourceVersion=175205&watch=true
I1011 18:20:03.840848   31610 round_trippers.go:469] Request Headers:
I1011 18:20:03.840884   31610 round_trippers.go:473]     Accept: application/json
I1011 18:20:03.840907   31610 round_trippers.go:473]     User-Agent: oc/4.11.0 (linux/amd64) kubernetes/fcf512e
I1011 18:20:03.840928   31610 round_trippers.go:473]     Authorization: Bearer <masked>
I1011 18:20:03.972219   31610 round_trippers.go:574] Response Status: 200 OK in 131 milliseconds
error: timed out waiting for the condition on consoleplugins/test472

and in kube-controller-manager logs we see:

2022-10-11T16:25:32.192864016Z I1011 16:25:32.192788       1 garbagecollector.go:501] "Processing object" object="test472" objectUID=0cc46a01-113b-4bbe-9c7a-829a97d6867c kind="ConsolePlugin" virtual=false
2022-10-11T16:25:32.282303274Z I1011 16:25:32.282161       1 garbagecollector.go:623] remove DeleteDependents finalizer for item [console.openshift.io/v1/ConsolePlugin, namespace: , name: test472, uid: 0cc46a01-113b-4bbe-9c7a-829a97d6867c]
2022-10-11T16:25:32.304835330Z E1011 16:25:32.304730       1 garbagecollector.go:379] error syncing item &garbagecollector.node{identity:garbagecollector.objectReference{OwnerReference:v1.OwnerReference{APIVersion:"console.openshift.io/v1", Kind:"ConsolePlugin", Name:"test472", UID:"0cc46a01-113b-4bbe-9c7a-829a97d6867c", Controller:(*bool)(nil), BlockOwnerDeletion:(*bool)(nil)}, Namespace:""}, dependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:1, readerWait:0}, dependents:map[*garbagecollector.node]struct {}{}, deletingDependents:true, deletingDependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, beingDeleted:true, beingDeletedLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, virtual:false, virtualLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, owners:[]v1.OwnerReference(nil)}: ConsolePlugin.console.openshift.io "test472" is invalid: spec.i18n.loadType: Unsupported value: "": supported values: "Preload", "Lazy"

Version-Release number of selected component (if applicable):

OCP 4.12.0 ec4

How reproducible:

100%

Steps to Reproduce:

1. cat <<EOF | oc apply -f -
apiVersion: console.openshift.io/v1
kind: ConsolePlugin
metadata:
  name: test472
spec:
  backend:
    service:
      basePath: /
      name: test472-service
      namespace: kubevirt-hyperconverged
      port: 9443
    type: Service
  displayName: Test 472 Plugin
EOF
2. oc delete consoleplugin test472 --timeout=30s --cascade='foreground' -v 7

Actual results:

2022-10-11T16:25:32.192864016Z I1011 16:25:32.192788       1 garbagecollector.go:501] "Processing object" object="test472" objectUID=0cc46a01-113b-4bbe-9c7a-829a97d6867c kind="ConsolePlugin" virtual=false
2022-10-11T16:25:32.282303274Z I1011 16:25:32.282161       1 garbagecollector.go:623] remove DeleteDependents finalizer for item [console.openshift.io/v1/ConsolePlugin, namespace: , name: test472, uid: 0cc46a01-113b-4bbe-9c7a-829a97d6867c]
2022-10-11T16:25:32.304835330Z E1011 16:25:32.304730       1 garbagecollector.go:379] error syncing item &garbagecollector.node{identity:garbagecollector.objectReference{OwnerReference:v1.OwnerReference{APIVersion:"console.openshift.io/v1", Kind:"ConsolePlugin", Name:"test472", UID:"0cc46a01-113b-4bbe-9c7a-829a97d6867c", Controller:(*bool)(nil), BlockOwnerDeletion:(*bool)(nil)}, Namespace:""}, dependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:1, readerWait:0}, dependents:map[*garbagecollector.node]struct {}{}, deletingDependents:true, deletingDependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, beingDeleted:true, beingDeletedLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, virtual:false, virtualLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, owners:[]v1.OwnerReference(nil)}: ConsolePlugin.console.openshift.io "test472" is invalid: spec.i18n.loadType: Unsupported value: "": supported values: "Preload", "Lazy"

Expected results:

Object correctly deleted

Additional info:

The issue doesn't happen with --cascade='background' which is the default on the CLI client
Story AGENT-349: Allow console login to agent image

Because the agent ISO is ephemeral, it is probably safe to allow a user to log in to it with a password. If the network configuration is broken, a user may have no other way to debug it other than to log in through the console, which is currently not possible.

The best password to set would be the kubeadmin password used for the OpenShift GUI, since we'll have generated that already.

We must take care to test that this does not result in the installed nodes on disk allowing login with a password.

Bug OCPBUGS-1351: health_statuses_insights metrics is showing disabled rules in "total"

Description of problem:

health_statuses_insights metrics is showing disabled rules in "total". In other fields, it shows the correct amount.
In the code linked below, we can see that the "Disabled" rules are only skipped during the value assigning of TotalRisk

https://github.com/openshift/insights-operator/blob/master/pkg/insights/insightsreport/insightsreport.go#L268

How reproducible:

Always

Steps to Reproduce:

1. Upload a fake archive to trigger health checks (for example with rule CVE_2020_8555_kubernetes)
2. Disable one of the rules through https://console.redhat.com/api/insights-results-aggregator/v1/clusters/{cluster.id}/rules/{rule}/error_key/{error_key}/disable
3. Create support secret and set endpoint="https://httpstat.us/200"
4. restart insights operator
5. wait for alerts to trigger
6. Check health_statuses_insights metrics.

rule:

ccx_rules_ocp.external.rules.ocp_version_end_of_life.report

error_key:

OCP4X_BEYOND_EOL

 

Actual results:

"moderate" health_statuses_insights shows 2 triggers
"total" shows 3. Therefore, it is accounting for the deactivated rule.

Expected results:

"moderate" health_statuses_insights shows 2 triggers
"total" health_statuses_insights shows 2 triggers (doesn't account for deactivated rule)

Additional info:

If there is any issue in triggering this events, you may contact me and I can help with the steps.

 

Bug OCPBUGS-2138: Get OSImageURL override related metric data available in telemetry
Bug OCPBUGS-1017: Can't cancel login when using multi-cluster

When multi-cluster is enabled, it possible to get in a situation where you can't cancel login. If you select a cluster you don't know the credentials for, console will remember the last cluster and repeatedly send you to the login page with no way to cancel or go back. If we decide to set the last cluster in the user's preferences, it might be possible to get stuck even if you clear cookies and localStorage.

There are similar issues logging into cluster that are hibernating. See attached video.

cc Scott Berens

Bug OCPBUGS-1321: node_exporter collects metrics for "virtual" network interfaces

Description of problem:

node_exporter collects network metrics for "virtual" interfaces like br-*. When OVN is used, it also reports metrics for ovs-*, ovn, and genev_sys_* interfaces.

Version-Release number of selected component (if applicable):

4.12 (and before)

How reproducible:

Always

Steps to Reproduce:

1. Launch a 4.12 cluster.
2. Run the following PromQL query: "group by(device) (node_network_info)"
3.

Actual results:

Expected results:

Only real host interfaces should be present.

Additional info:




            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1708:
            
             console.openshift.io/use-i18n false in v1alpha API is converted to "" in the v1 APi, which is not a valid value for the enum type declared in the code. 
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




console.openshift.io/use-i18n false in v1alpha API is converted to "" in the v1 APi, which is not a valid value for the enum type declared in the code. 




Version-Release number of selected component (if applicable):




 4.12.0-0.nightly-2022-09-25-071630




How reproducible:




Always




Steps to Reproduce:




1. Load a dynamic plugin with v1alpha API console.openshift.io/use-i18n set to 'false'
2. In the v1 API the {"spec":{"i18n":{"loadType":""}}} loadType is set to empty string, which is not a valid value defined here: https://github.com/jhadvig/api/blob/22d69793277ffeb618d642724515f249262959a5/console/v1/types_console_plugin.go#L46
https://github.com/openshift/api/pull/1186/files# 




Actual results:




{"spec":{"i18n":{"loadType":""}}}




Expected results:




{"spec":{"i18n":{"loadType":"Lazy"}}}




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Story 
            
            
            STOR-963:
            
             Enable CSI driver debug logging
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
As OpenShift user, I want ClusterCSIDriver.Spec.LogLevel to affect the vSphere CSI driver logs, so I can capture the logs with all details and send it to Red Hat for investigation.



As OpenShift developer, I want ClusterCSIDriver.Spec.LogLevel to affect the vShere CSI CSI driver logs, so I can debug the driver with all logs.



Exit criteria:


    
	
  • When ClusterCSIDriver.Spec.LogLevel is set to Debug or higher, vSphere CSI driver logs include DEBUG messages like:
    • 





2022-08-05T11:54:10.808Z DEBUG commonco/utils.go:102 Container Orchestrator init params: 


Unknown macro: {InternalFeatureStatesConfigInfo} 


ServiceMode:controller}

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3263:
            
             The terraform binaries shipped by the installer are not statically linked
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-3096. The following is the description of the original issue: 

—

While the installer binary is statically linked, the terraform binaries shipped with it are dynamically linked.



This could give issues when running the installer on Linux and depending on the GLIBC version the specific Linux distribution has installed. It becomes a risk when switching the base image of the builders from ubi8 to ubi9 and trying to run the installer in cs8 or rhel8.



For example, building the installer on cs9 and trying to run it in a cs8 distribution leads to:




time="2022-10-31T14:31:47+01:00" level=debug msg="[INFO] running Terraform command: /root/test/terraform/bin/terraform version -json"
time="2022-10-31T14:31:47+01:00" level=error msg="/root/test/terraform/bin/terraform: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /root/test/terraform/bin/terraform)"
time="2022-10-31T14:31:47+01:00" level=error msg="/root/test/terraform/bin/terraform: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /root/test/terraform/bin/terraform)"
time="2022-10-31T14:31:47+01:00" level=debug msg="[INFO] running Terraform command: /root/test/terraform/bin/terraform version -json"
time="2022-10-31T14:31:47+01:00" level=error msg="/root/test/terraform/bin/terraform: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /root/test/terraform/bin/terraform)"
time="2022-10-31T14:31:47+01:00" level=error msg="/root/test/terraform/bin/terraform: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /root/test/terraform/bin/terraform)"
time="2022-10-31T14:31:47+01:00" level=debug msg="[INFO] running Terraform command: /root/test/terraform/bin/terraform init -no-color -force-copy -input=false -backend=true -get=true -upgrade=false -plugin-dir=/root/test/terraform/plugins"
time="2022-10-31T14:31:47+01:00" level=error msg="/root/test/terraform/bin/terraform: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /root/test/terraform/bin/terraform)"
time="2022-10-31T14:31:47+01:00" level=error msg="/root/test/terraform/bin/terraform: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /root/test/terraform/bin/terraform)"
time="2022-10-31T14:31:47+01:00" level=error msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failure applying terraform for \"cluster\" stage: failed to create cluster: failed doing terraform init: exit status 1\n/root/test/terraform/bin/terraform: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /root/test/terraform/bin/terraform)\n/root/test/terraform/bin/terraform: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /root/test/terraform/bin/terraform)\n"






How reproducible:Always




Steps to Reproduce:{code:none}
1. Build the installer on cs9
2. Run the installer on cs8 until the terraform binary are started
3. Looking at the terrform binary with ldd or file, you can get it is not a statically linked binary and the error above might occur depending on the glibc version you are running on 






Actual results:




 




Expected results:




The terraform and providers binaries have to be statically linked as well as the installer is.





Additional info:




This comes from a build of OKD/SCOS that is happening outside of Prow on a cs9-based builder image.

One can use the Dockerfile at images/installer/Dockerfile.ci and replace the builder image with one like https://github.com/okd-project/images/blob/main/okd-builder.Dockerfile



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3696:
            
             Surface ClusterVersion RetrievedUpdates condition messages
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-3458. The following is the description of the original issue: 

—

Description of problem:



Since way back in 4.8, we've had a banner with To request update recommendations, configure a channel that supports your version when ClusterVersion has RetrievedUpdates=False .  But that's only one of several reasons we could be RetrievedUpdates=False.  Can we pivot to passing through the ClusterVersion condition message?



Version-Release number of selected component (if applicable):



4.8 and later.



How reproducible:



100%



Steps to Reproduce:



1. Launch a cluster-bot cluster like 4.11.12.

2. Set a channel with oc adm upgrade channel stable-4.11.

3. Scale down the CVO with oc scale --replicas 0 -n openshift-cluster-version deployments/cluster-version-operator.

4. Patch in a RetrievedUpdates condition with:





$ CONDITIONS="$(oc get -o json clusterversion version | jq -c '[.status.conditions[] | if .type == "RetrievedUpdates" then .status = "False" | .message = "Testing" else . end]')"
$ oc patch --subresource status clusterversion version --type json -p "[{\"op\": \"add\", \"path\": \"/status/conditions\", \"value\": ${CONDITIONS}}]"






5. View the admin console at /settings/cluster.



Actual results:



Advice about configuring the channel (but it's already configured).



Expected results:



See the message you patched into the RetrievedUpdates condition.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-575:
            
             The lacking securityContext.seccompProfile.type of OLM deployments is blocking OCP upgrade to 4.12
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



Get the below error when upgrading to OCP 4.12 from 4.9->4.10->4.11.




MacBook-Pro:~ jianzhang$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-08-24-091058   True        True          4h      Unable to apply 4.12.0-0.nightly-2022-08-24-053339: the workload openshift-operator-lifecycle-manager/package-server-manager cannot roll out
   - lastTransitionTime: "2022-08-25T04:47:36Z"
    lastUpdateTime: "2022-08-25T04:47:36Z"
    message: 'pods "package-server-manager-85b6dc4d89-sdzcc" is forbidden: violates
      PodSecurity "restricted:v1.24": seccompProfile (pod or container "package-server-manager"
      must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")'
    reason: FailedCreate
    status: "True"
    type: ReplicaFailure




 



Version-Release number of selected component (if applicable):




MacBook-Pro:~ jianzhang$ oc exec catalog-operator-c5c655d5c-b9lcn -- olm --version
OLM version: 0.19.0
git commit: 8a984d41acc67c0bc9bfe807fadeef23f83abd44 




How reproducible:



always



Steps to Reproduce:

1. Install OCP 4.11.0-0.nightly-2022-08-24-091058

2. Upgrade it to 4.12.0-0.nightly-2022-08-24-053339



Actual results:



The cluster upgrading is blocked. Get the above errors as described.



Expected results:



 Upgraded to 4.12 from old OCP versions 4.5, 4.9 successfully.



Additional info:




MacBook-Pro:~ jianzhang$ oc get deployment package-server-manager -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "5"
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
  creationTimestamp: "2022-08-25T00:14:08Z"
  generation: 5
  labels:
    app: package-server-manager
  name: package-server-manager
  namespace: openshift-operator-lifecycle-manager
  ownerReferences:
  - apiVersion: config.openshift.io/v1
    kind: ClusterVersion
    name: version
    uid: 3fd29082-0e76-4b09-988e-78cb5fc7c8b5
  resourceVersion: "169028"
  uid: c8f7cbe2-4f82-40ce-9468-817ffefa903f
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: package-server-manager
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        target.workload.openshift.io/management: '{"effect": "PreferredDuringScheduling"}'
      creationTimestamp: null
      labels:
        app: package-server-manager
    spec:
      containers:
      - args:
        - --name
        - $(PACKAGESERVER_NAME)
        - --namespace
        - $(PACKAGESERVER_NAMESPACE)
        command:
        - /bin/psm
        - start
        env:
        - name: PACKAGESERVER_NAME
          value: packageserver
        - name: PACKAGESERVER_IMAGE
          value: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d49e1e27114f4b719bc8f3c222b2c5934d3b8028c79ec8e2bd288f6e9b5b3d5c
        - name: PACKAGESERVER_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: RELEASE_VERSION
          value: 4.12.0-0.nightly-2022-08-24-053339
        image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d49e1e27114f4b719bc8f3c222b2c5934d3b8028c79ec8e2bd288f6e9b5b3d5c
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: package-server-manager
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 10m
            memory: 50Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: FallbackToLogsOnError
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
        node-role.kubernetes.io/master: ""
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsNonRoot: true
      serviceAccount: olm-operator-serviceaccount
      serviceAccountName: olm-operator-serviceaccount
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 120
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 120
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2022-08-25T03:14:20Z"
    lastUpdateTime: "2022-08-25T03:14:20Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2022-08-25T04:47:36Z"
    lastUpdateTime: "2022-08-25T04:47:36Z"
    message: 'pods "package-server-manager-85b6dc4d89-sdzcc" is forbidden: violates
      PodSecurity "restricted:v1.24": seccompProfile (pod or container "package-server-manager"
      must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")'
    reason: FailedCreate
    status: "True"
    type: ReplicaFailure
  - lastTransitionTime: "2022-08-25T04:57:37Z"
    lastUpdateTime: "2022-08-25T04:57:37Z"
    message: ReplicaSet "package-server-manager-85b6dc4d89" has timed out progressing.
    reason: ProgressDeadlineExceeded
    status: "False"
    type: Progressing
  observedGeneration: 5
  readyReplicas: 1
  replicas: 1
  unavailableReplicas: 1 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1490:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1533:
            
             sdn rebase to 1.25
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
We need to rebase openshift-sdn to kube 1.25's kube-proxy.



In particular, we need this to get https://github.com/kubernetes/kubernetes/pull/110334 into master because we will probably get asked to backport it.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1736:
            
             cncc crashloop in proxy deployments
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




cloud-network-config-controller pod crashloops in proxy deployments as it tries to reach Openstack keystone API directly (not through the proxy) and there is no connectivity.

NAMESPACE                                          NAME                                                         READY   STATUS             RESTARTS          AGE
openshift-cloud-network-config-controller          cloud-network-config-controller-c4867b748-vlq9h              0/1     CrashLoopBackOff   158 (2m10s ago)   13h

$ oc -n openshift-cloud-network-config-controller logs -p cloud-network-config-controller-c4867b748-vlq9h
W0927 05:48:18.678947       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0927 05:48:18.680269       1 leaderelection.go:248] attempting to acquire leader lease openshift-cloud-network-config-controller/cloud-network-config-controller-lock...
I0927 05:48:26.754377       1 leaderelection.go:258] successfully acquired lease openshift-cloud-network-config-controller/cloud-network-config-controller-lock
I0927 05:48:26.755413       1 openstack.go:121] Custom CA bundle found at location '/kube-cloud-config/ca-bundle.pem' - reading certificate information
F0927 05:48:28.233519       1 main.go:101] Error building cloud provider client, err: Get "https://10.46.44.10:13000/": dial tcp 10.46.44.10:13000: connect: no route to host
goroutine 51 [running]:
k8s.io/klog/v2.stacks(0x1)
        /go/src/github.com/openshift/cloud-network-config-controller/vendor/k8s.io/klog/v2/klog.go:860 +0x8a
k8s.io/klog/v2.(*loggingT).output(0x37696c0, 0x3, 0x0, 0xc000636000, 0x1, {0x2cbcbd8?, 0x1?}, 0xc000438400?, 0x0)
        /go/src/github.com/openshift/cloud-network-config-controller/vendor/k8s.io/klog/v2/klog.go:825 +0x686
k8s.io/klog/v2.(*loggingT).printfDepth(0x37696c0, 0x237798a?, 0x0, {0x0, 0x0}, 0x7fff81041af7?, {0x23a20d0, 0x2d}, {0xc00052c050, 0x1, ...})
        /go/src/github.com/openshift/cloud-network-config-controller/vendor/k8s.io/klog/v2/klog.go:630 +0x1f2
k8s.io/klog/v2.(*loggingT).printf(...)
        /go/src/github.com/openshift/cloud-network-config-controller/vendor/k8s.io/klog/v2/klog.go:612
k8s.io/klog/v2.Fatalf(...)
        /go/src/github.com/openshift/cloud-network-config-controller/vendor/k8s.io/klog/v2/klog.go:1516
main.main.func1({0x26e5638, 0xc00016c040})
        /go/src/github.com/openshift/cloud-network-config-controller/cmd/cloud-network-config-controller/main.go:101 +0x26d
created by k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run
        /go/src/github.com/openshift/cloud-network-config-controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:211 +0x11bgoroutine 1 [select]:
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00052bb60?, {0x26cee20, 0xc000581740}, 0x1, 0xc00052bb60)
        /go/src/github.com/openshift/cloud-network-config-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x135
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00016c080?, 0x60db88400, 0x0, 0x20?, 0x7fea470ec108?)
        /go/src/github.com/openshift/cloud-network-config-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/openshift/cloud-network-config-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew(0xc0000a8120, {0x26e5638?, 0xc00016c040?})
        /go/src/github.com/openshift/cloud-network-config-controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:268 +0xd0
k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc0000a8120, {0x26e5638, 0xc00025fcc0})
        /go/src/github.com/openshift/cloud-network-config-controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:212 +0x12f
k8s.io/client-go/tools/leaderelection.RunOrDie({0x26e5638, 0xc00025fcc0}, {{0x26e7430, 0xc00062afa0}, 0x1fe5d61a00, 0x18e9b26e00, 0x60db88400, {0xc00065e630, 0xc000634810, 0x0}, ...})
        /go/src/github.com/openshift/cloud-network-config-controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:226 +0x94
main.main()
        /go/src/github.com/openshift/cloud-network-config-controller/cmd/cloud-network-config-controller/main.go:86 +0x450





Version-Release number of selected component (if applicable):




4.12.0-0.nightly-2022-09-26-050728




How reproducible:




Always




Steps to Reproduce:




1. Install OCP with proxy





Actual results:




Bootstrap failure and pod crashloop




Expected results:




Successful installation




Additional info:



Please find the must-gather here.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2627:
            
             NTP configuration in AgentConfig does not get applied
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




In ZTP input, we can put AdditionalNTPSources in order to have assisted-service mix the provided sources with those the nodes receive from DHCP.

AdditionalNTPSources in AgentConfig needs to be generated in InfraEnv in order for it to be applied in the installation




Version-Release number of selected component (if applicable):




4.11 MVP patch 2




How reproducible:




100%




Steps to Reproduce:




1. Create AgentConfig with AdditionalNTPSources like for example "0.fedora.pool.ntp.org"
2. Generate ISO
3. Deploy
4. Check the resulting cluster nodes /etc/chrony.conf





Actual results:




chrony.conf only contains DHCP provided NTP sources (if not static network deplooyment)






Expected results:




/etc/chrony.conf in all the cluster nodes should have at least a server listed:
server 0.fedora.pool.ntp.org iburst




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Story 
            
            
            AGENT-354:
            
             Standard "agent-gather" script
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Create a script that gathers debug information from a host running the agent ISO and exports it in a standard format so that we can ask customers to provide it for debugging when something has gone wrong (and also use it in CI).



For now, it is fine to require the user to ssh into the host to run the script. The script should be already in place inside the agent ISO.



The output should probably be a compressed tar file. That file could be saved locally, or potentially piped to stdout so that a user only has to run a command like: ssh node0 -c agent-gather >node0.tgz



Things we need to collect:


    
	
  • systemctl status and journal for each of the systemd services created by the agent installer (ideally this should be determined programmatically so we can't forget to add any)
    • 
	
  • network information: ifconfig; ip -j -p addr
    • 
	
  • Data supplied by the agent installer in /etc/assisted/*
    • 
	
  • /etc/containers/registries.conf
    • 
	
  • /etc/assisted-service/node0 (if it exists)
    • 
	
  • /usr/local/share/assisted-service/*.env
    • 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-184:
            
             [OCP web console]  Wrong message "404: Not found" while the user selects an installed operator and navigates from operator hub to installed operator page.
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



When user selects a installed operator (for example, openshift elastic search) in operator hub and navigating to installed operator page from operator information page



with the help of "view it here" option, "404 Not found" information has wrongly shown/appeared although it navigates to the installed operator at the end.



 



Version-Release number of selected components (if applicable):

4.12.0-0.nightly-2022-08-15-150248

How reproducible:



 Always



 



Steps to Reproduce:


    
	
  1. Login to OCP web console.
    2. 
	
  2. Install Operator, For example,OpenShift Elasticsearch Operator- production operators if missing.
    3. 
	
  3. Go to the Operator hub and  search for OpenShift Elasticsearch Operator. (make sure Project filter sets to 'All projects')
    4. 
	
  4. Click on OpenShift Elasticsearch Operator- production operators.
    5. 
	
  5. Click on the link "View it here" from the installed operator section.
    6. 
	
  6. View the behavior.
    7. 





Actual results:



Wrong message "404: Not found" while the user selects an installed operator and navigates from operator hub to installed operator page.



 



Browser console log indicate as below





main-chunk-525818b154a57a9b220a.min.js:1 unhandled error: Uncaught TypeError: Cannot read properties of undefined (reading 'firstElementChild') TypeError: Cannot read properties of undefined (reading 'firstElementChild')
    at c (https://console-openshift-console.apps.jmekkatt-dob.ibmcloud.qe.devcluster.openshift.com/static/vendors~main-chunk-40fab65853dff2fbc413.min.js:118:125992)
    at HTMLDivElement.l (https://console-openshift-console.apps.jmekkatt-dob.ibmcloud.qe.devcluster.openshift.com/static/vendors~main-chunk-40fab65853dff2fbc413.min.js:118:126387) TypeError: Cannot read properties of undefined (reading 'firstElementChild')
    at c (vendors~main-chunk-40fab65853dff2fbc413.min.js:72303:1)
    at HTMLDivElement.l (vendors~main-chunk-40fab65853dff2fbc413.min.js:72303:1)
window.onerror @ main-chunk-525818b154a57a9b220a.min.js:1
vendors~main-chunk-40fab65853dff2fbc413.min.js:72303 Uncaught TypeError: Cannot read properties of undefined (reading 'firstElementChild')
    at c (vendors~main-chunk-40fab65853dff2fbc413.min.js:72303:1)
    at HTMLDivElement.l (vendors~main-chunk-40fab65853dff2fbc413.min.js:72303:1)
c @ vendors~main-chunk-40fab65853dff2fbc413.min.js:72303
l @ vendors~main-chunk-40fab65853dff2fbc413.min.js:72303
scroll (async)
componentWillUnmount @ vendor-patternfly-core-chunk-006bb1499791fa7cfea7.min.js:38397
hs @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171377
bs @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171377
hs @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171377
bs @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171377
Oc @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171377
t.unstable_runWithPriority @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171690
Hi @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171377
Ac @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171377
pc @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171377
(anonymous) @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171377
t.unstable_runWithPriority @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171690
Hi @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171377
Vi @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171377
qi @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171377
De @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171377
Yt @ vendors~main-chunk-40fab65853dff2fbc413.min.js:171377
main-chunk-525818b154a57a9b220a.min.js:1          GET https://console-openshift-console.apps.jmekkatt-dob.ibmcloud.qe.devcluster.openshift.com/api/kubernetes/apis/operators.coreos.com/v1alpha1/clusterserviceversions/elasticsearch-operator.5.5.0 404 (Not Found)
  




Expected results:



Installed operator details should show without any error when the user selects an installed operator and navigates from operator hub to installed operator page.



 



Additional info:



Reproduced in both chrome[103.0.5060.114 (Official Build) (64-bit)] and firefox[91.11.0esr (64-bit)] browsers



Attached screen share for the same issue InstalledOperatorNavigation404.mp4

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-705:
            
             vSphere privilege checking failing when providing user-defined folder and/or resource pool
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




vSphere privilege checking failing when providing user-defined folder and/or resource pool




Version-Release number of selected component (if applicable):




4.12.0-0.nightly-2022-08-30-054458




How reproducible:




consistently




Steps to Reproduce:




1. Provide pre-existing folder and/or resource pool to the install-config
2. Perform an installation with an account with read only privileges on the datacenter and cluster
3. The installer will fail with missing privileges for the cluster and datacenter.  When a pre-existing folder and resource pool are defined, the account can hold read only privileges on the datacenter and cluster .





Actual results:




Installer reports missing privileges




Expected results:




Installer should succeed




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3874:
            
             masters repeatedly losing connection to API and going NotReady
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-3508. The following is the description of the original issue: 

—

Exposed via the fact that the periodic-ci-openshift-release-master-nightly-4.12-e2e-metal-ipi-sdn-serial-ipv4 job is at 0% for at least the past two weeks over approximatesly 65 runs. 



Testgrid shows that this job started failing in a very consistent way on Oct 25th at about 8am UTC: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.12-informing#periodic-ci-openshift-release-master-nightly-4.12-e2e-metal-ipi-sdn-serial-ipv4



6 disruption tests fail, all with alarming consistency virtually always claiming exactly 8s of disruption, max allowed 1s. 



And then openshift-tests.[sig-arch] events should not repeat pathologically fails with an odd signature:





{  6 events happened too frequently

event happened 35 times, something is wrong: node/master-2 - reason/NodeHasNoDiskPressure roles/control-plane,master Node master-2 status is now: NodeHasNoDiskPressure
event happened 35 times, something is wrong: node/master-2 - reason/NodeHasSufficientMemory roles/control-plane,master Node master-2 status is now: NodeHasSufficientMemory
event happened 35 times, something is wrong: node/master-2 - reason/NodeHasSufficientPID roles/control-plane,master Node master-2 status is now: NodeHasSufficientPID
event happened 35 times, something is wrong: node/master-1 - reason/NodeHasNoDiskPressure roles/control-plane,master Node master-1 status is now: NodeHasNoDiskPressure
event happened 35 times, something is wrong: node/master-1 - reason/NodeHasSufficientMemory roles/control-plane,master Node master-1 status is now: NodeHasSufficientMemory
event happened 35 times, something is wrong: node/master-1 - reason/NodeHasSufficientPID roles/control-plane,master Node master-1 status is now: NodeHasSufficientPID}






The two types of tests started failing together exactly, and the disruption measurements are bizzarely consistent, every single time we see precisely 8s for kube-api, cache-kube-api, openshift-api, cache-openshift-api, oauth-api, cache-oauth-api. It's always these 6, and it seems to be always exactly 8 seconds. I cannot state enough how strange this is. It almost implies that something is happening on a very consistent schedule. 



Occasionally these are accompanied by 1-2s of disruption for those backends with new connections, but sometimes not as well.



It looks like all of the disruption consistently happens within two very long tests:



4s within: [sig-network] services when running openshift ipv4 cluster ensures external ip policy is configured correctly on the cluster [Serial] [Suite:openshift/conformance/serial]



4s within: [sig-network] services when running openshift ipv4 cluster on bare metal [apigroup:config.openshift.io] ensures external auto assign cidr is configured correctly on the cluster [Serial] [Suite:openshift/conformance/serial]



Both tests appear to have run prior to oct 25, so I don't think it's a matter of new tests breaking something or getting unskipped. Both tests also always pass, but appear to be impacting the cluster?



The master's going NotReady also appears to fall within the above two tests as well, though it does not seem to directly match with when we measure disruption, but bear in mind there's a 40s delay before the node goes NotReady.



Focusing on https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.12-e2e-metal-ipi-sdn-serial-ipv4/1590640492373086208 where the above are from:



Two of the three master nodes appear to be going NodeNotReady a couple times throughout the run, as visible in the spyglass chart under the node state row on the left. master-0 does not appear here, but it does exist. (I suspect it has leader and thus is the node reporting the others going not ready)



From the master-0 kubelet log in must-gather we can see one of these examples where it reports that master-2 has not checked in:





2022-11-10T10:38:35.874090961Z I1110 10:38:35.873975       1 node_lifecycle_controller.go:1137] node master-2 hasn't been updated for 40.00700561s. Last Ready is: &NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-10 1
0:36:10 +0000 UTC,LastTransitionTime:2022-11-10 10:29:11 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,}
2022-11-10T10:38:35.874090961Z I1110 10:38:35.874056       1 node_lifecycle_controller.go:1137] node master-2 hasn't been updated for 40.007097549s. Last MemoryPressure is: &NodeCondition{Type:MemoryPressure,Status:False,LastHeartb
eatTime:2022-11-10 10:36:10 +0000 UTC,LastTransitionTime:2022-11-10 10:29:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,}
2022-11-10T10:38:35.874090961Z I1110 10:38:35.874067       1 node_lifecycle_controller.go:1137] node master-2 hasn't been updated for 40.007110285s. Last DiskPressure is: &NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatT
ime:2022-11-10 10:36:10 +0000 UTC,LastTransitionTime:2022-11-10 10:29:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,}
2022-11-10T10:38:35.874090961Z I1110 10:38:35.874076       1 node_lifecycle_controller.go:1137] node master-2 hasn't been updated for 40.007119541s. Last PIDPressure is: &NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTim
e:2022-11-10 10:36:10 +0000 UTC,LastTransitionTime:2022-11-10 10:29:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,}
2022-11-10T10:38:35.881749410Z I1110 10:38:35.881705       1 controller_utils.go:181] "Recording status change event message for node" status="NodeNotReady" node="master-2"
2022-11-10T10:38:35.881749410Z I1110 10:38:35.881733       1 controller_utils.go:120] "Update ready status of pods on node" node="master-2"
2022-11-10T10:38:35.881820988Z I1110 10:38:35.881799       1 controller_utils.go:138] "Updating ready status of pod to false" pod="metal3-b7b69fdbb-rfbdj"
2022-11-10T10:38:35.881893234Z I1110 10:38:35.881858       1 topologycache.go:179] Ignoring node master-2 because it has an excluded label
2022-11-10T10:38:35.881893234Z W1110 10:38:35.881886       1 topologycache.go:199] Can't get CPU or zone information for worker-0 node
2022-11-10T10:38:35.881903023Z I1110 10:38:35.881892       1 topologycache.go:215] Insufficient node info for topology hints (0 zones, %!s(int64=0) CPU, false)
2022-11-10T10:38:35.881932172Z I1110 10:38:35.881917       1 controller.go:271] Node changes detected, triggering a full node sync on all loadbalancer services
2022-11-10T10:38:35.882290428Z I1110 10:38:35.882270       1 event.go:294] "Event occurred" object="master-2" fieldPath="" kind="Node" apiVersion="v1" type="Normal" reason="NodeNotReady" message="Node master-2 status is now: NodeNotReady"






Now from master-2's kubelet log around that time, 40 seconds earlier puts us at 10:37:55, so we'd be looking for something odd around there.



A few potential lines:





Nov 10 10:37:55.232537 master-2 kubenswrapper[1930]: I1110 10:37:55.232495    1930 patch_prober.go:29] interesting pod/kube-controller-manager-guard-master-2 container/guard namespace/openshift-kube-controller-manager: Readiness probe status=failure output="Get \"https://192.168.111.22:10257/healthz\": dial tcp 192.168.111.22:10257: connect: connection refused" start-of-body=

Nov 10 10:37:55.232537 master-2 kubenswrapper[1930]: I1110 10:37:55.232549    1930 prober.go:114] "Probe failed" probeType="Readiness" pod="openshift-kube-controller-manager/kube-controller-manager-guard-master-2" podUID=8be2c6c1-f8f6-4bf0-b26d-53ce487354bd containerName="guard" probeResult=failure output="Get \"https://192.168.111.22:10257/healthz\": dial tcp 192.168.111.22:10257: connect: connection refused"

Nov 10 10:38:12.238273 master-2 kubenswrapper[1930]: E1110 10:38:12.238229    1930 controller.go:187] failed to update lease, error: Put "https://api-int.ostest.test.metalkube.org:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/master-2?timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Nov 10 10:38:13.034109 master-2 kubenswrapper[1930]: E1110 10:38:13.034077    1930 kubelet_node_status.go:487] "Error updating node status, will retry" err="error getting node \"master-2\": Get \"https://api-int.ostest.test.metalkube.org:6443/api/v1/nodes/master-2?resourceVersion=0&timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"






At 10:38:40 all kinds of master-2 watches time out with messages like:





Nov 10 10:38:40.244399 master-2 kubenswrapper[1930]: W1110 10:38:40.244272    1930 reflector.go:347] object-"openshift-oauth-apiserver"/"kube-root-ca.crt": watch of *v1.ConfigMap ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding






And then suddenly we're back online:





Nov 10 10:38:40.252149 master-2 kubenswrapper[1930]: I1110 10:38:40.252131    1930 kubelet_node_status.go:590] "Recording event message for node" node="master-2" event="NodeHasSufficientMemory"
Nov 10 10:38:40.252149 master-2 kubenswrapper[1930]: I1110 10:38:40.252156    1930 kubelet_node_status.go:590] "Recording event message for node" node="master-2" event="NodeHasNoDiskPressure"
Nov 10 10:38:40.252268 master-2 kubenswrapper[1930]: I1110 10:38:40.252165    1930 kubelet_node_status.go:590] "Recording event message for node" node="master-2" event="NodeHasSufficientPID"
Nov 10 10:38:40.252268 master-2 kubenswrapper[1930]: I1110 10:38:40.252177    1930 kubelet_node_status.go:590] "Recording event message for node" node="master-2" event="NodeReady"
Nov 10 10:38:47.904430 master-2 kubenswrapper[1930]: I1110 10:38:47.904373    1930 kubelet.go:2229] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-kube-controller-manager/kube-controller-manager-master-2"
Nov 10 10:38:47.904842 master-2 kubenswrapper[1930]: I1110 10:38:47.904662    1930 kubelet.go:2229] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-kube-controller-manager/kube-controller-manager-master-2"
Nov 10 10:38:47.907900 master-2 kubenswrapper[1930]: I1110 10:38:47.907872    1930 kubelet.go:2229] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-kube-controller-manager/kube-controller-manager-master-2"
Nov 10 10:38:48.431448 master-2 kubenswrapper[1930]: I1110 10:38:48.431414    1930 kubelet.go:2229] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-kube-controller-manager/kube-controller-manager-master-2"
Nov 10 10:38:54.764069 master-2 kubenswrapper[1930]: I1110 10:38:54.764029    1930 kubelet_getters.go:182] "Pod status updated" pod="openshift-kube-scheduler/openshift-kube-scheduler-master-2" status=Running
Nov 10 10:38:54.764069 master-2 kubenswrapper[1930]: I1110 10:38:54.764059    1930 kubelet_getters.go:182] "Pod status updated" pod="openshift-kni-infra/keepalived-master-2" status=Running
Nov 10 10:38:54.764069 master-2 kubenswrapper[1930]: I1110 10:38:54.764077    1930 kubelet_getters.go:182] "Pod status updated" pod="openshift-kni-infra/coredns-master-2" status=Running
Nov 10 10:38:54.764069 master-2 kubenswrapper[1930]: I1110 10:38:54.764086    1930 kubelet_getters.go:182] "Pod status updated" pod="openshift-kni-infra/haproxy-master-2" status=Running
Nov 10 10:38:54.764492 master-2 kubenswrapper[1930]: I1110 10:38:54.764106    1930 kubelet_getters.go:182] "Pod status updated" pod="openshift-etcd/etcd-master-2" status=Running
Nov 10 10:38:54.764492 master-2 kubenswrapper[1930]: I1110 10:38:54.764113    1930 kubelet_getters.go:182] "Pod status updated" pod="openshift-kube-controller-manager/kube-controller-manager-master-2" status=Running






Also curious:





Nov 10 10:37:50.318237 master-2 ovs-vswitchd[1324]: ovs|00251|connmgr|INFO|br0<->unix#468: 2 flow_mods in the last 0 s (2 deletes)
Nov 10 10:37:50.342965 master-2 ovs-vswitchd[1324]: ovs|00252|connmgr|INFO|br0<->unix#471: 4 flow_mods in the last 0 s (4 deletes)
Nov 10 10:37:50.364271 master-2 ovs-vswitchd[1324]: ovs|00253|bridge|INFO|bridge br0: deleted interface vethcb8d36e6 on port 41

Nov 10 10:37:53.579562 master-2 NetworkManager[1336]: <info>  [1668076673.5795] dhcp4 (enp2s0): state changed new lease, address=192.168.111.22








These look like they could be related to the tests these problems appear to coincide with?


            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2344:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-89:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1678:
            
             CI: Backend unit tests fails because devfile registry was updated (mock response)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:

pkg/devfile/sample_test.go fails after devfile registry was updated (https://github.com/devfile/registry/pull/126)



OCPBUGS-1677 is about updating our assertion so that the CI job runs successfully again. We might want to backport this as well.



This is about updating the code that the test should use a mock response instead of the latest registry content OR check some specific attributes instead of comparing the full JSON response.



Version-Release number of selected component (if applicable):

4.12



How reproducible:

Always



Steps to Reproduce:

1. Clone openshift/console

2. Run ./test-backend.sh



Actual results:

Unit tests fail



Expected results:

Unit tests should pass again



Additional info:

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2210:
            
             ZTP with converged flow is too slow
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




While running scale tests with ACM provisioning 1200+ SNOs via ZTP, converged flow was enabled. With converged flow the rate at which clusters begin install is much slower than what was witnessed without converged flow.

Example:
Without converged flow - 1250/1269 SNOs completed install in 3hrs and 11m
With converged flow - 487/1250 SNOs completed install in 10hours

The test actually hit timeouts so we don't exactly know how long it took, but you can see we only managed 487 SNOs to be provisioned in 10 hours.

The concurrency measurement scripts show that converged flow ran at a concurrency of 68 SNOs installing at a time vs non-converged flow peaking at 507.  Something within the converged flow is bottlenecking the SNOs install.




Version-Release number of selected component (if applicable):




Hub/SNO OCP 4.11.8
ACM 2.6.1-DOWNSTREAM-2022-09-08-02-53-38




How reproducible:




 




Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




converged flow to match previous provisioning speeds/rates




Additional info:




Must gather will be provided.



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1916:
            
             Workloads list page has different HPA action items from details page when All Projects selected
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



When all projects are selected, workloads list page and details page shows inconsistent HorizontalPodAutoscaler actions



Version-Release number of selected component (if applicable):



4.12.0-0.nightly-2022-07-25-010250



How reproducible:



Always



Steps to Reproduce:


    
	
  1. cluster admin goes to All projects deployments list page, click the kebab button of deployment/api-server in openshift-apiserver namespace
    2. 
	
  2. goes to deployment details page /k8s/ns/openshift-apiserver/deployments/apiserver, click 'Actions' and check HorizontalPodAutoscaler related action items
    3. 
	
  3. goes to project deployment list page /k8s/ns/openshift-apiserver/deployments, check the action items
    4. 





Actual results:


    
	
  1. the HPA action is 'Add PodDisruptionBudget'
    2. 
	
  2. the HPA actions are 'Edit HorizontalPodAutoscaler' and 'Remove HorizontalPodAutoscaler'
    3. 
	
  3. the HPA actions are 'Edit HorizontalPodAutoscaler' and 'Remove HorizontalPodAutoscaler'
    4. 





Expected results:


    
	
  1. workloads list and details page should have consistent HPA action items when 'All projects' are selected
    2. 





Additional info:

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-451:
            
             Show Git icon in repository link in details page should be based on the git provider
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



Git icon shown in the repository details page should be based on the git provider.



Version-Release number of selected component (if applicable):

4.11



How reproducible:

Always



Steps to Reproduce:

1. Create a Repository with gitlab repo url

2. Navigate to the detail page.



Actual results:



github icon is displayed for the gitlab url.



Expected results:



gitlab icon should be displayed for the gitlab url.



Additional info:



use `GitLabIcon` and `BitBucketIcon` from patternfly react-icons.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Task 
            
            
            IR-299:
            
             Bump k8s to 1.25 in image-registry repo
          

          
          
          

            
            
            
            
            
          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3003:
            
             Ignore non-ready endpoints when processing endpointslices
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-1427. The following is the description of the original issue: 

—

Description of problem:



Jump looks the worst on gcp, but looking closer Azure and AWS both jumped as well just not as high.



Disruption data indicates that the image registry on GCP was averaging around 30-40 seconds of disruption during an upgrade, until Aug 27th when it jumped to 125-135 seconds and has remained there ever since.



We see similar spikes in ingress-to-console and ingress-to-oauth. NOTE: image registry backend is also behind ingress, so all three are ingress related disruption.



https://datastudio.google.com/s/uBC4zuBFdTE



These charts show the problem on Aug 27 for registry, ingress to console, and ingress to oauth.



sdn network type appears unaffected.



Something merged Aug 26-27 that caused a significant change for anything behind ingress using ovn on gcp.



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3464:
            
             IBM operator needs deployment manifest fixes
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




The IBM VPC block CSI driver was rebased to v5.0.0 in this PR:
https://github.com/openshift/ibm-vpc-block-csi-driver/pull/26

However, we're missing the manifest changes from this PR in 4.12 (delayed by CI issues):
https://github.com/openshift/ibm-vpc-block-csi-driver-operator/pull/45

That includes some important changes:
- add csi-snapshotter sidecar and snapshotter manifests
- only deploy volumesnapshotclass if CRD exists
- set consistent imagePullPolicy in deployment manifests
- enable topology tests




Version-Release number of selected component (if applicable):




 




How reproducible:




 




Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




 




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            
              
            
            Bug 
            
            
            OCPRHV-783:
            
             High performance worker nodes are created with wrong configuration
          

          
          
          

            
            
            
            
            
          

          
          

            

              
When installing OCP cluster with worker nodes VM type specified as high performance, some of the configuration settings of said VMs do not match the configuration settings a high performance VM should have. 



Specific configurations that do not match are described in subtasks.



 



Default configuration settings of high performance VMs:

https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html-single/virtual_machine_management_guide/index?extIdCarryOver=true&sc_cid=701f2000001Css5AAC#Configuring_High_Performance_Virtual_Machines_Templates_and_Pools

            

          

          
          
          
          
        

      

      

        

  

    

      

        

          

            
            Sub-task 
            
            OCPRHV-788:
             Manual and automatic migration is enabled in high performance worker nodes
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
When installing OCP cluster with worker nodes VM type specified as high performance, manual and automatic migration is enabled in the said VMs. 

However, high performance worker VMs are created with default values of the engine, so only manual migration should be enabled.



Default configuration settings of high performance VMs:

https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html-single/virtual_machine_management_guide/index?extIdCarryOver=true&sc_cid=701f2000001Css5AAC#Configuring_High_Performance_Virtual_Machines_Templates_and_Pools



How reproducible: 100%



How to reproduce:



1. Create install-config.yaml with a vmType field and set it to high performance, i.e.:




apiVersion: v1
baseDomain: basedomain.com
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    ovirt:
      affinityGroupsNames: []
      vmType: high_performance
  replicas: 2
...




2. Run installation




./openshift-install create cluster --dir=resources --log-level=debug




3. Check worker VM's configuration in the RHV webconsole.



Expected: 

Only manual migration (under Host) should be enabled.



Actual: 

Manual and automatic migration is enabled.

            

          

          
          
          
          
          
        

      

      

        

        

      

    

  



        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            MGMT-11170:
            
             No thread ID in assisted-installer logs
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of the problem:



Noticed there were no thread IDs in the assisted-installer logs when debugging 240 node cluster deployment with MCE (slack thread) making it difficult to debug.



How reproducible: 100%



 



Steps to reproduce:



1. Create cluster using assisted service and start the install 



2. Look at the assisted-installer logs 



Actual results:



Logs look like




time="2022-07-14T16:17:31Z" level=info msg="Start complete installation step, with params success: true, error info: " 




Expected results: Thread ID would also print so we can understand which thread it came from




Adding setReportCaller to true will also help

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3221:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3425:
            
             [release-4.12] Azure Disk CSI Driver Operator gets degraded without "CSISnapshot" capability
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
With CSISnapshot capability is disabled, all Azure Disk CSI Driver Operator gets Degraded.



The reason is that cluster-csi-snapshot-controller-operator does not create VolumeSnapshotClass CRD, which the operator expects to exist.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1080:
            
             It's not possible to share BMC secrets between BareMetalHosts
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




See https://github.com/metal3-io/baremetal-operator/issues/1045




Version-Release number of selected component (if applicable):




 




How reproducible:




 




Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




 




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1574:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-816:
            
             Extracted ISO images are cached forever
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
We cache images by filename, which works when downloading from the Internet as the filename always includes the CoreOS version.



However, when extracting an image from the release payload, it always has the same name. Therefore, we will never update it to a newer image even when running different versions of the installer.



A possible solution:


    
	
  1. Check that the cached ISO's checksum matches the RHCOS metadata.
    2. 
	
  2. If it doesn't, extract the expected checksum from the release payload and compare that to the cached ISO's checksum.
    3. 
	
  3. If it still doesn't match, extract the ISO from the release payload.
    4. 





An alternative might be to set the name of the cache file to something different. It's not clear how we'd guarantee a match between the release payload we've been given and the ISO unless the name was based on the release payload (which eliminates some of the point of the cache, since ordinarily most release payloads will point to a small number of images).

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Story 
            
            
            MON-1261:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-940:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3035:
            
             4.12 backport: Multiple extra manifests in the same file are not applied correctly
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




Custom manifest files can be placed in the /openshift folder so that they will be applied during cluster installation.
Anyhow, if a file contains more than one manifests, all but the first are ignored.




Version-Release number of selected component (if applicable):




 




How reproducible:




Always




Steps to Reproduce:




1.Create the following custom manifest file in the /openshift folder:

```
apiVersion: v1
kind: ConfigMap
metadata:  
  name: agent-test  
  namespace: openshift-config
data:  
  value: agent-test
---
apiVersion: v1
kind: ConfigMap
metadata: 
name: agent-test-2
namespace: openshift-config
data: 
  value: agent-test-2
```
2. Create the agent ISO image and deploy a cluster





Actual results:




ConfigMap agent-test-2 does not exist in the openshift-config namespace




Expected results:




ConfigMap agent-test-2 must exist in the openshift-config namespace




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Feature Request 
            
            
            RHDEVDOCS-3165:
            
             The details of this Jira Card are restricted (Only Red Hat employees and contractors)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Only Red Hat employees and contractors)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1912:
            
             downstream `opm alpha diff` moving to `oc-mirror`
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
 – NOT A BUG – 

This was a story, but CI is not working for OLM project, so moved to OCPBUGS where it is. 



----------------------------





upstream the `opm alpha diff` functionality moved to `oc-mirror` team by a non-RH actor.



This story is to track downstreaming the two PRs.



The only thing to verify here is that there is no more `opm alpha diff` command. 



Other changes in the PRs are to externalize some interfaces and implement an undocumented alpha-level internal channel-level property list.



 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2079:
            
             systemReserved:ephemeral-storage in KubeletConfig doesn't work as expected
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




The setting of systemReserved: ephemeral-storage in KubeletConfig is not working as expected. 





Version-Release number of selected component (if applicable):




4.10.z, may exist on other OCP versions as well. 





How reproducible:




always





Steps to Reproduce:




1. Create a KubeletConfig on the node:

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: system-reserved-config
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/master: ""
  kubeletConfig:
    systemReserved:
      cpu: 500m
      memory: 500Mi
      ephemeral-storage: 10Gi


2. Check node allocatable storage with command: oc describe node |grep -C 5 ephemeral-storage





Actual results:




The Allocatable:ephemeral-storage on the node is not capacity.ephemeral-storage - systemReserved.ephemeral-storage - eviction-thresholds (10% of the capacity.ephemeral-storage by default)  




Expected results:




The Allocatable:ephemeral-storage on the node should be capacity.ephemeral-storage - systemReserved.ephemeral-storage - eviction-thresholds (10% of the capacity.ephemeral-storage by default) 




Additional info:




The root cause might be: process argument '--system-reserved=cpu=500m,memory=500Mi' overwrote the setting in /etc/kubernetes/kubelet.conf, one example:

root        6824       1 27 Sep30 ?        1-09:00:24 kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --runtime-cgroups=/system.slice/crio.service --node-labels=node-role.kubernetes.io/master,node.openshift.io/os_id=rhcos --node-ip=192.168.58.47 --minimum-container-ttl-duration=6m0s --cloud-provider= --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --hostname-override= --register-with-taints=node-role.kubernetes.io/master=:NoSchedule --pod-infra-container-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7b6408460148cb73c59677dbc2c261076bc07226c43b0c9192cc70aef5ba62 --system-reserved=cpu=500m,memory=500Mi --v=2 --housekeeping-interval=30s







 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3944:
            
             Handle 0600 kubeconfig
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Clone of https://issues.redhat.com/browse/OCPBUGSM-44162.



Cannot use the original as the bot won't accept a security bug:




When the change merges, the Bugzilla associated with the CVE must be set to MODIFIED. Since the DPTP bugzilla bot is not permitted to scan bugs with the SECURITY group in Bugzilla, The REP will not be able to use the bot's public functionality of moving their bug to MODIFIED.




https://docs.google.com/document/d/1KuenDafC3Ukw19jY55tkVeH8nNVVAi8TEAfqynoVfzY/edit#heading=h.ikdk6suc575k

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Task 
            
            
            AGENT-318:
            
             Merge the agent installer branch into openshift/installer master
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
In order to start 4.12 development, we need to merge the agent-installer branch. We need to create a PR and engage the Installer team on getting it approved

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2265:
            
             Allow passing documentation links for alerts
          

          
          
          

            
            
            
            
            
            
            
          

          
          
          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3307:
            
             [gcp] when the optional Service Usage API is disabled, IPI installation cannot succeed
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-1704. The following is the description of the original issue: 

—

Description of problem:




According to OCP 4.11 doc (https://docs.openshift.com/container-platform/4.11/installing/installing_gcp/installing-gcp-account.html#installation-gcp-enabling-api-services_installing-gcp-account), the Service Usage API (serviceusage.googleapis.com) is an optional API service to be enabled. But, the installation cannot succeed if this API is disabled.




Version-Release number of selected component (if applicable):




4.12.0-0.nightly-2022-09-25-071630




How reproducible:




Always, if the Service Usage API is disabled in the GCP project.




Steps to Reproduce:




1. Make sure the Service Usage API (serviceusage.googleapis.com) is disabled in the GCP project.
2. Try IPI installation in the GCP project. 




Actual results:




The installation would fail finally, without any worker machines launched.




Expected results:




Installation should succeed, or the OCP doc should be updated.




Additional info:




Please see the attached must-gather logs (http://virt-openshift-05.lab.eng.nay.redhat.com/jiwei/jiwei-0926-03-cnxn5/) and the sanity check results. 
FYI if enabling the API, and without changing anything else, the installation could succeed. 




            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3557:
            
             [4.12] provisioning of baremetal nodes fails when using multipath device as rootDeviceHints
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




Customer is not able anymore to provision new baremetal nodes in 4.10.35 using the same rootDeviceHints used in 4.10.10.
Customer uses HP DL360 Gen10, with exteranal SAN storage that is seen by the system as a multipath device. Latest IPA versions are implementing some changes to avoid wiping shared disks and this seems to affect what we should provide as rootDeviceHints.
They used to put /dev/sda as rootDeviceHints, in 4.10.35 it doesn't make the IPA write the image to the disk anymore because it sees the disk as part of a multipath device, we tried using the on top multipath device /dev/dm-0, the system is then able to write the image to the disk but then it gets stuck when it tried to issue a partprobe command, rebooting the systems to boot from the disk does not seem to help complete the provisioning, no workaround so far.




 



Version-Release number of selected component (if applicable):




 




How reproducible:




by trying to provisioning a baremetal node with a multipath device.




Steps to Reproduce:




1. Create a new BMH using a multipath device as rootDeviceHints
2.
3.





Actual results:




The node does not get provisioned




Expected results:




the node gets provisioned correctly




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-498:
            
             Update console operator vendor with latest openshift/api
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
https://github.com/openshift/api/pull/1213 and https://github.com/openshift/api/pull/1202 PR's have been merged but the latest 4.12 OCP clusters do not show the changes . 



According to https://github.com/openshift/console-operator/blob/bd2a7c9077ccf214dd8a725a7660e86d96e045b0/Dockerfile.rhel7#L18-L23, we need to vendor the openshift/api in console operator repo so that the latest manifests get's applied.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-389:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-4121:
            
             [SNO] csi-snapshot-controller CO is degraded when upgrade from 4.12 to 4.13 and reports permissions issue. 
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-3973. The following is the description of the original issue: 

—

Description of problem:




Upgrade SNO cluster from 4.12 to 4.13, the csi-snapshot-controller is degraded with message (same with log from csi-snapshot-controller-operator): 
E1122 09:02:51.867727       1 base_controller.go:272] StaticResourceController reconciliation failed: ["csi_controller_deployment_pdb.yaml" (string): poddisruptionbudgets.policy "csi-snapshot-controller-pdb" is forbidden: User "system:serviceaccount:openshift-cluster-storage-operator:csi-snapshot-controller-operator" cannot delete resource "poddisruptionbudgets" in API group "policy" in the namespace "openshift-cluster-storage-operator", "webhook_deployment_pdb.yaml" (string): poddisruptionbudgets.policy "csi-snapshot-webhook-pdb" is forbidden: User "system:serviceaccount:openshift-cluster-storage-operator:csi-snapshot-controller-operator" cannot delete resource "poddisruptionbudgets" in API group "policy" in the namespace "openshift-cluster-storage-operator"]





Version-Release number of selected component (if applicable):




4.12.0-0.nightly-2022-11-19-191518 to 4.13.0-0.nightly-2022-11-19-182111




How reproducible:




1/1




Steps to Reproduce:




Upgrade SNO cluster from 4.12 to 4.13 




Actual results:




csi-snapshot-controller is degraded




Expected results:




csi-snapshot-controller should be healthy




Additional info:




It also happened on from scratch cluster on 4.13: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.13-ocp-e2e-aws-ovn-arm64-single-node/1594946128904720384



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            SDN-3515:
            
             Targets for service monitor in the hosted control plane are not up
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
The issue found while testing HOSTEDCP-400 and HOSTEDCP-401.



Hypershift operator installed with flags:



 




--platform-monitoring=operator-only
--enable-uwm-telemetry-remote-write=true
--metrics-set=telemetry




 



Service monitors and pod monitors in the control plane:



 




[jiezhao@cube hypershift]$ oc get servicemonitor -n clusters-jz-test
NAME                                  AGE
catalog-operator                      45m
cluster-version-operator              45m
etcd                                  46m
kube-apiserver                        46m
kube-controller-manager               45m
monitor-multus-admission-controller   43m
monitor-ovn-master-metrics            43m
node-tuning-operator                  45m
olm-operator                          45m
openshift-apiserver                   45m
openshift-controller-manager          45m

[jiezhao@cube hypershift]$ oc get podmonitor -n clusters-jz-test
NAME                              AGE
cluster-image-registry-operator   46m
controlplane-operator             47m
hosted-cluster-config-operator    46m
ignition-server                   47m





 



In OCP management web console, go to Observe->Targets:



 




1. Status of service monitor 'monitor-multus-admission-controller' is Down, error:
   Scraped failed: server returned HTTP status 401 Unauthorized.
   It doesn't have cluster id in target labels
2. Target of pod monitor 'cluster-image-registry-operator' is missing, not shown





 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-169:
            
             Console e2e tests broken due to pod security admission controller
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
CI is failing due to the updated pod security admission controller. We need to update the console test pods with the correct security values.



Error: Command failed: echo '{"apiVersion":"v1","kind":"Pod","metadata":

{"name":"test-jxlpt-event-test-pod","namespace":"test-jxlpt"}

,"spec":{"containers":[

{"name":"httpd","image":"image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest"}

]}}' | kubectl create -n test-jxlpt -f -

Error from server (Forbidden): error when creating "STDIN": pods "test-jxlpt-event-test-pod" is forbidden: violates PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (container "httpd" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "httpd" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "httpd" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "httpd" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1776:
            
             Duplicate "Getting Started" notification will show on Search page for normal user
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




Duplicate notification "Getting started" would be shown on Search page 




Version-Release number of selected component (if applicable):




4.12.0-0.nightly-2022-09-26-111919




How reproducible:




Always




Steps to Reproduce:




1. Login OCP as normal user, and change to developer prespective, create a new project
2. Delete the project on page (switch to Administator prespective, go to Home -> Projects page)
3. Switch to Developer prespective, and go to Search page, check the notification "Getting Started"





Actual results:




Two notification shown on page




Expected results:




Only one should exist




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3289:
            
             [IBMCloud] Worker machines unreachable during initial bring up
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-1327. The following is the description of the original issue: 

—

See this comment for some updated information



—



Description of problem:

During IPI installation on IBM Cloud (x86_64), some of the worker machines have been seen to have no network connectivity during their initial bootup. Investigations were performed with IBM Cloud VPC to attempt to identify the issue, but in all appearances, all virtualization appears to be working.



Unfortunately due to this issue, no network traffic, no access to these worker machines is available to help identify the issue (Ignition is stuck without network traffic), so no SSH or console login is available to collect logs, or perform any testing on these machines.



The only content available is the console output, showing ignition is stuck due to the network issue.






Version-Release number of selected component (if applicable):

4.12.0



How reproducible:

About 60%



Steps to Reproduce:

1. Create an IPI cluster on IBM Cloud

2. Wait for the worker machines to be provisioned, causing IPI to fail waiting on machine-api operator

3. Check console of worker machines failing to report in to cluster (in this case 2 of 3 failed)



Actual results:

IPI creation failed waiting on machine-api operator to complete all worker node deployment



Expected results:

Successful IPI creation on IBM Cloud



Additional info:

As stated, investigation was performed by IBM Cloud VPC, but no further investigation could be performed since no access to these worker machines is available. Any further details that could be provided to help identify the issue would be helpful.



This appears to have become more prominent recently as well, causing concern for IBM Cloud's IPI GA support on the 4.12 release.



The only solution to restore network connectivity is rebooting the machine, which loses ignition bring up (I assume it must be triggered manually now), and in the case of IPI, isn't a great mitigation.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-183:
            
             Log line numbers overlap with cut-off rule when number is too big
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



When log line number is too big, the number will overlap with cut-off line in the log viewer.



Version-Release number of selected component (if applicable):



4.12.0-0.nightly-2022-08-15-150248



How reproducible:



Always



Steps to Reproduce:

1.Go to a pod log page with lots of logs, such as pod in openshift-cluster-version namespace. Check log line numbers.

2.

3.



Actual results:



1. When line number is too big, it will overlap with cut-off line.



Expected results:



1. Should have no overlaps in logs



Additional info:




            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3786: