Problem:

Certain Insights Advisor features differentiate between RHEL and OCP advisor

Goal:

Address top priority UI misalignments between RHEL and OCP advisor. Address UI features dropped from Insights ADvisor for OCP GA.

Scope:

Specific tasks and priority of them tracked in https://issues.redhat.com/browse/CCXDEV-7432

Managing PVs at scale for a fleet creates difficulties where "one size does not fit all". The ability for SRE to deploy prometheus with PVs and have retention based an on a desired size would enable easier management of these volumes across the fleet. 

The prometheus-operator exposes retentionSize.

This is a feature request to enable this configuration option via CMO cluster-monitoring-config ConfigMap.

cc Simon Pasquier  

Problem Alignment

The Problem

Today, all configuration for setting individual, for example, routing configuration is done via a single configuration file that only admins have access to. If an environment uses multiple tenants and each tenant, for example, has different systems that they are using to notify teams in case of an issue, then someone needs to file a request w/ an admin to add the required settings.

That can be bothersome for individual teams, since requests like that usually disappear in the backlog of an administrator. At the same time, administrators might get tons of requests that they have to look at and prioritize, which takes them away from more crucial work.

We would like to introduce a more self service approach whereas individual teams can create their own configuration for their needs w/o the administrators involvement.

Last but not least, since Monitoring is deployed as a Core service of OpenShift there are multiple restrictions that the SRE team has to apply to all OSD and ROSA clusters. One restriction is the ability for customers to use the central Alertmanager that is owned and managed by the SRE team. They can't give access to the central managed secret due to security concerns so that users can add their own routing information.

High-Level Approach

Provide a new API (based on the Operator CRD approach) as part of the Prometheus Operator that allows creating a subset of the Alertmanager configuration without touching the central Alertmanager configuration file.

Please note that we do not plan to support additional individual webhooks with this work. Customers will need to deploy their own version of the third party webhooks.

Goal & Success

• Allow users to deploy individual configurations that allow setting up Alertmanager for their needs without an administrator.

Solution Alignment

Key Capabilities

• As an OpenShift administrator, I want to control who can CRUD individual configuration so that I can make sure that any unknown third person can touch the central Alertmanager instance shipped within OpenShift Monitoring.
• As a team owner, I want to deploy a routing configuration to push notifications for alerts to my system of choice.

Key Flows

Team A wants to send all their important notifications to a specific Slack channel.

• Administrator gives permission to Team A to allow creating a new configuration CR in their individual namespace.
• Team A creates a new configuration CR.
• Team A configures what alerts should go into their Slack channel.
• Open Questions & Key Decisions (optional)
• Do we want to improve anything inside the developer console to allow configuration?

Copy/paste from [_https://github.com/openshift-cs/managed-openshift/issues/60_]

Which service is this feature request for?
OpenShift Dedicated and Red Hat OpenShift Service on AWS

What are you trying to do?
Allow ROSA/OSD to integrate with AWS Managed Prometheus.

Describe the solution you'd like
Remote-write of metrics is supported in OpenShift but it does not work with AWS Managed Prometheus since AWS Managed Prometheus requires AWS SigV4 auth.

• Note that Prometheus supports AWS SigV4 since v2.26 and OpenShift 4.9 uses v2.29.

Describe alternatives you've considered
There is the workaround to use the "AWS SigV4 Proxy" but I'd think this is not properly supported by RH.
https://mobb.ninja/docs/rosa/cluster-metrics-to-aws-prometheus/

Additional context
The customer wants to use an open and portable solution to centralize metrics storage and analysis. If they also deploy to other clouds, they don't want to have to re-configure. Since most clouds offer a Prometheus service (or it's easy to self-manage Prometheus), app migration should be simplified.

OCP/Telco Definition of Done
Feature Template descriptions and documentation.

Feature Overview.

Early customer feedback is that they see SNO as a great solution covering smaller footprint deployment, but are wondering what is the evolution story OpenShift is going to provide where more capacity or high availability are needed in the future.

While migration tooling (moving workload/config to new cluster) could be a mid-term solution, customer desire is not to include extra hardware to be involved in this process.

 For Telecommunications Providers, at the Far Edge they intend to start small and then grow. Many of these operators will start with a SNO-based DU deployment as an initial investment, but as DUs evolve, different segments of the radio spectrum are added, various radio hardware is provisioned and features delivered to the Far Edge, the Telecommunication Providers desire the ability for their Far Edge deployments to scale up from 1 node to 2 nodes to n nodes. On the opposite side of the spectrum from SNO is MMIMO where there is a robust cluster and workloads use HPA.

Goals

• Provide the capability to expand a single replica control plane topology to host more workloads capacity - add worker
• Provide the capability to expand a single replica control plane to be a highly available control plane
• To satisfy MMIMO Telecommunications providers will want the ability to scale a SNO to a multi-node cluster that can support HPA.
• Telecommunications providers do not want workload (DU specifically) downtime when migrating from SNO to a multi-node cluster.
• Telecommunications providers wish to be able to scale from one to two or more nodes to support a variety of radio hardware.
• Support CP scaling (CP HA) for 2 node cluster, 3 node cluster and n node cluster. As the number of nodes in the cluster increases so does the failure domain of the cluster. The cluster is now supporting more cell sectors and therefore has more of a need for HA and resiliency including the cluster CP.

Requirements

• TBD

(Optional) Use Cases

This Section:

• Main success scenarios - high-level user stories
• Alternate flow/scenarios - high-level user stories
• ...

Questions to answer…

• ...

Out of Scope

• …

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

• ...

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview

• As an infrastructure owner, I want a repeatable method to quickly deploy the initial OpenShift cluster.

• As an infrastructure owner, I want to install the first (management, hub, “cluster 0”) cluster to manage other (standalone, hub, spoke, hub of hubs) clusters.

Goals

• Enable customers and partners to successfully deploy a single “first” cluster in disconnected, on-premises settings

Requirements

4.11 MVP Requirements

• Customers and partners needs to be able to download the installer
• Enable customers and partners to deploy a single “first” cluster (cluster 0) using single node, compact, or highly available topologies in disconnected, on-premises settings
• Installer must support advanced network settings such as static IP assignments, VLANs and NIC bonding for on-premises metal use cases, as well as DHCP and PXE provisioning environments.
• Installer needs to support automation, including integration with third-party deployment tools, as well as user-driven deployments.
• In the MVP automation has higher priority than interactive, user-driven deployments.
• For bare metal deployments, we cannot assume that users will provide us the credentials to manage hosts via their BMCs.
• Installer should prioritize support for platforms None, baremetal, and VMware.
• The installer will focus on a single version of OpenShift, and a different build artifact will be produced for each different version.
• The installer must not depend on a connected registry; however, the installer can optionally use a previously mirrored registry within the disconnected environment.

Use Cases

• As a Telco partner engineer (Site Engineer, Specialist, Field Engineer), I want to deploy an OpenShift cluster in production with limited or no additional hardware and don’t intend to deploy more OpenShift clusters [Isolated edge experience].
• As a Enterprise infrastructure owner, I want to manage the lifecycle of multiple clusters in 1 or more sites by first installing the first  (management, hub, “cluster 0”) cluster to manage other (standalone, hub, spoke, hub of hubs) clusters [Cluster before your cluster].
• As a Partner, I want to package OpenShift for large scale and/or distributed topology with my own software and/or hardware solution.
• As a large enterprise customer or Service Provider, I want to install a “HyperShift Tugboat” OpenShift cluster in order to offer a hosted OpenShift control plane at scale to my consumers (DevOps Engineers, tenants) that allows for fleet-level provisioning for low CAPEX and OPEX, much like AKS or GKE [Hypershift].
• As a new, novice to intermediate user (Enterprise Admin/Consumer, Telco Partner integrator, RH Solution Architect), I want to quickly deploy a small OpenShift cluster for Poc/Demo/Research purposes.

Questions to answer…

Out of Scope

Out of scope use cases (that are part of the Kubeframe/factory project):

• As a Partner (OEMs, ISVs), I want to install and pre-configure OpenShift with my hardware/software in my disconnected factory, while allowing further (minimal) reconfiguration of a subset of capabilities later at a different site by different set of users (end customer) [Embedded OpenShift].
• As an Infrastructure Admin at an Enterprise customer with multiple remote sites, I want to pre-provision OpenShift centrally prior to shipping and activating the clusters in remote sites.

Background, and strategic fit

• This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

1. The user has only access to the target nodes that will form the cluster and will boot them with the image presented locally via a USB stick. This scenario is common in sites with restricted access such as government infra where only users with security clearance can interact with the installation, where software is allowed to enter in the premises (in a USB, DVD, SD card, etc.) but never allowed to come back out. Users can't enter supporting devices such as laptops or phones.
2. The user has access to the target nodes remotely to their BMCs (e.g. iDrac, iLo) and can map an image as virtual media from their computer. This scenario is common in data centers where the customer provides network access to the BMCs of the target nodes.
3. We cannot assume that we will have access to a computer to run an installer or installer helper software.

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

References

• Bootable Ephemeral Installer Image for Disconnected Cluster 0 Deployments

Feature Overview

The AWS-specific code added in OCPPLAN-6006 needs to become GA and with this we want to introduce a couple of Day2 improvements.
Currently the AWS tags are defined and applied at installation time only and saved in the infrastructure CRD's status field for further operator use, which in turn just add the tags during creation.

Saving in the status field means it's not included in Velero backups, which is a crucial feature for customers and Day2.
Thus the status.resourceTags field should be deprecated in favour of a newly created spec.resourceTags with the same content. The installer should only populate the spec, consumers of the infrastructure CRD must favour the spec over the status definition if both are supplied, otherwise the status should be honored and a warning shall be issued.

Being part of the spec, the behaviour should also tag existing resources that do not have the tags yet and once the tags in the infrastructure CRD are changed all the AWS resources should be updated accordingly.

On AWS this can be done without re-creating any resources (the behaviour is basically an upsert by tag key) and is possible without service interruption as it is a metadata operation.

Tag deletes continue to be out of scope, as the customer can still have custom tags applied to the resources that we do not want to delete.

Due to the ongoing intree/out of tree split on the cloud and CSI providers, this should not apply to clusters with intree providers (!= "external").

Once confident we have all components updated, we should introduce an end2end test that makes sure we never create resources that are untagged.

After that, we can remove the experimental flag and make this a GA feature.

Goals

• Inclusion in the cluster backups
• Flexibility of changing tags during cluster lifetime, without recreating the whole cluster

Requirements

• This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.

List any affected packages or components.

• Installer
• Cluster Infrastructure
• Storage
• Node
• NetworkEdge
• Internal Registry
• CCO

Feature Overview  

Much like core OpenShift operators, a standardized flow exists for OLM-managed operators to interact with the cluster in a specific way to leverage AWS STS authorization when using AWS APIs as opposed to insecure static, long-lived credentials. OLM-managed operators can implement integration with the CloudCredentialOperator in well-defined way to support this flow.

Goals:

Enable customers to easily leverage OpenShift's capabilities around AWS STS with layered products, for increased security posture. Enable OLM-managed operators to implement support for this in well-defined pattern.

Requirements:

• CCO gets a new mode in which it can reconcile STS credential request for OLM-managed operators
• A standardized flow is leveraged to guide users in discovering and preparing their AWS IAM policies and roles with permissions that are required for OLM-managed operators 
• A standardized flow is defined in which users can configure OLM-managed operators to leverage AWS STS
• An example operator is used to demonstrate the end2end functionality
• Clear instructions and documentation for operator development teams to implement the required interaction with the CloudCredentialOperator to support this flow

Use Cases:

See Operators & STS slide deck.

Out of Scope:

• handling OLM-managed operator updates in which AWS IAM permission requirements might change from one version to another (which requires user awareness and intervention)

Background:

The CloudCredentialsOperator already provides a powerful API for OpenShift's cluster core operator to request credentials and acquire them via short-lived tokens. This capability should be expanded to OLM-managed operators, specifically to Red Hat layered products that interact with AWS APIs. The process today is cumbersome to none-existent based on the operator in question and seen as an adoption blocker of OpenShift on AWS.

Customer Considerations

This is particularly important for ROSA customers. Customers are expected to be asked to pre-create the required IAM roles outside of OpenShift, which is deemed acceptable.

Documentation Considerations

• Internal documentation needs to exists to guide Red Hat operator developer teams on the requirements and proposed implementation of integration with CCO and the proposed flow
• External documentation needs to exist to guide users on:
  • how to become aware that the cluster is in STS mode
  • how to become aware of operators that support STS and the proposed CCO flow
  • how to become aware of the IAM permissions requirements of these operators
  • how to configure an operator in the proposed flow to interact with CCO

Interoperability Considerations

• this needs to work with ROSA
• this needs to work with self-managed OCP on AWS

OCP/Telco Definition of Done
Feature Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Feature --->
<--- Remove the descriptive text as appropriate --->

Feature Overview

• As RH OpenShift Product Owners, we want to enable new providers/platforms/service with varying levels of capabilities and integration with minimal reliance on OpenShift Engineering.
• As a new provider/platform partner, I want to enable my solution (hardware and/or software) with OpenShift with minimal effort.

Problem

• It is currently challenging for us to enable new platforms / providers without taking the heavy burden on doing the platform specific development ourselves.

Goals

• We want to enable the long-tail new platforms/providers to expand our reach into new markets and/or support new use cases.
• We want to remove strict dependencies we have on Engineering teams to review, support and test new providers.
• We want to lower the effort required for onboarding new platforms/providers.
• We want to enable new platform/providers to self-certify.
• We want to define tiered model for provider/platform integration that delineates ownership and responsibilities throughout new provider/platform development lifecycle and support model.
• We want to reduce time to onboard new provider/platform – ideally to a single release.
• We want to maintain consistent customer experience across all providers/platforms.

Requirements

• Step-by-step guide on how to add a new platform/provider for each tier
• Certification tool for partner to self-certify
• Certification tool results for (at least) each Y/minor release submitted by partner to Red Hat for acknowledgement
• DCI program to enable partners to run CI with OpenShift on their platform
• Well documented, accessible, and up-to-date test suites for providing the test coverage of the partner
• CI includes upgrade testing of OpenShift with partner's components
• Partner component upgrade failure should not block OpenShift upgrade
• Partner code is available in repositories in the openshift org on github with an open source license compatible with OpenShift

(Optional) Use Cases

This Section:

• Main success scenarios - high-level user stories
• Alternate flow/scenarios - high-level user stories
• ...

Questions to answer…

• ...

Out of Scope

• …

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

• ...

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

References

• [ Notes - OpenShift Everywhere - Onboarding new providers/platforms|https://www.google.com/url?q=[https://docs.google.com/document/d/1M1zLG2-wmkeCWTS8omN5G8vsDKvDeZfmegsi7cCVh-U/edit?usp%3Dsharing&sa=D&source=calendar&ust=1641920246517943&usg=AOvVaw2ZIl_XuE7OsSr4wCmRjOp3]] 
• OKD Installer Architecture Brainstorming Working Group Meeting Minutes
• New-Platform Documentation Working Group Meeting Minutes

Goal

Increase integration of Shipwright, Tekton, Argo CD in OpenShift GitOps with OpenShift platform and related products such as ACM.

Feature Overview

We drive OpenShift cross-market customer success and new customer adoption with constant improvements and feature additions to the existing capabilities of our OpenShift Core Networking (SDN and Network Edge). This feature captures that natural progression of the product.

Goals

• Feature enhancements (performance, scale, configuration, UX, ...)
• Modernization (incorporation and productization of new technologies)

Requirements

• Core Networking Stability
• Core Networking Performance and Scale
• Core Neworking Extensibility (Multus CNIs)
• Core Networking UX (Observability)
• Core Networking Security and Compliance

In Scope

• Network Edge (ingress, DNS, LB)
• SDN (CNI plugins, openshift-sdn, OVN, network policy, egressIP, egress Router, ...)
• Networking Observability

Out of Scope

There are definitely grey areas, but in general:

• CNV
• Service Mesh
• CNF

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

tldr: three basic claims, the rest is explanation and one example

1. We cannot improve long term maintainability solely by fixing bugs.
2. Teams should be asked to produce designs for improving maintainability/debugability.
3. Specific maintenance items (or investigation of maintenance items), should be placed into planning as peer to PM requests and explicitly prioritized against them.

While bugs are an important metric, fixing bugs is different than investing in maintainability and debugability. Investing in fixing bugs will help alleviate immediate problems, but doesn't improve the ability to address future problems. You (may) get a code base with fewer bugs, but when you add a new feature, it will still be hard to debug problems and interactions. This pushes a code base towards stagnation where it gets harder and harder to add features.

One alternative is to ask teams to produce ideas for how they would improve future maintainability and debugability instead of focusing on immediate bugs. This would produce designs that make problem determination, bug resolution, and future feature additions faster over time.

I have a concrete example of one such outcome of focusing on bugs vs quality. We have resolved many bugs about communication failures with ingress by finding problems with point-to-point network communication. We have fixed the individual bugs, but have not improved the code for future debugging. In so doing, we chase many hard to diagnose problem across the stack. The alternative is to create a point-to-point network connectivity capability. this would immediately improve bug resolution and stability (detection) for kuryr, ovs, legacy sdn, network-edge, kube-apiserver, openshift-apiserver, authentication, and console. Bug fixing does not produce the same impact.

We need more investment in our future selves. Saying, "teams should reserve this" doesn't seem to be universally effective. Perhaps an approach that directly asks for designs and impacts and then follows up by placing the items directly in planning and prioritizing against PM feature requests would give teams the confidence to invest in these areas and give broad exposure to systemic problems.

Relevant links:

• Documentation:
  • Edge Diagnostics Scratchpad, our team's internal diagnostic guide.
  • Troubleshooting OCP networking issues - The complete guide, the SDN team's diagnostic guide.
  • Linux Performance, Brendan Gregg's guide to analyzing Linux performance issues.
  • RFC: A proper feedback loop on Alerts.
  • OpenShift Router Reload Technical Overview on Access.
  • Performance Scaling HAProxy with OpenShift on Access.
  • How to collect worker metrics to troubleshoot CPU load, memory pressure and interrupt issues and networking on worker nodes in OCP 4 on Access.
  • OpenShift Performance and Scale Knowledge Base on Mojo, results from OpenShift scalability testing.
  • Scalability and performance, OCP 4.5 documentation about the router's currently known scalability limits.
  • Scaling OpenShift Container Platform HAProxy Router, OCP 3.11 documentation about the manual performance configuration that was possible in OCP 3.
  • Timing web requests with cURL and Chrome from the Cloudflare blog.
  • tcpdump advanced filters, some useful tcpdump commands.
  • OpenShift SDN - Networking, OCP 3.11 documentation on the SDN (useful background reading).
  • Ingress Operator and Controller Status Conditions, design document for improved status condition reporting.
  • Observability tips for HAProxy, a slide deck by Willy Tarreau.
  • Interesting Traces - Out of Order versus Retransmissions, analysis using tshark.
  • The PCP Book: A Complete Documentation of Performance Co-Pilot, by Yogesh Babar.
  • Debugging kernel networking bug, brief guide to using SystemTap on RHCOS.
  • Troubleshooting throughput issues from the OCP 4.5 documentation.
  • Troubleshooting OpenShift Clusters and Workloads.
  • Red Hat Enterprise Linux Network Performance Tuning Guide (PDF).
  • openshift/enhancements#289 stability: point to point network check, a diagnostic built into the kube-apiserver operator.
• Diagnostic tools:
  • dropwatch to watch for packet drops.
  • ethtool to check NIC configuration.
  • iovisor/bcc: BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more to trace and diagnose various issues in the networking stack.
  • r-curler to gather timing information about HTTP/HTTPS connections.
  • route-monitor, to monitor routes for reachability.
  • hping(3), a programmable packet generator.
  • OpenTracing / Jaeger in OpenShift.
  • node-problem-detector, a possible integration point for new diagnostics.
  • Using SystemTap by Brendan Gregg.
  • DTrace SystemTap cheatsheet (PDF).
• Visualization and more sophisticated diagnostic tools:
  • eldadru/ksniff, kubectl plugin for tcpdump & Wireshark.
  • ironcladlou/ditm, Dan's "Dan in the Middle" tool.
  • Skydive, network diagnostic and visualization tool.
  • ali, a "load testing tool capable of performing real-time analysis" with visualization.
• Testing tools:
  • stress-ng, a general stress-loading tool (CPU, filesystem, network, ...).
  • mb, the networking benchmarking tool written and used by Jiri Mencak from our Perf+Scale team.
• Case studies:
  • BZ1763206 is an example of diagnosing DNS latency/timeouts.
  • BZ1829779 Investigation details the diagnosis of route latency.
  • BZ1845545 is an example of diagnosing misconfigured DNS for an external LB.
  • Debugging network stalls on Kubernetes, from the GitHub Blog, about diagnosing Kubernetes performance issues related to ksoftirqd.

Feature Overview

• This Section:* High-Level description of the feature ie: Executive Summary

• Note: A Feature is a capability or a well defined set of functionality that delivers business value. Features can include additions or changes to existing functionality. Features can easily span multiple teams, and multiple releases.

Goals

• This Section:* Provide high-level goal statement, providing user context and expected user outcome(s) for this feature

• …

Requirements

• This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.

(Optional) Use Cases

This Section: 

• Main success scenarios - high-level user stories

• Alternate flow/scenarios - high-level user stories

• ...

Questions to answer…

• ...

Out of Scope

• …

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

• ...

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?

• Does this feature have doc impact?  

• New Content, Updates to existing content,  Release Note, or No Doc Impact

• If unsure and no Technical Writer is available, please contact Content Strategy.

• What concepts do customers need to understand to be successful in [action]?

• How do we expect customers will use the feature? For what purpose(s)?

• What reference material might a customer want/need to complete [action]?

• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.

• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview

Customers are asking for improvements to the upgrade experience (both over-the-air and disconnected). This is a feature tracking epics required to get that work done.  

Goals

1. Have an option to do upgrades in more discrete steps under admin control. Specifically, these steps are: 
   • Control plane upgrade
   • Worker nodes upgrade
   • Workload enabling upgrade (i..e. Router, other components) or infra nodes
2. Better visibility into any errors during the upgrades and documentation of what they error means and how to recover. 
3. An user experience around an end-2-end back-up and restore after a failed upgrade 
4. OTA-810  - Better Documentation: 
   • Backup procedures before upgrades. 
   • More control over worker upgrades (with tagged pools between user Vs admin)
   • The kinds of pre-upgrade tests that are run, the errors that are flagged and what they mean and how to address them. 
   • Better explanation of each discrete step in upgrades, and what each CVO Operator is doing and potential errors, troubleshooting and mitigating actions.

References

• OpenShift Upgrades Customer Issues Discussion -6/22/2023
• Select Customer Experiences and Feedback on Upgrades

Feature Overview

Enable sharing ConfigMap and Secret across namespaces

Requirements

Questions to answer…

NA

Out of Scope

NA

Background, and strategic fit

Consumption of RHEL entitlements has been a challenge on OCP 4 since it moved to a cluster-based entitlement model compared to the node-based (RHEL subscription manager) entitlement mode. In order to provide a sufficiently similar experience to OCP 3, the entitlement certificates that are made available on the cluster (OCPBU-93) should be shared across namespaces in order to prevent the need for cluster admin to copy these entitlements in each namespace which leads to additional operational challenges for updating and refreshing them. 

Documentation Considerations

Questions to be addressed:
 * What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
 * Does this feature have doc impact?
 * New Content, Updates to existing content, Release Note, or No Doc Impact
 * If unsure and no Technical Writer is available, please contact Content Strategy.
 * What concepts do customers need to understand to be successful in [action]?
 * How do we expect customers will use the feature? For what purpose(s)?
 * What reference material might a customer want/need to complete [action]?
 * Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
 * What is the doc impact (New Content, Updates to existing content, or Release Note)?

Summary (PM+lead)

https://issues.redhat.com/browse/AUTH-2 revealed that, in prinicipal, Pod Security Admission is possible to integrate into OpenShift while retaining SCC functionality.

This epic is about the concrete steps to enable Pod Security Admission by default in OpenShift

Motivation (PM+lead)

Goals (lead)

• Enable Pod Security Admission in "restricted" policy level by default
• Migrate existing core workloads to comply to the "restricted" pod security policy level

Non-Goals (lead)

• Other OpenShift workloads must be migrated by the individual responsible teams.

Deliverables

Proposal (lead)

Enhancement - https://github.com/openshift/enhancements/pull/1010

User Stories (PM)

Dependencies (internal and external, lead)

Previous Work (lead)

Open questions (lead)

1. ...

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

• https://github.com/openshift/cluster-authentication-operator/pull/570
• https://github.com/openshift/cluster-csi-snapshot-controller-operator/pull/120
• https://github.com/openshift/cluster-samples-operator/pull/425

Epic Goal

HyperShift provisions OpenShift clusters with externally managed control-planes. It follows a slightly different process for provisioning clusters. For example, HyperShift uses cluster API as a backend and moves all the machine management bits to the management cluster.  

Why is this important?

showing machine management/cluster auto-scaling tabs in the console is likely to confuse users and cause unnecessary side effects. 

Definition of Done

• MachineConfig and MachineConfigPool should not be present, they should be either removed or hidden when the cluster is spawned using HyperShift. 
• Cluster Settings show say the control plane is externally managed and be read-only.
• Cluster Settings -> Configuration resources should be read-only, maybe hide the tab
• Some resources should go in an allowlist. Most will be hidden
• Review getting started steps

See Design Doc: https://docs.google.com/document/d/1k76JtRRHBdCCEjHPqKcYvbNVsuaGmRhWDLESWIm0mbo/edit#

Setup / Testing

It's based on the SERVER_FLAG controlPlaneTopology being set to External is really the driving factor here; this can be done in one of two ways:

• Locally via a Bridge Variable, export BRIDGE_CONTROL_PLANE_TOPOLOGY_MODE="External"
• Locally / OnCluster via modifying the window.SERVER_FLAGS.controlPlaneTopology to External in the dev tools

To test work related to cluster upgrade process, use a 4.10.3 cluster set on the candidate-4.10 upgrade channel using 4.11 frontend code.

Epic Goal

• Ensure that OCP Admin console can enable dark mode
• Mechanism for switching between light/dark mode, based on user preferences or User Settings in OCP
• Identify changes needed for OCP console's dynamic plugins
• Status as we work through the stories: https://docs.google.com/document/d/18ib0A9ZtSYo1e2aOtM2LtZcsNFs1aRCHXx6NAPsIAso/edit#

Why is this important?

• So the UX satisfies the current trands, where dark mode is becoming a standard for modern services.  

Acceptance Criteria

• OCP admin console must be rendered in a preferred mode based on `prefers-color-scheme` media query
• OCP admin console must be rendered in a preferred mode selected in the User Setting page
• Create an followup epic/story for and listing and tracking changes needed in OCP console's dynamic plugins

Dependencies (internal and external)

1. PatternFly - Dark mode PF variables

Previous Work (Optional):

1. Mike Coker has worked on a POC from the PF point of view on both the admin and dev console, and the screenshot results are listed below along with the repo branch. Also listed is a document covering some of the common issues found when putting together the admin console POC. https://github.com/mcoker/console/tree/dark-theme
   Background POC work completed for reference:

PatternFly Dark Theme Handbook: https://docs.google.com/document/d/1mRYEfUoOjTsSt7hiqjbeplqhfo3_rVDO0QqMj2p67pw/edit

Admin Console -> Workloads & Pods

• Screenshots: https://docs.google.com/presentation/d/1BoOpXpX96_uNUhVJiEqGUCZ-JMYh1A78fXJhk3H86K8/edit#slide=id.p
• Github branch: https://github.com/mcoker/console/tree/dark-theme
• Github PR for Dark Theme WIP by Vikram: https://github.com/openshift/console/pull/10947{}

Dev Console -> Gotcha pages: Observe Dashboard and Metrics, Add, Pipelines: builder, list, log, and run

• Screenshots: https://docs.google.com/presentation/d/13jagAf_JEu82hd_KpEHRmj25xCEqwImWpNufzVopuUs/edit#slide=id.p
• Github branch: https://github.com/openshift/console/pull/10815
• Evaluation: https://docs.google.com/spreadsheets/d/1fBCPPb4Z1sqUDKgQkpueGe8VQZo4pBUA9PV4vnFCnDw/edit#gid=0

Open questions::

1. Who should be responsible for updating DynamicPlugins to be able to render in dark mode?

An epic we can duplicate for each release to ensure we have a place to catch things we ought to be doing regularly but can tend to fall by the wayside.

This epic contains all the Dynamic Plugins related stories for OCP release-4.11 

Epic Goal

• Track all the stories under a single epic

Acceptance Criteria

Goal

• Add the ability for users to select supported but not recommended updates.
• Refine workflow when both "upgradeable=false" and "supported-but-not-recommended" updates occur

Background
RFE: for 4.10, Cincinnati and the cluster-version operator are adding conditional updates (a.k.a. targeted edge blocking): https://issues.redhat.com/browse/OTA-267

High-level plans in https://github.com/openshift/enhancements/blob/master/enhancements/update/targeted-update-edge-blocking.md#update-client-support-for-the-enhanced-schema

Example of what the oc adm upgrade UX will be in https://github.com/openshift/enhancements/blob/master/enhancements/update/targeted-update-edge-blocking.md#cluster-administrator.

The oc implementation landed via https://github.com/openshift/oc/pull/961.

Design

• Use case 01: "supported but not recommended" occurs to the latest version:
  • Add an info icon next to the version on update path with a pop-over to explain about why updating to this version is supported, but not recommended and a link to known risks
  • Identify the difference in "recommended" versions, "supported but not recommended" versions, and "blocked" versions (upgradeable=false) in the + more modal.
  • The latest version is pre-selected in the dropdown in the update modal with an inline alert to inform users about supported-but-not-recommended version with link to known risks. Users can choose to update to another recommended versions, update to a supported-but-not-recommended one, or wait.
  • The "recommended" and "supported but not recommended" updates are separated in the dropdown.
  • If a user selects a "recommended" update, the inline alert disappears.
• Use case 02: When both "upgradeable=false" and "supported but not recommended" occur:
  • Add an alert banner to explain why users shouldn’t update to the latest version and link to how to resolve on the cluster settings details page. Users have the options to resolve the issue, update to a patch version, or wait.
  • If users open the update modal without resolving the "upgradeable=false" issue, the next recommended version is pre-selected. An expandable link "View blocked versions (#)" is included under the dropdown to show "upgradeable=false" versions with resolve link.
  • If users resolve the "upgradeable=false" issue, the cluster settings page will change to use case 01

• • Question: Priority on changing the upgradeable=false alert banner in update modal and blocked versions in dropdown

See design doc: https://docs.google.com/document/d/1Nja4whdsI5dKmQNS_rXyN8IGtRXDJ8gXuU_eSxBLMIY/edit#

See marvel: https://marvelapp.com/prototype/h3ehaa4/screen/86077932

Epic Goal

• Add telemetry so that we know how image stream features are used.

Why is this important?

• We have a long standing epic to create image streams v2. We need to better understand how image streams are used today.

Scenarios

1. ...

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Epic Goal

• Make the image registry distributed across availability zones.

Why is this important?

• The registry should be highly available and zone failsafe.

Scenarios

1. As an administrator I want to rely on a default configuration that spreads image registry pods across topology zones so that I don't suffer from a long recovery time (>6 mins) in case of a complete zone failure if all pods are impacted.

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.

Dependencies (internal and external)

1. Pod's topologySpreadConstraints

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: https://github.com/openshift/cluster-image-registry-operator/pull/730
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Goal

Remove Jenkins from the OCP Payload.

Problem

• Jenkins images are "non-trival in size, impact experience around OCP payloads
• Security advisories cannot be handled once, but against all actively supported OCP releases, adding to response time for handling said advisories
• Some customers may now want to upgrade Jenkins as OCP upgrades (making this configurable is more ideal)

Why is this important

• This is an engineering motivated item to reduce costs so we have more cycles for strategic work
• Aside from the team itself, top level OCP architects want this to reduce the image size, improve general OCP upgrade experience
• Sends a mix message with respect to what is startegic CI/CI when Jenkins is baked into OCP, but Tekton/Pipelines is an add-on, day 2 install sort of thing

Dependencies (internal and external)

See epic linking - need alternative non payload image available to provide relatively seamless migration

Also, the EP for this is approved and merged at https://github.com/openshift/enhancements/blob/master/enhancements/builds/remove-jenkins-payload.md

Estimate (xs, s, m, l, xl, xxl):

Questions:

• in addition to needing the CPaaS Image available first, have we confirmed the "deprecate first, then remove" requirements?  Per grooming we've assigned Rob Gormley the task https://issues.redhat.com/browse/JKNS-274 for tracking this down.  I will be reaching out to Ben Parees (he is currently on vacation) to confirm from an OCP staff engineer / architect perspective if his approval of https://github.com/openshift/enhancements/pull/841 is sufficient signoff from that end.

       PARTIAL ANSWER ^^:  confirmed with Ben Parees in https://coreos.slack.com/archives/C014MHHKUSF/p1646683621293839 that EP merging is currently sufficient OCP "technical leadership" approval.

Previous work

Customers

assuming none

Epic Goal

• Remove this UI from our stack that we cannot support.

Why is this important?

• Reduce support burden.
• Remove Bugzilla burden of addressing continuous CVEs found in this project.

Acceptance Criteria

• All Prometheus upstream UI links are removed
• Related documentation is updated
• Ports/routes etc configured to expose access to this UI are removed such that no configuration we provide enables access to this UI or its codepaths.
• There is no reason any CVEs found in this UI would ever require intervention by the Monitoring Team.

Dependencies (internal and external)

1. Make the Prometheus Targets information available in Console UI (https://issues.redhat.com/browse/MON-1079)

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Epic Goal

When users configure CMO to interact with systems outside of an OpenShift cluster, we want to provide an easy way to add the cluster ID to the data send.

Why is this important?

Technically this can be achieved today, by adding an identifying label to the remote_write configuration for a given cluster. The operator adding the remote_write integration needs to take care that the label is unique over the managed fleet of clusters. This however adds management complexity. Any given cluster already has a pseudo-unique datum, that can be used for this purpose.

• Starting in 4.9 we support the Prometheus remote_write feature to send metric data to a storage integration outside of the cluster similar to our own Telemetry service.
• In Telemetry we already use the cluster ID to distinguish the various clusters.
• For users of remote_write this could add an easy way to add such distinguishing information.

Scenarios

1. An organisation with multiple OpenShift clusters want to store their metric data centralized in a dedicated system and use remote_write in all their clusters to send this data. When querying their centralized storage, metadata (here a label) is needed to separate the data of the various clusters.
2. Service providers who manage multiple clusters for multiple customers via a centralized storage system need distinguishing metadata too. See https://issues.redhat.com/browse/OSD-6573 for example

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• Document how to use this feature

Dependencies (internal and external)

1. none

Previous Work (Optional):

1. none

Open questions::

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Implementation proposal:

Expose a flag in the CMO configuration, that is false by default (keeps backward compatibility) and when set to true will add the _id label to a remote_write configuration. More specifically it will be added to the top of a remote_write relabel_config list via the replace action. This will add the label as expect, but additionally a user could alter this label in a later relabel config to suit any specific requirements (say rename the label or add additional information to the value).
The location of this flag is the remote_write Spec, so this can be set for individual remote_write configurations.

Epic Goal

• Offer the option to double the scrape intervals for CMO controlled ServiceMonitors in single node deployments
• Alternatively automatically double the same scrape intervals if CMO detects an SNO setup

The potential target ServiceMonitors are:

• kubelet
• kube-state-metrics
• node-exporter
• etcd
• openshift-state-metrics

Why is this important?

• Reduce CPU usage in SNO setups
• Specifically doubling the scrape interval is important because:

1. we are confident that this will have the least chance to interfere with existing rules. We typically have rate queries over the last 2 minutes (no shorter time window). With 30 second scrape intervals (the current default) this gives us 4 samples in any 2 minute window. rate needs at least 2 samples to work, we want another 2 for failure tolerance. Doubling the scrape interval will still give us 2 samples in most 2 minute windows. If a scrape fails, a few rule evaluations might fail intermittently.
2. We expect a measureable reduction of CPU resources (see previous work)

Scenarios

1. RAN deployments (Telco Edge) are SNO deployments. In these setups a full CMO deployment is often not needed and the default setup consumes too many resources. OpenShift as a whole has only very limited CPU cycles available and too many cycles are spend on Monitoring

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Previous Work (Optional):

1. https://issues.redhat.com/browse/MON-1569

Open questions:

1. Whether doubling some scrape intervals reduces CPU usage to fit into the assigned budget

Non goals

• Allow arbitrarily long scrape intervals. This will interfere with alert and recoring rules
• Implement a global override to scrape intervals.

Problem:

This epic is mainly focused on the 4.10 Release QE activities

Goal:

1. Identify the scenarios for automation
2. Segregate the test Scenarios into smoke, Regression and other user stories
a. Update the https://docs.jboss.org/display/ODC/Automation+Status+Report
3. Align with layered operator teams for updating scripts
3. Work closely with dev team for epic automation
4. Create the automation scripts using cypress
5. Implement CI for nightly builds
6. Execute scripts on sprint basis

Why is it important?

To the track the QE progress at one place in 4.10 Release Confluence page

Use cases:

1. <case>

Acceptance criteria:

1. <criteria>

Dependencies (External/Internal):

Design Artifacts:

Exploration:

Note:

Goal:

This epic covers a number of customer requests(RFEs) as well as increases usability.

Why is it important?

Customer satisfaction as well as improved usability.

Acceptance Criteria

1. Allow user to re-arrange the resources which have ben added to nav by the user
2. Improved user experience (form based experience)
   1. Form based editing of Routes
   2. Form based creation and editing of Config Maps
   3. Form base creation of Deployments
3. Improved discovery
   1. Include Share my project on the Add page to increase discoverability
   2. NS Helm Chart Repo
      1. Add tile to Add page for discoverability
      2. Provide a form driven creation experience
      3. User should be able to switch back and forth from Form/YAML
      4. change the intro text to the below & have the link in the intro text bring up the full page form
         1. Browse for charts that help manage complex installations and upgrades. Cluster administrators can customize the content made available in the catalog. Alternatively, developers can try to configure their own custom Helm Chart repository.

Dependencies (External/Internal):

None

Exploration:

Miro board from Epic Exploration

Problem:

Currently we are only able to get limited telemetry from the Dev Sandbox, but not from any of our managed clusters or on prem clusters.

Goals:

1. Enable gathering segment telemetry whenever cluster telemetry is enabled on OSD clusters
2. Have our OSD clusters opt into telemetry by default
3. Work with PM & UX to identify additional metrics to capture in addition to what we have enabled currently on Sandbox.
4. Ability to get a single report from woopra across all of our Sandbox and OSD clusters.
5. Be able to generate a report including metrics of a single cluster or all clusters of a certain type ( sandbox, or OSD)

Why is it important?

In order to improve properly analyze usage and the user experience, we need to be able to gather as much data as possible.

Goal:
Enhance oc adm release new (and related verbs info, extract, mirror) with heterogeneous architecture support

Epic Goal

• Update image registry dependencies (Kubernetes and OpenShift) to the latest versions.

Why is this important?

• New versions usually bring improvements that are needed by the registry and help with getting updates for z-stream.

Scenarios

1. As an OpenShift engineer, I want my components to use the versions of dependencies, so that they get fixes for known issues and can be easily updated in z-stream.

Acceptance Criteria

• CI - MUST be running successfully with tests automated

Dependencies (internal and external)

1. Kubernetes 1.24

Previous Work (Optional):

1. IR-210

Done Checklist

• CI - CI is running, tests are automated and merged.
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• QE - Test plans in Polarion: <link or reference to Polarion>

Epic Goal

• Provide a dedicated dashboard for NVIDIA GPU usage visualization in the OpenShift Console.

Why is this important?

• Customers that use GPUs in their clusters usually have the GPU workloads as the main purpose of their cluster. As such, it makes much more sense to have the details about the usage they are doing of GPGPU resources AND CPU/RAM rather than just CPU/RAM

Scenarios

1. As an admin of a cluster dedicated to data science, I want to quickly find out how much of my very costly resources are currently in use and if things are getting queued due to lack of resources

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.

Dependencies (internal and external)

1. The NVIDIA GPU Operator must export to prometheus the relevant data

Open questions::

1. Will NVIDIA agree to these extra data exports in their GPU Operator?

I asked Zvonko Kaiser and he seemed open to it. I need to confirm with Shiva Merla

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Description of problem:
Users on a fully-disconnected cluster could not see Devfiles in the developer catalog or import a Devfiles. That's fine.

But the API calls /api/devfile/samples/ and /api/devfile/ takes 30 seconds until they fail with a 504 Gateway timeout error.

If possible they should fail immediately.

Version-Release number of selected component (if applicable):
This might happen since 4.8

Tested this yet only on 4.12.0-0.nightly-2022-09-07-112008

How reproducible:
Always

Steps to Reproduce:

1. Start a disconnected cluster with a proxy
2. Open the browser network inspector and filter for /api/devfile
3. Switch to Developer perspective
   1. Navigate to Add > Developer Catalog (All Services) > Devfiles
   2. Or Add > Import from Git > and enter https://github.com/devfile-samples/devfile-sample-go-basic.git

Actual results:

• Network call fails after 30 seconds
• Import doesn't work

Expected results:

• Network calls should fail immediately
• We doesn't expect that the import will work

Additional info:
The console Pod log contains this error:

E0909 10:28:18.448680 1 devfile-handler.go:74] Failed to parse devfile: failed to populateAndParseDevfile: Get "https://registry.devfile.io/devfiles/go": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

• https://github.com/openshift/console/pull/12186

This fix contains the following changes coming from updated version of kubernetes up to v1.24.10:

Changelog:
v1.24.11: https://github.com/kubernetes/kubernetes/blob/release-1.24/CHANGELOG/CHANGELOG-1.24.md#changelog-since-v12410
v1.24.10: https://github.com/kubernetes/kubernetes/blob/release-1.24/CHANGELOG/CHANGELOG-1.24.md#changelog-since-v1249
v1.24.9: https://github.com/kubernetes/kubernetes/blob/release-1.24/CHANGELOG/CHANGELOG-1.24.md#changelog-since-v1248
v1.24.8: https://github.com/kubernetes/kubernetes/blob/release-1.24/CHANGELOG/CHANGELOG-1.24.md#changelog-since-v1247
v1.24.7: https://github.com/kubernetes/kubernetes/blob/release-1.24/CHANGELOG/CHANGELOG-1.24.md#changelog-since-v1246

• https://github.com/openshift/kubernetes/pull/1500

Description of problem:

Manual backport of 
* https://github.com/openshift/cluster-dns-operator/pull/336
* https://github.com/openshift/cluster-dns-operator/pull/339

Version-Release number of selected component (if applicable):

4.11

• https://github.com/openshift/cluster-dns-operator/pull/347

Description of problem:

Disconnected IPI OCP 4.10.22 cluster install on baremetal fails when hostname of master nodes does not include "master"    

Version-Release number of selected component (if applicable): 4.10.22

How reproducible:  Perform disconnected IPI install of OCP 4.10.22 on bare metal with master nodes that do not contain the text "master"

Steps to Reproduce:

Perform disconnected IPI install of OCP 4.10.22 on bare metal with master nodes that do not contain the text "master"

Actual results: master nodes do come up.

Expected results: master nodes should come up despite that the text "master" is not in their hostname.

Additional info:

Disconnected IPI OCP 4.10.22 cluster install on baremetal fails when hostname of master nodes does not include "master"    

The code for the cluster-baremetal-operator at the following link: 

https://github.com/openshift/cluster-baremetal-operator/blob/49d7b249c5dcef8228f206eff4530a25f03b201f/controllers/provisioning_controller.go#L441

The following condition is concerning:

if strings.Contains(bmh.Name, "master") && len(bmh.Spec.BootMACAddress) > 0

The packages reveal that bmh.Name references the name inside the metadata of the BMH object. 

Should a customer have masters with names that do not include the text "master", the above condition can never become true, and so, the following slice is never created :

macs = append(macs, bmh.Spec.BootMACAddress)

• https://github.com/openshift/cluster-baremetal-operator/pull/307
• https://github.com/openshift/cluster-baremetal-operator/pull/283

This is a clone of issue OCPBUGS-7960. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-7780. The following is the description of the original issue:
—
Description of problem:

4.9 and 4.10 oc calls to oc adm upgrade channel ... for 4.11+ clusters would clear spec.capabilities. Not all that many clusters try to restrict capabilities, but folks will need to bump their channel for at least every other minor (if their using EUS channels), and while we recommend folks use an oc from the 4.y they're heading towards, we don't have anything in place to enforce that.

Version-Release number of selected component (if applicable):

4.9 and 4.10 oc are exposed vs. the new-in-4.11 spec.capabilities. Newer oc could theoretically be exposed vs. any new ClusterVersion spec capabilities.

How reproducible:

100%

Steps to Reproduce:

1. Install a 4.11+ cluster with None capabilities.
2. Set the channel with a 4.10.51 oc, like oc adm upgrade channel fast-4.11.
3. Check the capabilities with oc get -o json clusterversion version | jq -c .spec.capabilities.

Actual results:

null

Expected results:

{"baselineCapabilitySet":"None"}

• https://github.com/openshift/oc/pull/1360

This is a clone of issue OCPBUGS-10943. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-10661. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-10591. The following is the description of the original issue:
—
Description of problem:

Starting with 4.12.0-0.nightly-2023-03-13-172313, the machine API operator began receiving an invalid version tag either due to a missing or invalid VERSION_OVERRIDE(https://github.com/openshift/machine-api-operator/blob/release-4.12/hack/go-build.sh#L17-L20) value being passed tot he build.

This is resulting in all jobs invoked by the 4.12 nightlies failing to install.

Version-Release number of selected component (if applicable):

4.12.0-0.nightly-2023-03-13-172313 and later

How reproducible:

consistently in 4.12 nightlies only(ci builds do not seem to be impacted).

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

Example of failure https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.12-e2e-aws-csi/1635331349046890496/artifacts/e2e-aws-csi/gather-extra/artifacts/pods/openshift-machine-api_machine-api-operator-866d7647bd-6lhl4_machine-api-operator.log

• https://github.com/openshift/machine-api-operator/pull/1133

Tracker issue for bootimage bump in 4.11. This issue should block issues which need a bootimage bump to fix.

The previous bump was OCPBUGS-3362.

• https://github.com/openshift/installer/pull/6875

This is a clone of issue OCPBUGS-8044. The following is the description of the original issue:
—

Description of problem:

rhbz#2089199, backported to 4.11.5, shifted the etcd Grafana dashboard from the monitoring operator to the etcd operator. During the shift, the ConfigMap was renamed from grafana-dashboard-etcd to etcd-dashboard. However, we did not include logic for garbage-collecting the obsolete dasboard, so clusters that update from 4.11.1 and similar into 4.11.>=5 or 4.12+ currently end up with both the obsolete and new ConfigMaps. We should grow code to remove the obsolete ConfigMap.

Version-Release number of selected component (if applicable):

4.11.>=5 and 4.12+ are currently exposed.

How reproducible:

100%

Steps to Reproduce:

1. Install 4.11.1.
2. Update to a release that defines the etcd-dashboard ConfigMap.
3. Check for etcd dashboards with oc -n openshift-config-managed get configmaps | grep etcd.

Actual results:

Both etcd-dashboard and grafana-dashboard-etcd exist:

$ oc -n openshift-config-managed get configmaps | grep etcd
etcd-dashboard                                        1      196d
grafana-dashboard-etcd                                1      2y282d

Another example is 4.11.1 to 4.11.5 CI:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/1570415394001260544/artifacts/e2e-aws-upgrade/configmaps.json | jq -r '.items[].metadata | select(.namespace == "openshift-config-managed" and (.name | contains("etcd"))) | .name'
etcd-dashboard
grafana-dashboard-etcd

Expected results:

Only etcd-dashboard still exists.

Additional info:

A new manifest for the outgoing ConfigMap that sets the release.openshift.io/delete: "true" annotation would ask the cluster-version operator to reap the obsolete ConfigMap.

• https://github.com/openshift/cluster-etcd-operator/pull/1022

Description of problem:

IPI installation failed against 4.11 nightly build on Azure Stack Hub WWT.

workers machines failed to be provisioned as using the wrong image which does not exist.
$ oc get machine -n openshift-machine-api
NAME                                  PHASE     TYPE              REGION   ZONE   AGE
jima411az-xvwvg-master-0              Running   Standard_DS4_v2   mtcazs          3h10m
jima411az-xvwvg-master-1              Running   Standard_DS4_v2   mtcazs          3h10m
jima411az-xvwvg-master-2              Running   Standard_DS4_v2   mtcazs          3h10m
jima411az-xvwvg-worker-mtcazs-4k6hs   Failed                                      3h3m
jima411az-xvwvg-worker-mtcazs-fc6s5   Failed                                      3h3m
jima411az-xvwvg-worker-mtcazs-sclgf   Failed                                      3h3m

$ oc describe machine jima411az-xvwvg-worker-mtcazs-4k6hs -n openshift-machine-api
...
Events:
  Type     Reason        Age   From              Message
  ----     ------        ----  ----              -------
  Warning  FailedCreate  3h4m  azure-controller  InvalidConfiguration: failed to reconcile machine "jima411az-xvwvg-worker-mtcazs-4k6hs": failed to create vm jima411az-xvwvg-worker-mtcazs-4k6hs: failure sending request for machine jima411az-xvwvg-worker-mtcazs-4k6hs: cannot create vm: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=404 -- Original Error: Code="NotFound" Message="The Image '/subscriptions/de7e09c3-b59a-4c7d-9c77-439c11b92879/resourceGroups/jima411az-xvwvg-rg/providers/Microsoft.Compute/images/jima411az-xvwvg-gen2' cannot be found in 'mtcazs' region."

The image created on ASH is `/subscriptions/de7e09c3-b59a-4c7d-9c77-439c11b92879/resourceGroups/jima411az-xvwvg-rg/providers/Microsoft.Compute/images/jima411az-xvwvg`

# az resource list -g jima411az-xvwvg-rg --query "[?type=='Microsoft.Compute/images']"
[
  {
    "changedTime": "2023-08-03T01:08:03.369233+00:00",
    "createdTime": "2023-08-03T00:57:34.228352+00:00",
    "id": "/subscriptions/de7e09c3-b59a-4c7d-9c77-439c11b92879/resourceGroups/jima411az-xvwvg-rg/providers/Microsoft.Compute/images/jima411az-xvwvg",
    "identity": null,
    "kind": null,
    "location": "mtcazs",
    "managedBy": null,
    "name": "jima411az-xvwvg",
    "plan": null,
    "properties": null,
    "provisioningState": "Succeeded",
    "resourceGroup": "jima411az-xvwvg-rg",
    "sku": null,
    "tags": {},
    "type": "Microsoft.Compute/images"
  }
]

Version-Release number of selected component (if applicable):

4.11.0-0.nightly-2023-07-29-013834

How reproducible:

Always on 4.11

Steps to Reproduce:

1. Deploy IPI with 4.11 on ASH WWT
2. 
3.

Actual results:

worker nodes failed to be provisioned

Expected results:

Installation is successful. 

Additional info:

Detect issue on 4.11, IPI installation works well on 4.12+.

• https://github.com/openshift/installer/pull/7393

This is a clone of issue OCPBUGS-1417. The following is the description of the original issue:
—
Description of problem:

Egress IP is not being assigned to primary interface of node as per hostsubnet definition. The issue being observed at an Openshift cluster hosted on Disconnected AWS environment.  Following steps were performed at AWS end:

- Disconnected VPC was created and installation of Openshift was done as per documentation.
- Elastic IP could not be used as it is a disconnected environment. Customer identified a free IP from same subnet as the node and modified interface of the node to add a secondary IP.

It seems cloud.network.openshift.io/egress-ipconfig annotation is need on the node to attach IP to primary interface but its missing. From SDN POD log on the same node I  could see its complaining about 'an incomplete annotation "cloud.network.openshift.io/egress-ipconfig"'. Will share more details over comments.

Version-Release number of selected component (if applicable):

Openshift 4.10.28

How reproducible:

Always

Steps to Reproduce:

1. Create a disconnected environment on AWS
2. find a free IP from subnet where a worker node is hosted and add that as secondary  IP to NIC of that node.
3. Configure hostsubnet and netnamespace on Openshift cluster

Actual results:

- Eress IP is not being attached to primary interface of node for which hostsubnet has been configured

Expected results:

- Egress IP should get configured without any issue.

Additional info:

• https://github.com/openshift/cloud-network-config-controller/pull/84

This is a clone of issue OCPBUGS-6766. The following is the description of the original issue:
—
This is a clone of https://bugzilla.redhat.com/show_bug.cgi?id=2083087 (OCPBUGSM-44070) to backport this issue.

Description of problem:
"Delete dependent objects of this resource" is a bit of confusing for some users because when creating the Application in Dev console not only the deployment but also IS, route, svc, secret objects will be created as well. When deleting the Application (in fact it is deployment), there is an option called "Delete dependent objects of this resource" and some users might think this means the IS, route, svc and any other objects which are created alongside with the deployment will be deleted as well

Version-Release number of selected component (if applicable):
4.8

How reproducible:
Always

Steps to Reproduce:
1. Create Application in Dev console
2. Delete the deployment
3. Check "Delete dependent objects of this resource"

Actual results:
Only deployment will be deleted and IS, svc, route will not be deleted

Expected results:
We either change the description of this option, or we really delete IS, svc, route and any other objects created under this Application.

Additional info:

• https://github.com/openshift/console/pull/12532

• https://github.com/openshift/multus-cni/pull/145

This is a clone of issue OCPBUGS-3235. The following is the description of the original issue:
—

Description of problem:

Frequently we see the loading state of the topology view, even when there aren't many resources in the project.

Including an example

Prerequisites (if any, like setup, operators/versions):

Steps to Reproduce

1. load topology
2. if it loads successfully, keep trying  until it fails to load

Actual results:

topology will sometimes hang with the loading indicator showing indefinitely

Expected results:

topology should load consistently without fail

Reproducibility (Always/Intermittent/Only Once):

intermittent

Build Details:

4.9

Additional info:

• https://github.com/openshift/console/pull/12311

• https://github.com/openshift/installer/pull/6743

• https://github.com/openshift/gcp-pd-csi-driver/pull/34

This bug is a backport clone of [Bugzilla Bug 2094174](https://bugzilla.redhat.com/show_bug.cgi?id=2094174). The following is the description of the original bug:
—
Created attachment 1887340
CVO log file

Description of problem:
Clearing upgrade after signature verification fails, ReleaseAccepted=False keeps complaining about the update cannot be verified blah blah.

1. oc get clusterversion/version -ojson | jq -r '.spec, .status.conditions'
   {
   "channel": "stable-4.11",
   "clusterID": "d740b8f3-bb49-40cf-86e8-5df4a755111a"
   }
   [
   { "lastTransitionTime": "2022-06-07T01:31:43Z", "message": "Unable to retrieve available updates: currently reconciling cluster version 4.11.0-0.nightly-2022-06-06-025509 not found in the \"stable-4.11\" channel", "reason": "VersionNotFound", "status": "False", "type": "RetrievedUpdates" }

   ,
   

   { "lastTransitionTime": "2022-06-07T01:31:43Z", "message": "Capabilities match configured spec", "reason": "AsExpected", "status": "False", "type": "ImplicitlyEnabledCapabilities" }

   ,
   

   { "lastTransitionTime": "2022-06-07T02:44:54Z", "message": "Retrieving payload failed version=\"\" image=\"registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100\" failure=The update cannot be verified: unable to verify sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 against keyrings: verifier-public-key-redhat", "reason": "RetrievePayload", "status": "False", "type": "ReleaseAccepted" }

   ,
   

   { "lastTransitionTime": "2022-06-07T01:56:17Z", "message": "Done applying 4.11.0-0.nightly-2022-06-06-025509", "status": "True", "type": "Available" }

   ,
   

   { "lastTransitionTime": "2022-06-07T01:55:47Z", "status": "False", "type": "Failing" }

   ,

   { "lastTransitionTime": "2022-06-07T01:56:17Z", "message": "Cluster version is 4.11.0-0.nightly-2022-06-06-025509", "status": "False", "type": "Progressing" }

   ]

Version-Release number of the following components:
4.11.0-0.nightly-2022-06-06-025509

How reproducible:
1/1

Steps to Reproduce:
1. Upgrade to a fake release

1. oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 --allow-explicit-upgrade
   warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway
   Requesting update to release image registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100

2. Check ReleaseAccepted=False due to target image signature verification failure

1. oc adm upgrade
   Cluster version is 4.11.0-0.nightly-2022-06-04-014713

ReleaseAccepted=False

Reason: RetrievePayload
Message: Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100" failure=The update cannot be verified: unable to verify sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 against keyrings: verifier-public-key-redhat

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.11
warning: Cannot display available updates:
Reason: VersionNotFound
Message: Unable to retrieve available updates: currently reconciling cluster version 4.11.0-0.nightly-2022-06-04-014713 not found in the "stable-4.11" channel

3. Clear the upgrade

1. oc adm upgrade --clear
   Cancelled requested upgrade to registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100

4. Check oc adm upgrade info

1. oc adm upgrade
   Cluster version is 4.11.0-0.nightly-2022-06-04-014713

ReleaseAccepted=False

Reason: RetrievePayload
Message: Retrieving payload failed version="" image="registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100" failure=The update cannot be verified: unable to verify sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f100 against keyrings: verifier-public-key-redhat

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.11
warning: Cannot display available updates:
Reason: VersionNotFound
Message: Unable to retrieve available updates: currently reconciling cluster version 4.11.0-0.nightly-2022-06-04-014713 not found in the "stable-4.11" channel

Actual results:
After upgrade is cleared, cv condition ReleaseAccepted keeps to false with message The update cannot be verified

Expected results:
After upgrade is cleared, cv condition ReleaseAccepted should stop complaining about the target image

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

• https://github.com/openshift/cluster-version-operator/pull/935

This is a clone of issue OCPBUGS-4250. The following is the description of the original issue:
—
Description of problem:

Added a script to collect PodNetworkConnectivityChecks to able to view the overall status of the pod network connectivity.

Current must-gather collects the contents of `openshift-network-diagnostics` but does not collect the PodNetworkConnectivityCheck.

Version-Release number of selected component (if applicable):

4.12, 4.11, 4.10

• https://github.com/openshift/must-gather/pull/338

Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/driver-toolkit/pull/77

Description of problem:

Upgrade to 4.10 is stuck looping in syncEgressFirewall

We see transacting operations with context deadline exceeded.

It looks to be trying to process 2.8 million records is one go.

2023-02-21T19:55:06.514097513Z I0221 19:55:06.435220 1 client.go:781] "msg"="transacting operations" "database"="OVN_Northbound" "operations"="[{Op:mutate Table:Logical_Switch Row:map[] Rows:[] Columns:[] Mutations:[{Column:acls Mutator:delete Value:{GoSet:[{GoUUID:6a3ad543-a77d-4700-83b8-5ccae6b2d067} ID:1c5297ff-8588-467a-93f4-22f22d609563} {GoUUID:f6288ed3-3928-45a8-ae57-40ed94cfa249} {GoUUID:04bf90c2-fde1-4a10-baaa-6a3f1d8e2931} {GoUUID:c6609536-857c-48ae-9125-9505753180 a8} {GoUUID:c79b4398-d7cc-4dcf-8c1d-11484f318324} {GoUUID:4323ac2c-033e-43c3-885b-e951cd7a4159} {GoUUID:7b316a80-076f-4266-b7d2-bd69b1d4b874} {GoUUID:57dfecb2-2f94-4cd8-a277-8 b28205e1048} {GoUUID:2c039f15-ff11-4ceb-aa82-bcbe82fc86d1} {GoUUID:063c4121-73c3-4d53-a89d-1063e775146b} {GoUUID:25c788e3-6146-4571-98bf-61010100a22a} {GoUUID:3d3c150f-1296-4d 91-b334-506f28bff4bd}]}}] Timeout:<nil> Where:[where column _uuid == {ba9652de-5aae-4a74-a512-29f775e38c19}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:}]: context deadline exceeded 

2023-02-21T19:55:18.739739417Z E0221 19:55:18.643127       1 master.go:1369] Failed (will retry) in syncing syncEgressFirewall: failed to remove reject acl from node logical switches: error while removing ACLS: [6a3ad543-a77d-4700-83b8-5ccae6b2d067 8e004991-0382-455f-9901-33ef724acbc2 

Everything is built into one operation via:
https://github.com/openshift/ovn-kubernetes/blob/release-4.10/go-controller/pkg/libovsdbops/switch.go#L243

TrandactAndCheck is being called with a 10s timeout and this operation never completes. 
 

Version-Release number of selected component (if applicable):

4.10.50

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Upgrade completes

Additional info:

• https://github.com/openshift/ovn-kubernetes/pull/1629

Description of problem:

The ovn-kubernetes ovnkube-master containers are continuously crashlooping since we updated to 4.11.0-0.okd-2022-10-15-073651.

Log Excerpt:

] [] []  [{kubectl-client-side-apply Update networking.k8s.io/v1 2022-09-12 12:25:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:ingress":{},"f:policyTypes":{}}} }]},Spec:NetworkPolicySpec{PodSelector:{map[] []},Ingress:[]NetworkPolicyIngressRule{NetworkPolicyIngressRule{Ports:[]NetworkPolicyPort{},From:[]NetworkPolicyPeer{NetworkPolicyPeer{PodSelector:&v1.LabelSelector{MatchLabels:map[string]string{access: true,},MatchExpressions:[]LabelSelectorRequirement{},},NamespaceSelector:nil,IPBlock:nil,},},},},Egress:[]NetworkPolicyEgressRule{},PolicyTypes:[Ingress],},} &NetworkPolicy{ObjectMeta:{allow-from-openshift-ingress  compsci-gradcentral  a405f843-c250-40d7-8dd4-a759f764f091 217304038 1 2022-09-22 14:36:38 +0000 UTC <nil> <nil> map[] map[] [] []  [{openshift-apiserver Update networking.k8s.io/v1 2022-09-22 14:36:38 +0000 UTC FieldsV1 {"f:spec":{"f:ingress":{},"f:policyTypes":{}}} }]},Spec:NetworkPolicySpec{PodSelector:{map[] []},Ingress:[]NetworkPolicyIngressRule{NetworkPolicyIngressRule{Ports:[]NetworkPolicyPort{},From:[]NetworkPolicyPeer{NetworkPolicyPeer{PodSelector:nil,NamespaceSelector:&v1.LabelSelector{MatchLabels:map[string]string{policy-group.network.openshift.io/ingress: ,},MatchExpressions:[]LabelSelectorRequirement{},},IPBlock:nil,},},},},Egress:[]NetworkPolicyEgressRule{},PolicyTypes:[Ingress],},}]: cannot clean up egress default deny ACL name: error in transact with ops [{Op:mutate Table:Port_Group Row:map[] Rows:[] Columns:[] Mutations:[{Column:acls Mutator:delete Value:{GoSet:[{GoUUID:60cb946a-46e9-4623-9ba4-3cb35f018ed6}]}}] Timeout:<nil> Where:[where column _uuid == {ccdd01bf-3009-42fb-9672-e1df38190cd7}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:mutate Table:Port_Group Row:map[] Rows:[] Columns:[] Mutations:[{Column:acls Mutator:delete Value:{GoSet:[{GoUUID:60cb946a-46e9-4623-9ba4-3cb35f018ed6}]}}] Timeout:<nil> Where:[where column _uuid == {10bbf229-8c1b-4c62-b36e-4ba0097722db}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:delete Table:ACL Row:map[] Rows:[] Columns:[] Mutations:[] Timeout:<nil> Where:[where column _uuid == {7b55ba0c-150f-4a63-9601-cfde25f29408}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:delete Table:ACL Row:map[] Rows:[] Columns:[] Mutations:[] Timeout:<nil> Where:[where column _uuid == {60cb946a-46e9-4623-9ba4-3cb35f018ed6}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:}] results [{Count:1 Error: Details: UUID:{GoUUID:} Rows:[]} {Count:1 Error: Details: UUID:{GoUUID:} Rows:[]} {Count:1 Error: Details: UUID:{GoUUID:} Rows:[]} {Count:1 Error: Details: UUID:{GoUUID:} Rows:[]} {Count:0 Error:referential integrity violation Details:cannot delete ACL row 7b55ba0c-150f-4a63-9601-cfde25f29408 because of 1 remaining reference(s) UUID:{GoUUID:} Rows:[]}] and errors []: referential integrity violation: cannot delete ACL row 7b55ba0c-150f-4a63-9601-cfde25f29408 because of 1 remaining reference(s)

Additional info:

https://github.com/okd-project/okd/issues/1372

Issue persisted through update to 4.11.0-0.okd-2022-10-28-153352

must-gather: https://nbc9-snips.cloud.duke.edu/snips/must-gather.local.2859117512952590880.zip

• https://github.com/openshift/ovn-kubernetes/pull/1444

Description of problem:

After setting the appsDomain flag in the OpenShift Ingress configuration, all Routes that are generated are based off this domain.

This includes OpenShift-core Routes, which can result in failing health-checks, such as seen with the OpenShift Web Console.

According to the documentation[0], the appsDomain value should only be used by user-created Routes and is not intended to be seen when Openshift components are reconciled.

Version-Release number of selected component (if applicable):

OpenShift 4.12

How reproducible:

Everytime

Steps to Reproduce:

1. Provision a new cluster
2. Configure appsDomain value
3. Delete the OpenShift Console Route:
     oc delete routes -n openshift-console console
4. The Route is automatically reconciled

Actual results:

The re-created Route uses the appsDomain value, resulting in failed health-checks throughout the cluster.

Expected results:

The Console and all other OpenShift-core components should use the default domain.

Additional info:

Reviewing the appsDomain code, if the appsDomain value is configured this takes precedent for all Routes. [3][1][2]


Resources:
[0] https://docs.openshift.com/container-platform/4.9/networking/ingress-operator.html#nw-ingress-configuring-application-domain_configuring-ingress
[1] Sets Route Subdomain to appsdomain if it's present, otherwise uses the default domain
    https://github.com/openshift/cluster-openshift-apiserver-operator/blob/d2182396d647839f61b027602578ac31c9437e7d/pkg/operator/configobservation/ingresses/observe_ingresses.go#L16-L60
[2] Sets up the Route generator
    https://github.com/openshift/openshift-apiserver/blob/9c381fc873b35c074f7c0c485893e038828c1405/pkg/cmd/openshift-apiserver/openshiftapiserver/config.go#L217-L218
[3] Where Route.spec.host is generated:
    https://github.com/openshift/openshift-apiserver/blob/9c381fc873b35c074f7c0c485893e038828c1405/vendor/github.com/openshift/library-go/pkg/route/hostassignment/plugin.go#L22-L47

• https://github.com/openshift/console-operator/pull/753

• https://github.com/openshift/ovn-kubernetes/pull/1734

Description of problem:

Cluster running 4.10.52 had three aws-ebs-csi-driver-node pods begin to consume multiple GB of memory, causing heavy node memory pressure as the pods have no memory limit. 

All other aws-ebs-csi-driver-node pods were still in the 50-70MB range:

NAME                                            CPU(cores)   MEMORY(bytes)   
aws-ebs-csi-driver-controller-59867579b-d6s2q   0m           397Mi           
aws-ebs-csi-driver-controller-59867579b-t4wgq   0m           276Mi           
aws-ebs-csi-driver-node-4rmvk                   0m           53Mi            
aws-ebs-csi-driver-node-5799f                   0m           50Mi            
aws-ebs-csi-driver-node-6dpvg                   0m           59Mi            
aws-ebs-csi-driver-node-6ldzk                   0m           65Mi            
aws-ebs-csi-driver-node-6mbk5                   0m           54Mi            
aws-ebs-csi-driver-node-bkvsr                   0m           50Mi            
aws-ebs-csi-driver-node-c2fb2                   0m           62Mi            
aws-ebs-csi-driver-node-f422m                   0m           61Mi            
aws-ebs-csi-driver-node-lwzbb                   6m           1940Mi          
aws-ebs-csi-driver-node-mjznt                   0m           53Mi            
aws-ebs-csi-driver-node-pczsj                   0m           62Mi            
aws-ebs-csi-driver-node-pmskn                   0m           3493Mi          
aws-ebs-csi-driver-node-qft8w                   0m           68Mi            
aws-ebs-csi-driver-node-v5bpx                   11m          2076Mi          
aws-ebs-csi-driver-node-vn8km                   0m           84Mi            
aws-ebs-csi-driver-node-ws6hx                   0m           73Mi            
aws-ebs-csi-driver-node-xsk7k                   0m           59Mi            
aws-ebs-csi-driver-node-xzwlh                   0m           55Mi            
aws-ebs-csi-driver-operator-8c5ffb6d4-fk6zk     5m           88Mi            

Deleting the pods caused them to recreate, with normal memory consumption levels.

Version-Release number of selected component (if applicable):

4.10.52

How reproducible:

Unknown

• https://github.com/openshift/csi-livenessprobe/pull/42

Description of problem:

[OVN][OSP] After reboot egress node, egress IP cannot be applied anymore.

Version-Release number of selected component (if applicable):

4.12.0-0.nightly-2022-11-07-181244

How reproducible:

Frequently happened in automation. But didn't reproduce it in manual.

Steps to Reproduce:

1. Label one node as egress node

2.
Config one egressIP object
STEP: Check  one EgressIP assigned in the object.

Nov  8 15:28:23.591: INFO: egressIPStatus: [{"egressIP":"192.168.54.72","node":"huirwang-1108c-pg2mt-worker-0-2fn6q"}]

3.
Reboot the node, wait for the node ready.

Actual results:

EgressIP cannot be applied anymore. Waited more than 1 hour.
 oc get egressip
NAME             EGRESSIPS       ASSIGNED NODE   ASSIGNED EGRESSIPS
egressip-47031   192.168.54.72    

Expected results:

The egressIP should be applied correctly.

Additional info:

Some logs
E1108 07:29:41.849149       1 egressip.go:1635] No assignable nodes found for EgressIP: egressip-47031 and requested IPs: [192.168.54.72]
I1108 07:29:41.849288       1 event.go:285] Event(v1.ObjectReference{Kind:"EgressIP", Namespace:"", Name:"egressip-47031", UID:"", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'NoMatchingNodeFound' no assignable nodes for EgressIP: egressip-47031, please tag at least one node with label: k8s.ovn.org/egress-assignable


W1108 07:33:37.401149       1 egressip_healthcheck.go:162] Could not connect to huirwang-1108c-pg2mt-worker-0-2fn6q (10.131.0.2:9107): context deadline exceeded
I1108 07:33:37.401348       1 master.go:1364] Adding or Updating Node "huirwang-1108c-pg2mt-worker-0-2fn6q"
I1108 07:33:37.437465       1 egressip_healthcheck.go:168] Connected to huirwang-1108c-pg2mt-worker-0-2fn6q (10.131.0.2:9107)

After this log, seems like no logs related to "192.168.54.72" happened.

• https://github.com/openshift/ovn-kubernetes/pull/1445

Description of problem:

Looks like a regression was introduced and the Windows networking tests started to fail accross all 4.11 CI jobs for both Linux-to-Windows and Windows-to-Windows pods communication. 

https://github.com/openshift/ovn-kubernetes/pull/1415 merged without passing Windows networking tests.

https://github.com/openshift/windows-machine-config-operator/pull/1359  shows networking tests failing accross all CI jobs.

Version-Release number of selected component (if applicable):

4.11

How reproducible:

Always 

Steps to Reproduce:

1. Trigger a job retest in any of the above mentioned PRs
 

Actual results:

Windows networking test fail

Expected results:

Windows networking test pass

Additional info:

Direct link to a release-4.11 aws-e2e-operator CI job showing the failed networking test. 

• https://github.com/openshift/ovn-kubernetes/pull/1459

This is a clone of issue OCPBUGS-5016. The following is the description of the original issue:
—
Description of problem:

When editing any pipeline in the openshift console, the correct content cannot be obtained (the obtained information is the initial information).

Version-Release number of selected component (if applicable):

How reproducible:

100%

Steps to Reproduce:

Developer -> Pipeline -> select pipeline -> Details -> Actions -> Edit Pipeline -> YAML view -> Cancel ->  Actions -> Edit Pipeline -> YAML view 

Actual results:

displayed content is incorrect.

Expected results:

Get the content of the current pipeline, not the "pipeline create" content.

Additional info:

If cancel or save in the "Pipeline Builder" interface after "Edit Pipeline", can get the expected content.
～
Developer -> Pipeline -> select pipeline -> Details -> Actions -> Edit Pipeline -> Pipeline builder -> Cancel ->  Actions -> Edit Pipeline -> YAML view :Display resource content normally
～

• https://github.com/openshift/console/pull/12540

This is a clone of issue OCPBUGS-884. The following is the description of the original issue:
—
Description of problem:

Since the decomissioning of the psi cluster, and subsequent move of the rhcos release browser, product builds machine-os-images builds have been failing. See e.g. https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=47565717

Version-Release number of selected component (if applicable):

4.12, 4.11, 4.10.

How reproducible:

Have ART build the image

Steps to Reproduce:

1. Have ART build the image

Actual results:

Build failure

Expected results:

Build succesful

Additional info:

• https://github.com/openshift/machine-os-images/pull/22

Description of problem:

Sometimes we see VMs fail to power on when the land on a host that does not have enough resources. The current power on does not retry or leverage DRS to power on the node on a suitable host.

https://github.com/vmware/govmomi/issues/1026

Our code is still making calls to PowerOnVM_Task which, according to the vsphere docs, is deprecated and we should use PowerOnMultiVM_Task instead.

PowerOnVM_Task does not return a DRS ClusterRecommendation, no vmotion nor host power operations will be done as part of a DRS-facilitated power on. To have DRS consider such operations use PowerOnMultiVM_Task.

https://vdc-download.vmware.com/vmwb-repository/dcr-public/b50dcbbf-051d-4204-a3e7-e1b618c1e384/538cf2ec-b34f-4bae-a332-3820ef9e7773/vim.VirtualMachine.html#powerOn:
As of vSphere API 5.1, use of this method with vCenter Server is deprecated; use PowerOnMultiVM_Task instead.

Version-Release number of selected component (if applicable):
4.8.x

How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:
Sometimes powers on fails requiring manual intervention.

Expected results:
PowerOn should use DRS to ensure it's always successful.

Additional info:

• https://github.com/openshift/machine-api-operator/pull/1064

• https://github.com/openshift/sdn/pull/544

This is a clone of issue OCPBUGS-4805. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-4101. The following is the description of the original issue:
—
Description of problem:

We experienced two separate upgrade failures relating to the introduction of the SYSTEM_RESERVED_ES node sizing parameter, causing kubelet to stop running.

One cluster (clusterA) upgraded from 4.11.14 to 4.11.17. It experienced an issue whereby 
   /etc/node-sizing.env 
on its master nodes contained an empty SYSTEM_RESERVED_ES value:

---
cat /etc/node-sizing.env 
SYSTEM_RESERVED_MEMORY=5.36Gi
SYSTEM_RESERVED_CPU=0.11
SYSTEM_RESERVED_ES=
---

causing the kubelet to not start up. To restore service, this file was manually updated to set a value (1Gi), and kubelet was restarted.

We are uncertain what conditions led to this occuring on the clusterA master nodes as part of the upgrade.

A second cluster (clusterB) upgraded from 4.11.16 to 4.11.17. It experienced an issue whereby worker nodes were impacted by a similar problem, however this was because a custom node-sizing-enabled.env MachineConfig which did not set SYSTEM_RESERVED_ES

This caused existing worker nodes to go into a NotReady state after the ugprade, and additionally new nodes did not join the cluster as their kubelet would become impacted. 

For clusterB the conditions are more well-known of why the value is empty.

However, for both clusters, if SYSTEM_RESERVED_ES ends up as empty on a node it can cause the kubelet to not start. 

We have some asks as a result:
- Can MCO be made to recover from this situation if it occurs, perhaps  through application of a safe default if none exists, such that kubelet would start correctly?
- Can there possibly be alerting that could indicate and draw attention to the misconfiguration?

Version-Release number of selected component (if applicable):

4.11.17

How reproducible:

Have not been able to reproduce it on a fresh cluster upgrading from 4.11.16 to 4.11.17

Expected results:

If SYSTEM_RESERVED_ES is empty in /etc/node-sizing*env then a default should be applied and/or kubelet able to continue running.

Additional info:

• https://github.com/openshift/machine-config-operator/pull/3459

This is a clone of issue OCPBUGS-13739. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-13692. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-13549. The following is the description of the original issue:
—
Description of problem:

Incorrect AWS ARN [1] is used for GovCloud and AWS China regions, which will cause the command `ccoctl aws create-all` to fail:

Failed to create Identity provider: failed to apply public access policy to the bucket ci-op-bb5dgq54-77753-oidc: MalformedPolicy: Policy has invalid resource
	status code: 400, request id: VNBZ3NYDH6YXWFZ3, host id: pHF8v7C3vr9YJdD9HWamFmRbMaOPRbHSNIDaXUuUyrgy0gKCO9DDFU/Xy8ZPmY2LCjfLQnUDmtQ=

Correct AWS ARN prefix:
GovCloud (us-gov-east-1 and us-gov-west-1): arn:aws-us-gov
AWS China (cn-north-1 and cn-northwest-1): arn:aws-cn

[1] https://github.com/openshift/cloud-credential-operator/pull/526/files#diff-1909afc64595b92551779d9be99de733f8b694cfb6e599e49454b380afc58876R211


 

Version-Release number of selected component (if applicable):

4.12.0-0.nightly-2023-05-11-024616

How reproducible:

Always
 

Steps to Reproduce:

1. Run command: `aws create-all --name="${infra_name}" --region="${REGION}" --credentials-requests-dir="/tmp/credrequests" --output-dir="/tmp"` on GovCloud regions
2.
3.

Actual results:

Failed to create Identity provider
 

Expected results:

Create resources successfully.
 

Additional info:

Related PRs:
4.10: https://github.com/openshift/cloud-credential-operator/pull/531
4.11: https://github.com/openshift/cloud-credential-operator/pull/530
4.12: https://github.com/openshift/cloud-credential-operator/pull/529
4.13: https://github.com/openshift/cloud-credential-operator/pull/528
4.14: https://github.com/openshift/cloud-credential-operator/pull/526
 

• https://github.com/openshift/cloud-credential-operator/pull/540

This is a clone of issue OCPBUGS-2181. The following is the description of the original issue:
—
Description of problem:

E2E test Installs Red Hat Integration - 3scale operator test is failing due to change of Operator name

CI Search: https://search.ci.openshift.org/?search=Installs+Red+Hat+Integration+-+3scale+operator+in+test+namespace+and+creates+3scale+Backend+Schema+operand+instance&maxAge=24h&context=1&type=bug%2Bissue%2Bjunit&name=pull-ci-openshift-console-master-e2e-gcp-console&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

• https://github.com/openshift/console/pull/12175

Description of problem:

For OVNK to become CNCF complaint, we need to support session affinity timeout feature and enable the e2e's on OpenShift side. This bug tracks the efforts to get this into 4.12 OCP.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/ovn-kubernetes/pull/1522
• https://github.com/openshift/origin/pull/27722

This is a clone of issue OCPBUGS-12914. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-12878. The following is the description of the original issue:
—
We want to add the dual-stack tests to the CNI plugin conformance test suite, for the currently supported releases.

(This has no impact on OpenShift itself. We're just modifying a test suite that OCP does not use.)

• https://github.com/openshift/origin/pull/27904

• https://github.com/openshift/ironic-image/pull/381

This is a clone of issue OCPBUGS-13335. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-13334. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-11753. The following is the description of the original issue:
—
Description of problem:

During the IPI bm provisioning process the deployment fails with the following error:

Error: could not inspect: could not inspect node, node is currently 'inspect failed' , last error was 'Redfish exception occurred. Error: Failed to get network interface information on node 995431b0-14c8-4aae-9733-e02b42da29ac: The attribute EthernetInterfaces is missing from the resource /redfish/v1/Systems/<redacted>'

Version-Release number of selected component (if applicable):

4.12

How reproducible:

Snippet from my install-config.yaml
      - name: ocp-test-2
        role: master
        bmc:
          address: redfish-virtualmedia://<redacted>/redfish/v1/Systems/<redacted>
          username: admin
          password: <redacted>
          disableCertificateVerification: True
        bootMACAddress: <redacted>

Steps to Reproduce:

see above

Actual results:

Error: could not inspect: could not inspect node, node is currently 'inspect failed' , last error was 'Redfish exception occurred. Error: Failed to get network interface information on node 995431b0-14c8-4aae-9733-e02b42da29ac: The attribute EthernetInterfaces is missing from the resource /redfish/v1/Systems/<redacted>'

Expected results:

Because I already specified the bootMACAddress in the install-config.yaml I would like the installer to continue and don't fail when the Redfish API doesn't provide the EthernetInterfaces

Additional info:

The blades in this specific chassis don't have access to the NIC adapter.
Therefore they can't get the MAC address and populate this field in the Redfish API.
These blades are currently still selling and will need to be supported for another 5 years atleast.
Our new servers in the new chassis don't have this problem (X-Series).

Please keep the bootMACAddress as an optional field in the install-config.yaml to support these blades.

• https://github.com/openshift/ironic-image/pull/380

Description of problem:

PROXY protocol cannot be enabled for the "Private" endpoint publishing strategy type.

Version-Release number of selected component (if applicable):

4.10.0+. PROXY protocol was made configurable for the "HostNetwork" and "NodePortService" endpoint publishing strategy types, but not for "Private", in this release.

How reproducible:

Always.

Steps to Reproduce:

1. Create an ingresscontroller with the "Private" endpoint publishing strategy type:

oc create -f - <<'EOF'
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: example
  namespace: openshift-ingress-operator
spec:
  domain: example.com
  endpointPublishingStrategy:
    type: Private
    private:
      protocol: PROXY
EOF

2. Check the ingresscontroller's status:

oc -n openshift-ingress-operator get ingresscontrollers/example -o 'jsonpath={.status.endpointPublishingStrategy}'

3. Check whether the resulting router deployment has PROXY protocol enabled.

oc -n openshift-ingress get deployments/router-example -o 'jsonpath={.spec.template.spec.containers[0].env[?(@.name=="ROUTER_USE_PROXY_PROTOCOL")]}'

Actual results:

The ingresscontroller is created:

ingresscontroller.operator.openshift.io/example created

The status shows that the spec.endpointPublishingStrategy.private.protocol setting was ignored:

{"private":{},"type":"Private"}

The deployment does not enable PROXY protocol; the oc get command prints no output.

Expected results:

The ingresscontroller's status should indicate that PROXY protocol is enabled:

{"private":{"protocol":"PROXY"},"type":"Private"}

The deployment should have PROXY protocol enabled:

{"name":"ROUTER_USE_PROXY_PROTOCOL","value":"true"}

Additional info:

This bug report duplicates https://bugzilla.redhat.com/show_bug.cgi?id=2104481 in order to facilitate backports.

• https://github.com/openshift/cluster-ingress-operator/pull/914

• https://github.com/openshift/ovn-kubernetes/pull/1272

• https://github.com/openshift/console/pull/12956

Description of problem:

There is a bug affecting verify steps functionality on iDRAC hardware in OpenShift 4.11 and 4.10. Original bug report has been made against 4.10:

https://issues.redhat.com/browse/OCPBUGS-1740

While I am not aware of this issue being reported against 4.11, due to the fact that the fix is only present in 4.12 codebase, 4.11 versions will also be affected by this issue.

This bug is created to meet automation requirements for backporting the fixes from 4.12 version to 4.11 (and then to 4.11 in the bug quoted above).

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/ironic-image/pull/302

In order to delete the correct GCP cloud resources, the "--credentials-requests-dir" parameter must be passed to "ccoctl gcp delete". This was fixed for 4.12 as part of https://github.com/openshift/cloud-credential-operator/pull/489 but must be backported for previous releases. See https://github.com/openshift/cloud-credential-operator/pull/489#issuecomment-1248733205 for discussion regarding this bug.

To reproduce, create GCP infrastructure with a name parameter that is a subset of another set of GCP infrastructure's name parameter. I will "ccoctl gcp create all" with "name=abutcher-gcp" and "name=abutcher-gcp1".

$ ./ccoctl gcp create-all \
--name=abutcher-gcp \
--region=us-central1 \
--project=openshift-hive-dev \
--credentials-requests-dir=./credrequests

$ ./ccoctl gcp create-all \
--name=abutcher-gcp1 \
--region=us-central1 \
--project=openshift-hive-dev \
--credentials-requests-dir=./credrequests

Running "ccoctl gcp delete --name=abutcher-gcp" will result in GCP infrastructure for both "abutcher-gcp" and "abutcher-gcp1" being deleted. 

$ ./ccoctl gcp delete --name abutcher-gcp --project openshift-hive-dev
2022/10/24 11:30:06 Credentials loaded from file "/home/abutcher/.gcp/osServiceAccount.json"
2022/10/24 11:30:06 Deleted object .well-known/openid-configuration from bucket abutcher-gcp-oidc
2022/10/24 11:30:07 Deleted object keys.json from bucket abutcher-gcp-oidc
2022/10/24 11:30:07 OIDC bucket abutcher-gcp-oidc deleted
2022/10/24 11:30:09 IAM Service account abutcher-gcp-openshift-image-registry-gcs deleted
2022/10/24 11:30:10 IAM Service account abutcher-gcp-openshift-gcp-ccm deleted
2022/10/24 11:30:11 IAM Service account abutcher-gcp1-openshift-cloud-network-config-controller-gcp deleted
2022/10/24 11:30:12 IAM Service account abutcher-gcp-openshift-machine-api-gcp deleted
2022/10/24 11:30:13 IAM Service account abutcher-gcp-openshift-ingress-gcp deleted
2022/10/24 11:30:15 IAM Service account abutcher-gcp-openshift-gcp-pd-csi-driver-operator deleted
2022/10/24 11:30:16 IAM Service account abutcher-gcp1-openshift-ingress-gcp deleted
2022/10/24 11:30:17 IAM Service account abutcher-gcp1-openshift-image-registry-gcs deleted
2022/10/24 11:30:19 IAM Service account abutcher-gcp-cloud-credential-operator-gcp-ro-creds deleted
2022/10/24 11:30:20 IAM Service account abutcher-gcp1-openshift-gcp-pd-csi-driver-operator deleted
2022/10/24 11:30:21 IAM Service account abutcher-gcp1-openshift-gcp-ccm deleted
2022/10/24 11:30:22 IAM Service account abutcher-gcp1-cloud-credential-operator-gcp-ro-creds deleted
2022/10/24 11:30:24 IAM Service account abutcher-gcp1-openshift-machine-api-gcp deleted
2022/10/24 11:30:25 IAM Service account abutcher-gcp-openshift-cloud-network-config-controller-gcp deleted
2022/10/24 11:30:25 Workload identity pool abutcher-gcp deleted

• https://github.com/openshift/cloud-credential-operator/pull/506

This bug is a backport clone of [Bugzilla Bug 2034883](https://bugzilla.redhat.com/show_bug.cgi?id=2034883). The following is the description of the original bug:
—
Description of problem:

Situation (starting point):

• There is an ongoing change to the machine-config-daemon daemonset being applied by the machine-config-operator pod. It is waiting for the daemonset to roll out.
• There are some nodes not ready, so daemonset rollout never ends and waiting on that ends in timeout error.

Problem:

• Machine-config-operator pods stops trying to reconcile stuff whenever it finds timeout error in waiting for the machine-config-daemon rollout
• This implies that the `spec.kubeAPIServerServingCAData` field of controllerconfig/machine-config-controller object is not updated when the kube-apiserver-operator updates kube-apiserver-to-kubelet-client-ca configmap.
• Without that field updated, a kube-apiserver-to-kubelet-client-ca change is never rolled out to the nodes.
• That ultimately leads to cluster-wide unavailability of "oc logs", "oc rsh" etc. commands when the kube-apiserver-operator starts using a client cert signed by the new kube-apiserver-to-kubelet-client-ca to access kubelet ports.

Version-Release number of MCO (Machine Config Operator) (if applicable):

4.7.21

Platform (AWS, VSphere, Metal, etc.): (not relevant)

Are you certain that the root cause of the issue being reported is the MCO (Machine Config Operator)?
(Y/N/Not sure): Y

How reproducible:

Always if the said conditions are met.

Steps to Reproduce:
1. Have some nodes not ready
2. Force a change that requires machine-config-daemon daemonset rollout (I think that changing proxy settings would work for this)
3. Wait until a new kube-apiserver-to-kubelet-client-ca is rolled out by kube-apiserver-operator

Actual results:

New kube-apiserver-to-kubelet-client-ca not forwarded to controllerconfig, kube-apiserver-to-kubelet-client-ca not deployed on nodes

Expected results:

kube-apiserver-to-kubelet-client-ca forwarded to controllerconfig, kube-apiserver-to-kubelet-client-ca deployed to nodes.

Additional info:

In comments

• https://github.com/openshift/machine-config-operator/pull/3421

• https://github.com/openshift/cluster-baremetal-operator/pull/321

Description of problem:

During an upgrade from 4.10.0-0.nightly-2022-09-06-081345 -> 4.11.0-0.nightly-2022-09-06-074353

DNS operator stuck in progressing state with following error in dns-defalt pod

2022-09-07T00:47:35.959931289Z [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://172.30.0.1:443/version": dial tcp 172.30.0.1:443: i/o timeout

must-gather: http://shell.lab.bos.redhat.com/~anusaxen/must-gather-136172-050448657.tar.gz

Version-Release number of selected component (if applicable):

4.11.0-0.nightly-2022-09-06-074353

How reproducible:

Intermittent

Steps to Reproduce:

1. Perform upgrade from 4.10->4.11
2.
3.

Actual results:

Upgrade unsuccessful due to API svc unreachability

Expected results:

Upgrade should be successful

Additional info:

$ omg get co | grep -v "True.*False.*False"
NAME                                      VERSION                             AVAILABLE  PROGRESSING  DEGRADED  SINCE
dns                                       4.10.0-0.nightly-2022-09-06-081345  True       True         False     3h4m
network                                   4.11.0-0.nightly-2022-09-06-074353  True       True         False     3h5m

$ omg logs dns-default-5mqz4 -c dns -n openshift-dns
/home/anusaxen/Downloads/must-gather-136208-318613238/must-gather.local.4033693973186012455/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-d995e1608ba83a1f71b5a1c49705aa3cef38618e0674ee256332d1b0e2cb0e23/namespaces/openshift-dns/pods/dns-default-5mqz4/dns/dns/logs/current.log
2022-09-07T00:45:36.355161941Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:36.854403897Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:37.354367197Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:37.861427861Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:38.355095996Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:38.855063626Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:39.355170472Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:39.855296624Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:40.355481413Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:40.855455283Z [WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API
2022-09-07T00:45:40.856166731Z .:5353
2022-09-07T00:45:40.856166731Z [INFO] plugin/reload: Running configuration SHA512 = 7c3db3182389e270fe3af971fef9e7157a34bde01eb750d58c2d4895a965ecb2e2d00df34594b6a819653088b6efa2590e748ce23f9e16cbb0f44d74a4fc3587
2022-09-07T00:45:40.856166731Z CoreDNS-1.9.2
2022-09-07T00:45:40.856166731Z linux/amd64, go1.18.4, 
2022-09-07T00:46:05.957503195Z [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://172.30.0.1:443/version": dial tcp 172.30.0.1:443: i/o timeout
2022-09-07T00:47:25.444876534Z [INFO] SIGTERM: Shutting down servers then terminating
2022-09-07T00:47:25.444984367Z [INFO] plugin/health: Going into lameduck mode for 20s
2022-09-07T00:47:35.959931289Z [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://172.30.0.1:443/version": dial tcp 172.30.0.1:443: i/o timeout

• https://github.com/openshift/sdn/pull/488

This is a clone of issue OCPBUGS-1549. The following is the description of the original issue:
—
Description of problem:

The cluster-dns-operator does not reconcile the openshift-dns namespace, which has been exposed as an issue in 4.12 due to the requirement for the namespace to have pod-security labels.

If a cluster has been incrementally updated from a version less than or equal to 4.9, the openshift-dns namespace will most likely not contain the required pod-security labels since the namespace was statically created when the cluster was installed with old namespace configuration.

Version-Release number of selected component (if applicable):

4.12

How reproducible:

Always if cluster originally installed with v4.9 or less

Steps to Reproduce:

1. Install v4.9
2. Upgrade to v4.12 (incrementally if required for upgrade path)
3. openshift-dns namespace will be missing pod-security labels

Actual results:

"oc get ns openshift-dns -o yaml" will show missing pod-security labels: 

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    openshift.io/node-selector: ""
    openshift.io/sa.scc.mcs: s0:c15,c0
    openshift.io/sa.scc.supplemental-groups: 1000210000/10000
    openshift.io/sa.scc.uid-range: 1000210000/10000
  creationTimestamp: "2020-05-21T19:36:15Z"
  labels:
    kubernetes.io/metadata.name: openshift-dns
    olm.operatorgroup.uid/3d42c0c1-01cd-4c55-bf88-864f041c7e7a: ""
    openshift.io/cluster-monitoring: "true"
    openshift.io/run-level: "0"
  name: openshift-dns
  resourceVersion: "3127555382"
  uid: 0fb4571e-952f-4bea-bc45-461beec54369
spec:
  finalizers:
  - kubernetes

Expected results:

pod-security labels should exist:
 
 labels:
    kubernetes.io/metadata.name: openshift-dns
    olm.operatorgroup.uid/3d42c0c1-01cd-4c55-bf88-864f041c7e7a: ""
    openshift.io/cluster-monitoring: "true"
    openshift.io/run-level: "0"
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/warn: privileged

Additional info:

Issue found in CI during upgrade

https://coreos.slack.com/archives/C03G7REB4JV/p1663676443155839 

• https://github.com/openshift/cluster-dns-operator/pull/346

Discovered in the must gather kubelet_service.log from https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-gcp-sdn-upgrade/1586093220087992320

It appears the guard pod names are too long, and being truncated down to where they will collide with those from the other masters.

From kubelet logs in this run:

❯ grep openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-maste kubelet_service.log
Oct 28 23:58:55.693391 ci-op-3hj6pnwf-4f6ab-lv57z-master-1 kubenswrapper[1657]: E1028 23:58:55.693346    1657 kubelet_pods.go:413] "Hostname for pod was too long, truncated it" podName="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-master-1" hostnameMaxLen=63 truncatedHostname="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-maste"
Oct 28 23:59:03.735726 ci-op-3hj6pnwf-4f6ab-lv57z-master-0 kubenswrapper[1670]: E1028 23:59:03.735671    1670 kubelet_pods.go:413] "Hostname for pod was too long, truncated it" podName="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-master-0" hostnameMaxLen=63 truncatedHostname="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-maste"
Oct 28 23:59:11.168082 ci-op-3hj6pnwf-4f6ab-lv57z-master-2 kubenswrapper[1667]: E1028 23:59:11.168041    1667 kubelet_pods.go:413] "Hostname for pod was too long, truncated it" podName="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-master-2" hostnameMaxLen=63 truncatedHostname="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-maste"

This also looks to be happening for openshift-kube-scheduler-guard, kube-controller-manager-guard, possibly others.

Looks like they should be truncated further to make room for random suffixes in https://github.com/openshift/library-go/blame/bd9b0e19121022561dcd1d9823407cd58b2265d0/pkg/operator/staticpod/controller/guard/guard_controller.go#L97-L98

Unsure of the implications here, it looks a little scary.

• https://github.com/openshift/cluster-kube-apiserver-operator/pull/1429
• https://github.com/openshift/cluster-kube-controller-manager-operator/pull/690
• https://github.com/openshift/cluster-kube-scheduler-operator/pull/458

Description of problem:

The storageclass "thin-csi" is created by vsphere-CSI-Driver-Operator, after deleting it manually, it should be re-created immediately. 

Version-Release number of selected component (if applicable):

4.11.4

How reproducible:

Always

Steps to Reproduce:

1. Check storageclass in running cluster, thin-csi is present:
$ oc get sc 
NAME             PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
thin (default)   kubernetes.io/vsphere-volume   Delete          Immediate              false                  41m
thin-csi         csi.vsphere.vmware.com         Delete          WaitForFirstConsumer   true                   38m

2. Delete thin-csi storageclass:
$ oc delete sc thin-csi
storageclass.storage.k8s.io "thin-csi" deleted

3. Check storageclass again, thin-csi is not present:
$ oc get sc
NAME             PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
thin (default)   kubernetes.io/vsphere-volume   Delete          Immediate           false                  50m

4. Check vmware-vsphere-csi-driver-operator log:
......
I0909 03:47:42.172866       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1662695014\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1662695014\" (2022-09-09 02:43:34 +0000 UTC to 2023-09-09 02:43:34 +0000 UTC (now=2022-09-09 03:47:42.172853123 +0000 UTC))"I0909 03:49:38.294962       
1 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOFI0909 03:49:38.295468       
1 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOFI0909 03:49:38.295765       
1 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF

5. Only first time creating in vmware-vsphere-csi-driver-operator log:
$ oc -n openshift-cluster-csi-drivers logs vmware-vsphere-csi-driver-operator-7cc6d44b5c-c8czw | grep -i "storageclass"I0909 03:46:31.865926   1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-csi-drivers", Name:"vmware-vsphere-csi-driver-operator", UID:"9e0c3e2d-d403-40a1-bf69-191d7aec202b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'StorageClassCreated' Created StorageClass.storage.k8s.io/thin-csi because it was missing 

Actual results:

The storageclass "thin-csi" could not be re-created after deleting

Expected results:

The storageclass "thin-csi" should be re-created after deleting

Additional info:

• https://github.com/openshift/vmware-vsphere-csi-driver-operator/pull/127

• https://github.com/openshift/sdn/pull/543

• https://github.com/openshift/installer/pull/6350