Feature Overview
Insights Advisor for OpenShift is integrated within OpenShift Cluster Manager. This has some limitations for adding new features and also for sharing codebase between RHEL Advisor and OCM Insights Advisor tab. Insights Advisor for OpenShift lacks certain features from the RHEL UI, the codebase is not 1:1 clone.
As a customer of Insights I will have same/very similar user experience with Insights for OpenShift and Insights for RHEL. The workflows will share the main concepts, the UI elements will be same and features introduced to Advisor will be automatically considered for both all supported platforms.
As OpenShift users I will still see integrations of Insights Advisor within OpenShift Cluster Manager that shows aggregated information for customer account and single cluster view on Advisor data. These integration will point to new Insights Advisor for OpenShift app that will be tightly integrated into OpenShift Cluster Manager.

• Note: The application will be reusing the codebase but will run as a separate app for OpenShift. THere's no intent to merge RHEL and OpenShift workflows into a single app.

Goals

• Q2CY21: Explore possibility to unify codebase between RHEL Advisor and OCM Insights Advisor tab. Identify architecture misalignments, create UI mockups to merge the two existing UIs.
• Q3CY21: Integrate OpenShift into Advisor codebase, standup the Insights Advisor for OpenShift application and change integration in OpenShift Cluster manager to point at the new app
• Q4CY21: Deliver missing screen of Insights Advisor for OpenShift (Systems and Recommendations views)

Requirements

• UX overview of UI elements in both UIs - Marie Doruskova
• Architecture overview/misalignments for both UIs - Jan Zeleny [~fjansen]

Benefits

• Feature parity between RHEL and OpenShift
• Adopting new features developed by RHEL Advisor team quicker
• Smaller maintenance cost

Questions to answer...

• Possible deviations between OpenShift and RHEL
• Remediation workflow different between OpenShift and RHEL

Out of Scope

• Single app that combines RHEL hosts and OpenShift clusters. Goal is still to differentiate between platforms and offer view only for a single platform.
• Direct/Supervised remediations and integration of remediations with Advanced Cluster Manager (as a Service)

Background, and strategic fit

• Insights Advisor for OpenShift follows the goal to introduce multiple applications that add value for OpenShift customers under the Insights brand. The current UI and integration of Advisor into OpenShift cluster manager doesn't follow pattern that other Insights for OpenShift applications can/will follow.

Documentation Considerations

• OCM documentation is impacted, existing workflows described in OCM documentation will persist. The placement of the application within OCM will be different.

Problem Alignment

The Problem

Today, all configuration for setting individual, for example, routing configuration is done via a single configuration file that only admins have access to. If an environment uses multiple tenants and each tenant, for example, has different systems that they are using to notify teams in case of an issue, then someone needs to file a request w/ an admin to add the required settings.

That can be bothersome for individual teams, since requests like that usually disappear in the backlog of an administrator. At the same time, administrators might get tons of requests that they have to look at and prioritize, which takes them away from more crucial work.

We would like to introduce a more self service approach whereas individual teams can create their own configuration for their needs w/o the administrators involvement.

Last but not least, since Monitoring is deployed as a Core service of OpenShift there are multiple restrictions that the SRE team has to apply to all OSD and ROSA clusters. One restriction is the ability for customers to use the central Alertmanager that is owned and managed by the SRE team. They can't give access to the central managed secret due to security concerns so that users can add their own routing information.

High-Level Approach

Provide a new API (based on the Operator CRD approach) as part of the Prometheus Operator that allows creating a subset of the Alertmanager configuration without touching the central Alertmanager configuration file.

Please note that we do not plan to support additional individual webhooks with this work. Customers will need to deploy their own version of the third party webhooks.

Goal & Success

• Allow users to deploy individual configurations that allow setting up Alertmanager for their needs without an administrator.

Solution Alignment

Key Capabilities

• As an OpenShift administrator, I want to control who can CRUD individual configuration so that I can make sure that any unknown third person can touch the central Alertmanager instance shipped within OpenShift Monitoring.
• As a team owner, I want to deploy a routing configuration to push notifications for alerts to my system of choice.

Key Flows

Team A wants to send all their important notifications to a specific Slack channel.

• Administrator gives permission to Team A to allow creating a new configuration CR in their individual namespace.
• Team A creates a new configuration CR.
• Team A configures what alerts should go into their Slack channel.
• Open Questions & Key Decisions (optional)
• Do we want to improve anything inside the developer console to allow configuration?

Feature Overview  

Much like core OpenShift operators, a standardized flow exists for OLM-managed operators to interact with the cluster in a specific way to leverage AWS STS authorization when using AWS APIs as opposed to insecure static, long-lived credentials. OLM-managed operators can implement integration with the CloudCredentialOperator in well-defined way to support this flow.

Goals:

Enable customers to easily leverage OpenShift's capabilities around AWS STS with layered products, for increased security posture. Enable OLM-managed operators to implement support for this in well-defined pattern.

Requirements:

• CCO gets a new mode in which it can reconcile STS credential request for OLM-managed operators
• A standardized flow is leveraged to guide users in discovering and preparing their AWS IAM policies and roles with permissions that are required for OLM-managed operators 
• A standardized flow is defined in which users can configure OLM-managed operators to leverage AWS STS
• An example operator is used to demonstrate the end2end functionality
• Clear instructions and documentation for operator development teams to implement the required interaction with the CloudCredentialOperator to support this flow

Use Cases:

See Operators & STS slide deck.

Out of Scope:

• handling OLM-managed operator updates in which AWS IAM permission requirements might change from one version to another (which requires user awareness and intervention)

Background:

The CloudCredentialsOperator already provides a powerful API for OpenShift's cluster core operator to request credentials and acquire them via short-lived tokens. This capability should be expanded to OLM-managed operators, specifically to Red Hat layered products that interact with AWS APIs. The process today is cumbersome to none-existent based on the operator in question and seen as an adoption blocker of OpenShift on AWS.

Customer Considerations

This is particularly important for ROSA customers. Customers are expected to be asked to pre-create the required IAM roles outside of OpenShift, which is deemed acceptable.

Documentation Considerations

• Internal documentation needs to exists to guide Red Hat operator developer teams on the requirements and proposed implementation of integration with CCO and the proposed flow
• External documentation needs to exist to guide users on:
  • how to become aware that the cluster is in STS mode
  • how to become aware of operators that support STS and the proposed CCO flow
  • how to become aware of the IAM permissions requirements of these operators
  • how to configure an operator in the proposed flow to interact with CCO

Interoperability Considerations

• this needs to work with ROSA
• this needs to work with self-managed OCP on AWS

Feature Overview

Enable sharing ConfigMap and Secret across namespaces

Requirements

Questions to answer…

NA

Out of Scope

NA

Background, and strategic fit

Consumption of RHEL entitlements has been a challenge on OCP 4 since it moved to a cluster-based entitlement model compared to the node-based (RHEL subscription manager) entitlement mode. In order to provide a sufficiently similar experience to OCP 3, the entitlement certificates that are made available on the cluster (OCPBU-93) should be shared across namespaces in order to prevent the need for cluster admin to copy these entitlements in each namespace which leads to additional operational challenges for updating and refreshing them. 

Documentation Considerations

Questions to be addressed:
 * What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
 * Does this feature have doc impact?
 * New Content, Updates to existing content, Release Note, or No Doc Impact
 * If unsure and no Technical Writer is available, please contact Content Strategy.
 * What concepts do customers need to understand to be successful in [action]?
 * How do we expect customers will use the feature? For what purpose(s)?
 * What reference material might a customer want/need to complete [action]?
 * Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
 * What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview (aka. Goal Summary)  

Upstream Kuberenetes is following other SIGs by moving it's intree cloud providers to an out of tree plugin format, Cloud Controller Manager, at some point in a future Kubernetes release. OpenShift needs to be ready to action this change  

Goals (aka. expected user outcomes)

Bring together all the cloud controller managers (AWS, GCP, Azure), complete testing and prepare for final GA

Requirements (aka. Acceptance Criteria):

A list of specific needs or objectives that a feature must deliver in order to be considered complete.  Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc.  Initial completion during Refinement status.

Use Cases (Optional):

Include use case diagrams, main success scenarios, alternative flow scenarios.  Initial completion during Refinement status.

Questions to Answer (Optional):

Include a list of refinement / architectural questions that may need to be answered before coding can begin.  Initial completion during Refinement status.

Out of Scope

High-level list of items that are out of scope.  Initial completion during Refinement status.

Background

Provide any additional context is needed to frame the feature.  Initial completion during Refinement status.

Customer Considerations

Provide any additional customer-specific considerations that must be made when designing and delivering the Feature.  Initial completion during Refinement status.

Documentation Considerations

Provide information that needs to be considered and planned so that documentation will meet customer needs.  Initial completion during Refinement status.

Interoperability Considerations

Which other projects and versions in our portfolio does this feature impact?  What interoperability test scenarios should be factored by the layered products?  Initial completion during Refinement status.

Feature Overview

This Feature is a general "catch all" for the time being. There are a number of existing priorities from Q1 that should be aligned with existing priorities below but if not, assign to this feature as needed.

Goals

In order to get a better overall portfolio view, we'll leverage this Feature to gather work that doesn't fall into other existing priorities on this board. As this list grows, the portfolio priority grooming team will look to split out or handle appropriately.

Requirements

A list of specific needs or objectives that a Feature must deliver to satisfy the Feature. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts.  If a non MVP requirement slips, it does not shift the feature.

(Optional) Use Cases

< How will the user interact with this feature? >

< Which users will use this and when will they use it? >

< Is this feature used as part of current user interface? >

Out of Scope

Background, and strategic fit

< What does the person writing code, testing, documenting need to know? >

Assumptions

< Are there assumptions being made regarding prerequisites and dependencies?>

< Are there assumptions about hardware, software or people resources?>

Customer Considerations

< Are there specific customer environments that need to be considered (such as working with existing h/w and software)?>

Documentation Considerations

< What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)? >

<What does success look like?>

< Does this feature have doc impact? Possible values are: New Content, Updates to existing content,  Release Note, or No Doc Impact?>

 <If unsure and no Technical Writer is available, please contact Content Strategy. If yes, complete the following.>

• <What concepts do customers need to understand to be successful in [action]?>
• <How do we expect customers will use the feature? For what purpose(s)?>
• <What reference material might a customer want/need to complete [action]?>
• <Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available. >
• <What is the doc impact (New Content, Updates to existing content, or Release Note)?>

Questions

tldr: three basic claims, the rest is explanation and one example

1. We cannot improve long term maintainability solely by fixing bugs.
2. Teams should be asked to produce designs for improving maintainability/debugability.
3. Specific maintenance items (or investigation of maintenance items), should be placed into planning as peer to PM requests and explicitly prioritized against them.

While bugs are an important metric, fixing bugs is different than investing in maintainability and debugability. Investing in fixing bugs will help alleviate immediate problems, but doesn't improve the ability to address future problems. You (may) get a code base with fewer bugs, but when you add a new feature, it will still be hard to debug problems and interactions. This pushes a code base towards stagnation where it gets harder and harder to add features.

One alternative is to ask teams to produce ideas for how they would improve future maintainability and debugability instead of focusing on immediate bugs. This would produce designs that make problem determination, bug resolution, and future feature additions faster over time.

I have a concrete example of one such outcome of focusing on bugs vs quality. We have resolved many bugs about communication failures with ingress by finding problems with point-to-point network communication. We have fixed the individual bugs, but have not improved the code for future debugging. In so doing, we chase many hard to diagnose problem across the stack. The alternative is to create a point-to-point network connectivity capability. this would immediately improve bug resolution and stability (detection) for kuryr, ovs, legacy sdn, network-edge, kube-apiserver, openshift-apiserver, authentication, and console. Bug fixing does not produce the same impact.

We need more investment in our future selves. Saying, "teams should reserve this" doesn't seem to be universally effective. Perhaps an approach that directly asks for designs and impacts and then follows up by placing the items directly in planning and prioritizing against PM feature requests would give teams the confidence to invest in these areas and give broad exposure to systemic problems.

Relevant links:

• Documentation:
  • Edge Diagnostics Scratchpad, our team's internal diagnostic guide.
  • Troubleshooting OCP networking issues - The complete guide, the SDN team's diagnostic guide.
  • Linux Performance, Brendan Gregg's guide to analyzing Linux performance issues.
  • RFC: A proper feedback loop on Alerts.
  • OpenShift Router Reload Technical Overview on Access.
  • Performance Scaling HAProxy with OpenShift on Access.
  • How to collect worker metrics to troubleshoot CPU load, memory pressure and interrupt issues and networking on worker nodes in OCP 4 on Access.
  • OpenShift Performance and Scale Knowledge Base on Mojo, results from OpenShift scalability testing.
  • Scalability and performance, OCP 4.5 documentation about the router's currently known scalability limits.
  • Scaling OpenShift Container Platform HAProxy Router, OCP 3.11 documentation about the manual performance configuration that was possible in OCP 3.
  • Timing web requests with cURL and Chrome from the Cloudflare blog.
  • tcpdump advanced filters, some useful tcpdump commands.
  • OpenShift SDN - Networking, OCP 3.11 documentation on the SDN (useful background reading).
  • Ingress Operator and Controller Status Conditions, design document for improved status condition reporting.
  • Observability tips for HAProxy, a slide deck by Willy Tarreau.
  • Interesting Traces - Out of Order versus Retransmissions, analysis using tshark.
  • The PCP Book: A Complete Documentation of Performance Co-Pilot, by Yogesh Babar.
  • Debugging kernel networking bug, brief guide to using SystemTap on RHCOS.
  • Troubleshooting throughput issues from the OCP 4.5 documentation.
  • Troubleshooting OpenShift Clusters and Workloads.
  • Red Hat Enterprise Linux Network Performance Tuning Guide (PDF).
  • openshift/enhancements#289 stability: point to point network check, a diagnostic built into the kube-apiserver operator.
• Diagnostic tools:
  • dropwatch to watch for packet drops.
  • ethtool to check NIC configuration.
  • iovisor/bcc: BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more to trace and diagnose various issues in the networking stack.
  • r-curler to gather timing information about HTTP/HTTPS connections.
  • route-monitor, to monitor routes for reachability.
  • hping(3), a programmable packet generator.
  • OpenTracing / Jaeger in OpenShift.
  • node-problem-detector, a possible integration point for new diagnostics.
  • Using SystemTap by Brendan Gregg.
  • DTrace SystemTap cheatsheet (PDF).
• Visualization and more sophisticated diagnostic tools:
  • eldadru/ksniff, kubectl plugin for tcpdump & Wireshark.
  • ironcladlou/ditm, Dan's "Dan in the Middle" tool.
  • Skydive, network diagnostic and visualization tool.
  • ali, a "load testing tool capable of performing real-time analysis" with visualization.
• Testing tools:
  • stress-ng, a general stress-loading tool (CPU, filesystem, network, ...).
  • mb, the networking benchmarking tool written and used by Jiri Mencak from our Perf+Scale team.
• Case studies:
  • BZ1763206 is an example of diagnosing DNS latency/timeouts.
  • BZ1829779 Investigation details the diagnosis of route latency.
  • BZ1845545 is an example of diagnosing misconfigured DNS for an external LB.
  • Debugging network stalls on Kubernetes, from the GitHub Blog, about diagnosing Kubernetes performance issues related to ksoftirqd.

Feature Overview

Plugin teams need a mechanism to extend the OCP console that is decoupled enough so they can deliver at the cadence of their projects and not be forced in to the OCP Console release timelines.

The OCP Console Dynamic Plugin Framework will enable all our plugin teams to do the following:

• Extend the Console
• Deliver UI code with their Operator
• Work in their own git Repo
• Deliver at their own cadence

Goals

• • Operators can deliver console plugins separate from the console image and update plugins when the operator updates.
  • The dynamic plugin API is similar to the static plugin API to ease migration.
  • Plugins can use shared console components such as list and details page components.
  • Shared components from core will be part of a well-defined plugin API.
  • Plugins can use Patternfly 4 components.
  • Cluster admins control what plugins are enabled.
  • Misbehaving plugins should not break console.
  • Existing static plugins are not affected and will continue to work as expected.

Out of Scope

• • Initially we don't plan to make this a public API. The target use is for Red Hat operators. We might reevaluate later when dynamic plugins are more mature.
  • We can't avoid breaking changes in console dependencies such as Patternfly even if we don't break the console plugin API itself. We'll need a way for plugins to declare compatibility.
  • Plugins won't be sandboxed. They will have full JavaScript access to the DOM and network. Plugins won't be enabled by default, however. A cluster admin will need to enable the plugin.
  • This proposal does not cover allowing plugins to contribute backend console endpoints.

Requirements

 
 Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?

• Does this feature have doc impact?  

• New Content, Updates to existing content,  Release Note, or No Doc Impact

• If unsure and no Technical Writer is available, please contact Content Strategy.

• What concepts do customers need to understand to be successful in [action]?

• How do we expect customers will use the feature? For what purpose(s)?

• What reference material might a customer want/need to complete [action]?

• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.

• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview

• This Section:* High-Level description of the feature ie: Executive Summary

• Note: A Feature is a capability or a well defined set of functionality that delivers business value. Features can include additions or changes to existing functionality. Features can easily span multiple teams, and multiple releases.

Goals

• This Section:* Provide high-level goal statement, providing user context and expected user outcome(s) for this feature

• …

Requirements

• This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.

(Optional) Use Cases

This Section: 

• Main success scenarios - high-level user stories

• Alternate flow/scenarios - high-level user stories

• ...

Questions to answer…

• ...

Out of Scope

• …

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

• ...

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?

• Does this feature have doc impact?  

• New Content, Updates to existing content,  Release Note, or No Doc Impact

• If unsure and no Technical Writer is available, please contact Content Strategy.

• What concepts do customers need to understand to be successful in [action]?

• How do we expect customers will use the feature? For what purpose(s)?

• What reference material might a customer want/need to complete [action]?

• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.

• What is the doc impact (New Content, Updates to existing content, or Release Note)?

OCP/Telco Definition of Done
Feature Template descriptions and documentation.
Feature Overview

• Connect OpenShift workloads to Google services with Google Workload Identity

• Enable customers to access Google services from workloads on OpenShift clusters using Google Workload Identity (aka WIF)
  https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity 

Goals

• Customers want to be able to manage and operate OpenShift on Google Cloud Platform with workload identity, much like they do with AWS + STS or Azure + workload identity.
• Customers want to be able to manage and operate operators and customer workloads on top of OCP on GCP with workload identity.

Requirements

• Add support to CCO for the Installation and Upgrade using both UPI and IPI methods with GCP workload identity.
• Support install and upgrades for connected and disconnected/restriction environments.
• Support the use of Operators with GCP workload identity with minimal friction.
• Support for HyperShift and non-HyperShift clusters.
• This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.

(Optional) Use Cases

This Section:

• Main success scenarios - high-level user stories
• Alternate flow/scenarios - high-level user stories
• ...

Questions to answer…

• ...

Out of Scope

• …

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

• ...

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

An epic we can duplicate for each release to ensure we have a place to catch things we ought to be doing regularly but can tend to fall by the wayside.

Epic Goal

• Improve CI testing of the image registry components.

Why is this important?

• The image registry, image API and the image pruner had a lot of tests removed during transition 4.0. This may make the platform less stable and/or slow down the team.

Scenarios

1. ...

Acceptance Criteria

• CI - tests should be more stable and have broader coverage

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

• ...

Why is this important?

• …

Scenarios

1. ...

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

Epic Goal

• As a CFE team, we would like to enable query logging for all Prometheus read paths
• As part of this, we would like to enable audit & query logging for Prometheus Adapter(aggregated server audit log), Prometheus(query log) and ThanosQuerier(query log)

Why is this important?

• This would help all parties(customers, app-sres, CCX, monitoring team,..) to debug an overloaded Prometheus instance.

Scenarios

1. When a customer faces a high cpu consumption in any of the Prometheus instance, they can enable audit logging in Prometheus Adapter to see which component is calling metrics API
2. When a customer faces a high cpu consumption in any of the Prometheus instance, they can enable query logging in all Prometheus instances(PM & UWM) and ThanosQuerier to see which query is frequently executed
3. https://bugzilla.redhat.com/show_bug.cgi?id=1982302

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• Prometheus Adapter audit logs must be enabled by default
• Prometheus Adapter audit logs must be preserved after each CI run

Open questions::

1. Should we enable ThanosRuler query logs?

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Description:

Openshift on RHV is composed of the following subproject the team maintains:

• installer
• cluster-api-provider-ovirt
• ovirt-csi-driver
• ovirt-csi-driver-operator
• terraform-provider-ovirt 

Each of those projects currently uses the generated oVirt API project go-ovirt.

This leads to a number of issues:

1. Duplicated code between the subprojects: Since the go-ovirt is a thin layer around the API then a lot of the code which interacts with oVirt is duplicated between the projects, which leads to all the classic duplication problems such as maintaining the project, lack of clear conventions, and so on.
2. Bad error handling and unclear errors:
   1. Since the go-ovirt is a thin layer there is a lot of error handling and checking which needs to be done, since a lot of the times it looks like a certain error should be ignored, it is never checked which could lead to unexpected situations.
   2. Since the errors which are returned from the oVirt Engine are sometimes unclear, when we return those errors to the users or log them is hard to understand what is the actual issue.
3. Lack of retries: sometimes an operation can take some time due to some condition that needs to be met, or an operation can fail due to infrastructure issues, the go-ovirt library doesn't contain any retry logic which means each client needs to implement its own retry logic which is not done at the moment and will cause more duplicated code.
4. Poor logging: The current go-ovirt library doesn't log anything, and all the logs come from the subprojects, this leads to:
   1. Inconsistent logging between the projects.
   2. Lack of logs.
5. Almost no test coverage:
   1. It's very hard to mock and write tests with go-ovirt since there are so many calls, but will be much easier to mock and write tests with go-ovirt-clent.
   2. go-ovirt only has rudimentary tests.

Then came go-ovirt-client, go-ovirt-client-log, go-ovirt-client-log-klog and k8sOVirtCredentialsMonitor to the rescue!

The go-ovirt-client is a wrapper around the go-ovirt which contains all the error handling/retry logic/logs/tests needed to provide a decent user experience and an easy-to-use API to the oVirt engine.

go-ovirt-client-log is a library to unify the logging logic between the projects, it is used by go-ovirt-client and should be used by all the sub-projects.

go-ovirt-client-log-klog is a companion library to go-ovirt-client-log enabling logging via the Kubernetes "klog" facility.

k8sOVirtCredentialsMonitor is a utility for monitoring the oVirt credentials secret, which will automatically update the ovirt credentials is they are changed. 

We aim to move all projects which are using the go-ovirt to use go-ovirt-client, go-ovirt-client-log and k8sOVirtCredentialsMonitor instead.

Benefits for the eng:

• Possible to write unit tests.
• Easier to maintain since less code duplication - reduce the amount of code.
• Test coverage exists on the ovirt-client as well.
• No(Less) bugs regarding operations that needed a retry or polling logic.
• Solves a number of existing bugs

Benefits for the customers:

• Clearer error messages and logs.
• Fewer bugs.

Acceptance criteria:

1. All sub-projects are not using go-ovirt directly - at least 90% of the calls to go-ovirt should be migrated to go-ovirt-client.
2. All sub-projects should use the corresponding go-ovirt-client-log for logging.
3. All csi-driver and cluster provide use k8sOVirtCredentialsMonitor.
4. CI tests are green for all components.

How to test:

1. QE regression - make sure all flows are still working.
2. Green CI on all jobs.
3. Keep an eye out for log messages that might confuse customers.

T-shirt size: M

Goal:

Provide an easy and successful experience for front end developers to build and deploy their applications

Why is it important?

Currently, the front end dev experience is not positive. It's much easier for them to use other platforms. Improving the front end dev experience will enable us to gain more marketshare

Use cases:

1. Need to be able to override the npm command when using Node Builder Image
2. Need to expose target port
3. Need access to the URL to access my application

Although we provide the ability for 2 & 3 today, the current journey does not match with the mental model of the front end developer

Acceptance criteria:

1. When importing an app, I should be able to easily provide the npm build and run commands
2. When opting in to create a route, the target port should be exposed without having to open any Advanced Options
3. After importing my app, if a route is exposed, I should be able to access/copy that URL

Dependencies (External/Internal):

Design Artifacts:

Desired UX experience

• enable user to provide the *Build Command* when Node Builder image is being used
• enable user to provide the *Run Command* when Node Builder image is being used
• 

• expose the Target Port under the *Create a route to the Application *rather than inside Show advanced Routing options

• NEED TO FINALIZE HOW TO PROVIDE THE ROUTE TO EASILY COPY – Inline Notification maybe? As well as side panel?

Note:

Problem:

This epic is mainly focused on the 4.10 Release QE activities

Goal:

1. Identify the scenarios for automation
2. Segregate the test Scenarios into smoke, Regression and other user stories
a. Update the https://docs.jboss.org/display/ODC/Automation+Status+Report
3. Align with layered operator teams for updating scripts
3. Work closely with dev team for epic automation
4. Create the automation scripts using cypress
5. Implement CI for nightly builds
6. Execute scripts on sprint basis

Why is it important?

To the track the QE progress at one place in 4.10 Release Confluence page

Use cases:

1. <case>

Acceptance criteria:

1. <criteria>

Dependencies (External/Internal):

Design Artifacts:

Exploration:

Note:

Epic Goal

• Port all remaining Protractor tests to Cypress

Why is this important?

• Protractor is very hard to debug when tests fail/flake
• Once all protractor tests are ported we can remove all Protractor dependencies, scripts, and configuration files.
• Cypress has better debugging, plug-ins, and reporting tools

Acceptance Criteria

• CI - MUST be running successfully with tests automated

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Problem:

Complete all the 4.9 epic features automation user stories and merge it to master branch.

Goal:

4.9 epics automation completion

Why is it important?

Tech debt should be completed

Use cases:

1. <case>

Acceptance criteria:

Create the pr's for 4.9 epic user stories automation
Review it
Merge it to 4.10 master branch and 4.9 master branch

Dependencies (External/Internal):

Design Artifacts:

Exploration:

Note:

• https://github.com/openshift/jenkins/pull/1480

This is a clone of issue OCPBUGS-4851. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-4850. The following is the description of the original issue:
—
Description of problem:

Kuryr might take a while to create Pods because it has to create Neutron ports for the pods. If a pod gets deleted while this is being processed, a
warning Event will be generated causing the "[sig-network] pods should successfully create sandboxes by adding pod to network" to fail.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/origin/pull/27878

• https://github.com/openshift/console-operator/pull/768

This is a clone of issue OCPBUGS-1237. The following is the description of the original issue:
—
job=pull-ci-openshift-origin-master-e2e-gcp-builds=all

This test has started permafailing on e2e-gcp-builds:

[sig-builds][Feature:Builds][Slow] s2i build with environment file in sources Building from a template should create a image from "test-env-build.json" template and run it in a pod [apigroup:build.openshift.io][apigroup:image.openshift.io]

The error in the test says

Sep 13 07:03:30.345: INFO: At 2022-09-13 07:00:21 +0000 UTC - event for build-test-pod: {kubelet ci-op-kg1t2x13-4e3c6-7hrm8-worker-a-66nwd} Pulling: Pulling image "image-registry.openshift-image-registry.svc:5000/e2e-test-build-sti-env-nglnt/test@sha256:262820fd1a94d68442874346f4c4024fdf556631da51cbf37ce69de094f56fe8"
Sep 13 07:03:30.345: INFO: At 2022-09-13 07:00:23 +0000 UTC - event for build-test-pod: {kubelet ci-op-kg1t2x13-4e3c6-7hrm8-worker-a-66nwd} Pulled: Successfully pulled image "image-registry.openshift-image-registry.svc:5000/e2e-test-build-sti-env-nglnt/test@sha256:262820fd1a94d68442874346f4c4024fdf556631da51cbf37ce69de094f56fe8" in 1.763914719s
Sep 13 07:03:30.345: INFO: At 2022-09-13 07:00:23 +0000 UTC - event for build-test-pod: {kubelet ci-op-kg1t2x13-4e3c6-7hrm8-worker-a-66nwd} Created: Created container test
Sep 13 07:03:30.345: INFO: At 2022-09-13 07:00:23 +0000 UTC - event for build-test-pod: {kubelet ci-op-kg1t2x13-4e3c6-7hrm8-worker-a-66nwd} Started: Started container test
Sep 13 07:03:30.345: INFO: At 2022-09-13 07:00:24 +0000 UTC - event for build-test-pod: {kubelet ci-op-kg1t2x13-4e3c6-7hrm8-worker-a-66nwd} Pulled: Container image "image-registry.openshift-image-registry.svc:5000/e2e-test-build-sti-env-nglnt/test@sha256:262820fd1a94d68442874346f4c4024fdf556631da51cbf37ce69de094f56fe8" already present on machine
Sep 13 07:03:30.345: INFO: At 2022-09-13 07:00:25 +0000 UTC - event for build-test-pod: {kubelet ci-op-kg1t2x13-4e3c6-7hrm8-worker-a-66nwd} Unhealthy: Readiness probe failed: Get "http://10.129.2.63:8080/": dial tcp 10.129.2.63:8080: connect: connection refused
Sep 13 07:03:30.345: INFO: At 2022-09-13 07:00:26 +0000 UTC - event for build-test-pod: {kubelet ci-op-kg1t2x13-4e3c6-7hrm8-worker-a-66nwd} BackOff: Back-off restarting failed container

• https://github.com/openshift/origin/pull/27479

This is a clone of issue OCPBUGS-2083. The following is the description of the original issue:
—
Description of problem:
Currently we are running VMWare CSI Operator in OpenShift 4.10.33. After running vulnerability scans, the operator was discovered to be running a known weak cipher 3DES. We are attempting to upgrade or modify the operator to customize the ciphers available. We were looking at performing a manual upgrade via Quay.io but can't seem to pull the image and was trying to steer away from performing a custom install from scratch. Looking for any suggestions into mitigated the weak cipher in the kube-rbac-proxy under VMware CSI Operator.

Version-Release number of selected component (if applicable):

 

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/vmware-vsphere-csi-driver-operator/pull/120

• https://github.com/openshift/cloud-provider-openstack/pull/158

• https://github.com/openshift/kubernetes/pull/1510
• https://github.com/openshift/origin/pull/27800

• https://github.com/openshift/csi-driver-manila-operator/pull/178

• https://github.com/openshift/ironic-image/pull/348

• https://github.com/openshift/jenkins/pull/1530

This is a clone of issue  OCPBUGS-11465. The following is the description of the original issue:

—

This is a clone of issue OCPBUGS-11404. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-11333. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-10690. The following is the description of the original issue:
—

Description of problem:

according to PR: https://github.com/openshift/cluster-monitoring-operator/pull/1824, startupProbe for UWM prometheus/platform prometheus should be 1 hour, but startupProbe for UWM prometheus is still 15m after enabled UWM, platform promethues does not have issue, startupProbe is increased to 1 hour

$ oc -n openshift-user-workload-monitoring get pod prometheus-user-workload-0 -oyaml | grep startupProbe -A20
    startupProbe:
      exec:
        command:
        - sh
        - -c
        - if [ -x "$(command -v curl)" ]; then exec curl --fail http://localhost:9090/-/ready;
          elif [ -x "$(command -v wget)" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready;
          else exit 1; fi
      failureThreshold: 60
      periodSeconds: 15
      successThreshold: 1
      timeoutSeconds: 3
...

$ oc -n openshift-monitoring get pod prometheus-k8s-0 -oyaml | grep startupProbe -A20
    startupProbe:
      exec:
        command:
        - sh
        - -c
        - if [ -x "$(command -v curl)" ]; then exec curl --fail http://localhost:9090/-/ready;
          elif [ -x "$(command -v wget)" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready;
          else exit 1; fi
      failureThreshold: 240
      periodSeconds: 15
      successThreshold: 1
      timeoutSeconds: 3

Version-Release number of selected component (if applicable):

4.13.0-0.nightly-2023-03-19-052243

How reproducible:

always

Steps to Reproduce:

1. enable UWM, check startupProbe for UWM prometheus/platform prometheus
2.
3.

Actual results:

startupProbe for UWM prometheus is still 15m

Expected results:

startupProbe for UWM prometheus should be 1 hour

Additional info:

since startupProbe for platform prometheus is increased to 1 hour, and no similar bug for UWM prometheus, won't fix the issue is OK.

• https://github.com/openshift/cluster-monitoring-operator/pull/1942

Description of problem:

Tracking this for backport of https://bugzilla.redhat.com/show_bug.cgi?id=2072710

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/cluster-network-operator/pull/1563

plus smaller fixes
this is a backport of https://github.com/openshift/cluster-baremetal-operator/pull/263

• https://github.com/openshift/cluster-baremetal-operator/pull/291

This is a clone of issue OCPBUGS-533. The following is the description of the original issue:
—
Description of problem:

customer is using Azure AD as openid provider and groups synchronization from the provider.

The scenario is the following:

1)

• user A login.
  groups are created with the membership.
  User A is member of a group with admin rights and it's cluster-admin

2)

• user B login:
  groups are updated with membership
  UserB is also member of the group with admin rights and it's cluster admin

3)

• user A login:
  groups are identical as in the former step.
  user A has no administration rights.

The groups memberships are the same in step 2 and 3.
The cluster role bindings of the groups have never changed.

the only way to have user A again the admin rights is to delete the membership from the group and have user A login again.

I have not managed to reproduce this using RH SSO. Neither Azure AD.

But my configuration is not exactly the same yet.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/oauth-apiserver/pull/85

+++ This bug was initially created as a clone of OCPBUGSM-46761 +++

Description of problem:
When using the admin console, under "Cluster Settings" and choosing Upstream Configuration, the "window" that appears has a dead link to documentation for how to create a local (disconnected) update server.

The link in the window is
https://access.redhat.com/documentation/en-us/openshift_container_platform/4.10/html/updating_clusters/installing-update-service

Assume the right link should be something like:
https://docs.openshift.com/container-platform/4.10/updating/updating-restricted-network-cluster.html#update-restricted-network-cluster-update-service

Version-Release number of selected component (if applicable):

4.10.*

— Additional comment from plarsen@redhat.com on 2022-06-17 16:33:53 UTC —

Created attachment 1890947
Screenshot showing the "window" with the 404 link

— Additional comment from rhamilto@redhat.com on 2022-06-28 19:56:17 UTC —

Thank you, Peter, for wonderfully documenting the bug!

• https://github.com/openshift/console/pull/11974

This is a clone of issue OCPBUGS-5258. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-5191. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-5164. The following is the description of the original issue:
—
Description of problem:

It looks like the ODC doesn't register KNATIVE_SERVING and KNATIVE_EVENTING flags. Those are based on KnativeServing and KnativeEventing CRs, but they are looking for v1alpha1 version of those: https://github.com/openshift/console/blob/f72519fdf2267ad91cc0aa51467113cc36423a49/frontend/packages/knative-plugin/console-extensions.json#L6-L8
This PR https://github.com/openshift-knative/serverless-operator/pull/1695 moved the CRs to v1beta1, and that breaks that ODC discovery.

Version-Release number of selected component (if applicable):

Openshift 4.8, Serverless Operator 1.27

Additional info:

https://coreos.slack.com/archives/CHGU4P8UU/p1671634903447019

• https://github.com/openshift/console/pull/12441

This is a clone of issue OCPBUGS-7409. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-7374. The following is the description of the original issue:
—
Originally reported by lance5890 in issue https://github.com/openshift/cluster-etcd-operator/issues/1000

The controllers sometimes get stuck on listing members in failure scenarios, this is known and can be mitigated by simply restarting the CEO. 

similar BZ 2093819 with stuck controllers was fixed slightly different in https://github.com/openshift/cluster-etcd-operator/commit/4816fab709e11e0681b760003be3f1de12c9c103

This fix was contributed by lance5890, thanks a lot!

• https://github.com/openshift/cluster-etcd-operator/pull/1010

• https://github.com/openshift/origin/pull/27872

This is a clone of issue OCPBUGS-4960. The following is the description of the original issue:
—
Description of problem:
This is a follow up on OCPBUGSM-47202 (https://bugzilla.redhat.com/show_bug.cgi?id=2110570)

While OCPBUGSM-47202 fixes the issue specific for Set Pod Count, many other actions aren't fixed. When the user updates a Deployment with one of this options, and selects the action again, the old values are still shown.

Version-Release number of selected component (if applicable)
4.8-4.12 as well as master with the changes of OCPBUGSM-47202

How reproducible:
Always

Steps to Reproduce:

1. Import a deployment
2. Select the deployment to open the topology sidebar
3. Click on actions and one of the 4 options to update the deployment with a modal
   1. Edit labels
   2. Edit annotatations
   3. Edit update strategy
   4. Edit resource limits
4. Click on the action again and check if the data in the modal reflects the changes from step 3

Actual results:
Old data (labels, annotations, etc.) was shown.

Expected results:
Latest data should be shown

Additional info:

• https://github.com/openshift/console/pull/12533

This is a clone of issue OCPBUGS-11182. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-8497. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-8442. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-8437. The following is the description of the original issue:
—
Description of problem:

Jenkins images based on rhel8 are wrongly tagged with rhel7

Version-Release number of selected component (if applicable):

How reproducible:

Always

Steps to Reproduce:

1.Check the OS version.
$ podman run -it --rm --entrypoint bash quay.io/openshift/origin-jenkins-agent-base:latest
bash-4.4# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.6 (Ootpa)

2.Check the image labels.
$ podman inspect quay.io/openshift/origin-jenkins-agent-base:latest | grep rhel7
                    "com.redhat.component": "jenkins-slave-base-rhel7-container",
                    "name": "openshift4/jenkins-slave-base-rhel7",
               "com.redhat.component": "jenkins-slave-base-rhel7-container",
               "name": "openshift4/jenkins-slave-base-rhel7",

Actual results:

OS is rhel8, but labels are rhel7

Expected results:

Labels should match the OS version

Additional info:

• https://github.com/openshift/jenkins/pull/1643

• https://github.com/openshift/machine-config-operator/pull/3436

Acceptance criteria:

• All tests (including e2e) pass
• No regressions are introduced
• openshift/api points to a recent commit on the master branch

• https://github.com/openshift/cluster-image-registry-operator/pull/728

Description of problem:

  intra namespace allow network policy doesn't work after applying ingress&egress deny all network policy

Version-Release number of selected component (if applicable):

  OpenShift 4.10.12

How reproducible:

Always

Steps to Reproduce:
  1. Define deny all network policy for egress an ingress in a namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

2. Define the following network policy to allow the traffic between the pods in the namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-intra-namespace-001
spec:
  egress:
  - to:
    - podSelector: {}
  ingress:
  - from:
    - podSelector: {}
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress 

3. Test the connectivity between two pods from the namespace.

Actual results:

   The connectivity is not allowed

Expected results:

  The connectivity should be allowed between pods from the same namespace.

Additional info:

  After performing a test and analyzing SDN flows for the namespace: 

sh-4.4# ovs-ofctl dump-flows -O OpenFlow13 br0 | grep --color 0x964376 
 cookie=0x0, duration=99375.342s, table=20, n_packets=14, n_bytes=588, priority=100,arp,in_port=21,arp_spa=10.128.2.20,arp_sha=00:00:0a:80:02:14/00:00:ff:ff:ff:ff actions=load:0x964376->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=1681.845s, table=20, n_packets=11, n_bytes=462, priority=100,arp,in_port=24,arp_spa=10.128.2.23,arp_sha=00:00:0a:80:02:17/00:00:ff:ff:ff:ff actions=load:0x964376->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=99375.342s, table=20, n_packets=135610, n_bytes=759239814, priority=100,ip,in_port=21,nw_src=10.128.2.20 actions=load:0x964376->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=1681.845s, table=20, n_packets=2006, n_bytes=12684967, priority=100,ip,in_port=24,nw_src=10.128.2.23 actions=load:0x964376->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=99375.342s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.128.2.20 actions=load:0x964376->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=1681.845s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.128.2.23 actions=load:0x964376->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=975.129s, table=27, n_packets=0, n_bytes=0, priority=150,reg0=0x964376,reg1=0x964376 actions=goto_table:30
 cookie=0x0, duration=99375.342s, table=70, n_packets=145260, n_bytes=11722173, priority=100,ip,nw_dst=10.128.2.20 actions=load:0x964376->NXM_NX_REG1[],load:0x15->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=1681.845s, table=70, n_packets=2336, n_bytes=191079, priority=100,ip,nw_dst=10.128.2.23 actions=load:0x964376->NXM_NX_REG1[],load:0x18->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=975.129s, table=80, n_packets=0, n_bytes=0, priority=150,reg0=0x964376,reg1=0x964376 actions=output:NXM_NX_REG2[]

We see that the following rule doesn't match because `reg1` hasn't been defined:

 cookie=0x0, duration=975.129s, table=27, n_packets=0, n_bytes=0, priority=150,reg0=0x964376,reg1=0x964376 actions=goto_table:30 

• https://github.com/openshift/sdn/pull/465

• https://github.com/openshift/ovn-kubernetes/pull/1492

• https://github.com/openshift/cluster-autoscaler-operator/pull/226

A bug prevents the package server cert from being rotated. This was fixed for 4.11 with the release of RHSA-2022:5069 but not fixed in 4.10 or earlier. See bz 2020484 or OCPBUGSM-36848 for details.

• https://github.com/openshift/operator-framework-olm/pull/428

This is a clone of issue OCPBUGS-5766. The following is the description of the original issue:
—
Description of problem:

Data race seen in unit tests:
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_ovn-kubernetes/1448/pull-ci-openshift-ovn-kubernetes-release-4.11-unit/1604898712423763968/artifacts/test/build-log.txt

• https://github.com/openshift/ovn-kubernetes/pull/1480

• https://github.com/openshift/machine-api-provider-azure/pull/34

• https://github.com/openshift/sdn/pull/472

Update blueocean-autofavorite to version 1.2.5, which does not bundle the groovy-sandbox library

• https://github.com/openshift/jenkins/pull/1478

Description of problem:

Currently when installing Openshift on the Openstack cluster name length limit is allowed to  14 characters.
Customer wants to know if is it possible to override this validation when installing Openshift on Openstack and create a cluster name that is greater than 14 characters.

Version : OCP 4.8.5 UPI Disconnected 
Environment : Openstack 16 

Issue:
User reports that they are getting error for OCP cluster in Openstack UPI, where the name of the cluster is > 14 characters.

Error events :
~~~
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/usr/local/bin/openshift-install", "create", "manifests", "--dir=/home/gitlab-runner/builds/WK8mkokN/0/CPE/SKS/pipelines/non-prod/ocp4-openstack-build/ocpinstaller/install-upi"], "delta": "0:00:00.311397", "end": "2022-09-03 21:38:41.974608", "msg": "non-zero return code", "rc": 1, "start": "2022-09-03 21:38:41.663211", "stderr": "level=fatal msg=failed to fetch Master Machines: failed to load asset \"Install Config\": invalid \"install-config.yaml\" file: metadata.name: Invalid value: \"sks-osp-inf-cpe-1-cbr1a\": cluster name is too long, please restrict it to 14 characters", "stderr_lines": ["level=fatal msg=failed to fetch Master Machines: failed to load asset \"Install Config\": invalid \"install-config.yaml\" file: metadata.name: Invalid value: \"sks-osp-inf-cpe-1-cbr1a\": cluster name is too long, please restrict it to 14 characters"], "stdout": "", "stdout_lines": []}
~~~

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Users are getting error "cluster name is too long" when clustername contains more than 14 characters for OCP on Openstack

Expected results:

The 14 characters limit should be change for the OCP clustername on Openstack

Additional info:

• https://github.com/openshift/installer/pull/6420

Our Prometheus alerts are inconsistent with both upstream and sometimes our own vendor folder. Let's do a clean update run before the next release is branched off.

• https://github.com/openshift/cluster-etcd-operator/pull/956

This is a clone of issue OCPBUGS-2451. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-2181. The following is the description of the original issue:
—
Description of problem:

E2E test Installs Red Hat Integration - 3scale operator test is failing due to change of Operator name

CI Search: https://search.ci.openshift.org/?search=Installs+Red+Hat+Integration+-+3scale+operator+in+test+namespace+and+creates+3scale+Backend+Schema+operand+instance&maxAge=24h&context=1&type=bug%2Bissue%2Bjunit&name=pull-ci-openshift-console-master-e2e-gcp-console&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

• https://github.com/openshift/console/pull/12184

• https://github.com/openshift/cluster-storage-operator/pull/311

This is a clone of issue OCPBUGS-4137. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-3824. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-2598. The following is the description of the original issue:
—
Description of problem:

Liveness probe of ipsec pods fail with large clusters. Currently the command that is executed in the ipsec container is
ovs-appctl -t ovs-monitor-ipsec ipsec/status && ipsec status
The problem is with command "ipsec/status". In clusters with high node count this command will return a list with all the node daemons of the cluster. This means that as the node count raises the completion time of the command raises too. 

This makes the main command 

ovs-appctl -t ovs-monitor-ipsec

To hang until the subcommand is finished.

As the liveness and readiness probe values are hardcoded in the manifest of the ipsec container herehttps//github.com/openshift/cluster-network-operator/blob/9c1181e34316d34db49d573698d2779b008bcc20/bindata/network/ovn-kubernetes/common/ipsec.yaml] the liveness timeout of the container probe of 60 seconds start to be  insufficient as the node count list is growing. This resulted in a cluster with 170 + nodes to have 15+ ipsec pods in a crashloopbackoff state.

Version-Release number of selected component (if applicable):

Openshift Container Platform 4.10 but i think the same will be visible to other versions too.

How reproducible:

I was not able to reproduce due to an extreamely high amount of resources are needed and i think that there is no point as we have spotted the issue.

Steps to Reproduce:

1. Install an Openshift cluster with IPSEC enabled
2. Scale to 170+ nodes or more
3. Notice that the ipsec pods will start getting in a Crashloopbackoff state with failed Liveness/Readiness probes.

Actual results:

Ip Sec pods are stuck in a Crashloopbackoff state

Expected results:

Ip Sec pods to work normally

Additional info:

We have provided a workaround where CVO and CNO operators are scaled to 0 replicas in order for us to be able to increase the liveness probe limit to a value of 600 that recovered the cluster. 
As a next step the customer will try to reduce the node count and restore the default liveness timeout value along with bringing the operators back to see if the cluster will stabilize.

• https://github.com/openshift/cluster-network-operator/pull/1651

Description of problem:

With every pod update we are executing a mutate operation to add the pod port to the port group or add the pod IP to an address set. This functionally doesn't hurt, since mutate will not add duplicate values to the same set. However, this is bad for performance. For example, with a 730 network policies affecting a pod, and issuing 7 pod updates would result in over 5k transactions.

• https://github.com/openshift/ovn-kubernetes/pull/1452

• https://github.com/openshift/machine-config-operator/pull/3500

This is a clone of issue OCPBUGS-536. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-268. The following is the description of the original issue:
—
The linux kernel was updated:
https://lkml.org/lkml/2020/3/20/1030
to include steal

accounting

This would greatly assist in troubleshooting vSphere performance issues
caused by over-provisioned ESXi hosts.

• https://github.com/openshift/machine-api-operator/pull/1143

Description of problem:

Whereabouts doesn't allow the use of network interface names that are not preceded by the prefix "net", see https://github.com/k8snetworkplumbingwg/whereabouts/issues/130.

Version-Release number of selected component (if applicable):

How reproducible:

Always

Steps to Reproduce:

1. Define two Pods, one with the interface name 'port1' and the other with 'net-port1':

test-ip-removal-port1:	
              k8s.v1.cni.cncf.io/networks:
                [
                  {
                    "name": "test-sriovnd",
                    "interface": "port1",
                    "namespace": "default"
                  }
                ]

test-ip-removal-net-port1:
              k8s.v1.cni.cncf.io/networks:
                [
                  {
                    "name": "test-sriovnd",
                    "interface": "net-port1",
                    "namespace": "default"
                  }
                ]

2. IP allocated in the IPPool:

kind: IPPool
...
spec:
  allocations:
    "16":
      id: ...
      podref: test-ecoloma-1/test-ip-removal-port1
    "17":
      id: ...
      podref: test-ecoloma-1/test-ip-removal-net-port1

3. When the ip-reconciler job is run, the allocation for the port with the interface name 'port1' is removed:

[13:29][]$ oc get cronjob -n openshift-multus
NAME            SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
ip-reconciler   */15 * * * *   False     0        14m             11d

[13:29][]$ oc get ippools.whereabouts.cni.cncf.io -n openshift-multus   2001-1b70-820d-2610---64 -o yaml
apiVersion: whereabouts.cni.cncf.io/v1alpha1
kind: IPPool
metadata:
...
spec:
  allocations:
    "17":
      id: ...
      podref: test-ecoloma-1/test-ip-removal-net-port1
  range: 2001:1b70:820d:2610::/64

[13:30][]$ oc get cronjob -n openshift-multus
NAME            SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
ip-reconciler   */15 * * * *   False     0        9s              11d

Actual results:

The network interface with a name that doesn't have a 'net' prefix is removed from the ip-reconciler cronjob.

Expected results:

The network interface must not be removed, regardless of the name.

Additional info:

Upstream PR @ https://github.com/k8snetworkplumbingwg/whereabouts/pull/147 master PR @ https://github.com/openshift/whereabouts-cni/pull/94

• https://github.com/openshift/whereabouts-cni/pull/98

This is a manual clone of https://bugzilla.redhat.com/show_bug.cgi?id=2093597 to backport this to 4.10.

Description of problem:
When importing a component from git or from a container image, and open one or more advanced options, the sentence "Click on the names to access advanced options for ..." is splited into two parts. And the headlines have no padding and everything looks squashed.

Version-Release number of selected component (if applicable):
4.10+

How reproducible:
Always

Steps to Reproduce:
1. Switch to dev perspective
2. Navigate to the add page > Import from container
3. Scroll down and open one or more of the advanced options

Actual results:
1. The sentence "Click on the names to access advanced options for ..." is shown before the opened option. The other available options are shown below the selected option.
2. The headline is displayed directly below "Click on the names to access advanced options for"
3. Another section is also shown directly under the first one.

Expected results:
1. The sentence "Click on the names to access advanced options for ..." and the options should be "one sentence" again.
2+3. Some padding for the header and/or between the sections, similar to 4.9. It must not look exactly as in 4.9, but there should be some padding between independent sections.

Additional info:
none

• https://github.com/openshift/console/pull/12204

Description of problem:

The IPI installation in some regions got bootstrap failure, and without any node available/ready.

Version-Release number of selected component (if applicable):

12-22 16:22:27.970  ./openshift-install 4.12.0-0.nightly-2022-12-21-202045
12-22 16:22:27.970  built from commit 3f9c38a5717c638f952df82349c45c7d6964fcd9
12-22 16:22:27.970  release image registry.ci.openshift.org/ocp/release@sha256:2d910488f25e2638b6d61cda2fb2ca5de06eee5882c0b77e6ed08aa7fe680270
12-22 16:22:27.971  release architecture amd64

How reproducible:

Always

Steps to Reproduce:

1. try the IPI installation in the problem regions (so far tried and failed with ap-southeast-2, ap-south-1, eu-west-1, ap-southeast-6, ap-southeast-3, ap-southeast-5, eu-central-1, cn-shanghai, cn-hangzhou and cn-beijing) 

Actual results:

Bootstrap failed to complete

Expected results:

Installation in those regions should succeed.

Additional info:

FYI the QE flexy-install job: https://mastern-jenkins-csb-openshift-qe.apps.ocp-c1.prod.psi.redhat.com/job/ocp-common/job/Flexy-install/166672/

No any node available/ready, and no any operator available.
$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version             False       True          30m     Unable to apply 4.12.0-0.nightly-2022-12-21-202045: an unknown error has occurred: MultipleErrors
$ oc get nodes
No resources found
$ oc get machines -n openshift-machine-api -o wide
NAME                         PHASE   TYPE   REGION   ZONE   AGE   NODE   PROVIDERID   STATE
jiwei-1222f-v729x-master-0                                  30m                       
jiwei-1222f-v729x-master-1                                  30m                       
jiwei-1222f-v729x-master-2                                  30m                       
$ oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication
baremetal
cloud-controller-manager                                                                          
cloud-credential                                                                                  
cluster-autoscaler                                                                                
config-operator                                                                                   
console                                                                                           
control-plane-machine-set                                                                         
csi-snapshot-controller                                                                           
dns                                                                                               
etcd                                                                                              
image-registry                                                                                    
ingress                                                                                           
insights                                                                                          
kube-apiserver                                                                                    
kube-controller-manager                                                                           
kube-scheduler                                                                                    
kube-storage-version-migrator                                                                     
machine-api                                                                                       
machine-approver                                                                                  
machine-config                                                                                    
marketplace                                                                                       
monitoring                                                                                        
network                                                                                           
node-tuning                                                                                       
openshift-apiserver                                                                               
openshift-controller-manager                                                                      
openshift-samples                                                                                 
operator-lifecycle-manager                                                                        
operator-lifecycle-manager-catalog                                                                
operator-lifecycle-manager-packageserver
service-ca
storage
$

Mater nodes don't run for example kubelet and crio services.
[core@jiwei-1222f-v729x-master-0 ~]$ sudo crictl ps
FATA[0000] unable to determine runtime API version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/crio/crio.sock: connect: no such file or directory" 
[core@jiwei-1222f-v729x-master-0 ~]$ 

The machine-config-daemon firstboot tells "failed to update OS".
[jiwei@jiwei log-bundle-20221222085846]$ grep -Ei 'error|failed' control-plane/10.0.187.123/journals/journal.log 
Dec 22 16:24:16 localhost kernel: GPT: Use GNU Parted to correct GPT errors.
Dec 22 16:24:16 localhost kernel: GPT: Use GNU Parted to correct GPT errors.
Dec 22 16:24:18 localhost ignition[867]: failed to fetch config: resource requires networking
Dec 22 16:24:18 localhost ignition[891]: GET error: Get "http://100.100.100.200/latest/user-data": dial tcp 100.100.100.200:80: connect: network is unreachable
Dec 22 16:24:18 localhost ignition[891]: GET error: Get "http://100.100.100.200/latest/user-data": dial tcp 100.100.100.200:80: connect: network is unreachable
Dec 22 16:24:19 localhost.localdomain NetworkManager[919]: <info>  [1671726259.0329] hostname: hostname: hostnamed not used as proxy creation failed with: Could not connect: No such file or directory
Dec 22 16:24:19 localhost.localdomain NetworkManager[919]: <warn>  [1671726259.0464] sleep-monitor-sd: failed to acquire D-Bus proxy: Could not connect: No such file or directory
Dec 22 16:24:19 localhost.localdomain ignition[891]: GET error: Get "https://api-int.jiwei-1222f.alicloud-qe.devcluster.openshift.com:22623/config/master": dial tcp 10.0.187.120:22623: connect: connection refused
...repeated logs omitted...
Dec 22 16:27:46 jiwei-1222f-v729x-master-0 ovs-ctl[1888]: 2022-12-22T16:27:46Z|00001|dns_resolve|WARN|Failed to read /etc/resolv.conf: No such file or directory
Dec 22 16:27:46 jiwei-1222f-v729x-master-0 ovs-vswitchd[1888]: ovs|00001|dns_resolve|WARN|Failed to read /etc/resolv.conf: No such file or directory
Dec 22 16:27:46 jiwei-1222f-v729x-master-0 dbus-daemon[1669]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.resolve1.service': Unit dbus-org.freedesktop.resolve1.service not found.
Dec 22 16:27:46 jiwei-1222f-v729x-master-0 nm-dispatcher[1924]: Error: Device '' not found.
Dec 22 16:27:46 jiwei-1222f-v729x-master-0 nm-dispatcher[1937]: Error: Device '' not found.
Dec 22 16:27:46 jiwei-1222f-v729x-master-0 nm-dispatcher[2037]: Error: Device '' not found.
Dec 22 08:35:32 jiwei-1222f-v729x-master-0 machine-config-daemon[2181]: Warning: failed, retrying in 1s ... (1/2)I1222 08:35:32.477770    2181 run.go:19] Running: nice -- ionice -c 3 oc image extract --path /:/run/mco-extensions/os-extensions-content-910221290 --registry-config /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:259d8c6b9ec714d53f0275db9f2962769f703d4d395afb9d902e22cfe96021b0
Dec 22 08:56:06 jiwei-1222f-v729x-master-0 rpm-ostree[2288]: Txn Rebase on /org/projectatomic/rpmostree1/rhcos failed: remote error: Get "https://quay.io/v2/openshift-release-dev/ocp-v4.0-art-dev/blobs/sha256:27f262e70d98996165748f4ab50248671d4a4f97eb67465cd46e1de2d6bd24d0": net/http: TLS handshake timeout
Dec 22 08:56:06 jiwei-1222f-v729x-master-0 machine-config-daemon[2181]: W1222 08:56:06.785425    2181 firstboot_complete_machineconfig.go:46] error: failed to update OS to quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:411e6e3be017538859cfbd7b5cd57fc87e5fee58f15df19ed3ec11044ebca511 : error running rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:411e6e3be017538859cfbd7b5cd57fc87e5fee58f15df19ed3ec11044ebca511: Warning: The unit file, source configuration file or drop-ins of rpm-ostreed.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Dec 22 08:56:06 jiwei-1222f-v729x-master-0 machine-config-daemon[2181]: error: remote error: Get "https://quay.io/v2/openshift-release-dev/ocp-v4.0-art-dev/blobs/sha256:27f262e70d98996165748f4ab50248671d4a4f97eb67465cd46e1de2d6bd24d0": net/http: TLS handshake timeout
Dec 22 08:57:31 jiwei-1222f-v729x-master-0 machine-config-daemon[2181]: Warning: failed, retrying in 1s ... (1/2)I1222 08:57:31.244684    2181 run.go:19] Running: nice -- ionice -c 3 oc image extract --path /:/run/mco-extensions/os-extensions-content-4021566291 --registry-config /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:259d8c6b9ec714d53f0275db9f2962769f703d4d395afb9d902e22cfe96021b0
Dec 22 08:59:20 jiwei-1222f-v729x-master-0 systemd[2353]: /usr/lib/systemd/user/podman-kube@.service:10: Failed to parse service restart specifier, ignoring: never
Dec 22 08:59:21 jiwei-1222f-v729x-master-0 podman[2437]: Error: open default: no such file or directory
Dec 22 08:59:21 jiwei-1222f-v729x-master-0 podman[2450]: Error: failed to start API service: accept unixgram @00026: accept4: operation not supported
Dec 22 08:59:21 jiwei-1222f-v729x-master-0 systemd[2353]: podman-kube@default.service: Failed with result 'exit-code'.
Dec 22 08:59:21 jiwei-1222f-v729x-master-0 systemd[2353]: Failed to start A template for running K8s workloads via podman-play-kube.
Dec 22 08:59:21 jiwei-1222f-v729x-master-0 systemd[2353]: podman.service: Failed with result 'exit-code'.
[jiwei@jiwei log-bundle-20221222085846]$ 

• https://github.com/openshift/installer/pull/7130

Description of problem:

Jenkins install-plugins.sh script does not ignore update requests for locked versions of plugins, and does not verify that the locked version was actually included in the bundle-plugins.txt file.

Version-Release number of selected component (if applicable):

How reproducible:

Run make plugins-list

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/jenkins/pull/1484

• https://github.com/openshift/console/pull/12110

Currently, Telemeter is not equipped with configurable request limit for receive endpoint (for full context see: https://github.com/openshift/cluster-monitoring-operator/pull/1416). It is using the default limit defined in the code base, however it seems this limit might not be suitable for our usage.

As a part of this ticket, it should be:

1) Understood what is the appropriate limit for request size for our use cases

2) Make the limit configurable in Telemeter via a flag

3) Deploy the changes, initially to the staging environment, to enable our team to test it.

• https://github.com/openshift/telemeter/pull/392
• https://github.com/openshift/telemeter/pull/391

• https://github.com/openshift/sdn/pull/524

The kube-state-metric pod inside the openshift-monitoring namespace is not running as expected.

On checking the logs I am able to see that there is a memory panic

~~~
2022-11-22T09:57:17.901790234Z I1122 09:57:17.901768 1 main.go:199] Starting kube-state-metrics self metrics server: 127.0.0.1:8082
2022-11-22T09:57:17.901975837Z I1122 09:57:17.901951 1 main.go:66] levelinfomsgTLS is disabled.http2false
2022-11-22T09:57:17.902389844Z I1122 09:57:17.902291 1 main.go:210] Starting metrics server: 127.0.0.1:8081
2022-11-22T09:57:17.903191857Z I1122 09:57:17.903133 1 main.go:66] levelinfomsgTLS is disabled.http2false
2022-11-22T09:57:17.906272505Z I1122 09:57:17.906224 1 builder.go:191] Active resources: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments
2022-11-22T09:57:17.917758187Z E1122 09:57:17.917560 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
2022-11-22T09:57:17.917758187Z goroutine 24 [running]:
2022-11-22T09:57:17.917758187Z k8s.io/apimachinery/pkg/util/runtime.logPanic(

)
2022-11-22T09:57:17.917758187Z /usr/lib/golang/src/runtime/panic.go:1038 +0x215
2022-11-22T09:57:17.917758187Z k8s.io/kube-state-metrics/v2/internal/store.ingressMetricFamilies.func6(0x40)
2022-11-22T09:57:17.917758187Z /go/src/k8s.io/kube-state-metrics/internal/store/ingress.go:136 +0x189
2022-11-22T09:57:17.917758187Z k8s.io/kube-state-metrics/v2/internal/store.wrapIngressFunc.func1(

)
2022-11-22T09:57:17.917758187Z /go/src/k8s.io/kube-state-metrics/pkg/metric_generator/generator.go:107 +0xd8
~~~

Logs are attached to the support case

• https://github.com/openshift/kube-state-metrics/pull/86

This is a clone of issue OCPBUGS-5206. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-4897. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-2500. The following is the description of the original issue:
—
Description of problem:

When the Ux switches to the Dev console the topology is always blank in a Project that has a large number of components.

Version-Release number of selected component (if applicable):

How reproducible:

Always occurs

Steps to Reproduce:

1.Create a project with at least 12 components (Apps, Operators, knative Brokers)
2. Go to the Administrator Viewpoint
3. Switch to Developer Viewpoint/Topology
4. No components displayed
5. Click on 'fit to screen'
6. All components appear

Actual results:

Topology renders with all controls but no components visible (see screenshot 1)

Expected results:

All components should be visible

Additional info:

• https://github.com/openshift/console/pull/12485

Description of problem:

Description of problem:
The startup script for ovnkube-master container starts ovn-nbctl daemon that logs to ovn-nbctl.log file[1]. However, apart from tailing it, nothing is done with that file. Concretely, it is never rotated.
Recently, one 4.8 cluster has hit the situation where this file grew up to ~24GB size. It was necessary to manually remove that file and restart ovnkube-master.
Version-Release number of selected component (if applicable):
4.8. It seems that 4.10 can hit this problem as well[2] but latest master branch has changed so much that I am not sure if it will be possible for this to happen in 4.11.
How reproducible:
Always (but may take very long and requires no ovnkube-master container restart)
Steps to Reproduce:
1. Let /run/ovn/ovn-nbctl.log grow for long enough
Actual results:
File growing without control
Expected results:
Log file to be eventually rotated.
References:
[1] - https://github.com/openshift/cluster-network-operator/blob/release-4.8/bindata/network/ovn-kubernetes/ovnkube-master.yaml#L813
[2] - https://github.com/openshift/cluster-network-operator/blob/release-4.10/bindata/network/ovn-kubernetes/ovnkube-master.yaml#L750

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/cluster-network-operator/pull/1587

This is a clone of issue OCPBUGS-7127. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-5016. The following is the description of the original issue:
—
Description of problem:

When editing any pipeline in the openshift console, the correct content cannot be obtained (the obtained information is the initial information).

Version-Release number of selected component (if applicable):

How reproducible:

100%

Steps to Reproduce:

Developer -> Pipeline -> select pipeline -> Details -> Actions -> Edit Pipeline -> YAML view -> Cancel ->  Actions -> Edit Pipeline -> YAML view 

Actual results:

displayed content is incorrect.

Expected results:

Get the content of the current pipeline, not the "pipeline create" content.

Additional info:

If cancel or save in the "Pipeline Builder" interface after "Edit Pipeline", can get the expected content.
～
Developer -> Pipeline -> select pipeline -> Details -> Actions -> Edit Pipeline -> Pipeline builder -> Cancel ->  Actions -> Edit Pipeline -> YAML view :Display resource content normally
～

• https://github.com/openshift/console/pull/12576

The static authorizer feature has landed in upstream kube-rbac-proxy. Lets use it by configuring a static authorizer for all requests that hit a /metrics endpoint.

DoD:

• Downstream kube-rbac-proxy is synced.
• All CMO operands are configured with static authorization.
• Bugzillas created for all non-monitoring components using kube-rbac-proxy for metrics authn/authz.

• https://github.com/openshift/cluster-monitoring-operator/pull/1318

This is a clone of issue OCPBUGS-2508. The following is the description of the original issue:
—
Description of problem:

Installer fails due to Neutron policy error when creating Openstack servers for OCP master nodes.

$ oc get machines -A
NAMESPACE               NAME                          PHASE          TYPE   REGION   ZONE   AGE
openshift-machine-api   ostest-kwtf8-master-0         Running                               23h
openshift-machine-api   ostest-kwtf8-master-1         Running                               23h
openshift-machine-api   ostest-kwtf8-master-2         Running                               23h
openshift-machine-api   ostest-kwtf8-worker-0-g7nrw   Provisioning                          23h
openshift-machine-api   ostest-kwtf8-worker-0-lrkvb   Provisioning                          23h
openshift-machine-api   ostest-kwtf8-worker-0-vwrsk   Provisioning                          23h

$ oc -n openshift-machine-api logs machine-api-controllers-7454f5d65b-8fqx2 -c machine-controller
[...]
E1018 10:51:49.355143       1 controller.go:317] controller/machine_controller "msg"="Reconciler error" "error"="error creating Openstack instance: Failed to create port err: Request forbidden: [POST https://overcloud.redhat.local:13696/v2.0/ports], error message: {\"NeutronError\": {\"type\": \"PolicyNotAuthorized\", \"message\": \"(rule:create_port and (rule:create_port:allowed_address_pairs and (rule:create_port:allowed_address_pairs:ip_address and rule:create_port:allowed_address_pairs:ip_address))) is disallowed by policy\", \"detail\": \"\"}}" "name"="ostest-kwtf8-worker-0-lrkvb" "namespace"="openshift-machine-api"

Version-Release number of selected component (if applicable):

4.10.0-0.nightly-2022-10-14-023020

How reproducible:

Always

Steps to Reproduce:

1. Install 4.10 within provider networks (in primary or secondary interface)

Actual results:

Installation failure:
4.10.0-0.nightly-2022-10-14-023020: some cluster operators have not yet rolled out

Expected results:

Successful installation

Additional info:

Please find must-gather for installation on primary interface link here and for installation on secondary interface link here.

• https://github.com/openshift/cluster-api-provider-openstack/pull/249

This bug is a backport clone of [Bugzilla Bug 2081119](https://bugzilla.redhat.com/show_bug.cgi?id=2081119). The following is the description of the original bug:
—
Description of problem:

`oc explain` output of default overlaySize is 10G. The outdated doc confused the customer with default overlaySize. The description should be updated since the default was removed from the implementation of storage.conf.

Version-Release number of selected component (if applicable):

How reproducible:

Actual results:

$ oc explain containerRuntimeConfig.spec.containerRuntimeConfig.logLevel
KIND: ContainerRuntimeConfig
VERSION: machineconfiguration.openshift.io/v1

FIELD: logLevel <string>

DESCRIPTION:
logLevel specifies the verbosity of the logs based on the level it is set
to. Options are fatal, panic, error, warn, info, and debug.
[qiwan@qiwan ~]$ oc explain containerRuntimeConfig.spec.containerRuntimeConfig.overlaySize
KIND: ContainerRuntimeConfig
VERSION: machineconfiguration.openshift.io/v1

FIELD: overlaySize <string>

DESCRIPTION:
overlaySize specifies the maximum size of a container image. This flag can
be used to set quota on the size of container images. (default: 10GB).

Expected results:

$ oc explain containerRuntimeConfig.spec.containerRuntimeConfig.logLevel
KIND: ContainerRuntimeConfig
VERSION: machineconfiguration.openshift.io/v1

FIELD: logLevel <string>

DESCRIPTION:
logLevel specifies the verbosity of the logs based on the level it is set
to. Options are fatal, panic, error, warn, info, and debug.
[qiwan@qiwan ~]$ oc explain containerRuntimeConfig.spec.containerRuntimeConfig.overlaySize
KIND: ContainerRuntimeConfig
VERSION: machineconfiguration.openshift.io/v1

FIELD: overlaySize <string>

DESCRIPTION:
overlaySize specifies the maximum size of a container image. This flag can
be used to set quota on the size of container images.

Additional info:

• https://github.com/openshift/machine-config-operator/pull/3395

This is a clone of issue OCPBUGS-11163. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-10976. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-10934. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-10917. The following is the description of the original issue:
—
Description of problem:

Product security has set a required Jenkins version to 2.387.1 for June 6th, 2023

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/jenkins/pull/1642

• https://github.com/openshift/jenkins/pull/1642

This is a clone of issue OCPBUGS-1354. The following is the description of the original issue:
—
This was originally reported in BZ as https://bugzilla.redhat.com/show_bug.cgi?id=2046335

—

Description of problem:

The issue reported here https://bugzilla.redhat.com/show_bug.cgi?id=1954121 still occur (tested on OCP 4.8.11, the CU also verified that the issue can happen even with OpenShift 4.7.30, 4.8.17 and 4.9.11)

How reproducible:

Attach a NIC to a master node will trigger the issue

Steps to Reproduce:
1. Deploy an OCP cluster (I've tested it IPI on AWS)
2. Attach a second NIC to a running master node (in my case "ip-10-0-178-163.eu-central-1.compute.internal")

Actual results:

~~~
$ oc get node ip-10-0-178-163.eu-central-1.compute.internal -o json | jq ".status.addresses"
[

,

,

,

]

$ oc get co etcd
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
etcd 4.8.11 True False True 31h

$ oc get co etcd -o json | jq ".status.conditions[0]"

~~~

Expected results:

To have the certificate valid also for the second IP (the newly created one "10.0.187.247")

Additional info:

Deleting the following secrets seems to solve the issue:
~~~
$ oc get secret n openshift-etcd | grep kubernetes.io/tls | grep ^etcd
etcd-client kubernetes.io/tls 2 61s
etcd-peer-ip-10-0-132-49.eu-central-1.compute.internal kubernetes.io/tls 2 61s
etcd-peer-ip-10-0-178-163.eu-central-1.compute.internal kubernetes.io/tls 2 61s
etcd-peer-ip-10-0-202-187.eu-central-1.compute.internal kubernetes.io/tls 2 60s
etcd-serving-ip-10-0-132-49.eu-central-1.compute.internal kubernetes.io/tls 2 60s
etcd-serving-ip-10-0-178-163.eu-central-1.compute.internal kubernetes.io/tls 2 59s
etcd-serving-ip-10-0-202-187.eu-central-1.compute.internal kubernetes.io/tls 2 59s
etcd-serving-metrics-ip-10-0-132-49.eu-central-1.compute.internal kubernetes.io/tls 2 58s
etcd-serving-metrics-ip-10-0-178-163.eu-central-1.compute.internal kubernetes.io/tls 2 59s
etcd-serving-metrics-ip-10-0-202-187.eu-central-1.compute.internal kubernetes.io/tls 2 58s

$ oc get secret n openshift-etcd | grep kubernetes.io/tls | grep ^etcd | awk '

' | xargs -I {} oc delete secret {} -n openshift-etcd
secret "etcd-client" deleted
secret "etcd-peer-ip-10-0-132-49.eu-central-1.compute.internal" deleted
secret "etcd-peer-ip-10-0-178-163.eu-central-1.compute.internal" deleted
secret "etcd-peer-ip-10-0-202-187.eu-central-1.compute.internal" deleted
secret "etcd-serving-ip-10-0-132-49.eu-central-1.compute.internal" deleted
secret "etcd-serving-ip-10-0-178-163.eu-central-1.compute.internal" deleted
secret "etcd-serving-ip-10-0-202-187.eu-central-1.compute.internal" deleted
secret "etcd-serving-metrics-ip-10-0-132-49.eu-central-1.compute.internal" deleted
secret "etcd-serving-metrics-ip-10-0-178-163.eu-central-1.compute.internal" deleted
secret "etcd-serving-metrics-ip-10-0-202-187.eu-central-1.compute.internal" deleted

$ oc get co etcd -o json | jq ".status.conditions[0]"

~~~

• https://github.com/openshift/cluster-etcd-operator/pull/936

• https://github.com/openshift/vmware-vsphere-csi-driver-operator/pull/114

Description of problem:

[ovn] [ocp 4.10.z] Service `spec.externalTrafficPolicy` does not trigger rules update in ovnkube-node pod handlers on edit, even though it does successfully update the rules if deployed explicitly with that spec value set, or if you delete the handler pods for ovn (forces a refresh).

Version-Release number of selected component (if applicable):

observed in 4.10.32 and 4.10.40, tested on azure platform.

How reproducible:

every time

Steps to Reproduce:

1. Deploy a test pod with curlable resource in a test namespace

2. create a service from yaml exposing pod at internal clusterIP (example yaml provided by customer below)
~~~ 
apiVersion: v1
kind: Service
metadata:
  labels:
    run: test
  name: test
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    service.beta.kubernetes.io/azure-load-balancer-internal-subnet: paas1
spec:
  allocateLoadBalancerNodePorts: true
  externalTrafficPolicy: Cluster ##MODIFY THIS SPEC VALUE AND OBSERVE FAIL CONDITION
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    run: test
  sessionAffinity: None
  type: LoadBalancer
~~~

3. curl against service succeeds
4. edit service to change `spec.externalTrafficPolicy: local`
5. observe externalIP does not change, but healthz port updates
6. curl against same externalIP:port time out indefinitely, no response.

//workaround: 

delete service and redeploy with spec line set already to `local`, or delete ovnkube-node pod serving pod(s) to force refresh the local ruleset and allow traffic (curls subsequently will succeed).

Actual results:

spec change appears to update properly in the database but does not send a notification to update the ovnkube-node pod handlers (or similar) to allow traffic through once the externalTrafficPolicy spec value is changed.

Expected results:

spec change to service yaml should be immediately updated in DB AND update ovnkube-node handlers for same.

Additional info:

Attachments available and case number with specifics in next internal comment. 

• https://github.com/openshift/ovn-kubernetes/pull/1463

• https://github.com/openshift/csi-driver-manila-operator/pull/169

As per [1], the jsonnet code for managing thanos-ruler resources should reuse the upstream kube-thanos project.

[1] https://github.com/openshift/cluster-monitoring-operator/blob/399c84dbca596b611b0c30a0d2df63a5d2b0b8cc/jsonnet/components/thanos-ruler.libsonnet#L1

• https://github.com/openshift/cluster-monitoring-operator/pull/1478

Description of problem:
Previously we had a bug opened for "Reduce buildah log level for default build log level [NEEDINFO]":

https://bugzilla.redhat.com/show_bug.cgi?id=1996883

We suggested customer using secrets, however, customer confirmed that they are using secrets as per:
https://docs.openshift.com/container-platform/4.8/cicd/builds/creating-build-inputs.html#builds-input-secrets-configmaps_creating-build-inputs

But not able to use "--quiet" build argument since they are using openshift s2i config:

strategy:
type: Source
sourceStrategy:
from:
kind: ImageStreamTag
namespace: xxxxxxx
name: 's2i-xxxxxx-xxxxxxx:v1.0.0'

Actual results:
Under this setting, the secret as well as every other openshift secret are printed.

Expected results:
Sensitive information (ENV) should not appear in build logs

Additional info:
Maybe pass the --quiet option via the buildconfig fir s2i?

— Additional comment from taxu@redhat.com on 2022-07-01 03:42:46 UTC —

Hi Build team,

I see that the target release for this bug is 4.11.0

Please let us know where to add the LOG_LEVEL for s2i and/or passing "--quiet" options once the fix is deployed.

Kind regards,

Tao Xu

— Additional comment from aos-team-art-private@redhat.com on 2022-07-13 06:44:30 UTC —

Elliott changed bug status from MODIFIED to ON_QA.
This bug is expected to ship in the next 4.12 release.

— Additional comment from rdlugyhe@redhat.com on 2022-07-26 20:34:06 UTC —

Please approve the updated Doc Text.

— Additional comment from cdaley@redhat.com on 2022-07-26 20:49:05 UTC —

Approved

— Additional comment from jitsingh@redhat.com on 2022-07-27 04:48:18 UTC —

verified

— Additional comment from oarribas@redhat.com on 2022-08-15 12:11:23 UTC —

@Corey, I can see this BZ now verified for 4.12. Can this be backported to previous OCP versions?

— Additional comment from cdaley@redhat.com on 2022-08-15 13:57:02 UTC —

What version are you interested in getting it backported to?

— Additional comment from oarribas@redhat.com on 2022-08-15 15:46:10 UTC —

@Corey, I can see only 4.10 and 4.11 as "Full Support" in [1], so at least to those versions.

• https://github.com/openshift/builder/pull/310

• https://github.com/openshift/cluster-monitoring-operator/pull/1338

This is a clone of issue OCPBUGS-1786. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-1677. The following is the description of the original issue:
—
Description of problem:
pkg/devfile/sample_test.go fails after devfile registry was updated (https://github.com/devfile/registry/pull/126)

This issue is about updating our assertion so that the CI job runs successfully again. We might want to backport this as well.

OCPBUGS-1678 is about updating the code that the test should use a mock response instead of the latest registry content OR check some specific attributes instead of comparing the full JSON response.

Version-Release number of selected component (if applicable):
4.12

How reproducible:
Always

Steps to Reproduce:
1. Clone openshift/console
2. Run ./test-backend.sh

Actual results:
Unit tests fail

Expected results:
Unit tests should pass again

Additional info:

• https://github.com/openshift/console/pull/12112

Description of problem:

When adding new nodes to the existing cluster, the newly allocated node-subnet can be overlapped with the existing node.

Version-Release number of selected component (if applicable):

openshift 4.10.30

How reproducible:

It's quite hard to reproduce but  there is a possibility it can happen any time. 

Steps to Reproduce:

1. Create a OVN dual-stack cluster
2. add nodes to the existing cluster
3. check the allocated node subnet 

Actual results:

Some newly added nodes have the same node-subnet and ovn-k8s-mp0 IP as some existing nodes.

Expected results:

Should have duplicated node-subnet and ovn-k8s-mp0 IP

Additional info:

Additional info can be found at the case 03329155 and the must-gather attached(comment #1) 

% omg logs ovnkube-master-v8crc -n openshift-ovn-kubernetes -c ovnkube-master | grep '2022-09-30T06:42:50.857'
2022-09-30T06:42:50.857031565Z W0930 06:42:50.857020       1 master.go:1422] Did not find any logical switches with other-config
2022-09-30T06:42:50.857112441Z I0930 06:42:50.857099       1 master.go:1003] Allocated Subnets [10.131.0.0/23 fd02:0:0:4::/64] on Node worker01.ss1.samsung.local
2022-09-30T06:42:50.857122455Z I0930 06:42:50.857105       1 master.go:1003] Allocated Subnets [10.129.4.0/23 fd02:0:0:a::/64] on Node oam04.ss1.samsung.local
2022-09-30T06:42:50.857130289Z I0930 06:42:50.857122       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.131.0.0/23","fd02:0:0:4::/64"]}] on node worker01.ss1.samsung.local
2022-09-30T06:42:50.857140773Z I0930 06:42:50.857132       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.129.4.0/23","fd02:0:0:a::/64"]}] on node oam04.ss1.samsung.local
2022-09-30T06:42:50.857166726Z I0930 06:42:50.857156       1 master.go:1003] Allocated Subnets [10.128.2.0/23 fd02:0:0:5::/64] on Node oam01.ss1.samsung.local
2022-09-30T06:42:50.857176132Z I0930 06:42:50.857157       1 master.go:1003] Allocated Subnets [10.131.0.0/23 fd02:0:0:4::/64] on Node rhel01.ss1.samsung.local
2022-09-30T06:42:50.857176132Z I0930 06:42:50.857167       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.128.2.0/23","fd02:0:0:5::/64"]}] on node oam01.ss1.samsung.local
2022-09-30T06:42:50.857185257Z I0930 06:42:50.857157       1 master.go:1003] Allocated Subnets [10.128.6.0/23 fd02:0:0:d::/64] on Node call03.ss1.samsung.local
2022-09-30T06:42:50.857192996Z I0930 06:42:50.857183       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.131.0.0/23","fd02:0:0:4::/64"]}] on node rhel01.ss1.samsung.local
2022-09-30T06:42:50.857200017Z I0930 06:42:50.857190       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.128.6.0/23","fd02:0:0:d::/64"]}] on node call03.ss1.samsung.local
2022-09-30T06:42:50.857282717Z I0930 06:42:50.857258       1 master.go:1003] Allocated Subnets [10.130.2.0/23 fd02:0:0:7::/64] on Node call01.ss1.samsung.local
2022-09-30T06:42:50.857304886Z I0930 06:42:50.857293       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.130.2.0/23","fd02:0:0:7::/64"]}] on node call01.ss1.samsung.local
2022-09-30T06:42:50.857338896Z I0930 06:42:50.857314       1 master.go:1003] Allocated Subnets [10.128.4.0/23 fd02:0:0:9::/64] on Node f501.ss1.samsung.local
2022-09-30T06:42:50.857349485Z I0930 06:42:50.857329       1 master.go:1003] Allocated Subnets [10.131.2.0/23 fd02:0:0:8::/64] on Node call02.ss1.samsung.local
2022-09-30T06:42:50.857371344Z I0930 06:42:50.857354       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.128.4.0/23","fd02:0:0:9::/64"]}] on node f501.ss1.samsung.local
2022-09-30T06:42:50.857371344Z I0930 06:42:50.857361       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.131.2.0/23","fd02:0:0:8::/64"]}] on node call02.ss1.samsung.local

• https://github.com/openshift/ovn-kubernetes/pull/1332

This is a clone of issue OCPBUGS-6622. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-6018. The following is the description of the original issue:
—
This is a public clone of OCPBUGS-3821

The MCO can sometimes render a rendered-config in the middle of an upgrade with old MCs, e.g.:

1. the containerruntimeconfigcontroller creates a new containerruntimeconfig due to the update
2. the template controller finishes re-creating the base configs
3. the kubeletconfig errors long enough and doesn't finish until after 2

This will cause the render controller to create a new rendered MC that uses the OLD kubeletconfig-MC, which at best is a double reboot for 1 node, and at worst block the update and break maxUnavailable nodes per pool.

• https://github.com/openshift/machine-config-operator/pull/3517

This is a clone of issue OCPBUGS-13792. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-13739. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-13692. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-13549. The following is the description of the original issue:
—
Description of problem:

Incorrect AWS ARN [1] is used for GovCloud and AWS China regions, which will cause the command `ccoctl aws create-all` to fail:

Failed to create Identity provider: failed to apply public access policy to the bucket ci-op-bb5dgq54-77753-oidc: MalformedPolicy: Policy has invalid resource
	status code: 400, request id: VNBZ3NYDH6YXWFZ3, host id: pHF8v7C3vr9YJdD9HWamFmRbMaOPRbHSNIDaXUuUyrgy0gKCO9DDFU/Xy8ZPmY2LCjfLQnUDmtQ=

Correct AWS ARN prefix:
GovCloud (us-gov-east-1 and us-gov-west-1): arn:aws-us-gov
AWS China (cn-north-1 and cn-northwest-1): arn:aws-cn

[1] https://github.com/openshift/cloud-credential-operator/pull/526/files#diff-1909afc64595b92551779d9be99de733f8b694cfb6e599e49454b380afc58876R211


 

Version-Release number of selected component (if applicable):

4.12.0-0.nightly-2023-05-11-024616

How reproducible:

Always
 

Steps to Reproduce:

1. Run command: `aws create-all --name="${infra_name}" --region="${REGION}" --credentials-requests-dir="/tmp/credrequests" --output-dir="/tmp"` on GovCloud regions
2.
3.

Actual results:

Failed to create Identity provider
 

Expected results:

Create resources successfully.
 

Additional info:

Related PRs:
4.10: https://github.com/openshift/cloud-credential-operator/pull/531
4.11: https://github.com/openshift/cloud-credential-operator/pull/530
4.12: https://github.com/openshift/cloud-credential-operator/pull/529
4.13: https://github.com/openshift/cloud-credential-operator/pull/528
4.14: https://github.com/openshift/cloud-credential-operator/pull/526
 

• https://github.com/openshift/cloud-credential-operator/pull/541

This is a clone of issue OCPBUGS-1890. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-1765. The following is the description of the original issue:
—
Description of problem:

If a customer creates a machine with a networks section like this

networks:
- filter: {}
  noAllowedAddressPairs: false
  subnets:
  - filter: {}
    uuid: primary-subnet-uuid
- filter: {}
  noAllowedAddressPairs: true
  subnets:
  - filter: {}
    uuid: other-subnet-uuid
primarySubnet: primary-subnet-uuid

Then all the ports are created without the allowed address pairs.

Doing some research in the source code, I have found that:
- For each entry on the networks: section, networks are filtered as per its filter: section[1]
- Then, if the subnets: section of the network entry is not empty, for each of the network IDs found above[2], 2 things are done that are relevant for this situatoin:
  - The net ID is saved on a netsWithoutAllowedAddressPairs[3]. That map is later checked while creating any port[4].
  - For each subnet entry that matches the network ID, a port is created[5].

So, the problematic behavior happens due to the following:

- Both entries in the networks array have empty filters. This means that both entries selected all the neutron networks.
- This configuration results in one port per subnet as expected because, in the later traversal of the subnets array of each entry[5], it is filtering by subnet and creating a single port as expected.
- However, the entry with "noAllowedAddressPairs: true" is selecting all the neutron networks, so it adds all of them to the netsWithoutAllowedAddressPairs map[3], regardless of the subnets filtering.
- As all the networks are in noAllowedAddressPairs: true array, all the ports created for the VM have their allowed address pairs removed[4].

Why do we consider this behavior undesired?

I understand that, if we create a port for a network that has no allowed pairs, we create all the other ports in the same networks without the pairs. However, it is surprising that a port in a network is removed the allowed address pairs due to a setting in an entry that yielded no port on that network. In other words, one would expect that the same subnet filtering that happens on each network entry in what regards yielding ports for the VM would also work for the noAllowedPairs parameter.

Version-Release number of selected component (if applicable):

4.10.30

How reproducible:

Always

Steps to Reproduce:

1. Create a machineset like in the description
2.
3.

Actual results:

All ports have no address pairs

Expected results:

Only the port on the secondary subnet has no address pairs.

Additional info:

A simple workaround would be to just fill the filter so that a single network is selected for each network entry.

References:
[1] - https://github.com/openshift/cluster-api-provider-openstack/blob/f6b51710d4f395ded401347589447f5f41dd5c4c/pkg/cloud/openstack/clients/machineservice.go#L576
[2] - https://github.com/openshift/cluster-api-provider-openstack/blob/f6b51710d4f395ded401347589447f5f41dd5c4c/pkg/cloud/openstack/clients/machineservice.go#L580
[3] - https://github.com/openshift/cluster-api-provider-openstack/blob/f6b51710d4f395ded401347589447f5f41dd5c4c/pkg/cloud/openstack/clients/machineservice.go#L581-L583
[4] - https://github.com/openshift/cluster-api-provider-openstack/blob/f6b51710d4f395ded401347589447f5f41dd5c4c/pkg/cloud/openstack/clients/machineservice.go#L658-L660
[5] - https://github.com/openshift/cluster-api-provider-openstack/blob/f6b51710d4f395ded401347589447f5f41dd5c4c/pkg/cloud/openstack/clients/machineservice.go#L610-L625

• https://github.com/openshift/cluster-api-provider-openstack/pull/245

• https://github.com/openshift/aws-ebs-csi-driver/pull/210

Description of problem:

Sometimes we see VMs fail to power on when the land on a host that does not have enough resources. The current power on does not retry or leverage DRS to power on the node on a suitable host.

https://github.com/vmware/govmomi/issues/1026

Our code is still making calls to PowerOnVM_Task which, according to the vsphere docs, is deprecated and we should use PowerOnMultiVM_Task instead.

PowerOnVM_Task does not return a DRS ClusterRecommendation, no vmotion nor host power operations will be done as part of a DRS-facilitated power on. To have DRS consider such operations use PowerOnMultiVM_Task.

https://vdc-download.vmware.com/vmwb-repository/dcr-public/b50dcbbf-051d-4204-a3e7-e1b618c1e384/538cf2ec-b34f-4bae-a332-3820ef9e7773/vim.VirtualMachine.html#powerOn:
As of vSphere API 5.1, use of this method with vCenter Server is deprecated; use PowerOnMultiVM_Task instead.

Version-Release number of selected component (if applicable):
4.8.x

How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:
Sometimes powers on fails requiring manual intervention.

Expected results:
PowerOn should use DRS to ensure it's always successful.

Additional info:

• https://github.com/openshift/machine-api-operator/pull/1065

• https://github.com/openshift/ovn-kubernetes/pull/1423
• https://github.com/openshift/ovn-kubernetes/pull/1293

Description of problem:

After an incident occurred on one of the 2 egress nodes the router policies in the ovn_cluster_router are not correct, and are not reconciled making the pods created before the incident to use only one egress node, while new pods use both.

Version-Release number of selected component (if applicable):

4.10.40

How reproducible:

Not know, visible at the customer cluster after an incident.

Steps to Reproduce:

1.
2.
3.

Actual results:

Pods created before the incident are not using one of the 2 egress nodes.

Expected results:

The configuration should be reconciled to makes the old pods to use all the egress nodes configured.

Additional info:

Could it be a regression of https://issues.redhat.com/browse/OCPBUGSM-33570 fix in 4.10.3? 

• https://github.com/openshift/ovn-kubernetes/pull/1542

This is a clone of issue OCPBUGS-709. The following is the description of the original issue:
—
Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/jenkins/pull/1477

Description of problem:

Intended to backport the corresponding https://bugzilla.redhat.com/show_bug.cgi?id=2095852 which has been fixed already for this version.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/ovn-kubernetes/pull/1269
• https://github.com/openshift/origin/pull/27429

This is a clone of issue OCPBUGS-14931. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-13778. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-13427. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-12780. The following is the description of the original issue:
—
Description of problem:

023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health [-] Component KuryrPortHandler is dead. Last caught exception below: openstack.exceptions.InvalidRequest: Request requires an ID but none was found
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health Traceback (most recent call last):
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/kuryrport.py", line 169, in on_finalize
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health     pod = self.k8s.get(f"{constants.K8S_API_NAMESPACES}"
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 121, in get
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health     self._raise_from_response(response)
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/k8s_client.py", line 99, in _raise_from_response
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health     raise exc.K8sResourceNotFound(response.text)
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health kuryr_kubernetes.exceptions.K8sResourceNotFound: Resource not found: '{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \\"mygov-tuo-microservice-dev2-59fffbc58c-l5b79\\" not found","reason":"NotFound","details":{"name":"mygov-tuo-microservice-dev2-59fffbc58c-l5b79","kind":"pods"},"code":404}\n'
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health During handling of the above exception, another exception occurred:
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health Traceback (most recent call last):
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/logging.py", line 38, in __call__
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health     self._handler(event, *args, **kwargs)
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 85, in __call__
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health     self._handler(event, *args, retry_info=info, **kwargs)
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 98, in __call__
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health     self.on_finalize(obj, *args, **kwargs)
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/kuryrport.py", line 184, in on_finalize
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health     pod = self._mock_cleanup_pod(kuryrport_crd)
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/kuryrport.py", line 160, in _mock_cleanup_pod
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health     host_ip = utils.get_parent_port_ip(port_id)
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/utils.py", line 830, in get_parent_port_ip
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health     parent_port = os_net.get_port(port_id)
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health   File "/usr/lib/python3.6/site-packages/openstack/network/v2/_proxy.py", line 1987, in get_port
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health     return self._get(_port.Port, port)
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health   File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 48, in check
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health     return method(self, expected, actual, *args, **kwargs)
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health   File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 513, in _get
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health     resource_type=resource_type.__name__, value=value))
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health   File "/usr/lib/python3.6/site-packages/openstack/resource.py", line 1472, in fetch
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health     base_path=base_path)
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health   File "/usr/lib/python3.6/site-packages/openstack/network/v2/_base.py", line 26, in _prepare_request
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health     base_path=base_path, params=params)
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health   File "/usr/lib/python3.6/site-packages/openstack/resource.py", line 1156, in _prepare_request
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health     "Request requires an ID but none was found")
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health openstack.exceptions.InvalidRequest: Request requires an ID but none was found
2023-04-20 02:08:09.770 1 ERROR kuryr_kubernetes.controller.managers.health
2023-04-20 02:08:09.918 1 INFO kuryr_kubernetes.controller.service [-] Service 'KuryrK8sService' stopping
2023-04-20 02:08:09.919 1 INFO kuryr_kubernetes.watcher [-] Stopped watching '/apis/openstack.org/v1/kuryrnetworks'
2023-04-20 02:08:10.026 1 INFO kuryr_kubernetes.watcher [-] Stopped watching '/apis/machine.openshift.io/v1beta1/machines'
2023-04-20 02:08:10.152 1 INFO kuryr_kubernetes.watcher [-] Stopped watching '/api/v1/pods'
2023-04-20 02:08:10.174 1 INFO kuryr_kubernetes.watcher [-] Stopped watching '/apis/networking.k8s.io/v1/networkpolicies'
2023-04-20 02:08:10.857 1 INFO kuryr_kubernetes.watcher [-] Stopped watching '/api/v1/namespaces'
2023-04-20 02:08:10.877 1 WARNING kuryr_kubernetes.controller.drivers.utils [-] Namespace dev-health-air-ids not yet ready: kuryr_kubernetes.exceptions.K8sResourceNotFound: Resource not found: '{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"kuryrnetworks.openstack.org \\"dev-health-air-ids\\" not found","reason":"NotFound","details":{"name":"dev-health-air-ids","group":"openstack.org","kind":"kuryrnetworks"},"code":404}\n'
2023-04-20 02:08:11.024 1 INFO kuryr_kubernetes.watcher [-] Stopped watching '/api/v1/services'
2023-04-20 02:08:11.078 1 INFO kuryr_kubernetes.watcher [-] Stopped watching '/api/v1/endpoints'
2023-04-20 02:08:11.170 1 INFO kuryr_kubernetes.watcher [-] Stopped watching '/apis/openstack.org/v1/kuryrports'
2023-04-20 02:08:11.344 1 INFO kuryr_kubernetes.watcher [-] Stopped watching '/apis/openstack.org/v1/kuryrnetworkpolicies'
2023-04-20 02:08:11.475 1 INFO kuryr_kubernetes.watcher [-] Stopped watching '/apis/openstack.org/v1/kuryrloadbalancers'
2023-04-20 02:08:11.475 1 INFO kuryr_kubernetes.watcher [-] No remaining active watchers, Exiting...
2023-04-20 02:08:11.475 1 INFO kuryr_kubernetes.controller.service [-] Service 'KuryrK8sService' stopping

Version-Release number of selected component (if applicable):

How reproducible:

Always

Steps to Reproduce:

1. Create a pod.
2. Stop kuryr-controller.
3. Delete the pod and the finalizer on it.
4. Delete pod's subport.
5. Start the controller.

Actual results:

Crash

Expected results:

Port cleaned up normally.

Additional info:

• https://github.com/openshift/kuryr-kubernetes/pull/736

• https://github.com/openshift/console/pull/12220

Description of problem:

Custonmer is facing this problem in OCP 4.10.40.
https://github.com/coredns/coredns/issues/5593

 Its root cause seems to be this bug in k8s code:

https://github.com/kubernetes/kubernetes/issues/109115
https://github.com/kubernetes/kubernetes/pull/109137

The issue seems to be fixed in OpenShift 4.11, but the customer can't update at this moment.

Can this fix be backported to OpenShift 4.10?

Version-Release number of selected component (if applicable):

4.10.40

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/openshift-apiserver/pull/323

This is a clone of issue OCPBUGS-1805. The following is the description of the original issue:
—
The vSphere CSI cloud.conf lists the single datacenter from platform workspace config but in a multi-zone setup (https://github.com/openshift/enhancements/pull/918 ) there may be more than the one datacenter.

This issue is resulting in PVs failing to attach because the virtual machines can't be find in any other datacenter. For example:

0s Warning FailedAttachVolume pod/image-registry-85b5d5db54-m78vp AttachVolume.Attach failed for volume "pvc-ab1a0611-cb3b-418d-bb3b-1e7bbe2a69ed" : rpc error: code = Internal desc = failed to find VirtualMachine for node:"rbost-zonal-ghxp2-worker-3-xm7gw". Error: virtual machine wasn't found  

The machine above lives in datacenter-2 but the CSI cloud.conf is only aware of the datacenter IBMCloud.

$ oc get cm vsphere-csi-config -o yaml  -n openshift-cluster-csi-drivers | grep datacenters
    datacenters = "IBMCloud" 

• https://github.com/openshift/vmware-vsphere-csi-driver-operator/pull/137
• https://github.com/openshift/vmware-vsphere-csi-driver-operator/pull/133

• https://github.com/openshift/cluster-api-provider-openstack/pull/251

Description of problem:

[4.10.z] Fix kubevirt-console tests

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/console/pull/12580

New release https://github.com/kubernetes/kube-state-metrics/releases

• https://github.com/openshift/kube-state-metrics/pull/61

This bug card represents work done in https://issues.redhat.com/browse/CCO-257 to set STS endpoints to regional in AWS credentials secrets and is created to facilitate backporting the change to previous releases as required by the backport process [1].

[1] https://docs.google.com/document/d/1qbKUl4K9PPW4tCbdDgWSXd5klosX77ifkxQLqzjztGE/edit#heading=h.mml0rz88o0nj

• https://github.com/openshift/cloud-credential-operator/pull/504

Description of problem:

Building the installer image started to fail on ppc64le

Version-Release number of selected component (if applicable):

4.10.0

How reproducible:

Always on the brew builders

Steps to Reproduce:

1.
2.
3.

Actual results:

2022-11-14 11:55:00,819 - atomic_reactor.tasks.binary_container_build - INFO - + go build -mod=vendor -ldflags ' -X github.com/openshift/installer/pkg/version.Raw=v4.10.0 -X github.com/openshift/installer/pkg/version.Commit=4e8922c14379dce4845f362ac3a83ff80f1dc655 -X github.com/openshift/installer/pkg/version.defaultArch=ppc64le -s -w' -tags ' release' -o bin/openshift-install ./cmd/openshift-install
2022-11-14 11:58:01,556 - atomic_reactor.tasks.binary_container_build - INFO - # github.com/openshift/installer/cmd/openshift-install
2022-11-14 11:58:01,556 - atomic_reactor.tasks.binary_container_build - INFO - github.com/aliyun/alibaba-cloud-sdk-go/services/cms.(*Client).DescribeMonitoringAgentHostsWithChan.func1: direct call too far: runtime.duffzero+1f0-tramp0 -20000e8

Expected results:

Successful build

Additional info:

This could potentially be a golang issue. After some preliminary investigation, we tried setting `-linkmode=external` in the ldflags which seems to allow the build to finish.
So while we investigate the real cause of the issue, changing the link mode will serve as a workaround to unblock the build pipeline.

• https://github.com/openshift/installer/pull/6597

This is a clone of issue OCPBUGS-268. The following is the description of the original issue:
—
The linux kernel was updated:
https://lkml.org/lkml/2020/3/20/1030
to include steal

accounting

This would greatly assist in troubleshooting vSphere performance issues
caused by over-provisioned ESXi hosts.

• https://github.com/openshift/installer/pull/6261

Description of problem:

When queried dns hostname from certain pod on the certain node, responded from random coredns pod, not prefer local one. Is it expected result ?

# In OCP v4.8.13 case
// Ran dig command on the certain node which is running the following test-7cc4488d48-tqc4m pod.
sh-4.4# while : ; do echo -n "$(date '+%H:%M:%S') :"; dig google.com +short; sleep 1; done
:
07:16:33 :172.217.175.238
07:16:34 :172.217.175.238 <--- Refreshed the upstream result
07:16:36 :142.250.207.46
07:16:37 :142.250.207.46

// The dig results is matched with the running node one as you can see the above one.
$ oc rsh  test-7cc4488d48-tqc4m bash -c 'while : ; do echo -n "$(date '+%H:%M:%S') :"; dig google.com +short; sleep 1; done'
:
07:16:35 :172.217.175.238 
07:16:36 :172.217.175.238 <--- At the same time, the pod dig result is also refreshed.
07:16:37 :142.250.207.46
07:16:38 :142.250.207.46


But in v4.10 case, in contrast, the dns query result is various and responded randomly regardless local dns results on the node as follows.

# In OCP v4.10.23 case, pod's response from DNS services are not consistent.
$ oc rsh test-848fcf8ddb-zrcbx  bash -c 'while : ; do echo -n "$(date '+%H:%M:%S') :"; dig google.com +short; sleep 1; done'
07:23:00 :142.250.199.110
07:23:01 :142.250.207.46
07:23:02 :142.250.207.46
07:23:03 :142.250.199.110
07:23:04 :142.250.199.110
07:23:05 :172.217.161.78

# Even though the node which is running the pod keep responding the same IP...
sh-4.4# while : ; do echo -n "$(date '+%H:%M:%S') :"; dig google.com +short; sleep 1; done
07:23:00 :172.217.161.78
07:23:01 :172.217.161.78
07:23:02 :172.217.161.78
07:23:03 :172.217.161.78
07:23:04 :172.217.161.78
07:23:05 :172.217.161.78

Version-Release number of selected component (if applicable):

v4.10.23 (ROSA)
SDN: OpenShiftSDN

How reproducible:

You can always reproduce this issue using "dig google.com" from both any pod and the node the pod running according to the above "Description" details.

Steps to Reproduce:

1. Run any usual pod, and check which node the pod is running on.
2. Run dig google.com on the pod and the node.
3. Check the IP is consistent with the running node each other. 

Actual results:

The response IPs are not consistent and random IP is responded.

Expected results:

The response IP is kind of consistent, and aware of prefer local dns.

Additional info:

This issue affects EgressNetworkPolicy dnsName feature.

• https://github.com/openshift/sdn/pull/467
• https://github.com/openshift/sdn/pull/457

• https://github.com/openshift/coredns/pull/80

Changelog between 3.5.5 and 3.5.4:
https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md#v355-tbd

Changelog between 3.5.3 and 3.5.4:
https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md#v354-2022-04-24

• https://github.com/openshift/etcd/pull/156

This is a clone of issue OCPBUGS-262. The following is the description of the original issue:
—
github rate limit failures for upi image downloading govc.

https://search.ci.openshift.org/?search=build+error%3A+error+building+at+STEP+%22RUN+curl+-L+-o&maxAge=48h&context=1&type=all&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

• https://github.com/openshift/installer/pull/6245

• https://github.com/openshift/jenkins/pull/1561

This is a clone of issue OCPBUGS-613. The following is the description of the original issue:
—

Description of problem:

The path used by --rotated-pod-logs to gather the rotated pod logs from /var/log/pods node folder via /api/v1/nodes/${NODE}/proxy/logs/${LOG_PATH} is only valid for regular pods but not for static pods.

The main problem is that, while normal pods have their rotated logs at this /var/log/pods/${POD_NAME}_${POD_UID_IN_API}/${CONTAINER_NAME}, static pods have them at /var/log/pods/${POD_NAME}_${CONFIG_HASH}/${CONTAINER_NAME} because the UID cannot be known at the time that the static pod is born (because static pods are created by kubelet before registering them in the kube-apiserver, and UID is assigned by the kube-apiserver).

The visible results of that are:

• Spurious errors of not found resources related to the pods.
• Rotated pod logs are not gathered even if present.

Version-Release number of selected component (if applicable):

4.10

How reproducible:

Always if there are static pods.

Steps to Reproduce:

1. oc adm inspect --rotated-pod-logs ns/openshift-etcd (or any other project with static pods).

Actual results:

• Rotated pods not gathered.
• Errors like these

  error: errors occurred while gathering data:
      one or more errors occurred while gathering pod-specific data for namespace: openshift-etcd
  
      [one or more errors occurred while gathering container data for pod etcd-master-0.example.net:
  
      the server could not find the requested resource, one or more errors occurred while gathering container data for pod etcd-master-1.example.net:
  
      the server could not find the requested resource, one or more errors occurred while gathering container data for pod etcd-master-2.example.net:
  
      the server could not find the requested resource]
  

Expected results:

No errors like the ones above and rotated pod logs to be gathered, if present.

Additional info:

Despite being marked as experimental, this --rotated-pod-logs is used in must-gather, so this issue can be easily reproduced by just running a default must-gather. I focused on bare oc adm inspect reproducers for simplicity.

• https://github.com/openshift/oc/pull/1246

• https://github.com/openshift/sdn/pull/545

Description of problem:

The storageclass "thin-csi" is created by vsphere-CSI-Driver-Operator, after deleting it manually, it should be re-created immediately. 

Version-Release number of selected component (if applicable):

4.11.4

How reproducible:

Always

Steps to Reproduce:

1. Check storageclass in running cluster, thin-csi is present:
$ oc get sc 
NAME             PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
thin (default)   kubernetes.io/vsphere-volume   Delete          Immediate              false                  41m
thin-csi         csi.vsphere.vmware.com         Delete          WaitForFirstConsumer   true                   38m

2. Delete thin-csi storageclass:
$ oc delete sc thin-csi
storageclass.storage.k8s.io "thin-csi" deleted

3. Check storageclass again, thin-csi is not present:
$ oc get sc
NAME             PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
thin (default)   kubernetes.io/vsphere-volume   Delete          Immediate           false                  50m

4. Check vmware-vsphere-csi-driver-operator log:
......
I0909 03:47:42.172866       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1662695014\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1662695014\" (2022-09-09 02:43:34 +0000 UTC to 2023-09-09 02:43:34 +0000 UTC (now=2022-09-09 03:47:42.172853123 +0000 UTC))"I0909 03:49:38.294962       
1 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOFI0909 03:49:38.295468       
1 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOFI0909 03:49:38.295765       
1 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF

5. Only first time creating in vmware-vsphere-csi-driver-operator log:
$ oc -n openshift-cluster-csi-drivers logs vmware-vsphere-csi-driver-operator-7cc6d44b5c-c8czw | grep -i "storageclass"I0909 03:46:31.865926   1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-csi-drivers", Name:"vmware-vsphere-csi-driver-operator", UID:"9e0c3e2d-d403-40a1-bf69-191d7aec202b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'StorageClassCreated' Created StorageClass.storage.k8s.io/thin-csi because it was missing 

Actual results:

The storageclass "thin-csi" could not be re-created after deleting

Expected results:

The storageclass "thin-csi" should be re-created after deleting

Additional info:

• https://github.com/openshift/vmware-vsphere-csi-driver-operator/pull/128

This is a clone of issue OCPBUGS-7885. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-7617. The following is the description of the original issue:
—
Description of problem:

Azure Disk volume is taking time to attach/detach
Version-Release number of selected component (if applicable):

Openshift ARO 4.10.30
How reproducible:

While performing scaledown and scaleup of statefulset pod takes time to attach and detach volume from nodes.

Reviewed must-gather and test output will share my findings in comments.

Steps to Reproduce:
1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/azure-disk-csi-driver-operator/pull/72

The CMO e2e tests create a bunch of resources. These should be cleaned up on a successful run. However:

• Some test failures leave the create resource behind, which have to be cleaned up before a re-run.
• There have been developer reports that even successful runs don't tidy up everything.

In a CI context this is rarely a problem, however running the tests locally can be made quite awkward, especially repeated runs on the same cluster.

We should tag all resources created by the e2e tests with a label (app.kubernetes.io/created-by: cmo-e2e-test).
This will allow easy cleanup by deleting all resources with that label and will allow for checking proper clean-up.

DoD:
All e2e resources get properly tagged.
It is straight forward to ensure that future code changes don't skip adding this tag.

• https://github.com/openshift/cluster-monitoring-operator/pull/1397

This bug is a backport clone of [Bugzilla Bug 2072040](https://bugzilla.redhat.com/show_bug.cgi?id=2072040). The following is the description of the original bug:
—
Description of problem:

configure-ovs resets network configuration on boot, and while doing so waits for all devices to become unmanaged in NetworkManager. Currently the patch port between br-int and br-ex created by ovn-kubernetes is managed by NetworkManager, never becomes unmanaged and causes an accumulated delay of 2 minutes on boot.

This patct port should never be managed by NetworkManager.

• https://github.com/openshift/machine-config-operator/pull/3435

This is a clone of issue OCPBUGS-1556. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-78. The following is the description of the original issue:
—
Copied from an upstream issue: https://github.com/operator-framework/operator-lifecycle-manager/issues/2830

What did you do?

When attempting to reinstall an operator that uses conversion webhooks by

• Deleting the operator subscription and any CSVs associated with it
• Recreating the operator subscription

The resulting InstallPlan enters a failed state with message similar to

error validating existing CRs against new CRD's schema for "devworkspaces.workspace.devfile.io": error listing resources in GroupVersionResource schema.GroupVersionResource{Group:"workspace.devfile.io", Version:"v1alpha1", Resource:"devworkspaces"}: conversion webhook for workspace.devfile.io/v1alpha2, Kind=DevWorkspace failed: Post "https://devworkspace-controller-manager-service.test-namespace.svc:443/convert?timeout=30s": service "devworkspace-controller-manager-service" not found

When the original CSVs are deleted, the operator's main deployment and service are removed, but CRDs are left in-cluster. However, since the service/CA bundle/deployment that serve the conversion webhook are removed, conversion webhooks are broken at that point. Eventually this impacts garbage collection on the cluster as well.

This can be reproduced by installing the DevWorkspace Operator from the Red Hat catalog. (I can provide yamls/upstream images that reproduce as well, if that's helpful). It may be necessary to create a DevWorkspace in the cluster before deletion, e.g. by oc apply -f https://raw.githubusercontent.com/devfile/devworkspace-operator/main/samples/plain.yaml

What did you expect to see?
Operator is able to be reinstalled without removing CRDs and all instances.

What did you see instead? Under which circumstances?
It's necessary to completely remove the operator including CRDs. For our operator (DevWorkspace), this also makes uninstall especially complicated as finalizers are used (so CRDs cannot be deleted if the controller is removed, and the controller cannot be restored by reinstalling)

Environment

operator-lifecycle-manager version: 4.10.24

Kubernetes version information: Kubernetes Version: v1.23.5+012e945 (OpenShift 4.10.24)

Kubernetes cluster kind: OpenShift

• https://github.com/openshift/operator-framework-olm/pull/458

This is a clone of issue OCPBUGS-10225. The following is the description of the original issue:
—
Description of problem:
Pipeline Repository (Pipeline-as-code) list never shows an Event type.

Version-Release number of selected component (if applicable):
4.9+

How reproducible:
Always

Steps to Reproduce:

1. Install Pipelines Operator and setup a Pipeline-as-code repository
2. Trigger an event and a build

Actual results:
Pipeline Repository list shows a column Event type but no value.

Expected results:
Pipeline Repository list should show the Event type from the matching Pipeline Run.

Similar to the Pipeline Run Details page based on the label.

Additional info:
The list page packages/pipelines-plugin/src/components/repository/list-page/RepositoryRow.tsx renders obj.metadata.namespace as event type.

I believe we should show the Pipeline Run event type instead. packages/pipelines-plugin/src/components/repository/RepositoryLinkList.tsx uses

to render it.

Also the Pipeline Repository details page tried to render the Branch and Event type from the Repository resource. My research says these properties doesn't exist on the Repository resource. The code should be removed from the Repository details page.

• https://github.com/openshift/console/pull/12833

 As mentioned in [1], the cluster monitoring operator doesn't define the relatedObjects field in the ClusterOperator manifest which is initially deployed by CVO [2].
If the CMO pod fails to start, the must-gather might miss information from the monitoring namespace. Note that once CMO runs, it will update the initial ClusterOperator object with the proper information [3].

[1] http://mailman-int.corp.redhat.com/archives/aos-devel/2021-May/msg00139.html
[2] https://github.com/openshift/cluster-monitoring-operator/blob/master/manifests/0000_50_cluster-monitoring-operator_06-clusteroperator.yaml
[3] https://github.com/openshift/cluster-monitoring-operator/blob/a6bc9824035ceb8dbfe7c53cf0c138bfb2ec5643/pkg/client/status_reporter.go#L49-L63

• https://github.com/openshift/cluster-monitoring-operator/pull/1483

• https://github.com/openshift/csi-driver-nfs/pull/114

This is a clone of issue OCPBUGS-8205. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-7960. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-7780. The following is the description of the original issue:
—
Description of problem:

4.9 and 4.10 oc calls to oc adm upgrade channel ... for 4.11+ clusters would clear spec.capabilities. Not all that many clusters try to restrict capabilities, but folks will need to bump their channel for at least every other minor (if their using EUS channels), and while we recommend folks use an oc from the 4.y they're heading towards, we don't have anything in place to enforce that.

Version-Release number of selected component (if applicable):

4.9 and 4.10 oc are exposed vs. the new-in-4.11 spec.capabilities. Newer oc could theoretically be exposed vs. any new ClusterVersion spec capabilities.

How reproducible:

100%

Steps to Reproduce:

1. Install a 4.11+ cluster with None capabilities.
2. Set the channel with a 4.10.51 oc, like oc adm upgrade channel fast-4.11.
3. Check the capabilities with oc get -o json clusterversion version | jq -c .spec.capabilities.

Actual results:

null

Expected results:

{"baselineCapabilitySet":"None"}

• https://github.com/openshift/oc/pull/1379

This is a clone of issue OCPBUGS-14646. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-14642. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-14634. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-14609. The following is the description of the original issue:
—
Description of problem:

Update Jenkins plugins to mitigate existing CVEs

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/jenkins/pull/1696

Description of problem:

Customer is facing issue similar to https://github.com/devfile/api/issues/897

Version-Release number of selected component (if applicable):

OCP 4.10.17

How reproducible:
N/A
Steps to Reproduce:
1.
2.
3.

Actual results:

Expected results:

Additional info:

Tried working around it with ALL_PROXY but it did not help. Note because the console operator reverts changes pretty quickly testing this was a bit of a PITA

• https://github.com/openshift/console/pull/12040

console-operator codebase contains a lot of inline manifests. Instead we should put those manifests into a `/bindata` folder, from which they will be read and then updated per purpose.

• https://github.com/openshift/console-operator/pull/550

This is a clone of issue OCPBUGS-2250. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-2099. The following is the description of the original issue:
—
Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/jenkins/pull/1509

• https://github.com/openshift/machine-config-operator/pull/3315

This is a clone of issue OCPBUGS-7590. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-7458. The following is the description of the original issue:
—
Description of problem:

- After upgrading to OCP 4.10.41, thanos-ruler-user-workload-1 in the openshift-user-workload-monitoring namespace is consistently being created and deleted.
- We had to scale down the Prometheus operator multiple times so that the upgrade is considered as successful.
- This fix is temporary. After some time it appears again and Prometheus operator needs to be scaled down and up again.
- The issue is present on all clusters in this customer environment which are upgraded to 4.10.41.

Version-Release number of selected component (if applicable):

How reproducible:

N/A, I wasn't able to reproduce the issue.

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/prometheus-operator/pull/219

Description of problem:

Running the discovery cache every 10 minutes has a significant productivity impact on using kubectl on clusters with many CRDs as it is taking time to run these unnecessary requests.
The discovery cache doesn't really have to run every 10 minutes, as CRDs don't change that often. A lot of unnecessary load is created on clients and servers.

Version-Release number of selected component (if applicable):

4.10.z

How reproducible:

Cluster with a lot of CRDs

Steps to Reproduce:

https://github.com/kubernetes/kubernetes/issues/107130

Actual results:

Significant grow of kubectl request completion every 10 minutes

Expected results:

Significant grow of kubectl request completion every 24 hours

Additional info:

https://github.com/kubernetes/kubernetes/issues/107130

• https://github.com/openshift/oc/pull/1241

This bug is a backport clone of [Bugzilla Bug 2056519](https://bugzilla.redhat.com/show_bug.cgi?id=2056519). The following is the description of the original bug:
—
Version: 4.9

Platform:
azure

Please specify:

• IPI

What happened?
Issue: Customer reports unable to install IPI PRIVATE OpenShift cluster in Azure. They have orgnization policy which do not allow them to create storage account with public access. It should be disallowed.

What did you expect to happen?
Installer completes successfully.

• https://github.com/openshift/cluster-image-registry-operator/pull/804

Description of problem:

After upgrading the cluster to version 4.10.32 the web console and metrics tab  does not display any data.

we have to restart the thanos pods to get the functionality back to running state


Version-Release number of selected component (if applicable):

4.10.32

How reproducible:

NA

Steps to Reproduce:

NA

Actual results:

The data is not getting displayed and monitoring of the cluster is hampered

Expected results:

The data should be displayed properly without having to restart the pods

Additional info:

• https://github.com/openshift/cluster-monitoring-operator/pull/1811
• https://github.com/openshift/thanos/pull/92

This bug is a backport clone of [Bugzilla Bug 2092839](https://bugzilla.redhat.com/show_bug.cgi?id=2092839). The following is the description of the original bug:
—
Description of problem:

Application of customer uses the Downward API (/etc/podnetinfo/annotations) to get the pci-addresses for SR-IOV interfaces. However, when it adds the tuning metaplugin for sysctl configuration on the SR-IOV Networks, it does not get the pci-addresses for the SR-IOV interfaces.basically customer want to fetch the PCI-ID to bind the interface with the application and dpdk driver.

Version-Release number of selected component (if applicable):

OCP 4.7

Additional info:

OCP : 4.7
Customer: Ericsson
Case: 03221272
Customer has shared all the logs attached with the case.

Please do let us know, if you are able to access the attached log in the case.

For more information, please refer the case.

• https://github.com/openshift/multus-cni/pull/137

This is a clone of issue OCPBUGS-2113. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-1329. The following is the description of the original issue:
—
Description of problem:

etcd and kube-apiserver pods get restarted due to failed liveness probes while deleting/re-creating pods on SNO

Version-Release number of selected component (if applicable):

4.10.32

How reproducible:

Not always, after ~10 attempts

Steps to Reproduce:

1. Deploy SNO with Telco DU profile applied
2. Create multiple pods with local storage volumes attached(attaching yaml manifest)
3. Force delete and re-create pods 10 times

Actual results:

etcd and kube-apiserver pods get restarted, making to cluster unavailable for a period of time

Expected results:

etcd and kube-apiserver do not get restarted

Additional info:

Attaching must-gather.

Please let me know if any additional info is required. Thank you!

• https://github.com/openshift/cluster-etcd-operator/pull/953

This is a clone of issue OCPBUGS-4945. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-4805. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-4101. The following is the description of the original issue:
—
Description of problem:

We experienced two separate upgrade failures relating to the introduction of the SYSTEM_RESERVED_ES node sizing parameter, causing kubelet to stop running.

One cluster (clusterA) upgraded from 4.11.14 to 4.11.17. It experienced an issue whereby 
   /etc/node-sizing.env 
on its master nodes contained an empty SYSTEM_RESERVED_ES value:

---
cat /etc/node-sizing.env 
SYSTEM_RESERVED_MEMORY=5.36Gi
SYSTEM_RESERVED_CPU=0.11
SYSTEM_RESERVED_ES=
---

causing the kubelet to not start up. To restore service, this file was manually updated to set a value (1Gi), and kubelet was restarted.

We are uncertain what conditions led to this occuring on the clusterA master nodes as part of the upgrade.

A second cluster (clusterB) upgraded from 4.11.16 to 4.11.17. It experienced an issue whereby worker nodes were impacted by a similar problem, however this was because a custom node-sizing-enabled.env MachineConfig which did not set SYSTEM_RESERVED_ES

This caused existing worker nodes to go into a NotReady state after the ugprade, and additionally new nodes did not join the cluster as their kubelet would become impacted. 

For clusterB the conditions are more well-known of why the value is empty.

However, for both clusters, if SYSTEM_RESERVED_ES ends up as empty on a node it can cause the kubelet to not start. 

We have some asks as a result:
- Can MCO be made to recover from this situation if it occurs, perhaps  through application of a safe default if none exists, such that kubelet would start correctly?
- Can there possibly be alerting that could indicate and draw attention to the misconfiguration?

Version-Release number of selected component (if applicable):

4.11.17

How reproducible:

Have not been able to reproduce it on a fresh cluster upgrading from 4.11.16 to 4.11.17

Expected results:

If SYSTEM_RESERVED_ES is empty in /etc/node-sizing*env then a default should be applied and/or kubelet able to continue running.

Additional info:

• https://github.com/openshift/machine-config-operator/pull/3487

• https://github.com/openshift/cloud-provider-openstack/pull/177

Description of problem:
Follow-up of: https://issues.redhat.com/browse/SDN-2988

This failure is perma-failing in the e2e-metal-ipi-ovn-dualstack-local-gateway jobs.

Example: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.13-e2e-metal-ipi-ovn-dualstack-local-gateway/1597574181430497280
Search CI: https://search.ci.openshift.org/?search=when+using+openshift+ovn-kubernetes+should+ensure+egressfirewall+is+created&maxAge=336h&context=1&type=junit&name=e2e-metal-ipi-ovn-dualstack-local-gateway&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job
Sippy: https://sippy.dptools.openshift.org/sippy-ng/jobs/4.13/analysis?filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22equals%22%2C%22value%22%3A%22periodic-ci-openshift-release-master-nightly-4.13-e2e-metal-ipi-ovn-dualstack-local-gateway%22%7D%5D%7D

Version-Release number of selected component (if applicable):

4.12,4.13

How reproducible:

Every time

Steps to Reproduce:

1. Setup dualstack KinD cluster
2. Create egress fw policy with spec
Spec:
  Egress:
    To:
      Cidr Selector:  0.0.0.0/0
    Type:             Deny
3. create a pod and ping to 1.1.1.1

Actual results:

Egress policy does not block flows to external IP

Expected results:

Egress policy blocks flows to external IP

Additional info:

It seems mixing ip4 and ip6 operands in ACL matchs doesnt work

• https://github.com/openshift/ovn-kubernetes/pull/1508

Description of problem:

Cluster running 4.10.52 had three aws-ebs-csi-driver-node pods begin to consume multiple GB of memory, causing heavy node memory pressure as the pods have no memory limit. 

All other aws-ebs-csi-driver-node pods were still in the 50-70MB range:

NAME                                            CPU(cores)   MEMORY(bytes)   
aws-ebs-csi-driver-controller-59867579b-d6s2q   0m           397Mi           
aws-ebs-csi-driver-controller-59867579b-t4wgq   0m           276Mi           
aws-ebs-csi-driver-node-4rmvk                   0m           53Mi            
aws-ebs-csi-driver-node-5799f                   0m           50Mi            
aws-ebs-csi-driver-node-6dpvg                   0m           59Mi            
aws-ebs-csi-driver-node-6ldzk                   0m           65Mi            
aws-ebs-csi-driver-node-6mbk5                   0m           54Mi            
aws-ebs-csi-driver-node-bkvsr                   0m           50Mi            
aws-ebs-csi-driver-node-c2fb2                   0m           62Mi            
aws-ebs-csi-driver-node-f422m                   0m           61Mi            
aws-ebs-csi-driver-node-lwzbb                   6m           1940Mi          
aws-ebs-csi-driver-node-mjznt                   0m           53Mi            
aws-ebs-csi-driver-node-pczsj                   0m           62Mi            
aws-ebs-csi-driver-node-pmskn                   0m           3493Mi          
aws-ebs-csi-driver-node-qft8w                   0m           68Mi            
aws-ebs-csi-driver-node-v5bpx                   11m          2076Mi          
aws-ebs-csi-driver-node-vn8km                   0m           84Mi            
aws-ebs-csi-driver-node-ws6hx                   0m           73Mi            
aws-ebs-csi-driver-node-xsk7k                   0m           59Mi            
aws-ebs-csi-driver-node-xzwlh                   0m           55Mi            
aws-ebs-csi-driver-operator-8c5ffb6d4-fk6zk     5m           88Mi            

Deleting the pods caused them to recreate, with normal memory consumption levels.

Version-Release number of selected component (if applicable):

4.10.52

How reproducible:

Unknown

• https://github.com/openshift/csi-livenessprobe/pull/41

• https://github.com/openshift/machine-config-operator/pull/3278

Description of problem:

To avoid any potential bugs, the oVirt CSI driver should use the latest go-ovirt-client, preferably the tagged 1.0.0 version.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/ovirt-csi-driver/pull/116

• https://github.com/openshift/jenkins/pull/1644

This is a clone of issue OCPBUGS-15532. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-15099. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-14943. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-14668. The following is the description of the original issue:
—
Description of problem:

visiting global configurations page will return error after 'Red Hat OpenShift Serverless' is installed, the error persist even operator is uninstalled

Version-Release number of selected component (if applicable):

4.14.0-0.nightly-2023-06-06-212044

How reproducible:

Always

Steps to Reproduce:

1. Subscribe 'Red Hat OpenShift Serverless' from OperatorHub, wait for the operator to be successfully installed
2. Visit Administration -> Cluster Settings -> Configurations tab

Actual results:

react_devtools_backend_compact.js:2367 unhandled promise rejection: TypeError: Cannot read properties of undefined (reading 'apiGroup') 
    at r (main-chunk-e70ea3b3d562514df486.min.js:1:1)
    at main-chunk-e70ea3b3d562514df486.min.js:1:1
    at Array.map (<anonymous>)
    at main-chunk-e70ea3b3d562514df486.min.js:1:1
overrideMethod @ react_devtools_backend_compact.js:2367
window.onunhandledrejection @ main-chunk-e70ea3b3d562514df486.min.js:1

main-chunk-e70ea3b3d562514df486.min.js:1 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'apiGroup')
    at r (main-chunk-e70ea3b3d562514df486.min.js:1:1)
    at main-chunk-e70ea3b3d562514df486.min.js:1:1
    at Array.map (<anonymous>)
    at main-chunk-e70ea3b3d562514df486.min.js:1:1

Expected results:

no errors

Additional info:

• https://github.com/openshift/console/pull/12985

Description of problem:

I checked the log of the ironic-conductor container to get the shared lock of iDRAC, but the next installation does not proceed.

Provision Node
 ㄴ bootstrap
    ㄴ ironic-conductor    --- pending --> iDRAC8 (Baremetal master1,2,3)

Version-Release number of selected component (if applicable):

4.10.31

How reproducible:

1) baremetal inventory
   - Dell R630 firmware 2.75.75
   - iDRAC8

2) nodes
provisioning node, master 1,2,3

3) network : static IP(no DHCP) and bond/vlan
             disconnected installation

Steps to Reproduce:

1. Preparing to configure ipi on the provisioning node
   - RHEL 8 ( haproxy, named, mirror registry, rhcos_cache_server ..)

2. configuring the install-config.yaml (attached)
   
3. deploy the cluster

Actual results:

ironic does an iso mount to baremetal via idrac and starts coreos booting

Expected results:

ironic has established a session with idrac, but the installation does not proceed.

Additional info:

## curl <idrac api>
{
    "error": {
        "@Message.ExtendedInfo": [
            {
                "Message": "The request failed due to an internal service error.  The service is still operational.",
                "MessageArgs": [],
                "MessageArgs@odata.count": 0,
                "MessageId": "Base.1.2.InternalError",
                "RelatedProperties": [],
                "RelatedProperties@odata.count": 0,
                "Resolution": "Resubmit the request.  If the problem persists, consider resetting the service.",
                "Severity": "Critical"
            }
        ],
        "code": "Base.1.2.GeneralError",
        "message": "A general error has occurred. See ExtendedInfo for more information"
    }
}

• https://github.com/openshift/ironic-image/pull/303

This is a clone of issue OCPBUGS-6907. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-6517. The following is the description of the original issue:
—
Description of problem:

When the cluster is configured with Proxy the swift client in the image registry operator is not using the proxy to authenticate with OpenStack, so it's unable to reach the OpenStack API. This issue became evident since recently the support was added to not fallback to cinder in case swift is available[1].

[1]https://github.com/openshift/cluster-image-registry-operator/pull/819

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1. Deploy a cluster with proxy and restricted installation
2. 
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/cluster-image-registry-operator/pull/837

Add a Makefile rule in CMO to execute all the different rule that are used for verification and validation. Currenctly, some of them might not be at the right place, for example `check-assets` which is part of `generate` despite not being responsible of any generation. https://github.com/openshift/cluster-monitoring-operator/pull/1151/files#r629371735

DoD:

• Add a new rule in CMO to handle verification
• Add a CI job for this rule

• https://github.com/openshift/cluster-monitoring-operator/pull/1499
• https://github.com/openshift/cluster-monitoring-operator/pull/1492

• https://github.com/openshift/jenkins/pull/1587

This is a clone of issue OCPBUGS-515. The following is the description of the original issue:
—
When a thin provisioned COW format disk is created on OCP on RHV via CSI driver (a PVC -
https://github.com/openshift/ovirt-csi-driver/blob/master/deploy/example/storage-claim.yaml

• with for example requested storage 100 GB), the go-ovirt-client behaviour makes it so that the created disk has virtual size 100 GB and it's actual size is 110 GB.

But this is thin provisioned disk, so the initial size of the disk should be default of the engine and then grow as needed, it shouldn't be this big.

This causes all the disks created this way to be functionally preallocated (since it eats all that space), which is a real waste of space.

How reproducible: 100%

Steps to Reproduce:
1. Create a storage claim (PVC) in Openshift (
https://github.com/openshift/ovirt-csi-driver/blob/master/deploy/example/storage-claim.yaml
) using the default storage class (or any other storage class with thinProvisioning: "true") and with requested storage i.e. 100Gi

$ oc create -f storage-claim.yaml

2. In the RHV web console navigate to Storage -> Disks and check Virtual size and Actual size of the created disk (PVC)

Actual results:
Disk from our example with requested storage 100GB reports virtual size 100GB and actual size 110 GB.