Provide a simple way to get a VM-friendly networking setup, without having to configure the underlying physical network.

Feature Overview (aka. Goal Summary)

As a result of Hashicorp's license change to BSL, Red Hat OpenShift needs to remove the use of Hashicorp's Terraform from the installer – specifically for IPI deployments which currently use Terraform for setting up the infrastructure.

To avoid an increased support overhead once the license changes at the end of the year, we want to provision IBM Cloud VPC infrastructure without the use of Terraform.

Requirements (aka. Acceptance Criteria):

• The IBM Cloud VPC IPI Installer no longer contains or uses Terraform.
• The new provider should aim to provide the same results and have parity with the existing IBM Cloud VPC Terraform provider. Specifically, we should aim for feature parity against the install config and the cluster it creates to minimize impact on existing customers' UX.

Use Cases (Optional):

Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.

Questions to Answer (Optional):

Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.

Out of Scope

High-level list of items that are out of scope. Initial completion during Refinement status.

Background

Provide any additional context is needed to frame the feature. Initial completion during Refinement status.

Customer Considerations

Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.

Documentation Considerations

Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.

Interoperability Considerations

Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.

Feature Overview (aka. Goal Summary)  

• With this next-gen OLM GA release (graduated from ‘Tech Preview’), customers can: 
  • discover collections of k8s extension/operator contents released in the FBC format with richer visibility into their release channels, versions, update graphs, and the deprecation information (if any) to make informed decisions about installation and/or update them.
  • install a k8s extension/operator declaratively and potentially automate with GitOps to ensure predictable and reliable deployments.
  • update a k8s extension/operator to a desired target version or keep it updated within a specific version range for security fixes without breaking changes.
  • remove a k8s extension/operator declaratively and entirely including cleaning up its CRDs and other relevant on-cluster resources (with a way to opt out of this coming up in a later release).

• To address the security needs of 30% of our customers who run clusters in disconnected environments, the GA release will include cluster extension lifecycle management functionality for offline environments.

• [Tech Preview] (Cluster)Extension lifecycle management can handle runtime signature validation for container images to support OpenShift’s integration with the rising Sigstore project for secure validation of cloud-native artifacts,

Goals (aka. expected user outcomes)

1. Pre-installation:

• Customers can access a collection of k8s extension contents from a set of default catalogs leveraging the existing catalog images shipped with OpenShift (in the FBC format) with the new Catalog API from the OLM v1 GA release.

• With the new GAed Catalog API, customers get richer package content visibility in their release channels, versions, update graphs, and the deprecation information (if any) to help make informed decisions about installation and/or update.

• With the new GAed Catalog API, customers can render the catalog content in their clusters with fewer resources in terms of CPU and memory usage and faster performance.

• Customers can filter the available packages based on the package name and see the relevant information from the metadata shipped within the package. 

2. Installation:

• Customers using a ServiceAccount with sufficient permissions can install a k8s extension/operator with a desired target version or the latest version within a specific version range (from the associated channel) to get the latest security fixes.

• Customers can easily automate the installation flow declaratively with GitOps to ensure predictable and reliable deployments.

• Customers get protection from having two conflicting k8s extensions/operators owning the same API objects, i.e., no conflicting ownership, ensuring cluster stability.

• Customers can access the* metadata of the installed k8s extension/operator to see essential information such as its provided APIs, example YAMLs of its provided APIs, descriptions, infrastructure features, valid subscriptions, etc.

3. Update:

• Customers can see what updates are available for their k8s extension/operators in the form of immediate target versions and the associated update channels.

• Customers can trigger the update of a k8s extension/operator with a desired target version or the latest version within a specific version range (from the associated channel) to get the latest security fixes.

• Customers get protection from workload or k8s extension/operator breakage due to CustomResourceDefinition (CRD) being upgraded to a backward incompatible version during an update.

• During OpenShift cluster update, customers* get Informed when installed k8s extensions/operators ** do not support the next OpenShift version *(when annotated by the package author/provider).  Customers must update those k8s extensions/operators to a newer/compatible version before OLM unblocks the OpenShift cluster update. 

4. Uninstallation/Deletion:

• Customers can cleanly remove an installed k8s extension/operator including deleting CustomResourceDefinitions (CRDs), custom resource objects (CRs) of the CRDs, and other relevant resources to revert the cluster to its original state before the installation declaratively.

5. Disconnected Environments for High-Security Workloads:

• Approximately 30% of our customers prioritize high security by running their clusters in internet-disconnected environments, especially for mission-critical production workloads. To benefit these users, our supported GA release needs to include cluster extension lifecycle management functionality that functions within these disconnected environments.

6. [Tech Preview] Signature Validation for Secure Workflows:

• The Red Hat-sponsored Sigstore project is gaining traction in the Kubernetes community, aiming to simplify the signing of cloud-native artifacts. OpenShift leverages Sigstore tooling to enable scalable and flexible signature validation, including support for disconnected environments. This functionality will be available as a Tech Preview in 4.17 and is targeted for General Availability (GA) Tech Preview Phase 2 in the upcoming 4.18 release. To fully support this integration as a Tech Preview release, the (cluster)extension lifecycle management needs to (be prepared to) handle runtime validation of Sigstore signatures for container images.

Requirements (aka. Acceptance Criteria):

All the expected user outcomes and the acceptance criteria in the engineering epics are covered.

Background

OLM: Gateway to the OpenShift Ecosystem

Operator Lifecycle Manager (OLM) has been a game-changer for OpenShift Container Platform (OCP) 4.  Since its launch in 2019, OLM has fostered a rich ecosystem, expanding from a curated set of 25 operators to over 100 officially supported Red Hat operators and hundreds more from certified ISVs and the community.

OLM empowers users to manage diverse technologies with ease, including ACM, ACS, Quay, GitOps, Pipelines, Service Mesh, Serverless, and Virtualization.  It has also facilitated the introduction of groundbreaking operators for entirely new workloads, like Nvidia GPU, PTP, Windows Machine Config, SR-IOV networking, and more.  Today, a staggering 91% of our connected customers leverage OLM's capabilities.

OLM v0: A Stepping Stone

While OLM v0 has been instrumental, it has limitations.  The API design, not fully GitOps-friendly or entirely declarative, presents a steeper learning curve due to its complexity.  Furthermore, OLM v0 was designed with the assumption of namespace-scoped CRDs (Custom Resource Definitions), allowing for independent operator installations and parallel versions within a single cluster.  However, this functionality never materialized in core Kubernetes, and OLM v0's attempt to simulate it has introduced limitations and bugs.

The Operator Framework Team: Building the Future

The Operator Framework team is the cornerstone of the OpenShift ecosystem.  They build and manage OLM, the Operator SDK, operator catalog formats, and tooling (opm, file-based catalogs).  Their work directly impacts how operators are developed, packaged, delivered, and managed by users and SRE teams on OpenShift clusters.

A Streamlined Future with OLM v1

The Operator Framework team has undergone significant restructuring to focus on the next generation of OLM – OLM v1.  This transition includes moving the Operator SDK to a feature-complete state with ongoing maintenance for compatibility with the latest Kubernetes and controller-runtime libraries.  This strategic shift allows the team to dedicate resources to completely revamping OLM's API and management concepts for catalog content delivery.  

Leveraging learnings and customer feedback since OCP 4's inception, OLM v1 is designed to be a major overhaul, and it will be shipped as a Generally Available (GA) feature in OpenShift 4.17.

Customer Considerations

Provide any additional customer-specific considerations that must be made when designing and delivering the Feature.  Initial completion during Refinement status.

<your text here>

Documentation Considerations

1. Pre-installation:

• [GA release] Docs provide instructions on how to add Red Hat-provided Operator catalogs with the pull secret for catalogs hosted on a secure registry.

• [GA release] Docs provide instructions on how to discover the Operator packages from a catalog.

• [GA release] Docs provide instructions on how to query and inspect the metadata of Operator bundles and find feasible ones to be installed with the OLM v1.

2. Installation:

• [GA release] Docs provide instructions on how to use a ServiceAccount with sufficient permissions to install a k8s extension/operator with a desired target version or the latest version within a specific version range to get the latest security fixes.

• [GA release] Docs provide instructions on how to automate the installation flow declaratively with GitOps to ensure predictable and reliable deployments.

• [GA release] Docs mention the OLM v1’s protection from having two conflicting k8s extensions/operators owning the same API objects, i.e., no conflicting ownership, ensuring cluster stability.

• [GA release] Docs provide instructions on how to access the metadata of the installed k8s extension/operator to see essential information such as its provided APIs, example YAMLs of its provided APIs, descriptions, infrastructure features, valid subscriptions, etc.

• [GA release] Docs explain how to create RBACs from a CRD to grant cluster users access to the installed k8s extension/operator's provided APIs.

3. Update:

• [GA release] Docs provide instructions on how to see what updates are available for their k8s extension/operators in the form of immediate target versions and the associated update channels.

• [GA release] Docs provide instructions on how to trigger the update of a k8s extension/operator with a desired target version or the latest version within a specific version range to get the latest security fixes.

• [GA release] Docs mention OLM v1’s protection from workload or k8s extension/operator breakage due to CustomResourceDefinition (CRD) being upgraded to a backward incompatible version during an update.

• [GA release] Docs mention OLM v1 will block the OpenShift cluster update if installed k8s extensions/operators do not support the next OpenShift version (when annotated by the package author/provider).  Provide instructions on how to find and update to a newer/compatible version before OLM unblocks the OpenShift cluster update.

4. Uninstallation/Deletion:

• [GA release] Docs provide instructions on how to cleanly remove an installed k8s extension/operator including deleting CustomResourceDefinitions (CRDs), custom resource objects (CRs) of the CRDs, and other relevant resources.

• [GA release] Docs provide instructions to verify the cluster has been reverted to its original state after uninstalling a k8s extension/operator. 

Relevant upstream CNCF OLM v1 requirements, engineering brief, and epics:

1. Pre-installation:

• F1 - Extension catalogs
• F2 - Extension catalog discovery
• Brief​: Catalogd Content Storage and Serving 
• epic#242 Catalogd webserver uses HTTPS
• epic#239 Finalize Catalogd API Definitions for Phase 1 (API Review)

2. Installation:

• F7 - Extension installation
• Brief​: ClusterExtension Controller uses Helm for managing installed content
• Brief​: ClusterExtension support for simple registry+v1 bundles 
• epic#733 ClusterExtension uses Helm for deploying bundle content
• epic#734 ClusterExtension supports only: ‘Registry+v1 bundles’, ‘AllNamespaces mode’, ‘No webhooks’, ‘No dependencies’. If criteria not met, OLMv1 will block installation
• epic#735 Remove Extension API from main branch
• epic#736 Implement initial solution for no two (Cluster)Extension objects managing the same underlying object
• epic#737 ClusterExtension uses service account provided in spec
• epic#740 Finalize ClusterExtension API Definitions for Phase 1 (API Review)

3. Update:

• F10 - Extension updates
• F8 - Semver-based update policy
• Brief: CRD Upgrade Safety
• epic#657 CRD Upgrade Safety
• epic#738  Default and full support for Replaces, Skips, SkipRange
• epic#739 Support OperatorConditions expectations of registry+v1 bundles
• [Downstream epic] Inform when installed operators do not support the next Kubernetes minor version
• epic#740 Finalize ClusterExtension API Definitions for Phase 1 (API Review)

4. Uninstallation/Deletion:

• F17 - Extension cascading removal
• Brief​: ClusterExtension Controller uses Helm for managing installed content
• epic#733 ClusterExtension uses Helm for deploying bundle content
• epic#740 Finalize ClusterExtension API Definitions for Phase 1 (API Review)

Relevant documents:

• Next-gen OLM UX: Unifying workload management
• Next-gen OLM UX: Unifying workload management
• Operator Framework F2F - PM Session
• PRD OLM 1.0
• OLM F2F Discussion - Roadmap
• Post F2F Roadmap

Feature Overview (aka. Goal Summary)  

Customers who deploy a large number of OpenShift on OpenStack clusters want to minimise the resource requirements of their cluster control planes.

Customers deploying RHOSO (OpenShift services for OpenStack, i.e. OpenStack control plane on bare metal OpenShift) already have a bare metal management cluster capable of serving Hosted Control Planes.

We should enable self-hosted (i.e. on-prem) Hosted Control Planes to serve Hosted Control Planes to OpenShift on OpenStack clusters, with a specific focus of serving Hosted Control Planes from the RHOSO management cluster.

Goals (aka. expected user outcomes)

As an enterprise IT department and OpenStack customer, I want to provide self-managed OpenShift clusters to my internal customers with minimum cost to the business.

As an internal customer of said enterprise, I want to be able to provision an OpenShift cluster for myself using the business's existing OpenStack infrastructure.

Requirements (aka. Acceptance Criteria):

A list of specific needs or objectives that a feature must deliver in order to be considered complete.  Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc.  Initial completion during Refinement status.

TBD
 

Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed.  Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.

Use Cases (Optional):

Include use case diagrams, main success scenarios, alternative flow scenarios.  Initial completion during Refinement status.

<your text here>

Questions to Answer (Optional):

Include a list of refinement / architectural questions that may need to be answered before coding can begin.  Initial completion during Refinement status.

<your text here>

Out of Scope

High-level list of items that are out of scope.  Initial completion during Refinement status.

<your text here>

Background

Provide any additional context is needed to frame the feature.  Initial completion during Refinement status.

<your text here>

Customer Considerations

Provide any additional customer-specific considerations that must be made when designing and delivering the Feature.  Initial completion during Refinement status.

<your text here>

Documentation Considerations

Provide information that needs to be considered and planned so that documentation will meet customer needs.  If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.

<your text here>

Interoperability Considerations

Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact?  What interoperability test scenarios should be factored by the layered products?  Initial completion during Refinement status.

<your text here>

Feature Overview

Ability to install OpenShift on Nutanix with nodes having multiple NICs (multiple subnets) from IPI and for autoscaling with MachineSets.

Feature Overview

Implement authorization to secure API access for different user personas/actors in the agent-based installer.

User Personas:

• Read-Only Access: For "wait-for" and "monitor-add-nodes" commands.
• Read-Write Access: For systemd services and the agent service.

This is 

Goals

The agent-based installer APIs have implemented basic security measures through authentication, as covered in AGENT-145.

To further enhance security, it is crucial to implement user persona/actor-based authorization, allowing for differentiated access control, such as read-only or read-write permissions, based on the user's role.

The goal of this implementation is to provide a more robust and secure API framework, ensuring that users can only perform actions appropriate to their role.

Feature Overview

Improve the cluster expansion with the agent workflow added in OpenShift 4.16 (TP) and OpenShift 4.17 (GA) with:

• Caching RHCOS image for faster node addition, i.e. no extraction of image every time)
• Add a single node with just one command, no need to write config files describing node
• Support creating PXE artifacts 

Goals

Improve the user experience and functionality of the commands to add nodes to clusters using the image creation functionality.

Background

As part of being a first party Azure offering, ARO HCP needs to adhere to Microsoft secure supply chain software requirements. In order to do this, we require setting a label on all pods that run in the hosted cluster namespace.

Goal

Implement Mechanism for Labeling Hosted Cluster Control Plane Pods

Use-cases

• Adherance to Microsoft 1p Resource Provider Requirements

Components

• Any pods that hypershift deploys or run in the hosted cluster namespace.

Feature Overview (aka. Goal Summary)  

An elevator pitch (value statement) that describes the Feature in a clear, concise way.  Complete during New status.

<your text here>

Goals (aka. expected user outcomes)

The observable functionality that the user now has as a result of receiving this feature. Include the anticipated primary user type/persona and which existing features, if any, will be expanded. Complete during New status.

<your text here>

Requirements (aka. Acceptance Criteria):

A list of specific needs or objectives that a feature must deliver in order to be considered complete.  Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc.  Initial completion during Refinement status.

<enter general Feature acceptance here>

Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed.  Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.

Use Cases (Optional):

Include use case diagrams, main success scenarios, alternative flow scenarios.  Initial completion during Refinement status.

<your text here>

Questions to Answer (Optional):

Include a list of refinement / architectural questions that may need to be answered before coding can begin.  Initial completion during Refinement status.

<your text here>

Out of Scope

High-level list of items that are out of scope.  Initial completion during Refinement status.

<your text here>

Background

Provide any additional context is needed to frame the feature.  Initial completion during Refinement status.

<your text here>

Customer Considerations

Provide any additional customer-specific considerations that must be made when designing and delivering the Feature.  Initial completion during Refinement status.

<your text here>

Documentation Considerations

Provide information that needs to be considered and planned so that documentation will meet customer needs.  If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.

<your text here>

Interoperability Considerations

Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact?  What interoperability test scenarios should be factored by the layered products?  Initial completion during Refinement status.

<your text here>

Feature Overview (aka. Goal Summary)  

Add OpenStackLoadBalancerParameters and add an option for setting the load-balancer IP address for only those platforms where it can be implemented.

Goals (aka. expected user outcomes)

As a user of on-prem OpenShift, I need to manage DNS for my OpenShift cluster manually. I can already specify an IP address for the API server, but I cannot do this for Ingress. This means that I have to:

1. Manually create the API endpoint IP
2. Add DNS for the API endpoint
3. Create the cluster
4. Discover the created Ingress endpoint
5. Add DNS for the Ingress endpoint

I would like to simplify this workflow to:

1. Manually create the API and Ingress endpoint IPs
2. Add DNS for the API and Ingress endpoints
3. Create the cluster

Requirements (aka. Acceptance Criteria):

Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed.  Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.

Questions to Answer (Optional):

Out of Scope

• Although the Service API's loadBalancerIP API field was defined to be platform-agnostic, it wasn't consistently supported across platforms, and Kubernetes 1.24 has even deprecated it for this reason: https://github.com/kubernetes/kubernetes/pull/107235. We would not want to add a generic option to set loadBalancerIP given that it is deprecated and that it would work only on some platforms and not on others.

Background

• This request is similar to RFE-843 (for AWS), RFE-2238 (for GCP), RFE-2824 (for AWS and MetalLB, and maybe others), RFE-2884 (for AWS, Azure, and GCP), and RFE-3498 (for AWS). However, it might make sense to keep this RFE specifically for OpenStack.

Customer Considerations

Documentation Considerations

Interoperability Considerations

We need to do a lot of R&D and fix some known issues (e.g., see linked BZs). 

R&D targetted at 4.16 and productisation of this feature in 4.17

Feature Overview (aka. Goal Summary)  

Add support for the Installer to configure IPV4Subnet to customize internal OVN network in BYO VPC.

Goals (aka. expected user outcomes)

As an OpenShift user I'm able to provide IPv4 subnets to the Installer so I can customize the OVN networks at install time

Requirements (aka. Acceptance Criteria):

The Installer will allow the user to provide the information via the install config manifest and this information will be used at install time to configure the OVN network and deploy the cluster into an existing VPC provided by the user. 

Background

This is a requirement for ROSA, ARO and OSD

Documentation Considerations

As any other option for the Installer this will be documented as usual.

Implementation Considerations

Terraform is used for creating or referencing VPCs

Goal Summary

This feature aims to make sure that the HyperShift operator and the control-plane it deploys uses Managed Service Identities (MSI) and have access to scoped credentials (also via access to AKS's image gallery potentially). Additionally, for operators deployed in customers account (system components), they would be scoped with Azure workload identities. 

Goal:

As an administrator, I would like to use my own managed DNS solution instead of only specific openshift-install supported DNS services (such as AWS Route53, Google Cloud DNS, etc...) for my OpenShift deployment.

Problem:

While cloud-based DNS services provide convenient hostname management, there's a number of regulatory (ITAR) and operational constraints customers face prohibiting the use of those DNS hosting services on public cloud providers.

Why is this important:

• Provides customers with the flexibility to leverage their own custom managed ingress DNS solutions already in use within their organizations.
• Required for regions like AWS GovCloud in which many customers may not be able to use the Route53 service (only for commercial customers) for both internal or ingress DNS.
• OpenShift managed internal DNS solution ensures cluster operation and nothing breaks during updates.

Dependencies (internal and external):

• DNS work for KNI
• https://docs.google.com/document/d/1VsukDGafynKJoQV8Au-dvtmCfTjPd3X9Dn7zltPs8Cc/edit

• This is a prerequisite for the internal clusters epic:   https://docs.google.com/document/d/1gxtIW6OlasVQtQLTyOl6f9H9CMuxiDNM5hQFNd3xubE/edit#

Prioritized epics + deliverables (in scope / not in scope):

• Ability to bootstrap cluster without an OpenShift managed internal DNS service running yet
• Scalable, cluster (internal) DNS solution that’s not dependent on the operation of the control plane (in case it goes down)
• Ability to automatically propagate DNS record updates to all nodes running the DNS service within the cluster
• Option for connecting cluster to customers ingress DNS solution already in place within their organization

Estimate (XS, S, M, L, XL, XXL):

Previous Work:

Open questions:

Link to Epic: https://docs.google.com/document/d/1OBrfC4x81PHhpPrC5SEjixzg4eBnnxCZDr-5h3yF2QI/edit?usp=sharing

Motivation:

Content-Security-Policy (CSP) header provides a defense-in-depth measure in client-side security, as a second layer of protection against Cross-site Scripting (XSS) and clickjacking attacks.

It is not yet implemented in the OpenShift web console, however, there are some other related security headers present in the OpenShift console that cover some aspects of CSP functionality:

• X-Frame-Options: When set to DENY, this disallows allow attempts to iframe site (related CSP directive: `frame-ancestors`)
• X-XSS-Protection: Protects against reflected XSS attacks in Chrome and Internet Explorer (related CSP directive: `unsafe-inline`)
• X-Content-Type-Options: Protects against loading of external scripts and stylesheets unless the server indicates the correct MIME type, which can lead to some types of XSS attacks.

Epic Goal

• Implement CSP in a two phases
  • 1. Report only, in which the console will report the violations to the user and to the telemetry, so developers can have a better idea what type of violations are appearing.
    • This phase should ideally take place during several releases, at least two during which, data about the appeared violations would be gathered through telemetry. This will get plugin creators time to adapt to the change.
    • During this phase an API needs to be added to the ConsolePlugin CRD, to give plugin creators option how to provide list of their content sources, which console will register and take into account.
    • Also Console itself should remove any CSP violations which is causing.
  • 2. Enforcing, in which console will start enforcing the CSP, which will prevent plugins from loading assets from non-registered sources.

Why is this important?

• Add additional security level to console

Acceptance Criteria

• Implement 1 phase of the CSP
  • report only mode
  • surface the violations to the user
  • extend the ConsolePlugins API to give plugin creators a option how to extend plugin's source
  • extend CI to catch any CSP violations
  • update README and announce this feature so developers are aware of it

Open questions::

1. TBD

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

See proposal: https://docs.google.com/document/d/1-MgvfVamsv4vOWtM1RD49vCFMt2ZIr4QNKVTROkDEP4/edit?usp=sharing

Epic Goal
Through this epic, we will update our CI to use a have an available agent-based workflow instead of the libvirt openshift-installer, allowing us to eliminate the use of terraform in our deployments.

Why is this important?
There is an active initiative in openshift to remove terraform from the openshift installer.

Acceptance Criteria

• All tasks within the epic are completed.

Done Checklist

• CI - For new features (non-enablement), existing Multi-Arch CI jobs are not broken by the Epic
• All the stories, tasks, sub-tasks and bugs that belong to this epic need to have been completed and indicated by a status of 'Done'.

The history of this epic starts with this PR which triggered a lengthy conversation around the workings of the image  API with respect to importing imagestreams  images as single vs manifestlisted. The imagestreams today by default have the `importMode` flag set to `Legacy` to avoid breaking behavior of existing clusters in the field. This makes sense for single arch clusters deployed with a single  arch payload, but when users migrate to use the multi payload, more often than not, their intent is to add nodes of other architecture types. When this happens - it gives rise to problems when using imagestreams with the default behavior of importing a single manifest image. The oc commands do have a new flag to toggle the importMode, but this breaks functionality of existing users who just want to create an imagestream and use it with existing commands.

There was a discussion with David Eads and other staff engineers and it was decided that the approach to be taken is to default imagestreams' importMode to `preserveOriginal` if the cluster is installed with/ upgraded to a multi payload. So a few things need to happen to achieve this:

• CVO would need to expose a field in the status section indicative of the type of payload in the cluster (single vs multi)
• cluster-openshift-apiserver-operator would read this field and add it to the apiserver configmap. openshift-apiserver would use this value to determine the setting of importMode value.
• Document clearly that the behavior of imagestreams in a cluster with multi payload is different from the traditional single payload

Some open questions:

• What happens to existing imagestreams on upgrades
• How do we handle CVO managed imagestreams (IMO, CVO managed imagestreams should always set importMode to preserveOriginal as the images are associated with the payload)

Epic Goal

• Update all images that we ship with OpenShift to the latest upstream releases and libraries.
• Exact content of what needs to be updated will be determined as new images are released upstream, which is not known at the beginning of OCP development work. We don't know what new features will be included and should be tested and documented. Especially new CSI drivers releases may bring new, currently unknown features. We expect that the amount of work will be roughly the same as in the previous releases. Of course, QE or docs can reject an update if it's too close to deadline and/or looks too big.

Traditionally we did these updates as bugfixes, because we did them after the feature freeze (FF).

Why is this important?

• We want to ship the latest software that contains new features and bugfixes.

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.

Description of problem:

During the integration of Manila into csi-operator a new controller was added to csi-operator that checks if a precondition is valid in order to trigger all the other controllers. The precondition defined for manila checks that manila shares exists and if that is the case it syncs the CSI Driver and the Storage Classes. We need to handle the error returned in case those syncs fail.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

    1.
    2.
    3.
    

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/csi-operator/pull/318

Description of problem:

The finally tasks do not get removed and remain in the pipeline.    

Version-Release number of selected component (if applicable):

    In all supported OCP version

How reproducible:

    Always

Steps to Reproduce:

1. Create a finally task in a pipeline in pipeline builder
2. Save pipeline
3. Edit pipeline and remove finally task in pipeline builder
4. Save pipeline
5. Observe that the finally task has not been removed

Actual results:

The finally tasks do not get removed and remain in the pipeline.    

Expected results:

Finally task gets removed from pipeline when removing the finally tasks and saving the pipeline in the "pipeline builder" mode.    

Additional info:

• https://github.com/openshift/console/pull/14472

• https://github.com/openshift/insights-operator/pull/1031

Description of problem:

    `tag:UntagResources` is required for the AWS SDK call [UntagResourcesWithContext](https://github.com/openshift/installer/blob/master/pkg/destroy/aws/shared.go#L121) when removing the "shared" tag from the IAM profile.

Version-Release number of selected component (if applicable):

    4.17+

How reproducible:

    always

Steps to Reproduce:

    1.
    2.
    3.
    

Actual results:

    time="2024-11-19T12:22:19Z" level=debug msg="search for IAM instance profiles"
time="2024-11-19T12:22:19Z" level=debug msg="Search for and remove tags in us-east-1 matching kubernetes.io/cluster/ci-op-y8wbktiq-e515e-q6kvb: shared"
time="2024-11-19T12:22:19Z" level=debug msg="Nothing to clean for shared iam resource" arn="arn:aws:iam::460538899914:instance-profile/ci-op-y8wbktiq-e515e-byo-profile-worker"
time="2024-11-19T12:22:19Z" level=debug msg="Nothing to clean for shared iam resource" arn="arn:aws:iam::460538899914:instance-profile/ci-op-y8wbktiq-e515e-byo-profile-master"
time="2024-11-19T12:22:19Z" level=info msg="untag shared resources: AccessDeniedException: User: arn:aws:iam::460538899914:user/ci-op-y8wbktiq-e515e-minimal-perm is not authorized to perform: tag:UntagResources because no identity-based policy allows the tag:UntagResources action\n\tstatus code: 400, request id: 464de6ab-3de5-496d-a163-594dade11619"

See: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/58833/rehearse-58833-pull-ci-openshift-installer-release-4.18-e2e-aws-ovn-custom-iam-profile/1858807924600606720

Expected results:

    The perm is added to the required list when BYO IAM profile and the "shared" tag is removed from the profiles.

Additional info:

• https://github.com/openshift/installer/pull/9222

Component Readiness has found a potential regression in the following test:

install should succeed: infrastructure

installer fails with:

time="2024-10-20T04:34:57Z" level=error msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: control-plane machines were not provisioned within 15m0s: client rate limiter Wait returned an error: context deadline exceeded" 

Significant regression detected.
Fishers Exact probability of a regression: 99.96%.
Test pass rate dropped from 98.94% to 89.29%.

Sample (being evaluated) Release: 4.18
Start Time: 2024-10-14T00:00:00Z
End Time: 2024-10-21T23:59:59Z
Success Rate: 89.29%
Successes: 25
Failures: 3
Flakes: 0

Base (historical) Release: 4.17
Start Time: 2024-09-01T00:00:00Z
End Time: 2024-10-01T23:59:59Z
Success Rate: 98.94%
Successes: 93
Failures: 1
Flakes: 0

View the test details report at https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Aggregation=none&Architecture=amd64&FeatureSet=default&Installer=ipi&Network=ovn&NetworkAccess=default&Platform=azure&Scheduler=default&SecurityMode=default&Suite=serial&Topology=ha&Upgrade=none&baseEndTime=2024-10-01%2023%3A59%3A59&baseRelease=4.17&baseStartTime=2024-09-01%2000%3A00%3A00&capability=Other&columnGroupBy=Architecture%2CNetwork%2CPlatform&component=Installer%20%2F%20openshift-installer&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20azure%20serial%20ha%20none&ignoreDisruption=true&ignoreMissing=false&includeVariant=Architecture%3Aamd64&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Owner%3Aeng&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&sampleEndTime=2024-10-21%2023%3A59%3A59&sampleRelease=4.18&sampleStartTime=2024-10-14%2000%3A00%3A00&testId=cluster%20install%3A3e14279ba2c202608dd9a041e5023c4c&testName=install%20should%20succeed%3A%20infrastructure

• https://github.com/openshift/installer/pull/9226

Description of problem:

    If zones are not specified in the install-config.yaml, the installer will discover all the zones available for the region. Then it will try to filter those zones based on the instance type, which requires the `ec2:DescribeInstanceTypeOfferings` permission.

Version-Release number of selected component (if applicable):

    4.16+

How reproducible:

    Always by not specifying zones in the install-config.yaml and installing cluster with a minimal permissions user.

Steps to Reproduce:

    1.
    2.
    3.
    

Actual results:

    TBA

Expected results:

    A failure message indicating that `ec2:DescribeInstanceTypeOfferings` is need when zones are not set.

Additional info:

• https://github.com/openshift/installer/pull/9114

In payloads 4.18.0-0.ci-2024-11-01-110334 and 4.18.0-0.nightly-2024-11-01-101707 we observed GCP install failures

 Container test exited with code 3, reason Error
---
ails:
level=error msg=[
level=error msg=  {
level=error msg=    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
level=error msg=    "domain": "googleapis.com",
level=error msg=    "metadatas": {
level=error msg=      "consumer": "projects/711936183532",
level=error msg=      "quota_limit": "ListRequestsFilterCostOverheadPerMinutePerProject",
level=error msg=      "quota_limit_value": "75",
level=error msg=      "quota_location": "global",
level=error msg=      "quota_metric": "compute.googleapis.com/filtered_list_cost_overhead",
level=error msg=      "service": "compute.googleapis.com"
level=error msg=    },
level=error msg=    "reason": "RATE_LIMIT_EXCEEDED"
level=error msg=  },
level=error msg=  {
level=error msg=    "@type": "type.googleapis.com/google.rpc.Help",
level=error msg=    "links": [
level=error msg=      {
level=error msg=        "description": "The request exceeds API Quota limit, please see help link for suggestions.",
level=error msg=        "url": "https://cloud.google.com/compute/docs/api/best-practices#client-side-filter"
level=error msg=      }
level=error msg=    ]
level=error msg=  }
level=error msg=]
level=error msg=, rateLimitExceeded 

Patrick Dillon Noted ListRequestsFilterCostOverheadPerMinutePerProject can not have it's quota limit increased.

The problem subsided over the weekend presumably with fewer jobs run but has started to appear again. opening to track ongoing issue and potential work arounds.

This contributes to the following test failures for GCP

install should succeed: configuration
install should succeed: overall

• https://github.com/openshift/installer/pull/9202

• https://github.com/openshift/machine-config-operator/pull/4713

IBI and IBU use diffrent lables for var-lib-containers partitioin.
This results in failure to mount the partition in case of label mismatch (var-lib-containers vs varlibcontainers).
We should always use `var-lib-containers` as the label

See more details in the slack thread
https://redhat-internal.slack.com/archives/C05JHD9QYTC/p1731542185936629

Installer part

https://github.com/openshift/installer/blob/master/pkg/asset/imagebased/image/imagebased_config.go#L33

https://github.com/openshift/installer/blob/master/pkg/types/imagebased/imagebased_config_types.go#L65

Lca part (less interesting as config will be generated in installer)

https://github.com/openshift-kni/lifecycle-agent/blob/main/api/ibiconfig/ibiconfig.go#L20

• https://github.com/openshift/installer/pull/9217

Description of problem:

[vmware-vsphere-csi-driver-operator] driver controller/node/webhook update events repeat pathologically    

Version-Release number of selected component (if applicable):

4.18.0-0.nightly-2024-11-03-161006    

How reproducible:

Always    

Steps to Reproduce:

    1. Install an Openshift cluster on vSphere of version 4.17 nightly.
    2. Upgrade the cluster to 4.18 nightly.
    3. Check the driver controller/node/webhook update events should not repeat pathologically.     

CI failure record -> https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.18-upgrade-from-stable-4.17-e2e-vsphere-ovn-upgrade/1854191939318976512  

Actual results:

 In step 3: the driver controller/node/webhook update events repeat pathologically   

Expected results:

 In step 3: the driver controller/node/webhook update events should not repeat pathologically    

Additional info:

• https://github.com/openshift/api/pull/2105