Feature Based Change Log

Field	Description
retentionSize	Maximum amount of disk space used by blocks. Supported units: B, KB, MB, GB, TB, PB, EB. Ex: 512MB.

Requirement	Notes	isMvp?
CI - MUST be running successfully with test automation	This is a requirement for ALL features.	YES
Release Technical Enablement	Provide necessary release enablement details and documents.	YES

Requirement	Notes	isMvp?
CI - MUST be running successfully with test automation	This is a requirement for ALL features.	YES
Release Technical Enablement	Provide necessary release enablement details and documents.	YES

Requirement	Notes	isMvp?
CI - MUST be running successfully with test automation	This is a requirement for ALL features.	YES
Release Technical Enablement	Provide necessary release enablement details and documents.	YES

Requirement	Notes	isMvp?
Secrets and ConfigMaps can get shared across namespaces		YES

Bug OCPBUGS-1972: [IPI on Baremetal] ipv6 support issue in metal3-httpd

View the Description View the linked PRs

Description of problem:

The metal3 Pod of openshift-machine-api is in CrashLoopBackOff status.

Version-Release number of selected component (if applicable):

4.10.31

How reproducible:

Always reproduce in IPv6

Steps to Reproduce:

1. Preparing to configure ipi on the provisioning node
   - RHEL 8 ( haproxy, named, mirror registry, rhcos_cache_server ..)

2. configuring the install-config.yaml (attached)
   - provisioningNetwork: disabled
   - machine network: only IPv6
   - disconnected installation
   
3. deploy the cluster

Actual results:

It is not possible to add worker nodes because metal3 does not start normally.

Expected results:

metal3 starts normally in IPv6 environment

Additional info:

1. attached must-gather
https://drive.google.com/file/d/1GKxj3syROIMnURx_PYzOYhJdEuXBNXVW/view?usp=sharing

2. pod status
[kni@prov ~]$ oc get pod
NAME                                           READY   STATUS             RESTARTS          AGE
cluster-autoscaler-operator-6656bfd7b9-bt4j8   2/2     Running            0                 35h
cluster-baremetal-operator-6bbdd6758-rmxgq     2/2     Running            0                 35h
machine-api-controllers-55fb545b56-kl5sj       7/7     Running            0                 34h
machine-api-operator-845b6cf855-q7gdd          2/2     Running            0                 35h
metal3-574876cfdb-98fmz                        6/7     CrashLoopBackOff   179 (3m17s ago)   14h
metal3-image-cache-5mq2w                       1/1     Running            0                 14h
metal3-image-cache-nftpj                       1/1     Running            0                 14h
metal3-image-cache-t7whh                       1/1     Running            0                 14h
metal3-image-customization-68d4d6d99b-dbqgn    1/1     Running            0                 15h

[kni@prov ~]$ oc logs metal3-574876cfdb-98fmz -c metal3-httpd
AH00526: Syntax error on line 8 of /etc/httpd/conf.d/vmedia.conf:
The port number "2001:feed:101::102" is outside the appropriate range (i.e., 1..65535).

https://github.com/openshift/ironic-image/pull/306

Bug OCPBUGS-2112: [release-4.11] Address e2e failures due to pod security

View the Description View the linked PRs

Description of problem:

Manual backport of 
* https://github.com/openshift/cluster-dns-operator/pull/336
* https://github.com/openshift/cluster-dns-operator/pull/339

Version-Release number of selected component (if applicable):

4.11

https://github.com/openshift/cluster-dns-operator/pull/347

Bug OCPBUGS-2626: Worker creation fails within provider networks (as primary and secondary)

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-2508~~. The following is the description of the original issue:
—
Description of problem:

Installer fails due to Neutron policy error when creating Openstack servers for OCP master nodes.

$ oc get machines -A
NAMESPACE               NAME                          PHASE          TYPE   REGION   ZONE   AGE
openshift-machine-api   ostest-kwtf8-master-0         Running                               23h
openshift-machine-api   ostest-kwtf8-master-1         Running                               23h
openshift-machine-api   ostest-kwtf8-master-2         Running                               23h
openshift-machine-api   ostest-kwtf8-worker-0-g7nrw   Provisioning                          23h
openshift-machine-api   ostest-kwtf8-worker-0-lrkvb   Provisioning                          23h
openshift-machine-api   ostest-kwtf8-worker-0-vwrsk   Provisioning                          23h

$ oc -n openshift-machine-api logs machine-api-controllers-7454f5d65b-8fqx2 -c machine-controller
[...]
E1018 10:51:49.355143       1 controller.go:317] controller/machine_controller "msg"="Reconciler error" "error"="error creating Openstack instance: Failed to create port err: Request forbidden: [POST https://overcloud.redhat.local:13696/v2.0/ports], error message: {\"NeutronError\": {\"type\": \"PolicyNotAuthorized\", \"message\": \"(rule:create_port and (rule:create_port:allowed_address_pairs and (rule:create_port:allowed_address_pairs:ip_address and rule:create_port:allowed_address_pairs:ip_address))) is disallowed by policy\", \"detail\": \"\"}}" "name"="ostest-kwtf8-worker-0-lrkvb" "namespace"="openshift-machine-api"

Version-Release number of selected component (if applicable):

4.10.0-0.nightly-2022-10-14-023020

How reproducible:

Always

Steps to Reproduce:

1. Install 4.10 within provider networks (in primary or secondary interface)

Actual results:

Installation failure:
4.10.0-0.nightly-2022-10-14-023020: some cluster operators have not yet rolled out

Expected results:

Successful installation

Additional info:

Please find must-gather for installation on primary interface link here and for installation on secondary interface link here.

https://github.com/openshift/cluster-api-provider-openstack/pull/248

Bug OCPBUGS-2883: [release-4.11] Make ccoctl set sts endpoints to regional in AWS credentials secrets

View the Description View the linked PRs

This bug card represents work done in https://issues.redhat.com/browse/CCO-257 to set STS endpoints to regional in AWS credentials secrets and is created to facilitate backporting the change to previous releases as required by the backport process [1].

[1] https://docs.google.com/document/d/1qbKUl4K9PPW4tCbdDgWSXd5klosX77ifkxQLqzjztGE/edit#heading=h.mml0rz88o0nj

https://github.com/openshift/cloud-credential-operator/pull/503

Bug OCPBUGS-6684: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/cloud-provider-openstack/pull/176

Bug OCPBUGS-10487: Enable node healthz server for ovnk in CNO

View the Description View the linked PRs

Node healthz server was added in 4.13 with https://github.com/openshift/ovn-kubernetes/commit/c8489e3ff9c321e77f265dc9d484ed2549df4a6b and https://github.com/openshift/ovn-kubernetes/commit/9a836e3a547f3464d433ce8b9eef336624d51858. We need to configure it by default on 0.0.0.0:10256 on CNO for ovnk, just like we do for sdn.

https://github.com/openshift/cluster-network-operator/pull/1742

Feature Request RFE-2703: OCP should alarm/alert when the etcd container memory consumption goes beyond 90%

View the Description View the linked PRs

1. Proposed title of this feature request
--> Alert generation when the etcd container memory consumption goes beyond 90%

2. What is the nature and description of the request?
--> When the etcd database starts growing rapidly due to some high number of objects like secrets, events, or configmap generation by application/workload, the memory and CPU consumption of APIserver and etcd container (control plane component) spikes up and eventually the control plane nodes goes to hung/unresponsive or crash due to out of memory errors as some of the critical processes/services running on master nodes get killed. Hence we request an alert/alarm when the ETCD container's memory consumption goes beyond 90% so that the cluster administrator can take some action before the cluster/nodes go unresponsive.

I see we already have a etcdExcessiveDatabaseGrowth Prometheus rule which helps when the surge in etcd writes leading to a 50% increase in database size over the past four hours on etcd instance however it does not consider the memory consumption:

$ oc get prometheusrules etcd-prometheus-rules -o yaml|grep -i etcdExcessiveDatabaseGrowth -A 9

alert: etcdExcessiveDatabaseGrowth
annotations:
description: 'etcd cluster "{{ $labels.job }}": Observed surge in etcd writes
leading to 50% increase in database size over the past four hours on etcd
instance {{ $labels.instance }}, please check as it might be disruptive.'
expr: |
increase(((etcd_mvcc_db_total_size_in_bytes/etcd_server_quota_backend_bytes)*100)[240m:1m]) > 50
for: 10m
labels:
severity: warning

3. Why does the customer need this? (List the business requirements here)
--> Once the etcd memory consumption goes beyond 90-95% of total ram as it's system critical container, the OCP cluster goes unresponsive causing revenue loss to business and impacting the productivity of users of the openshift cluster.

4. List any affected packages or components.
--> etcd

https://github.com/openshift/machine-config-operator/pull/3124

Bug OCPBUGS-2077: List pages in pipelines is taking more time to load when there are too many items

View the Description View the linked PRs

Description of problem:

Pipeline list page fetches all the pipelineruns to find the last pipeline run and which results in more load time. This performance issue needs to be addressed in all the pieplines list pages wherever applicable.

Version-Release number of selected component (if applicable):
4.9

How reproducible:
Always

Steps to Reproduce:
1. Create 10+ pipelines in a namespace
2. Create more number of pipelineruns under each pipeline
3. navigate to piplines list page.

Actual results:
Pipelines list will take a long time to load the list.

Expected results:

Pipeline list should not take more time to load the list.

Additional info:

Reduce the amount to data fetched to find the last pipelinerun, maybe use PartialMetadata to find the latest pipeline run and to improve the performance.

https://github.com/openshift/console/pull/12144

Bug OCPBUGS-4081: Fails to deprovision cluster when swift omits 'content-type'

View the Description View the linked PRs

Description of problem: This is a follow-up to ~~OCPBUGS-2795~~ and ~~OCPBUGS-2941~~.

The installer fails to destroy the cluster when the OpenStack object storage omits 'content-type' from responses. This can happen on responses with HTTP status code 204, where a reverse proxy is truncating content-related headers (see this nginX bug report). In such cases, the Installer errors with:

level=error msg=Bulk deleting of container "5ifivltb-ac890-chr5h-image-registry-fnxlmmhiesrfvpuxlxqnkoxdbl" objects failed: Cannot extract names from response with content-type: []

Listing container object suffers from the same issue as listing the containers and this one isn't fixed in latest versions of gophercloud. I've reported https://github.com/gophercloud/gophercloud/issues/2509 and fixing it with https://github.com/gophercloud/gophercloud/issues/2510, however we likely won't be able to backport the bump to gophercloud master back to release-4.8 so we'll have to look for alternatives.

I'm setting the priority to critical as it's causing all our jobs to fail in master.

Version-Release number of selected component (if applicable):

4.8.z

How reproducible:

Likely not happening in customer environments where Swift is exposed directly. We're seeing the issue in our CI where we're using a non-RHOSP managed cloud.

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/installer/pull/6625

Bug OCPBUGS-955: [2087981] PowerOnVM_Task is deprecated use PowerOnMultiVM_Task for DRS ClusterRecommendation

View the Description View the linked PRs

Description of problem:

Sometimes we see VMs fail to power on when the land on a host that does not have enough resources. The current power on does not retry or leverage DRS to power on the node on a suitable host.

https://github.com/vmware/govmomi/issues/1026

Our code is still making calls to PowerOnVM_Task which, according to the vsphere docs, is deprecated and we should use PowerOnMultiVM_Task instead.

PowerOnVM_Task does not return a DRS ClusterRecommendation, no vmotion nor host power operations will be done as part of a DRS-facilitated power on. To have DRS consider such operations use PowerOnMultiVM_Task.

https://vdc-download.vmware.com/vmwb-repository/dcr-public/b50dcbbf-051d-4204-a3e7-e1b618c1e384/538cf2ec-b34f-4bae-a332-3820ef9e7773/vim.VirtualMachine.html#powerOn:
As of vSphere API 5.1, use of this method with vCenter Server is deprecated; use PowerOnMultiVM_Task instead.

Version-Release number of selected component (if applicable):
4.8.x

How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:
Sometimes powers on fails requiring manual intervention.

Expected results:
PowerOn should use DRS to ensure it's always successful.

Additional info:

https://github.com/openshift/machine-api-operator/pull/1064

Bug OCPBUGS-1511: [OCP 4.11] Fix generate script in CBO

View the Description View the linked PRs

plus smaller fixes
this is a backport of https://github.com/openshift/cluster-baremetal-operator/pull/263

https://github.com/openshift/cluster-baremetal-operator/pull/290

Bug OCPBUGS-4887: [4.11] Pods completed + deleted may leak

View the Description View the linked PRs

Description of problem:

When a pod runs to a completed state, we typically rely on the update event that will indicate to us that this pod is completed. At that point the pod IP is released and the port configuration is removed in OVN. The subsequent delete event for this pod will be ignored because it should have been cleaned up in the previous update.

However, there can be cases where the update event is missed with pod completed. In this case we will only receive a delete with pod completed event, and ignore tearing down the pod. The end result is the pod is not cleaned up in OVN and the IP address remains allocated, reducing the amount of address range available to launch another pod. This can lead to exhausting all IP addresses available for pod allocation on a node.

Version-Release number of selected component (if applicable):

4.10.24

How reproducible:

Not sure how to reproduce this. I'm guessing some lag in kapi updates can cause the completed update event and the final delete event to be combined into a single event.

Steps to Reproduce:

1.
2.
3.

Actual results:

Port still exists in OVN, IP remains allocated for a deleted pod.

Expected results:

IP should be freed, port should be removed from OVN.

Additional info:

https://github.com/openshift/ovn-kubernetes/pull/1455

Bug OCPBUGS-668: Prefer local dns does not work expectedly on OCPv4.11

View the Description View the linked PRs

Description of problem:

When queried dns hostname from certain pod on the certain node, responded from random coredns pod, not prefer local one. Is it expected result ?

# In OCP v4.8.13 case
// Ran dig command on the certain node which is running the following test-7cc4488d48-tqc4m pod.
sh-4.4# while : ; do echo -n "$(date '+%H:%M:%S') :"; dig google.com +short; sleep 1; done
:
07:16:33 :172.217.175.238
07:16:34 :172.217.175.238 <--- Refreshed the upstream result
07:16:36 :142.250.207.46
07:16:37 :142.250.207.46

// The dig results is matched with the running node one as you can see the above one.
$ oc rsh  test-7cc4488d48-tqc4m bash -c 'while : ; do echo -n "$(date '+%H:%M:%S') :"; dig google.com +short; sleep 1; done'
:
07:16:35 :172.217.175.238 
07:16:36 :172.217.175.238 <--- At the same time, the pod dig result is also refreshed.
07:16:37 :142.250.207.46
07:16:38 :142.250.207.46


But in v4.10 case, in contrast, the dns query result is various and responded randomly regardless local dns results on the node as follows.

# In OCP v4.10.23 case, pod's response from DNS services are not consistent.
$ oc rsh test-848fcf8ddb-zrcbx  bash -c 'while : ; do echo -n "$(date '+%H:%M:%S') :"; dig google.com +short; sleep 1; done'
07:23:00 :142.250.199.110
07:23:01 :142.250.207.46
07:23:02 :142.250.207.46
07:23:03 :142.250.199.110
07:23:04 :142.250.199.110
07:23:05 :172.217.161.78

# Even though the node which is running the pod keep responding the same IP...
sh-4.4# while : ; do echo -n "$(date '+%H:%M:%S') :"; dig google.com +short; sleep 1; done
07:23:00 :172.217.161.78
07:23:01 :172.217.161.78
07:23:02 :172.217.161.78
07:23:03 :172.217.161.78
07:23:04 :172.217.161.78
07:23:05 :172.217.161.78

Version-Release number of selected component (if applicable):

v4.10.23 (ROSA)
SDN: OpenShiftSDN

How reproducible:

You can always reproduce this issue using "dig google.com" from both any pod and the node the pod running according to the above "Description" details.

Steps to Reproduce:

1. Run any usual pod, and check which node the pod is running on.
2. Run dig google.com on the pod and the node.
3. Check the IP is consistent with the running node each other.

Actual results:

The response IPs are not consistent and random IP is responded.

Expected results:

The response IP is kind of consistent, and aware of prefer local dns.

Additional info:

This issue affects EgressNetworkPolicy dnsName feature.

Bug OCPBUGS-2123: [Azure]Availability Set will be created when vmSize is invalid in a region which has zones

View the Description View the linked PRs

Description of problem:

Availability Set will be created when vmSize is invalid in a region which has zones, but Availability Set should only be created in a region which don’t have zones.

Version-Release number of selected component (if applicable):

4.11.0-0.nightly-2022-10-07-174524
4.10.0-0.nightly-2022-10-07-205844

How reproducible:

Always

Steps to Reproduce:

1.Set up a cluster in a region which has zones. 
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME                                     PHASE     TYPE              REGION   ZONE   AGE
huliu-az410-99qcm-master-0               Running   Standard_D8s_v3   eastus   2      34m
huliu-az410-99qcm-master-1               Running   Standard_D8s_v3   eastus   3      34m
huliu-az410-99qcm-master-2               Running   Standard_D8s_v3   eastus   1      34m
huliu-az410-99qcm-worker-eastus1-xld58   Running   Standard_D4s_v3   eastus   1      27m
huliu-az410-99qcm-worker-eastus2-chzg8   Running   Standard_D4s_v3   eastus   2      27m
huliu-az410-99qcm-worker-eastus3-7g2mw   Running   Standard_D4s_v3   eastus   3      27m

2.Create a machineset with invalid vmSize 
liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms4.yaml 
machineset.machine.openshift.io/huliu-az410-99qcm-1 created
liuhuali@Lius-MacBook-Pro huali-test % oc get machine                                                       
NAME                                     PHASE     TYPE              REGION   ZONE   AGE
huliu-az410-99qcm-1-cfw6w                Failed                                      8s
huliu-az410-99qcm-master-0               Running   Standard_D8s_v3   eastus   2      35m
huliu-az410-99qcm-master-1               Running   Standard_D8s_v3   eastus   3      35m
huliu-az410-99qcm-master-2               Running   Standard_D8s_v3   eastus   1      35m
huliu-az410-99qcm-worker-eastus1-xld58   Running   Standard_D4s_v3   eastus   1      28m
huliu-az410-99qcm-worker-eastus2-chzg8   Running   Standard_D4s_v3   eastus   2      28m
huliu-az410-99qcm-worker-eastus3-7g2mw   Running   Standard_D4s_v3   eastus   3      28m
liuhuali@Lius-MacBook-Pro huali-test % oc get machine huliu-az410-99qcm-1-cfw6w  -o yaml
apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  annotations:
    machine.openshift.io/instance-state: Unknown
  creationTimestamp: "2022-10-08T07:42:28Z"
  finalizers:
  - machine.machine.openshift.io
  generateName: huliu-az410-99qcm-1-
  generation: 2
  labels:
    machine.openshift.io/cluster-api-cluster: huliu-az410-99qcm
    machine.openshift.io/cluster-api-machine-role: worker
    machine.openshift.io/cluster-api-machine-type: worker
    machine.openshift.io/cluster-api-machineset: huliu-az410-99qcm-1
  name: huliu-az410-99qcm-1-cfw6w
  namespace: openshift-machine-api
  ownerReferences:
  - apiVersion: machine.openshift.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: MachineSet
    name: huliu-az410-99qcm-1
    uid: bf8f7518-1fa9-4704-bdd7-6d0fde54e38e
  resourceVersion: "31287"
  uid: 303cf672-a2fa-44f3-8793-59801bb78902
spec:
  lifecycleHooks: {}
  metadata: {}
  providerSpec:
    value:
      apiVersion: machine.openshift.io/v1beta1
      credentialsSecret:
        name: azure-cloud-credentials
        namespace: openshift-machine-api
      image:
        offer: ""
        publisher: ""
        resourceID: /resourceGroups/huliu-az410-99qcm-rg/providers/Microsoft.Compute/images/huliu-az410-99qcm
        sku: ""
        version: ""
      kind: AzureMachineProviderSpec
      location: eastus
      managedIdentity: huliu-az410-99qcm-identity
      metadata:
        creationTimestamp: null
        name: huliu-az410-99qcm
      networkResourceGroup: huliu-az410-99qcm-rg
      osDisk:
        diskSettings: {}
        diskSizeGB: 128
        managedDisk:
          storageAccountType: Premium_LRS
        osType: Linux
      publicIP: false
      publicLoadBalancer: huliu-az410-99qcm
      resourceGroup: huliu-az410-99qcm-rg
      spotVMOptions: {}
      subnet: huliu-az410-99qcm-worker-subnet
      userDataSecret:
        name: worker-user-data
      vmSize: invalidStandard_D4s_v3
      vnet: huliu-az410-99qcm-vnet
      zone: "3"
status:
  conditions:
  - lastTransitionTime: "2022-10-08T07:42:28Z"
    status: "True"
    type: Drainable
  - lastTransitionTime: "2022-10-08T07:42:28Z"
    message: Instance has not been created
    reason: InstanceNotCreated
    severity: Warning
    status: "False"
    type: InstanceExists
  - lastTransitionTime: "2022-10-08T07:42:28Z"
    status: "True"
    type: Terminable
  errorMessage: 'failed to reconcile machine "huliu-az410-99qcm-1-cfw6w": failed to
    create vm huliu-az410-99qcm-1-cfw6w: failure sending request for machine huliu-az410-99qcm-1-cfw6w:
    cannot create vm: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending
    request: StatusCode=0 -- Original Error: Code="BadRequest" Message="Virtual Machine
    cannot be created because both Availability Zone and Availability Set were specified.
    Deploying an Availability Set to an Availability Zone isn’t supported."'
  errorReason: InvalidConfiguration
  lastUpdated: "2022-10-08T07:42:35Z"
  phase: Failed
  providerStatus:
    conditions:
    - lastProbeTime: "2022-10-08T07:42:35Z"
      lastTransitionTime: "2022-10-08T07:42:35Z"
      message: 'failed to create vm huliu-az410-99qcm-1-cfw6w: failure sending request
        for machine huliu-az410-99qcm-1-cfw6w: cannot create vm: compute.VirtualMachinesClient#CreateOrUpdate:
        Failure sending request: StatusCode=0 -- Original Error: Code="BadRequest"
        Message="Virtual Machine cannot be created because both Availability Zone
        and Availability Set were specified. Deploying an Availability Set to an Availability
        Zone isn’t supported."'
      reason: MachineCreationFailed
      status: "True"
      type: MachineCreated
    metadata: {}

Actual results:

Created Availability Set for it.

Expected results:

Should not create Availability Set, as the region has zones.

Additional info:

If provided correct vmSize, the machine get Running and will not create Availability Set for it. Not sure why it will create Availability Set for it when vmSize is invalid.

The issue can be reproduced both on 4.11 and 4.10 version, as Availability Set is introduced in 4.10. 
On 4.12, there is bug https://issues.redhat.com/browse/OCPBUGS-1871, will also check this on 4.12 when this bug get verified.

https://github.com/openshift/machine-api-provider-azure/pull/41

Bug OCPBUGS-3756: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/machine-config-operator/pull/3418

Bug OCPBUGS-5787: ClusterResourceQuota values are not reflecting.

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-723~~. The following is the description of the original issue:
—
Description of problem:
I have a customer who created clusterquota for one of the namespace, it got created but the values were not reflecting under limits or not displaying namespace details.
~~~
$ oc describe AppliedClusterResourceQuota
Name: test-clusterquota
Created: 19 minutes ago
Labels: size=custom
Annotations: <none>
Namespace Selector: []
Label Selector:
AnnotationSelector: map[openshift.io/requester:system:serviceaccount:application-service-accounts:test-sa]
Scopes: NotTerminating
Resource Used Hard
-------- ---- ----
~~~

WORKAROUND: They recreated the clusterquota object (cache it off, delete it, create new) after which it displayed values as expected.

In the past, they saw similar behavior on their test cluster, there it was heavily utilized the etcd DB was much larger in size (>2.5Gi), and had many more objects (at that time, helm secrets were being cached for all deployments, and keeping a history of 10, so etcd was being bombarded).

This cluster the same "symptom" was noticed however etcd was nowhere near that in size nor the amount of etcd objects and/or helm cached secrets.

Version-Release number of selected component (if applicable): OCP 4.9

How reproducible: Occurred only twice(once in test and in current cluster)

Steps to Reproduce:
1. Create ClusterQuota
2. Check AppliedClusterResourceQuota
3. The values and namespace is empty

Actual results: ClusterQuota should display the values

Expected results: ClusterQuota not displaying values

https://github.com/openshift/cluster-policy-controller/pull/96

Bug OCPBUGS-7539: Endless rerender loop and a stuck browser on the add and topology page when SBO is installed

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-4072~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-4026~~. The following is the description of the original issue:
—
Description of problem:
There is an endless re-render loop and a browser feels slow to stuck when opening the add page or the topology.

Saw also endless API calls to /api/kubernetes/apis/binding.operators.coreos.com/v1alpha1/bindablekinds/bindable-kinds

Version-Release number of selected component (if applicable):
1. Console UI 4.12-4.13 (master)
2. Service Binding Operator (tested with 1.3.1)

How reproducible:
Always with installed SBO

But the "stuck feeling" depends on the browser (Firefox feels more stuck) and your locale machine power

Steps to Reproduce:
1. Install Service Binding Operator
2. Create or update the BindableKinds resource "bindable-kinds"

apiVersion: binding.operators.coreos.com/v1alpha1
kind: BindableKinds
metadata:
  name: bindable-kinds

3. Open the browser console log
4. Open the console UI and navigate to the add page

Actual results:
1. Saw endless API calls to /api/kubernetes/apis/binding.operators.coreos.com/v1alpha1/bindablekinds/bindable-kinds
2. Browser feels slow and get stuck after some time
3. The page crashs after some time

Expected results:
1. The API call should be called just once
2. The add page should just work without feeling laggy
3. No crash

Additional info:
Get introduced after we watching the bindable-kinds resource with https://github.com/openshift/console/pull/11161

It looks like this happen only if the SBO is installed and the bindable-kinds resource exist, but doesn't contain any status.

The status list all available bindable resource types. I could not reproduce this by installing and uninstalling an operator, but you can manually create or update this resource as mentioned above.

https://github.com/openshift/console/pull/12570

Bug OCPBUGS-2278: [release-4.11] Add ccoctl option to create private s3 bucket with OIDC configurations served through public CloudFront URL

View the Description View the linked PRs

This bug represents a backport of ~~CCO-222~~ to release-4.11.

https://github.com/openshift/cloud-credential-operator/pull/500

Bug OCPBUGS-3971: CNO reporting incorrect status

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-3114~~. The following is the description of the original issue:
—
Description of problem:

When running a Hosted Cluster on Hypershift the cluster-networking-operator never progressed to Available despite all the components being up and running

Version-Release number of selected component (if applicable):

quay.io/openshift-release-dev/ocp-release:4.11.11-x86_64 for the hosted clusters
hypershift operator is quay.io/hypershift/hypershift-operator:4.11
4.11.9 management cluster

How reproducible:

Happened once

Steps to Reproduce:

1.
2.
3.

Actual results:

oc get co network reports False availability

Expected results:

oc get co network reports True availability

Additional info:

https://github.com/openshift/cluster-network-operator/pull/1635

Bug OCPBUGS-746: [4.11] Supermicro server FirmwareSchema CR does not contain allowable_values, attribute_type and read_only flag

View the Description View the linked PRs

Description of problem:

See https://bugzilla.redhat.com/show_bug.cgi?id=2104275

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/ironic-image/pull/297

Bug OCPBUGS-1658: Whereabouts should allow non default interfaces to Pod IP list [backport 4.11]

View the Description View the linked PRs

Description of problem:

Whereabouts doesn't allow the use of network interface names that are not preceded by the prefix "net", see https://github.com/k8snetworkplumbingwg/whereabouts/issues/130.

Version-Release number of selected component (if applicable):

How reproducible:

Always

Steps to Reproduce:

1. Define two Pods, one with the interface name 'port1' and the other with 'net-port1':

test-ip-removal-port1:	
              k8s.v1.cni.cncf.io/networks:
                [
                  {
                    "name": "test-sriovnd",
                    "interface": "port1",
                    "namespace": "default"
                  }
                ]

test-ip-removal-net-port1:
              k8s.v1.cni.cncf.io/networks:
                [
                  {
                    "name": "test-sriovnd",
                    "interface": "net-port1",
                    "namespace": "default"
                  }
                ]

2. IP allocated in the IPPool:

kind: IPPool
...
spec:
  allocations:
    "16":
      id: ...
      podref: test-ecoloma-1/test-ip-removal-port1
    "17":
      id: ...
      podref: test-ecoloma-1/test-ip-removal-net-port1

3. When the ip-reconciler job is run, the allocation for the port with the interface name 'port1' is removed:

[13:29][]$ oc get cronjob -n openshift-multus
NAME            SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
ip-reconciler   */15 * * * *   False     0        14m             11d

[13:29][]$ oc get ippools.whereabouts.cni.cncf.io -n openshift-multus   2001-1b70-820d-2610---64 -o yaml
apiVersion: whereabouts.cni.cncf.io/v1alpha1
kind: IPPool
metadata:
...
spec:
  allocations:
    "17":
      id: ...
      podref: test-ecoloma-1/test-ip-removal-net-port1
  range: 2001:1b70:820d:2610::/64

[13:30][]$ oc get cronjob -n openshift-multus
NAME            SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
ip-reconciler   */15 * * * *   False     0        9s              11d

Actual results:

The network interface with a name that doesn't have a 'net' prefix is removed from the ip-reconciler cronjob.

Expected results:

The network interface must not be removed, regardless of the name.

Additional info:

Upstream PR @ https://github.com/k8snetworkplumbingwg/whereabouts/pull/147 master PR @ https://github.com/openshift/whereabouts-cni/pull/94

https://github.com/openshift/whereabouts-cni/pull/97

Bug OCPBUGS-7756: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

Bug OCPBUGS-11663: AWS s3 policy changes block all OCP installs on AWS

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-11636~~. The following is the description of the original issue:
—
Description of problem:

The ACLs are disabled for all newly created s3 buckets, this causes all OCP installs to fail: the bootstrap ignition can not be uploaded:

level=info msg=Creating infrastructure resources...
level=error
level=error msg=Error: error creating S3 bucket ACL for yunjiang-acl413-4dnhx-bootstrap: AccessControlListNotSupported: The bucket does not allow ACLs
level=error msg=	status code: 400, request id: HTB2HSH6XDG0Q3ZA, host id: V6CrEgbc6eyfJkUbLXLxuK4/0IC5hWCVKEc1RVonSbGpKAP1RWB8gcl5dfyKjbrLctVlY5MG2E4=
level=error
level=error msg=  with aws_s3_bucket_acl.ignition,
level=error msg=  on main.tf line 62, in resource "aws_s3_bucket_acl" "ignition":
level=error msg=  62: resource "aws_s3_bucket_acl" ignition {
level=error
level=error msg=failed to fetch Cluster: failed to generate asset "Cluster": failure applying terraform for "bootstrap" stage: failed to create cluster: failed to apply Terraform: exit status 1
level=error
level=error msg=Error: error creating S3 bucket ACL for yunjiang-acl413-4dnhx-bootstrap: AccessControlListNotSupported: The bucket does not allow ACLs
level=error msg=	status code: 400, request id: HTB2HSH6XDG0Q3ZA, host id: V6CrEgbc6eyfJkUbLXLxuK4/0IC5hWCVKEc1RVonSbGpKAP1RWB8gcl5dfyKjbrLctVlY5MG2E4=
level=error
level=error msg=  with aws_s3_bucket_acl.ignition,
level=error msg=  on main.tf line 62, in resource "aws_s3_bucket_acl" "ignition":
level=error msg=  62: resource "aws_s3_bucket_acl" ignition {

Version-Release number of selected component (if applicable):

4.11+

How reproducible:

Always

Steps to Reproduce:

1.Create a cluster via IPI

Actual results:

install fail

Expected results:

install succeed

Additional info:

Heads-Up: Amazon S3 Security Changes Are Coming in April of 2023 - https://aws.amazon.com/blogs/aws/heads-up-amazon-s3-security-changes-are-coming-in-april-of-2023/

https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-ownership-error-responses.html - After you apply the bucket owner enforced setting for Object Ownership, ACLs are disabled.

https://github.com/openshift/installer/pull/7085

Bug OCPBUGS-2443: Help popovers cause error on Observe > Alerting pages

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-2438~~. The following is the description of the original issue:
—
Description of problem:

On the alert details page and alerting rule details page, clicking on a field that has a popover help throws an uncaught JavaScript error.

Version-Release number of selected component (if applicable):

How reproducible:

Always

Steps to Reproduce:

1. Go to Observe > Alerting pages
2. Click on an alert (or go to the rules tab then click on a rule)
3. Click on one of the underlined fields (those that have a popover help)

Actual results:

Expected results:

Additional info:

https://github.com/openshift/console/pull/12173

Bug OCPBUGS-6851: admin ack test nondeterministically does a check post-upgrade

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-6850~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-6503~~. The following is the description of the original issue:
—
Description of problem:

While looking into ~~OCPBUGS-5505~~ I discovered that some 4.10->4.11 upgrade job runs perform an Admin Ack check, while some do not. 4.11 has a ack-4.11-kube-1.25-api-removals-in-4.12 gate, so these upgrade jobs sometimes test that Upgradeable goes false after the ugprade, and sometimes they do not. This is only determined by the polling race condition: the check is executed once per 10 minutes, and we cancel the polling after upgrade is completed. This means that in some cases we are lucky and manage to run one check before the cancel, and sometimes we are not and only check while still on the base version.

Example job that checked admin acks post-upgrade:
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/openshift-cluster-version-operator-880-ci-4.11-upgrade-from-stable-4.10-e2e-azure-upgrade/1611444032104304640

$ curl --silent https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/openshift-cluster-version-operator-880-ci-4.11-upgrade-from-stable-4.10-e2e-azure-upgrade/1611444032104304640/artifacts/e2e-azure-upgrade/openshift-e2e-test/artifacts/e2e.log | grep 'Waiting for Upgradeable to be AdminAckRequired'
Jan  6 21:16:40.153: INFO: Waiting for Upgradeable to be AdminAckRequired ...

Example job that did not check admin acks post-upgrade:
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/openshift-cluster-version-operator-880-ci-4.11-upgrade-from-stable-4.10-e2e-azure-upgrade/1611444033509396480

$ curl --silent https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/openshift-cluster-version-operator-880-ci-4.11-upgrade-from-stable-4.10-e2e-azure-upgrade/1611444033509396480/artifacts/e2e-azure-upgrade/openshift-e2e-test/artifacts/e2e.log | grep 'Waiting for Upgradeable to be AdminAckRequired'

Version-Release number of selected component (if applicable):

4.11+ openshift-tests

How reproducible:

nondeterministic, wild guess is ~30% of upgrade jobs

Steps to Reproduce:

1. Inspect the E2E test log of an upgrade jobs and compare the time of the update ("Completed upgrade") with the time of the last check ( "Skipping admin ack", "Gate .* not applicable to current version", "Admin Ack verified') done by the admin ack test

Actual results:

Jan 23 00:47:43.842: INFO: Admin Ack verified
Jan 23 00:57:43.836: INFO: Admin Ack verified
Jan 23 01:07:43.839: INFO: Admin Ack verified
Jan 23 01:17:33.474: INFO: Completed upgrade to registry.build01.ci.openshift.org/ci-op-z09ll8fw/release@sha256:322cf67dc00dd6fa4fdd25c3530e4e75800f6306bd86c4ad1418c92770d58ab8

No check done after the upgrade

Expected results:

Jan 23 00:57:37.894: INFO: Admin Ack verified
Jan 23 01:07:37.894: INFO: Admin Ack verified
Jan 23 01:16:43.618: INFO: Completed upgrade to registry.build01.ci.openshift.org/ci-op-z8h5x1c5/release@sha256:9c4c732a0b4c2ae887c73b35685e52146518e5d2b06726465d99e6a83ccfee8d
Jan 23 01:17:57.937: INFO: Admin Ack verified

One or more checks done after upgrade

https://github.com/openshift/origin/pull/27685

Bug OCPBUGS-3462: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/cluster-network-operator/pull/1616

Bug OCPBUGS-5628: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/cluster-baremetal-operator/pull/321

Bug OCPBUGS-3300: Console shouldn't try to install dynamic plugins if permissions aren't available

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-3265~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-3172~~. The following is the description of the original issue:
—
Customer is trying to install the Logging operator, which appears to attempt to install a dynamic plugin. The operator installation fails in the console because permissions aren't available to "patch resource consoles".

We shouldn't block operator installation if permission issues prevent dynamic plugin installation.

This is an OSD cluster, presumably for a customer with "cluster-admin", although it may be a paired down permission set called "dedicated-admin".

See https://docs.google.com/document/d/1hYS-bm6aH7S6z7We76dn9XOFcpi9CGYcGoJys514YSY/edit for permissions investigation work on OSD

https://github.com/openshift/console/pull/12240

Bug OCPBUGS-4087: Kube-State-metrics pod fails to start due to panic

View the Description View the linked PRs

The kube-state-metric pod inside the openshift-monitoring namespace is not running as expected.

On checking the logs I am able to see that there is a memory panic

~~~
2022-11-22T09:57:17.901790234Z I1122 09:57:17.901768 1 main.go:199] Starting kube-state-metrics self metrics server: 127.0.0.1:8082
2022-11-22T09:57:17.901975837Z I1122 09:57:17.901951 1 main.go:66] levelinfomsgTLS is disabled.http2false
2022-11-22T09:57:17.902389844Z I1122 09:57:17.902291 1 main.go:210] Starting metrics server: 127.0.0.1:8081
2022-11-22T09:57:17.903191857Z I1122 09:57:17.903133 1 main.go:66] levelinfomsgTLS is disabled.http2false
2022-11-22T09:57:17.906272505Z I1122 09:57:17.906224 1 builder.go:191] Active resources: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments
2022-11-22T09:57:17.917758187Z E1122 09:57:17.917560 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
2022-11-22T09:57:17.917758187Z goroutine 24 [running]:
2022-11-22T09:57:17.917758187Z k8s.io/apimachinery/pkg/util/runtime.logPanic(

{0x1635600, 0x2696e10})
2022-11-22T09:57:17.917758187Z /go/src/k8s.io/kube-state-metrics/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x7d
2022-11-22T09:57:17.917758187Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xfffffffe})
2022-11-22T09:57:17.917758187Z /go/src/k8s.io/kube-state-metrics/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75
2022-11-22T09:57:17.917758187Z panic({0x1635600, 0x2696e10}

)
2022-11-22T09:57:17.917758187Z /usr/lib/golang/src/runtime/panic.go:1038 +0x215
2022-11-22T09:57:17.917758187Z k8s.io/kube-state-metrics/v2/internal/store.ingressMetricFamilies.func6(0x40)
2022-11-22T09:57:17.917758187Z /go/src/k8s.io/kube-state-metrics/internal/store/ingress.go:136 +0x189
2022-11-22T09:57:17.917758187Z k8s.io/kube-state-metrics/v2/internal/store.wrapIngressFunc.func1(

{0x17fe520, 0xc00063b590})
2022-11-22T09:57:17.917758187Z /go/src/k8s.io/kube-state-metrics/internal/store/ingress.go:175 +0x49
2022-11-22T09:57:17.917758187Z k8s.io/kube-state-metrics/v2/pkg/metric_generator.(*FamilyGenerator).Generate(...)
2022-11-22T09:57:17.917758187Z /go/src/k8s.io/kube-state-metrics/pkg/metric_generator/generator.go:67
2022-11-22T09:57:17.917758187Z k8s.io/kube-state-metrics/v2/pkg/metric_generator.ComposeMetricGenFuncs.func1({0x17fe520, 0xc00063b590}

)
2022-11-22T09:57:17.917758187Z /go/src/k8s.io/kube-state-metrics/pkg/metric_generator/generator.go:107 +0xd8
~~~

Logs are attached to the support case

https://github.com/openshift/kube-state-metrics/pull/85

Bug OCPBUGS-1551: Improve prometheus-adapter consistency

View the Description View the linked PRs

Description of problem:

This bugs purpose is to enable a feature backport of https://issues.redhat.com/browse/MON-1949.

https://github.com/openshift/cluster-monitoring-operator/pull/1774

Bug OCPBUGS-3807: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/machine-config-operator/pull/3419

Bug OCPBUGS-7720: set default timeouts in etcdcli

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-7409~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-7374~~. The following is the description of the original issue:
—
Originally reported by lance5890 in issue https://github.com/openshift/cluster-etcd-operator/issues/1000

The controllers sometimes get stuck on listing members in failure scenarios, this is known and can be mitigated by simply restarting the CEO.

similar BZ 2093819 with stuck controllers was fixed slightly different in https://github.com/openshift/cluster-etcd-operator/commit/4816fab709e11e0681b760003be3f1de12c9c103

This fix was contributed by lance5890, thanks a lot!

https://github.com/openshift/cluster-etcd-operator/pull/1007

Bug OCPBUGS-2303: Stale chassis are not removed

View the Description View the linked PRs

Missing backport from https://bugzilla.redhat.com/show_bug.cgi?id=2093826 in 4.11.

https://github.com/openshift/ovn-kubernetes/pull/1310

Bug OCPBUGS-4564: [4.11] Pod stuck in containerCreating state when the node on which it is running is Terminated

View the Description View the linked PRs

Description of problem:

The alertmanager pod is stuck on OCP 4.11 with OVN in container Creating State

From oc describe alertmanager pod:
...
Events:
  Type     Reason                  Age                  From     Message
  ----     ------                  ----                 ----     -------
  Warning  FailedCreatePodSandBox  16s (x459 over 17h)  kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_alertmanager-managed-ocs-alertmanager-0_openshift-storage_3a55ed54-4eaa-4f65-8a10-e5d21fad1ebc_0(88575547dc0b210307b89dd2bb8e379ece0962b607ac2707a1c2cf630b1aaa78): error adding pod openshift-storage_alertmanager-managed-ocs-alertmanager-0 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [openshift-storage/alertmanager-managed-ocs-alertmanager-0/3a55ed54-4eaa-4f65-8a10-e5d21fad1ebc:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[openshift-storage/alertmanager-managed-ocs-alertmanager-0 88575547dc0b210307b89dd2bb8e379ece0962b607ac2707a1c2cf630b1aaa78] [openshift

Version-Release number of selected component (if applicable):

OCP 4.11 with OVN

How reproducible:

100%

Steps to Reproduce:

1. Terminate the node on which alertmanager pod is running
2. pod will get stuck in container Creating state
3.

Actual results:

AlertManager pod is stuck in container Creating state

Expected results:

Alertmanager pod is ready

Additional info:

The workaround would be to terminate the alertmanager pod

https://github.com/openshift/ovn-kubernetes/pull/1422

Bug OCPBUGS-1251: Add admin acknowledgement in 4.11 for API removals in 4.12

View the Description View the linked PRs

Description of problem:

We need to have admin-ack in 4.11 so that admins can check the deprecated APIs and approve when they move to 4.12.Refer https://access.redhat.com/articles/6955381 for  more information. As planned we want to add the admin-ack around 4.12 feature freeze.

Version-Release number of selected component (if applicable):

4.11

How reproducible:

Always

Steps to Reproduce:

1. Install a cluster in 4.11. 
2. Run an application which uses the deprecated API. See https://access.redhat.com/articles/6955381 for more information.
3. Upgrade to 4.12

Actual results:

The upgrade happens without asking the admin to confirm that the worksloads do not use the deprecated APIs.

Expected results:

Upgrade should wait for the admin-ack.

Additional info:

We had admin-acks in the past too e.g. https://docs.openshift.com/container-platform/4.9/updating/updating-cluster-prepare.html#update-preparing-migrate_updating-cluster-prepare

https://github.com/openshift/cluster-version-operator/pull/836

Bug OCPBUGS-4685: CVE-2021-38561 ose-installer-container: golang: out-of-bounds read in golang.org/x/text/language leads to DoS [openshift-4]

View the Description View the linked PRs

Description of problem:

This is just a clone of https://bugzilla.redhat.com/show_bug.cgi?id=2105570 for purposes of cherry-picking.

Version-Release number of selected component (if applicable):

4.13

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/installer/pull/6685

Bug OCPBUGS-516: Setting a telemeter proxy in the cluster-monitoring-config config map does not work as expected

View the Description View the linked PRs

Description of problem:

Setting a telemeter proxy in the cluster-monitoring-config config map does not work as expected

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
the following KCS details steps to add a proxy.
The steps have been verified at 4.7 but do not work at 4.8, 4.9 or 4.10

https://access.redhat.com/solutions/6172402

When testing at 4.8, 4.9 and 4.10 the proxy setting where also nested under `telemeterClient`

which triggered a telemeter restart but the proxy setting do not get set in the deployment as they do in 4.7

Actual results:

4.8, 4.9 and 4.10 without the nested `telemeterClient`
does not trigger a restart of the telemeter pod

Expected results:

I think the proxy setting should be nested under telemeterClient
but should set the environment variables in the deployment

Additional info:

This is a backport of https://bugzilla.redhat.com/show_bug.cgi?id=2116382 from 4.12 to 4.11.z. Creating manually because as seen in https://github.com/openshift/cluster-monitoring-operator/pull/1743 `/cherry-pick` doesn't work for bugs originally created in bugzilla

https://github.com/openshift/cluster-monitoring-operator/pull/1743

Bug OCPBUGS-7494: Broken secret created while starting a Pipeline

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-6671~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-3228~~. The following is the description of the original issue:
—
While starting a Pipelinerun using UI, and in the process of providing the values on "Start Pipeline" , the IBM Power Customer (Deepak Shetty from IBM) has tried creating credentials under "Advanced options" with "Image Registry Credentials" (Authenticaion type). When the IBM Customer verified the credentials from Secrets tab (in Workloads) , the secret was found in broken state. Screenshot of the broken secret is attached.

The issue has been observed on OCP4.8, OCP4.9 and OCP4.10.

https://github.com/openshift/console/pull/12569

Bug OCPBUGS-9863: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/multus-cni/pull/152

Bug OCPBUGS-5665: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/installer/pull/6765

Bug OCPBUGS-7278: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/csi-driver-manila-operator/pull/168

Bug OCPBUGS-2801: [release-4.11] Add required parameter --credentials-requests-dir for ccoctl gcp delete

View the Description View the linked PRs

In order to delete the correct GCP cloud resources, the "--credentials-requests-dir" parameter must be passed to "ccoctl gcp delete". This was fixed for 4.12 as part of https://github.com/openshift/cloud-credential-operator/pull/489 but must be backported for previous releases. See https://github.com/openshift/cloud-credential-operator/pull/489#issuecomment-1248733205 for discussion regarding this bug.

To reproduce, create GCP infrastructure with a name parameter that is a subset of another set of GCP infrastructure's name parameter. I will "ccoctl gcp create all" with "name=abutcher-gcp" and "name=abutcher-gcp1".

$ ./ccoctl gcp create-all \
--name=abutcher-gcp \
--region=us-central1 \
--project=openshift-hive-dev \
--credentials-requests-dir=./credrequests

$ ./ccoctl gcp create-all \
--name=abutcher-gcp1 \
--region=us-central1 \
--project=openshift-hive-dev \
--credentials-requests-dir=./credrequests

Running "ccoctl gcp delete --name=abutcher-gcp" will result in GCP infrastructure for both "abutcher-gcp" and "abutcher-gcp1" being deleted.

$ ./ccoctl gcp delete --name abutcher-gcp --project openshift-hive-dev
2022/10/24 11:30:06 Credentials loaded from file "/home/abutcher/.gcp/osServiceAccount.json"
2022/10/24 11:30:06 Deleted object .well-known/openid-configuration from bucket abutcher-gcp-oidc
2022/10/24 11:30:07 Deleted object keys.json from bucket abutcher-gcp-oidc
2022/10/24 11:30:07 OIDC bucket abutcher-gcp-oidc deleted
2022/10/24 11:30:09 IAM Service account abutcher-gcp-openshift-image-registry-gcs deleted
2022/10/24 11:30:10 IAM Service account abutcher-gcp-openshift-gcp-ccm deleted
2022/10/24 11:30:11 IAM Service account abutcher-gcp1-openshift-cloud-network-config-controller-gcp deleted
2022/10/24 11:30:12 IAM Service account abutcher-gcp-openshift-machine-api-gcp deleted
2022/10/24 11:30:13 IAM Service account abutcher-gcp-openshift-ingress-gcp deleted
2022/10/24 11:30:15 IAM Service account abutcher-gcp-openshift-gcp-pd-csi-driver-operator deleted
2022/10/24 11:30:16 IAM Service account abutcher-gcp1-openshift-ingress-gcp deleted
2022/10/24 11:30:17 IAM Service account abutcher-gcp1-openshift-image-registry-gcs deleted
2022/10/24 11:30:19 IAM Service account abutcher-gcp-cloud-credential-operator-gcp-ro-creds deleted
2022/10/24 11:30:20 IAM Service account abutcher-gcp1-openshift-gcp-pd-csi-driver-operator deleted
2022/10/24 11:30:21 IAM Service account abutcher-gcp1-openshift-gcp-ccm deleted
2022/10/24 11:30:22 IAM Service account abutcher-gcp1-cloud-credential-operator-gcp-ro-creds deleted
2022/10/24 11:30:24 IAM Service account abutcher-gcp1-openshift-machine-api-gcp deleted
2022/10/24 11:30:25 IAM Service account abutcher-gcp-openshift-cloud-network-config-controller-gcp deleted
2022/10/24 11:30:25 Workload identity pool abutcher-gcp deleted

https://github.com/openshift/cloud-credential-operator/pull/506

Bug OCPBUGS-3290: WriteRequestBodies audit profile records routes/status events at RequestResponse level

View the Description View the linked PRs

This bug is a backport clone of [Bugzilla Bug 2073220](https://bugzilla.redhat.com/show_bug.cgi?id=2073220). The following is the description of the original bug:
—
Description of problem:

https://docs.openshift.com/container-platform/4.10/security/audit-log-policy-config.html#about-audit-log-profiles_audit-log-policy-config

Version-Release number of selected component (if applicable): 4.*

How reproducible: always

Steps to Reproduce:
1. Set audit profile to WriteRequestBodies
2. Wait for api server rollout to complete
3. tail -f /var/log/kube-apiserver/audit.log | grep routes/status

Actual results:

Write events to routes/status are recorded at the RequestResponse level, which often includes keys and certificates.

Expected results:

Events involving routes should always be recorded at the Metadata level, per the documentation at https://docs.openshift.com/container-platform/4.10/security/audit-log-policy-config.html#about-audit-log-profiles_audit-log-policy-config

Additional info:

Bug OCPBUGS-1607: [4.11] Investigate etcdGRPCRequestsSlow test

View the Description View the linked PRs

This is a clone of issue OCPBUGS-1130. The following is the description of the original issue:
—
The test results in sippy look really bad on our less common platforms, but still pretty unacceptable even on core clouds. It's reasonably often the only test that fails. We need to decide what to do here, and we're going to need input from the etcd team.

As of Sep 13th:

several vsphere and openstack variant combo's fail this test around 24-32% of the time
aws, amd64, ovn, upgrade, upgrade-micro, ha - fails 6% of the time
aws, amd64, ovn, upgrade, upgrade-minor, ha - fails 4% of the time
gcp, amd64, sdn, upgrade, upgrade-minor, ha - fails 8% of the time
globally across all jobs fails around 3% of the time.

Even on some major variant combos, a 4-8% failure rate is too high.
On Sep 13 arch call (no etcd present), Damien mentioned this might be an upstream alert that just isn't well suited for OpenShift's use cases, is this the case and it needs tuning?

Has the problem been getting worse?

I believe this link https://datastudio.google.com/s/urkKwmmzvgo indicates that this may be the case for 4.12, AWS and Azure are both getting worse in ways that I don't see if we change the release to 4.11 where it looks consistent. gcp seems fine on 4.12. We do not have data for vsphere for some reason.

This link shows the grpc_methods most commonly involved: https://search.ci.openshift.org/?search=etcdGRPCRequestsSlow+was+at+or+above&maxAge=48h&context=7&type=junit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

At a glance: LeaseGrant, MemberList, Txn, Status, Range.

Broken out of ~~TRT-401~~
For linking with sippy:
[bz-etcd][invariant] alert/etcdGRPCRequestsSlow should not be at or above info
[sig-arch][bz-etcd][Late] Alerts alert/etcdGRPCRequestsSlow should not be at or above info [Suite:openshift/conformance/parallel]

https://github.com/openshift/cluster-etcd-operator/pull/934

Bug OCPBUGS-5954: ip-reconciler removes the overlappingrangeipreservations whether the pod is alive or not [backport 4.11]

View the Description View the linked PRs

Description of problem:

The reconciler removes the overlappingrangeipreservations.whereabouts.cni.cncf.io resources whether the pod is alive or not.

Version-Release number of selected component (if applicable):

How reproducible:

Always

Steps to Reproduce:

1. Create pods and check the overlappingrangeipreservations.whereabouts.cni.cncf.io resources:

$ oc get overlappingrangeipreservations.whereabouts.cni.cncf.io -A
NAMESPACE          NAME                      AGE
openshift-multus   2001-1b70-820d-4b04--13   4m53s
openshift-multus   2001-1b70-820d-4b05--13   4m49s

2. Verify that when the ip-reconciler cronjob removes the overlappingrangeipreservations.whereabouts.cni.cncf.io resources when run:

$ oc get cronjob -n openshift-multus
NAME            SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
ip-reconciler   */15 * * * *   False     0        14m             4d13h

$ oc get overlappingrangeipreservations.whereabouts.cni.cncf.io -A
No resources found

$ oc get cronjob -n openshift-multus
NAME            SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
ip-reconciler   */15 * * * *   False     0        5s              4d13h

Actual results:

The overlappingrangeipreservations.whereabouts.cni.cncf.io resources are removed for each created pod by the ip-reconciler cronjob.
The "overlapping ranges" are not used.

Expected results:

The overlappingrangeipreservations.whereabouts.cni.cncf.io should not be removed regardless of if a pod has used an IP in the overlapping ranges.

Additional info:

https://github.com/openshift/cluster-network-operator/pull/1686

Bug OCPBUGS-1727: Regular user cannot open the debug container from pods they created

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-1522~~. The following is the description of the original issue:
—
Description of problem:

Normal user cannot open the debug container from the pods(crashLoopbackoff) they created, And would be got error message:
pods "<pod name>" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>

Version-Release number of selected component (if applicable):

4.12.0-0.nightly-2022-09-20-040107, 4.11.z, 4.10.z

How reproducible:

Always

Steps to Reproduce:

1. Login OCP as a normal user
   eg: flexy-htpasswd-provider
2. Create a project, go to Developer prespective -> +Add page
3. Click "Import from Git", and provide below data to get a Pods with CrashLoopBackOff state
   Git Repo URL: https://github.com/sclorg/nodejs-ex.git
   Name: nodejs-ex-git
   Run command: star a wktw
4. Navigate to /k8s/ns/<project name>/pods page, find the pod with CrashLoopBackOff status, and go to it details page -> Logs Tab
5. Click the link of "Debug container"
6. Check if the Debug container can be opened

Actual results:

6. Error message would be shown on page, user cannot open debug container via UI
   pods "nodejs-ex-git-6dd986d8bd-9h2wj-debug-tkqk2" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>

Expected results:

6. Normal user could use debug container without any error message

Additional info:

The debug container could be created for the normal user successfully via CommandLine
 $ oc debug <crashloopbackoff pod name> -n <project name>

https://github.com/openshift/console/pull/12100

Bug OCPBUGS-4945: Empty/missing node-sizing SYSTEM_RESERVED_ES parameter can result in kubelet not starting

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-4805~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-4101~~. The following is the description of the original issue:
—
Description of problem:

We experienced two separate upgrade failures relating to the introduction of the SYSTEM_RESERVED_ES node sizing parameter, causing kubelet to stop running.

One cluster (clusterA) upgraded from 4.11.14 to 4.11.17. It experienced an issue whereby
/etc/node-sizing.env
on its master nodes contained an empty SYSTEM_RESERVED_ES value:

---
cat /etc/node-sizing.env
SYSTEM_RESERVED_MEMORY=5.36Gi
SYSTEM_RESERVED_CPU=0.11
SYSTEM_RESERVED_ES=
---

causing the kubelet to not start up. To restore service, this file was manually updated to set a value (1Gi), and kubelet was restarted.

We are uncertain what conditions led to this occuring on the clusterA master nodes as part of the upgrade.

A second cluster (clusterB) upgraded from 4.11.16 to 4.11.17. It experienced an issue whereby worker nodes were impacted by a similar problem, however this was because a custom node-sizing-enabled.env MachineConfig which did not set SYSTEM_RESERVED_ES

This caused existing worker nodes to go into a NotReady state after the ugprade, and additionally new nodes did not join the cluster as their kubelet would become impacted.

For clusterB the conditions are more well-known of why the value is empty.

However, for both clusters, if SYSTEM_RESERVED_ES ends up as empty on a node it can cause the kubelet to not start.

We have some asks as a result:
- Can MCO be made to recover from this situation if it occurs, perhaps through application of a safe default if none exists, such that kubelet would start correctly?
- Can there possibly be alerting that could indicate and draw attention to the misconfiguration?

Version-Release number of selected component (if applicable):

4.11.17

How reproducible:

Have not been able to reproduce it on a fresh cluster upgrading from 4.11.16 to 4.11.17

Expected results:

If SYSTEM_RESERVED_ES is empty in /etc/node-sizing*env then a default should be applied and/or kubelet able to continue running.

Additional info:

https://github.com/openshift/machine-config-operator/pull/3459

Bug OCPBUGS-5258: Add support for API version v1beta1 for knativeServing and knativeEventing

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-5191~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-5164~~. The following is the description of the original issue:
—
Description of problem:

It looks like the ODC doesn't register KNATIVE_SERVING and KNATIVE_EVENTING flags. Those are based on KnativeServing and KnativeEventing CRs, but they are looking for v1alpha1 version of those: https://github.com/openshift/console/blob/f72519fdf2267ad91cc0aa51467113cc36423a49/frontend/packages/knative-plugin/console-extensions.json#L6-L8
This PR https://github.com/openshift-knative/serverless-operator/pull/1695 moved the CRs to v1beta1, and that breaks that ODC discovery.

Version-Release number of selected component (if applicable):

Openshift 4.8, Serverless Operator 1.27

Additional info:

https://coreos.slack.com/archives/CHGU4P8UU/p1671634903447019

https://github.com/openshift/console/pull/12403

Bug OCPBUGS-1523: Dev Catalog taking too much time to load in a complete disconnected cluster

View the Description View the linked PRs

Description of problem:
In a complete disconnected cluster, the dev catalog is taking too much time in loading

Version-Release number of selected component (if applicable):

How reproducible:
Always

Steps to Reproduce:
1. A complete disconnected cluster
2. In add page go to the All services page
3.

Actual results:
Taking too much time too load

Expected results:
Time taken should be reduced

Additional info:
Attached a gif for reference

https://github.com/openshift/console/pull/12070

Bug OCPBUGS-3764: Openshift/Thanos 4.11: Fix local-test for go 1.18

View the Description View the linked PRs

Description of problem:

The test local-test is failing on openshift/thanos when upgrading golang version to 1.18 on the branch release-4.11.

Please refer to this test log for details:
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_thanos/82/pull-ci-openshift-thanos-release-4.11-test-local/1541516614497734656

Version-Release number of selected component (if applicable):

4.11

How reproducible:

See local-test job on pull request on the repository Openshift/Thanos

Steps to Reproduce:

Actual results:

local-test fails on the following error:
level=error ts=2022-06-27T20:28:12.306Z caller=web.go:99 component=web msg="panic while serving request" client=127.0.0.1:37064 url=/api/v1/metadata err="runtime error: invalid memory address or nil pointer dereference" stack="goroutine 278 [running]:\ngithub.com/prometheus/prometheus/web.withStackTracer.func1.1()\n\t/go/pkg/mod/github.com/prometheus/prometheus@v1.8.2-0.20200724121523-657ba532e42f/web/web.go:98 +0x99\npanic({0x1c34760, 0x308ad40})\n\t/usr/lib/golang/src/runtime/panic.go:838 +0x207\nreflect.mapiternext(0xc000458540?)\n\t/usr/lib/golang/src/runtime/map.go:1378 +0x19\ngithub.com/modern-go/reflect2.(*UnsafeMapIterator).UnsafeNext(0x1bd62e0?)\n\t/go/pkg/mod/github.com/modern-go/reflect2@v1.0.1/unsafe_map.go:136 +0x32\ngithub.com/json-iterator/go.(*sortKeysMapEncoder).Encode(0xc000949d10, 0xc0002966b0, 0xc0006c7740)\n\t/go/pkg/mod/github.com/json-iterator/go@v1.1.9/reflect_map.go:297 +0x31a\ngithub.com/json-iterator/go.(*onePtrEncoder).Encode(0xc0008cb120, 0xc000948fc0, 0xc0001139c0?)\n\t/go/pkg/mod/github.com/json-iterator/go@v1.1.9/reflect.go:219 +0x82\ngithub.com/json-iterator/go.(*Stream).WriteVal(0xc0006c7740, {0x1c16da0, 0xc000948fc0})\n\t/go/pkg/mod/github.com/json-iterator/go@v1.1.9/reflect.go:98 +0x158\ngithub.com/json-iterator/go.(*dynamicEncoder).Encode(0xc00094cd58?, 0xfa9a07?, 0xc0006c7758?)\n\t/go/pkg/mod/github.com/json-iterator/go@v1.1.9/reflect_dynamic.go:15 +0x39\ngithub.com/json-iterator/go.(*structFieldEncoder).Encode(0xc000949620, 0x1a4aaba?, 0xc0006c7740)\n\t/go/pkg/mod/github.com/json-iterator/go@v1.1.9/reflect_struct_encoder.go:110 +0x56\ngithub.com/json-iterator/go.(*structEncoder).Encode(0xc000949740, 0x0?, 0xc0006c7740)\n\t/go/pkg/mod/github.com/json-iterator/go@v1.1.9/reflect_struct_encoder.go:158 +0x652\ngithub.com/json-iterator/go.(*OptionalEncoder).Encode(0xc0001afd60?, 0x0?, 0x0?)\n\t/go/pkg/mod/github.com/json-iterator/go@v1.1.9/reflect_optional.go:74 +0xa4\ngithub.com/json-iterator/go.(*onePtrEncoder).Encode(0xc0008cad40, 0xc0006c76e0, 0xc000949020?)\n\t/go/pkg/mod/github.com/json-iterator/go@v1.1.9/reflect.go:219 +0x82\ngithub.com/json-iterator/go.(*Stream).WriteVal(0xc0006c7740, {0x1ac56e0, 0xc0006c76e0})\n\t/go/pkg/mod/github.com/json-iterator/go@v1.1.9/reflect.go:98 +0x158\ngithub.com/json-iterator/go.(*frozenConfig).Marshal(0xc0001afd60, {0x1ac56e0, 0xc0006c76e0})\n\t/go/pkg/mod/github.com/json-iterator/go@v1.1.9/config.go:299 +0xc9\ngithub.com/prometheus/prometheus/web/api/v1.(*API).respond(0xc0002d7a40, {0x229a448, 0xc00022bd60}, {0x1c16da0?, 0xc000948fc0}, {0x0?, 0x7fe5a05a5b20?, 0x20?})\n\t/go/pkg/mod/github.com/prometheus/prometheus@v1.8.2-0.20200724121523-657ba532e42f/web/api/v1/api.go:1437 +0x162\ngithub.com/prometheus/prometheus/web/api/v1.(*API).Register.func1.1({0x229a448, 0xc00022bd60}, 0x7fe5982c5300?)\n\t/go/pkg/mod/github.com/prometheus/prometheus@v1.8.2-0.20200724121523-657ba532e42f/web/api/v1/api.go:273 +0x20b\nnet/http.HandlerFunc.ServeHTTP(0x7fe5982c5300?, {0x229a448?, 0xc00022bd60?}, 0xc00072b270?)\n\t/usr/lib/golang/src/net/http/server.go:2084 +0x2f\ngithub.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP({{0x2290780?, 0xc000856288?}}, {0x7fe5982c5300?, 0xc00072b270?}, 0x228fb20?)\n\t/go/pkg/mod/github.com/prometheus/prometheus@v1.8.2-0.20200724121523-657ba532e42f/util/httputil/compression.go:90 +0x69\ngithub.com/prometheus/prometheus/web.(*Handler).testReady.func1({0x7fe5982c5300?, 0xc00072b270?}, 0x7fe5982c5300?)\n\t/go/pkg/mod/github.com/prometheus/prometheus@v1.8.2-0.20200724121523-657ba532e42f/web/web.go:499 +0x39\nnet/http.HandlerFunc.ServeHTTP(0x7fe5982c5300?, {0x7fe5982c5300?, 0xc00072b270?}, 0x50?)\n\t/usr/lib/golang/src/net/http/server.go:2084 +0x2f\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1({0x7fe5982c5300?, 0xc00072b220?}, 0xc000250c00)\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.6.0/prometheus/promhttp/instrument_server.go:196 +0xa5\nnet/http.HandlerFunc.ServeHTTP(0x228fb80?, {0x7fe5982c5300?, 0xc00072b220?}, 0xc000948ed0?)\n\t/usr/lib/golang/src/net/http/server.go:2084 +0x2f\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2({0x7fe5982c5300, 0xc00072b220}, 0xc000250c00)\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.6.0/prometheus/promhttp/instrument_server.go:76 +0xa2\nnet/http.HandlerFunc.ServeHTTP(0x22a4a68?, {0x7fe5982c5300?, 0xc00072b220?}, 0x0?)\n\t/usr/lib/golang/src/net/http/server.go:2084 +0x2f\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x22a4a68?, 0xc00072b1d0?}, 0xc000250c00)\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.6.0/prometheus/promhttp/instrument_server.go:100 +0x94\ngithub.com/prometheus/prometheus/web.setPathWithPrefix.func1.1({0x22a4a68, 0xc00072b1d0}, 0xc000250b00)\n\t/go/pkg/mod/github.com/prometheus/prometheus@v1.8.2-0.20200724121523-657ba532e42f/web/web.go:1142 +0x290\ngithub.com/prometheus/common/route.(*Router).handle.func1({0x22a4a68, 0xc00072b1d0}, 0xc000250a00, {0x0, 0x0, 0xc00022c364?})\n\t/go/pkg/mod/github.com/prometheus/common@v0.10.0/route/route.go:83 +0x2ae\ngithub.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc0001cc780, {0x22a4a68, 0xc00072b1d0}, 0xc000250a00)\n\t/go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0x82b\ngithub.com/prometheus/common/route.(*Router).ServeHTTP(0x8?, {0x22a4a68?, 0xc00072b1d0?}, 0x203000?)\n\t/go/pkg/mod/github.com/prometheus/common@v0.10.0/route/route.go:121 +0x26\nnet/http.StripPrefix.func1({0x22a4a68, 0xc00072b1d0}, 0xc000250900)\n\t/usr/lib/golang/src/net/http/server.go:2127 +0x330\nnet/http.HandlerFunc.ServeHTTP(0x10?, {0x22a4a68?, 0xc00072b1d0?}, 0x7fe5c8423f18?)\n\t/usr/lib/golang/src/net/http/server.go:2084 +0x2f\nnet/http.(*ServeMux).ServeHTTP(0x413d87?, {0x22a4a68, 0xc00072b1d0}, 0xc000250900)\n\t/usr/lib/golang/src/net/http/server.go:2462 +0x149\ngithub.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5({0x22a3808?, 0xc000a282a0}, 0xc000250200)\n\t/go/pkg/mod/github.com/opentracing-contrib/go-stdlib@v0.0.0-20190519235532-cf7a6c988dc9/nethttp/server.go:140 +0x662\nnet/http.HandlerFunc.ServeHTTP(0x0?, {0x22a3808?, 0xc000a282a0?}, 0xffffffffffffffff?)\n\t/usr/lib/golang/src/net/http/server.go:2084 +0x2f\ngithub.com/prometheus/prometheus/web.withStackTracer.func1({0x22a3808?, 0xc000a282a0?}, 0xc0008ca850?)\n\t/go/pkg/mod/github.com/prometheus/prometheus@v1.8.2-0.20200724121523-657ba532e42f/web/web.go:103 +0x97\nnet/http.HandlerFunc.ServeHTTP(0x0?, {0x22a3808?, 0xc000a282a0?}, 0xc000100000?)\n\t/usr/lib/golang/src/net/http/server.go:2084 +0x2f\nnet/http.serverHandler.ServeHTTP({0xc000c55380?}, {0x22a3808, 0xc000a282a0}, 0xc000250200)\n\t/usr/lib/golang/src/net/http/server.go:2916 +0x43b\nnet/http.(*conn).serve(0xc0000d1540, {0x22a4e18, 0xc00061a0c0})\n\t/usr/lib/golang/src/net/http/server.go:1966 +0x5d7\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3071 +0x4db\n"
level=error ts=2022-06-27T20:28:12.306Z caller=stdlib.go:89 component=web caller="http: panic serving 127.0.0.1:37064" msg="runtime error: invalid memory address or nil pointer dereference"

Expected results:

local-test does no fail on the error above.

Additional info:

https://github.com/openshift/thanos/pull/87

Bug OCPBUGS-2528: dns-default pod missing "target.workload.openshift.io/management:" annotation

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-753~~. The following is the description of the original issue:
—
Description of problem:

The default dns-default pod is missing the "target.workload.openshift.io/management:" annotation. 

As a result when the workload partitioning feature is enabled on SNO, this pod resources will not get mutated and pinned to the reserved cpuset.

This is a regresion from 4.10. Pod spec from 4.10.17

Annotations:
...
   resources.workload.openshift.io/dns: {"cpushares": 51}
   resources.workload.openshift.io/kube-rbac-proxy: {"cpushares": 10}
   target.workload.openshift.io/management {"effect":"PreferredDuringScheduling"}

Version-Release number of selected component (if applicable):

4.11.0

How reproducible:

100%

Steps to Reproduce:

1. Install a SNO and check the annotation
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/cluster-dns-operator/pull/349

Bug OCPBUGS-9908: Garbage collect grafana-dashboard-etcd

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-8044~~. The following is the description of the original issue:
—

Description of problem:

rhbz#2089199, backported to 4.11.5, shifted the etcd Grafana dashboard from the monitoring operator to the etcd operator. During the shift, the ConfigMap was renamed from grafana-dashboard-etcd to etcd-dashboard. However, we did not include logic for garbage-collecting the obsolete dasboard, so clusters that update from 4.11.1 and similar into 4.11.>=5 or 4.12+ currently end up with both the obsolete and new ConfigMaps. We should grow code to remove the obsolete ConfigMap.

Version-Release number of selected component (if applicable):

4.11.>=5 and 4.12+ are currently exposed.

How reproducible:

100%

Steps to Reproduce:

1. Install 4.11.1.
2. Update to a release that defines the etcd-dashboard ConfigMap.
3. Check for etcd dashboards with oc -n openshift-config-managed get configmaps | grep etcd.

Actual results:

Both etcd-dashboard and grafana-dashboard-etcd exist:

$ oc -n openshift-config-managed get configmaps | grep etcd
etcd-dashboard                                        1      196d
grafana-dashboard-etcd                                1      2y282d

Another example is 4.11.1 to 4.11.5 CI:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/1570415394001260544/artifacts/e2e-aws-upgrade/configmaps.json | jq -r '.items[].metadata | select(.namespace == "openshift-config-managed" and (.name | contains("etcd"))) | .name'
etcd-dashboard
grafana-dashboard-etcd

Expected results:

Only etcd-dashboard still exists.

Additional info:

A new manifest for the outgoing ConfigMap that sets the release.openshift.io/delete: "true" annotation would ask the cluster-version operator to reap the obsolete ConfigMap.

https://github.com/openshift/cluster-etcd-operator/pull/1022

Bug OCPBUGS-1346: OpenStack UPI scripts do not create server group for Computes

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-1226~~. The following is the description of the original issue:
—
We added server groups for control plane and computes as part of ~~OSASINFRA-2570~~, except for UPI that only creates server group for the control plane.

We need to update the UPI scripts to create server group for computes to be consistent with IPI and have the instruction at https://docs.openshift.com/container-platform/4.11/machine_management/creating_machinesets/creating-machineset-osp.html work out of the box in case customers want to create MachineSets on their UPI clusters.

Related to OCPCLOUD-1135.

https://github.com/openshift/installer/pull/6351

Bug OCPBUGS-4863: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/ovn-kubernetes/pull/1448

Bug OCPBUGS-5078: Fails to deprovision cluster when swift omits 'content-type' and there are empty containers

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-5019~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-4941~~. The following is the description of the original issue:
—
Description of problem: This is a follow-up to ~~OCPBUGS-3933~~.

The installer fails to destroy the cluster when the OpenStack object storage omits 'content-type' from responses, and a container is empty.

Version-Release number of selected component (if applicable):

4.8.z

How reproducible:

Likely not happening in customer environments where Swift is exposed directly. We're seeing the issue in our CI where we're using a non-RHOSP managed cloud.

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/installer/pull/6725

Bug OCPBUGS-7515: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/sdn/pull/506

Bug OCPBUGS-1460: 4.11: [IBMCloud] The udevadm utility is missing in the IBM Cloud VPC block storage IPI image

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-416~~. The following is the description of the original issue:
—
Description of problem:
The pod fails to mount the PVC using IBM Cloud VPC block storage.

Version-Release number of selected component (if applicable):

How reproducible:
The steps can be followed here: from this link
https://cloud.ibm.com/docs/openshift?topic=openshift-vpc-block
.
The error occurs when the application pod tried to mount the VPC.

Steps to Reproduce:
Describe above.

Actual results:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 26m default-scheduler Successfully assigned default/test to a100-huge-m25p7-worker-3-with-secondary-xdwvl
Normal SuccessfulAttachVolume 26m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-8721c341-739d-4607-bbcb-9dcf66ef6dba"
Warning FailedMount 26m (x2 over 26m) kubelet MountVolume.MountDevice failed for volume "pvc-8721c341-739d-4607-bbcb-9dcf66ef6dba" : rpc error: code = Internal desc = {RequestID: ffbb97b4-e4d0-4016-87a9-dc46f80c5478 , Code: FormatAndMountFailed, Description: Failed to format '/dev/disk/by-id/virtio-0777-6872e22d-5c00-4' and mount it at '/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8721c341-739d-4607-bbcb-9dcf66ef6dba/globalmount', BackendError: format of disk "/dev/disk/by-id/virtio-0777-6872e22d-5c00-4" failed: type"ext4") target"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8721c341-739d-4607-bbcb-9dcf66ef6dba/globalmount") options"defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/disk/by-id/virtio-0777-6872e22d-5c00-4 does not exist and no size was specified.
) , Action: Please check if there is any error in POD describe related with volume attach}
Warning FailedMount 22m kubelet Unable to attach or mount volumes: unmounted volumes=[bs-pvc], unattached volumes=[kube-api-access-6bgvj bs-pvc]: timed out waiting for the condition
Warning FailedMount 4m11s (x9 over 24m) kubelet Unable to attach or mount volumes: unmounted volumes=[bs-pvc], unattached volumes=[bs-pvc kube-api-access-6bgvj]: timed out waiting for the condition
Warning FailedMount 3m51s (x17 over 26m) kubelet MountVolume.MountDevice failed for volume "pvc-8721c341-739d-4607-bbcb-9dcf66ef6dba" : rpc error: code = Internal desc = {RequestID: 1a12a7c5-3bd0-41cf-b8a9-90dd3224c2fb , Code: FormatAndMountFailed, Description: Failed to format '/dev/disk/by-id/virtio-0777-6872e22d-5c00-4' and mount it at '/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8721c341-739d-4607-bbcb-9dcf66ef6dba/globalmount', BackendError: format of disk "/dev/disk/by-id/virtio-0777-6872e22d-5c00-4" failed: type"ext4") target"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8721c341-739d-4607-bbcb-9dcf66ef6dba/globalmount") options"defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)

Expected results:
The pod should successfully mount the PVC

Additional info:
Had a debugging session with Sameer Shaikh and Arashad Ahamad from the IBM VPC block storage team. The conclusion is that the udevadm utility is missing in the IPI image used by the IBM Cloud VPC block storage CSI.

oc exec -it ibm-vpc-block-csi-controller-5fbb46bdc6-k7kpf -n openshift-cluster-csi-drivers -c iks-vpc-block-driver bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] – [COMMAND] instead.
bash-4.4$ which udevadm
which: no udevadm in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)

https://github.com/openshift/ibm-vpc-block-csi-driver/pull/19

Bug OCPBUGS-4761: [4.11] Bump OVS control plane to get "ovsdb/transaction.c: Refactor assess_weak_refs."

View the Description View the linked PRs

OVS 2.17+ introduced an optimization of "weak references" to substantially speed up database snapshots. in some cases weak references may leak memory; to aforementioned commit fixes that and has been pulled into ovs2.17-62 and later.

https://github.com/openshift/ovn-kubernetes/pull/1438

Bug OCPBUGS-8205: 'oc adm upgrade channel ...' should not clobber unrecognized spec properties

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-7960~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-7780~~. The following is the description of the original issue:
—
Description of problem:

4.9 and 4.10 oc calls to oc adm upgrade channel ... for 4.11+ clusters would clear spec.capabilities. Not all that many clusters try to restrict capabilities, but folks will need to bump their channel for at least every other minor (if their using EUS channels), and while we recommend folks use an oc from the 4.y they're heading towards, we don't have anything in place to enforce that.

Version-Release number of selected component (if applicable):

4.9 and 4.10 oc are exposed vs. the new-in-4.11 spec.capabilities. Newer oc could theoretically be exposed vs. any new ClusterVersion spec capabilities.

How reproducible:

100%

Steps to Reproduce:

1. Install a 4.11+ cluster with None capabilities.
2. Set the channel with a 4.10.51 oc, like oc adm upgrade channel fast-4.11.
3. Check the capabilities with oc get -o json clusterversion version | jq -c .spec.capabilities.

Actual results:

null

Expected results:

{"baselineCapabilitySet":"None"}

https://github.com/openshift/oc/pull/1360

Bug OCPBUGS-10671: Risk cache warming takes too long on channel changes

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-10514~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-10221~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-5469~~. The following is the description of the original issue:
—
Description of problem:

When changing channels it's possible that multiple new conditional update risks will need to be evaluated. For instance, a cluster running 4.10.34 in a 4.10 channel today only has to evaluate `OpenStackNodeCreationFails` but when the channel is changed to a 4.11 channel multiple new risks require evaluation and the evaluation of new risks is throttled at one every 10 minutes. This means if there are three new risks it may take up to 30 minutes after the channel has changed for the full set of conditional updates to be computed. This leads to a perception that no update paths are recommended because most will not wait 30 minutes, they expect immediate feedback.

Version-Release number of selected component (if applicable):

4.10.z, 4.11.z, 4.12, 4.13

How reproducible:

100%

Steps to Reproduce:

1. Install 4.10.34
2. Switch from stable-4.10 to stable-4.11
3.

Actual results:

Observe no recommended updates for 10-20 minutes because all available paths to 4.11 have a risk associated with them

Expected results:

Risks are computed in a timely manner for an interactive UX, lets say < 10s

Additional info:

This was intentional in the design, we didn't want risks to continuously re-evaluate or overwhelm the monitoring stack, however we didn't anticipate that we'd have long standing pile of risks and realize how confusing the user experience would be.

We intend to work around this in the deployed fleet by converting older risks from `type: promql` to `type: Always` avoiding the evaluation period but preserving the notification. While this may lead customers to believe they're exposed to a risk they may not be, as long as the set of outstanding risks to the latest version is limited to no more than one it's likely no one will notice. All 4.10 and 4.11 clusters currently have a clear path toward relatively recent 4.10.z or 4.11.z with no more than one risk to be evaluated.

https://github.com/openshift/cluster-version-operator/pull/915

Bug OCPBUGS-3049: [release-4.11] openshift-ingress-operator with mTLS does not download CRL

View the Description View the linked PRs

This is a copy of Bugzilla bug 2117524 for backport to 4.11.z

Original Text:

Description of problem:
On routers configured with mTLS and CRL defined in the CA with a CDP ; new CRL is downloaded only when restarting the ingress-operator.

2022-07-20T23:36:26.943Z	INFO	operator.clientca_configmap_controller	controller/controller.go:298	reconciling	{"request": "openshift-ingress-operator/service-bdrc"}
2022-07-20T23:36:26.943Z	INFO	operator.crl	crl/crl_configmap.go:69	certificate revocation list has expired	{"subject key identifier": "6aa909992e9890457b2a8de5659a44cab8e867a8"}
2022-07-20T23:36:26.943Z	INFO	operator.crl	crl/crl_configmap.go:69	retrieving certificate revocation list	{"subject key identifier": "6aa909992e9890457b2a8de5659a44cab8e867a8"}
2022-07-20T23:36:26.943Z	INFO	operator.crl	crl/crl_configmap.go:169	retrieving CRL distribution point	{"distribution point": "http://crl.domain.com/der/CN=XXXX,OU=XXX,O=XXX,C=XXX"}

Version-Release number of selected component (if applicable):
4.9.33

How reproducible:
Enable mTLS with a CRL

Actual results:
CRL is not download when expired
Clients get "SSL client certificate not trusted" errors while accessing resources

Expected results:
ingress-operator triggers CRL download when approaching expiration date so that the configmap is updated without manual action

https://github.com/openshift/cluster-ingress-operator/pull/853

Bug OCPBUGS-3819: Origin tests for bonds - 4.11 backport

View the Description View the linked PRs

Origin tests for the bond-cni

Backport of https://github.com/openshift/origin/pull/27405

https://github.com/openshift/origin/pull/27554

Bug OCPBUGS-4532: CSR are generated with incorrect Subject Alternate Names

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-4499~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-860~~. The following is the description of the original issue:
—
Description of problem:

In GCP, once an external IP address is assigned to master/infra node through GCP console, numbers of pending CSR from kubernetes.io/kubelet-serving is increasing, and the following error are reported:

I0902 10:48:29.254427       1 controller.go:121] Reconciling CSR: csr-q7bwd
I0902 10:48:29.365774       1 csr_check.go:157] csr-q7bwd: CSR does not appear to be client csr
I0902 10:48:29.371827       1 csr_check.go:545] retrieving serving cert from build04-c92hb-master-1.c.openshift-ci-build-farm.internal (10.0.0.5:10250)
I0902 10:48:29.375052       1 csr_check.go:188] Found existing serving cert for build04-c92hb-master-1.c.openshift-ci-build-farm.internal
I0902 10:48:29.375152       1 csr_check.go:192] Could not use current serving cert for renewal: CSR Subject Alternate Name values do not match current certificate
I0902 10:48:29.375166       1 csr_check.go:193] Current SAN Values: [build04-c92hb-master-1.c.openshift-ci-build-farm.internal 10.0.0.5], CSR SAN Values: [build04-c92hb-master-1.c.openshift-ci-build-farm.internal 10.0.0.5 35.211.234.95]
I0902 10:48:29.375175       1 csr_check.go:202] Falling back to machine-api authorization for build04-c92hb-master-1.c.openshift-ci-build-farm.internal
E0902 10:48:29.375184       1 csr_check.go:420] csr-q7bwd: IP address '35.211.234.95' not in machine addresses: 10.0.0.5
I0902 10:48:29.375193       1 csr_check.go:205] Could not use Machine for serving cert authorization: IP address '35.211.234.95' not in machine addresses: 10.0.0.5
I0902 10:48:29.379457       1 csr_check.go:218] Falling back to serving cert renewal with Egress IP checks
I0902 10:48:29.382668       1 csr_check.go:221] Could not use current serving cert and egress IPs for renewal: CSR Subject Alternate Names includes unknown IP addresses
I0902 10:48:29.382702       1 controller.go:233] csr-q7bwd: CSR not authorized

Version-Release number of selected component (if applicable):

4.11.2

Steps to Reproduce:

1. Assign external IPs to master/infra node in GCP
2. oc get csr | grep kubernetes.io/kubelet-serving

Actual results:

CSRs are not approved

Expected results:

CSRs are approved

Additional info:

This issue is only happen in GCP. Same OpenShift installations in AWS do not have this issue.

It looks like the CSR are created using external IP addresses once assigned.

Ref: https://coreos.slack.com/archives/C03KEQZC1L2/p1662122007083059

https://github.com/openshift/machine-api-provider-gcp/pull/27

Bug OCPBUGS-4785: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/cluster-etcd-operator/pull/973

Bug OCPBUGS-1750: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/ovn-kubernetes/pull/1282

Bug OCPBUGS-1782: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/console/pull/12107

Bug OCPBUGS-1790: [4.11] Users can't silence alerts from the dev console when dedicated UWM Alertmanager is deployed

View the Description View the linked PRs

Description of problem:

Version-Release number of selected component (if applicable):
4.11

How reproducible:
Always

Steps to Reproduce:
1. Enable UWM + dedicated UWM Alertmanager
2. Deploy an application + service monitor + alerting rule which fires always
3. Go to the OCP dev console and silence the alert.

Actual results:
Nothing happens

Expected results:
The alert notification is muted.

Additional info:
Copied from https://bugzilla.redhat.com/show_bug.cgi?id=2100860

Bug OCPBUGS-4137: [4.11] Ipsec pods restart due to liveness probes fail in cluster with more than 150 +

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-3824~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-2598~~. The following is the description of the original issue:
—
Description of problem:

Liveness probe of ipsec pods fail with large clusters. Currently the command that is executed in the ipsec container is
ovs-appctl -t ovs-monitor-ipsec ipsec/status && ipsec status
The problem is with command "ipsec/status". In clusters with high node count this command will return a list with all the node daemons of the cluster. This means that as the node count raises the completion time of the command raises too.

This makes the main command

ovs-appctl -t ovs-monitor-ipsec

To hang until the subcommand is finished.

As the liveness and readiness probe values are hardcoded in the manifest of the ipsec container herehttps//github.com/openshift/cluster-network-operator/blob/9c1181e34316d34db49d573698d2779b008bcc20/bindata/network/ovn-kubernetes/common/ipsec.yaml] the liveness timeout of the container probe of 60 seconds start to be insufficient as the node count list is growing. This resulted in a cluster with 170 + nodes to have 15+ ipsec pods in a crashloopbackoff state.

Version-Release number of selected component (if applicable):

Openshift Container Platform 4.10 but i think the same will be visible to other versions too.

How reproducible:

I was not able to reproduce due to an extreamely high amount of resources are needed and i think that there is no point as we have spotted the issue.

Steps to Reproduce:

1. Install an Openshift cluster with IPSEC enabled
2. Scale to 170+ nodes or more
3. Notice that the ipsec pods will start getting in a Crashloopbackoff state with failed Liveness/Readiness probes.

Actual results:

Ip Sec pods are stuck in a Crashloopbackoff state

Expected results:

Ip Sec pods to work normally

Additional info:

We have provided a workaround where CVO and CNO operators are scaled to 0 replicas in order for us to be able to increase the liveness probe limit to a value of 600 that recovered the cluster. 
As a next step the customer will try to reduce the node count and restore the default liveness timeout value along with bringing the operators back to see if the cluster will stabilize.

https://github.com/openshift/cluster-network-operator/pull/1640

Bug OCPBUGS-1540: [OCP 4.11] Inefficient use of SG rules when creating Service LBs leads to scale issues

View the Description View the linked PRs

Description of problem:

Each LB created for a Service type LoadBalancer results in 1 client rule and <# of public subnets> health rules being created.  The rules per SG quota in AWS is quite small; 60 by default, and 200 hard max.  OCP has about 40 rules OOTB. Assuming an HA cluster in 3 AZs, that is 4 rules per LB.  With default AWS quota, only ~5 LBs can be create and with the hard max of 200, only ~40 LBs can be created.

Version-Release number of selected component (if applicable):

4.12

How reproducible:

Always

Steps to Reproduce:

1.  Create Service type LoadBalancer and observe increase in master-sg and worker-sg rules sets
2.
3.

Actual results:

4 rules are created

Expected results:

1 rules is created when the client rule is a superset of the per-subnet health rules

Additional info:

This ~4x the number of Services of type LoadBalancer.  This is required for Hypershift.

https://github.com/openshift/kubernetes/pull/1375

Bug OCPBUGS-3911: Egress router POD creation is failing while using openshift-sdn network plugin

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-3889~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-3744~~. The following is the description of the original issue:
—
Description of problem:

Egress router POD creation on Openshift 4.11 is failing with below error.
~~~
Nov 15 21:51:29 pltocpwn03 hyperkube[3237]: E1115 21:51:29.467436    3237 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"stage-wfe-proxy-ext-qrhjw_stage-wfe-proxy(c965a287-28aa-47b6-9e79-0cc0e209fcf2)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"stage-wfe-proxy-ext-qrhjw_stage-wfe-proxy(c965a287-28aa-47b6-9e79-0cc0e209fcf2)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_stage-wfe-proxy-ext-qrhjw_stage-wfe-proxy_c965a287-28aa-47b6-9e79-0cc0e209fcf2_0(72bcf9e52b199061d6e651e84b0892efc142601b2442c2d00b92a1ba23208344): error adding pod stage-wfe-proxy_stage-wfe-proxy-ext-qrhjw to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus\\\" name=\\\"multus-cni-network\\\" failed (add): [stage-wfe-proxy/stage-wfe-proxy-ext-qrhjw/c965a287-28aa-47b6-9e79-0cc0e209fcf2:openshift-sdn]: error adding container to network \\\"openshift-sdn\\\": CNI request failed with status 400: 'could not open netns \\\"/var/run/netns/8c5ca402-3381-4935-baed-ea454161d669\\\": unknown FS magic on \\\"/var/run/netns/8c5ca402-3381-4935-baed-ea454161d669\\\": 1021994\\n'\"" pod="stage-wfe-proxy/stage-wfe-proxy-ext-qrhjw" podUID=c965a287-28aa-47b6-9e79-0cc0e209fcf2
~~~

I have checked SDN POD log from node where egress router POD is failing and I could see below error message.

~~~
2022-11-15T21:51:29.283002590Z W1115 21:51:29.282954  181720 pod.go:296] CNI_ADD stage-wfe-proxy/stage-wfe-proxy-ext-qrhjw failed: could not open netns "/var/run/netns/8c5ca402-3381-4935-baed-ea454161d669": unknown FS magic on "/var/run/netns/8c5ca402-3381-4935-baed-ea454161d669": 1021994
~~~

Crio is logging below event and looking at the log it seems the namespace has been created on node.

~~~
Nov 15 21:51:29 pltocpwn03 crio[3150]: time="2022-11-15 21:51:29.307184956Z" level=info msg="Got pod network &{Name:stage-wfe-proxy-ext-qrhjw Namespace:stage-wfe-proxy ID:72bcf9e52b199061d6e651e84b0892efc142601b2442c2d00b92a1ba23208344 UID:c965a287-28aa-47b6-9e79-0cc0e209fcf2 NetNS:/var/run/netns/8c5ca402-3381-4935-baed-ea454161d669 Networks:[] RuntimeConfig:map[multus-cni-network:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}] Aliases:map[]}"
~~~

Version-Release number of selected component (if applicable):

4.11.12

How reproducible:

Not Sure

Steps to Reproduce:

1.
2.
3.

Actual results:

Egress router POD is failing to create. Sample application could be created without any issue.

Expected results:

Egress router POD should get created

Additional info:

Egress router POD is created following below document and it does contain pod.network.openshift.io/assign-macvlan: "true" annotation.

https://docs.openshift.com/container-platform/4.11/networking/openshift_sdn/deploying-egress-router-layer3-redirection.html#nw-egress-router-pod_deploying-egress-router-layer3-redirection

https://github.com/openshift/cluster-network-operator/pull/1631

Bug OCPBUGS-5206: Developer Topology always blanks with large contents when first rendering

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-4897~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-2500~~. The following is the description of the original issue:
—
Description of problem:

When the Ux switches to the Dev console the topology is always blank in a Project that has a large number of components.

Version-Release number of selected component (if applicable):

How reproducible:

Always occurs

Steps to Reproduce:

1.Create a project with at least 12 components (Apps, Operators, knative Brokers)
2. Go to the Administrator Viewpoint
3. Switch to Developer Viewpoint/Topology
4. No components displayed
5. Click on 'fit to screen'
6. All components appear

Actual results:

Topology renders with all controls but no components visible (see screenshot 1)

Expected results:

All components should be visible

Additional info:

https://github.com/openshift/console/pull/12398

Bug OCPBUGS-7590: thanos-ruler-user-workload-1 pod is getting repeatedly re-created after upgrade do 4.10.41

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-7458~~. The following is the description of the original issue:
—
Description of problem:

- After upgrading to OCP 4.10.41, thanos-ruler-user-workload-1 in the openshift-user-workload-monitoring namespace is consistently being created and deleted.
- We had to scale down the Prometheus operator multiple times so that the upgrade is considered as successful.
- This fix is temporary. After some time it appears again and Prometheus operator needs to be scaled down and up again.
- The issue is present on all clusters in this customer environment which are upgraded to 4.10.41.

Version-Release number of selected component (if applicable):

How reproducible:

N/A, I wasn't able to reproduce the issue.

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

https://github.com/openshift/prometheus-operator/pull/218

Bug OCPBUGS-3023: GCP: missing multiple regions

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-3022~~. The following is the description of the original issue:
—
Description of problem:

** europe-west8
** europe-west9
** europe-southwest1
** southamerica-west1
** us-east5
** us-south1
are not listed in the survey

Steps to Reproduce:

1. run survey (openshift-install create install-config without an install config file)
2. go through prompts until regions
3.

Actual results:

listed regions are missing

Expected results:

regions above are listed (and install succeeds in the region)

https://github.com/openshift/installer/pull/6542

Bug OCPBUGS-3193: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/installer/pull/6350

Bug OCPBUGS-365: Update ose-baremetal-installer images to be consistent with ART

View the Description View the linked PRs

Description of problem:

This repository is out of sync with the downstream product builds for this component.
One or more images differ from those being used by ART to create product builds. This
should be addressed to ensure that the component's CI testing is accurately
reflecting what customers will experience.

The information within the following ART component metadata is driving this alignment
request: ose-baremetal-installer.yml.

The vast majority of these PRs are opened because a different Golang version is being
used to build the downstream component. ART compiles most components with the version
of Golang being used by the control plane for a given OpenShift release. Exceptions
to this convention (i.e. you believe your component must be compiled with a Golang
version independent from the control plane) must be granted by the OpenShift
architecture team and communicated to the ART team.

Version-Release number of selected component (if applicable): 4.11

Additional info: This issue is needed for the bot-created PR to merge after the 4.11 GA.

https://github.com/openshift/installer/pull/5897

Bug OCPBUGS-4304: Backport to 4.11 Specify resources.requests for operator pod

View the Description View the linked PRs

Description of problem:

To address: 'Static Pod is managed but errored" err="managed container xxx does not have Resource.Requests'

Version-Release number of selected component (if applicable):

4.11

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

Already merged in https://github.com/openshift/cluster-kube-controller-manager-operator/pull/660

https://github.com/openshift/cluster-kube-controller-manager-operator/pull/667

Bug OCPBUGS-4325: [gcp] when the optional Service Usage API is disabled, IPI installation cannot succeed

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-1704~~. The following is the description of the original issue:
—
Description of problem:

According to OCP 4.11 doc (https://docs.openshift.com/container-platform/4.11/installing/installing_gcp/installing-gcp-account.html#installation-gcp-enabling-api-services_installing-gcp-account), the Service Usage API (serviceusage.googleapis.com) is an optional API service to be enabled. But, the installation cannot succeed if this API is disabled.

Version-Release number of selected component (if applicable):

4.12.0-0.nightly-2022-09-25-071630

How reproducible:

Always, if the Service Usage API is disabled in the GCP project.

Steps to Reproduce:

1. Make sure the Service Usage API (serviceusage.googleapis.com) is disabled in the GCP project.
2. Try IPI installation in the GCP project.

Actual results:

The installation would fail finally, without any worker machines launched.

Expected results:

Installation should succeed, or the OCP doc should be updated.

Additional info:

Please see the attached must-gather logs (http://virt-openshift-05.lab.eng.nay.redhat.com/jiwei/jiwei-0926-03-cnxn5/) and the sanity check results. 
FYI if enabling the API, and without changing anything else, the installation could succeed.

Bug OCPBUGS-453: [4.11] update ironic to latest available

View the Description View the linked PRs

update ironic software to pick up latest bug fixes

update dependencies as needed

this is also related to https://bugzilla.redhat.com/show_bug.cgi?id=2115122

Bug OCPBUGS-4992: Windows East-West networking test fails for pod to pod communication

View the Description View the linked PRs

Description of problem:

Looks like a regression was introduced and the Windows networking tests started to fail accross all 4.11 CI jobs for both Linux-to-Windows and Windows-to-Windows pods communication. 

https://github.com/openshift/ovn-kubernetes/pull/1415 merged without passing Windows networking tests.

https://github.com/openshift/windows-machine-config-operator/pull/1359  shows networking tests failing accross all CI jobs.

Version-Release number of selected component (if applicable):

4.11

How reproducible:

Always

Steps to Reproduce:

1. Trigger a job retest in any of the above mentioned PRs

Actual results:

Windows networking test fail

Expected results:

Windows networking test pass

Additional info:

Direct link to a release-4.11 aws-e2e-operator CI job showing the failed networking test.

https://github.com/openshift/ovn-kubernetes/pull/1459

Bug OCPBUGS-5091: pods should successfully create sandboxes Test fails when using Kuryr

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-4851~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-4850~~. The following is the description of the original issue:
—
Description of problem:

Kuryr might take a while to create Pods because it has to create Neutron ports for the pods. If a pod gets deleted while this is being processed, a
warning Event will be generated causing the "[sig-network] pods should successfully create sandboxes by adding pod to network" to fail.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/origin/pull/27621

Bug OCPBUGS-8311: mapi_machinehealthcheck_short_circuit not properly reconciling causing MachineHealthCheckUnterminatedShortCircuit alert to fire

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-8286~~. The following is the description of the original issue:
—
Description of problem:

mapi_machinehealthcheck_short_circuit is not properly reconciling the state, when a MachineHealthCheck is failing because of unhealthy Machines but then is removed.

When doing two MachineSet (called blue and green and only one has running Machines at a specific point in time) with MachineAutoscaler and MachineHealthCheck, the mapi_machinehealthcheck_short_circuit will continue to report 1 for MachineHealth that actually was removed because of a switch from blue to green.

$ oc get machineset | egrep 'blue|green'
housiocp4-wvqbx-worker-blue-us-east-2a    0         0                             2d17h
housiocp4-wvqbx-worker-green-us-east-2a   1         1         1       1           2d17h

$ oc get machineautoscaler
NAME                      REF KIND     REF NAME                                   MIN   MAX   AGE
worker-green-us-east-1a   MachineSet   housiocp4-wvqbx-worker-green-us-east-2a   1     4     2d17h

$ oc get machinehealthcheck
NAME                              MAXUNHEALTHY   EXPECTEDMACHINES   CURRENTHEALTHY
machine-api-termination-handler   100%           0                  0
worker-green-us-east-1a           40%            1                  1

      {
        "name": "machine-health-check-unterminated-short-circuit",
        "file": "/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-machine-api-machine-api-operator-prometheus-rules-ccb650d9-6fc4-422b-90bb-70452f4aff8f.yaml",
        "rules": [
          { 
            "state": "firing",
            "name": "MachineHealthCheckUnterminatedShortCircuit",
            "query": "mapi_machinehealthcheck_short_circuit == 1",
            "duration": 1800,
            "labels": {
              "severity": "warning"
            },
            "annotations": {
              "description": "The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n",
              "summary": "machine health check {{ $labels.name }} has been disabled by short circuit for more than 30 minutes"
            },
            "alerts": [
              { 
                "labels": {
                  "alertname": "MachineHealthCheckUnterminatedShortCircuit",
                  "container": "kube-rbac-proxy-mhc-mtrc",
                  "endpoint": "mhc-mtrc",
                  "exported_namespace": "openshift-machine-api",
                  "instance": "10.128.0.58:8444",
                  "job": "machine-api-controllers",
                  "name": "worker-blue-us-east-1a",
                  "namespace": "openshift-machine-api",
                  "pod": "machine-api-controllers-779dcb8769-8gcn6",
                  "service": "machine-api-controllers",
                  "severity": "warning"
                },
                "annotations": {
                  "description": "The number of unhealthy machines has exceeded the `maxUnhealthy` limit for the check, you should check\nthe status of machines in the cluster.\n",
                  "summary": "machine health check worker-blue-us-east-1a has been disabled by short circuit for more than 30 minutes"
                },
                "state": "firing",
                "activeAt": "2022-12-09T15:59:25.1287541Z",
                "value": "1e+00"
              }
            ],
            "health": "ok",
            "evaluationTime": 0.000648129,
            "lastEvaluation": "2022-12-12T09:35:55.140174009Z",
            "type": "alerting"
          }
        ],
        "interval": 30,
        "limit": 0,
        "evaluationTime": 0.000661589,
        "lastEvaluation": "2022-12-12T09:35:55.140165629Z"
      },

As we can see above, worker-blue-us-east-1a is no longer available and active but rather worker-green-us-east-1a. But worker-blue-us-east-1a was there before the switch to green has happen and was actuall reporting some unhealthy Machines. But since it's now gone, mapi_machinehealthcheck_short_circuit should properly reconcile as otherwise this is a false/positive alert.

Version-Release number of selected component (if applicable):

OpenShift Container Platform 4.12.0-rc.3 (but is also seen on previous version)

How reproducible:

- Always

Steps to Reproduce:

1. Setup OpenShift Container Platform 4 on AWS for example
2. Create blue and green MachineSet with MachineAutoScaler and MachineHealthCheck
3. Have active Machines for blue only
4. Trigger unhealthy Machines in blue MachineSet
5. Switch to green MachineSet, by removing MachineHealthCheck, MachineAutoscaler and setting replicate of blue MachineSet to 0
6. Create green MachineHealthCheck, MachineAutoscaler and scale geen MachineSet to 1
7. Observe how mapi_machinehealthcheck_short_circuit continues to report unhealthy state for blue MachineHealthCheck which no longer exists.

Actual results:

mapi_machinehealthcheck_short_circuit reporting problematic MachineHealthCheck even though the faulty MachineHealthCheck does no longer exist.

Expected results:

mapi_machinehealthcheck_short_circuit to properly reconcile it's state and remove MachineHealthChecks that have been removed on OpenShift Container Platform level

Additional info:

It kind of looks like similar to the issue reported in https://bugzilla.redhat.com/show_bug.cgi?id=2013528 respectively https://bugzilla.redhat.com/show_bug.cgi?id=2047702 (although https://bugzilla.redhat.com/show_bug.cgi?id=2047702 may not be super relevant)

https://github.com/openshift/machine-api-operator/pull/1124

Bug OCPBUGS-3362: [4.11] Bootimage bump tracker

View the Description View the linked PRs

Tracker issue for bootimage bump in 4.11. This issue should block issues which need a bootimage bump to fix.

The previous bump was ~~OCPBUGS-562~~.

https://github.com/openshift/installer/pull/6720

Bug OCPBUGS-4167: [Azure] Some cloudprivateipconfig IP is error status

View the Description View the linked PRs

Description of problem:

Created two egressIP object, egressIPs in one egressIP object cannot be applied successfully

Version-Release number of selected component (if applicable):

4.11.0-0.nightly-2022-11-27-164248

How reproducible:

Frequently happen in auto case

Steps to Reproduce:

1.  Label two nodes as egress nodes
  oc get nodes -o wide
NAME                                   STATUS   ROLES    AGE    VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                        KERNEL-VERSION                 CONTAINER-RUNTIME
huirwang-1128a-s6j6t-master-0          Ready    master   154m   v1.24.6+5658434   10.0.0.8      <none>        Red Hat Enterprise Linux CoreOS 411.86.202211232221-0 (Ootpa)   4.18.0-372.32.1.el8_6.x86_64   cri-o://1.24.3-6.rhaos4.11.gitc4567c0.el8
huirwang-1128a-s6j6t-master-1          Ready    master   154m   v1.24.6+5658434   10.0.0.7      <none>        Red Hat Enterprise Linux CoreOS 411.86.202211232221-0 (Ootpa)   4.18.0-372.32.1.el8_6.x86_64   cri-o://1.24.3-6.rhaos4.11.gitc4567c0.el8
huirwang-1128a-s6j6t-master-2          Ready    master   153m   v1.24.6+5658434   10.0.0.6      <none>        Red Hat Enterprise Linux CoreOS 411.86.202211232221-0 (Ootpa)   4.18.0-372.32.1.el8_6.x86_64   cri-o://1.24.3-6.rhaos4.11.gitc4567c0.el8
huirwang-1128a-s6j6t-worker-westus-1   Ready    worker   135m   v1.24.6+5658434   10.0.1.5      <none>        Red Hat Enterprise Linux CoreOS 411.86.202211232221-0 (Ootpa)   4.18.0-372.32.1.el8_6.x86_64   cri-o://1.24.3-6.rhaos4.11.gitc4567c0.el8
huirwang-1128a-s6j6t-worker-westus-2   Ready    worker   136m   v1.24.6+5658434   10.0.1.4      <none>        Red Hat Enterprise Linux CoreOS 411.86.202211232221-0 (Ootpa)   4.18.0-372.32.1.el8_6.x86_64   cri-o://1.24.3-6.rhaos4.11.gitc4567c0.el8

 % oc get node huirwang-1128a-s6j6t-worker-westus-1 --show-labels
NAME                                   STATUS   ROLES    AGE    VERSION           LABELS
huirwang-1128a-s6j6t-worker-westus-1   Ready    worker   136m   v1.24.6+5658434   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_D4s_v3,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=westus,failure-domain.beta.kubernetes.io/zone=0,k8s.ovn.org/egress-assignable=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=huirwang-1128a-s6j6t-worker-westus-1,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,node.kubernetes.io/instance-type=Standard_D4s_v3,node.openshift.io/os_id=rhcos,topology.disk.csi.azure.com/zone=,topology.kubernetes.io/region=westus,topology.kubernetes.io/zone=0

 % oc get node huirwang-1128a-s6j6t-worker-westus-2 --show-labels
NAME                                   STATUS   ROLES    AGE    VERSION           LABELS
huirwang-1128a-s6j6t-worker-westus-2   Ready    worker   136m   v1.24.6+5658434   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_D4s_v3,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=westus,failure-domain.beta.kubernetes.io/zone=0,k8s.ovn.org/egress-assignable=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=huirwang-1128a-s6j6t-worker-westus-2,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,node.kubernetes.io/instance-type=Standard_D4s_v3,node.openshift.io/os_id=rhcos,topology.disk.csi.azure.com/zone=,topology.kubernetes.io/region=westus,topology.kubernetes.io/zone=0
2. Created two egressIP objects
3.

Actual results:

egressip-47032 was not applied to any egress node
% oc get egressip
NAME             EGRESSIPS    ASSIGNED NODE                          ASSIGNED EGRESSIPS
egressip-47032   10.0.1.166                                          
egressip-47034   10.0.1.181   huirwang-1128a-s6j6t-worker-westus-1   10.0.1.181

%  oc get cloudprivateipconfig                      
NAME         AGE
10.0.1.130   6m25s
10.0.1.138   6m34s
10.0.1.166   6m34s
10.0.1.181   6m25s
%  oc get cloudprivateipconfig  10.0.1.166  -o yaml
apiVersion: cloud.network.openshift.io/v1
kind: CloudPrivateIPConfig
metadata:
  annotations:
    k8s.ovn.org/egressip-owner-ref: egressip-47032
  creationTimestamp: "2022-11-28T10:27:37Z"
  finalizers:
  - cloudprivateipconfig.cloud.network.openshift.io/finalizer
  generation: 1
  name: 10.0.1.166
  resourceVersion: "87528"
  uid: 5221075a-35d0-4670-a6a7-ddfc6cbc700b
spec:
  node: huirwang-1128a-s6j6t-worker-westus-1
status:
  conditions:
  - lastTransitionTime: "2022-11-28T10:33:29Z"
    message: 'Error processing cloud assignment request, err: <nil>'
    observedGeneration: 1
    reason: CloudResponseError
    status: "False"
    type: Assigned
  node: huirwang-1128a-s6j6t-worker-westus-1
%  oc get cloudprivateipconfig  10.0.1.138  -o yaml 
apiVersion: cloud.network.openshift.io/v1
kind: CloudPrivateIPConfig
metadata:
  annotations:
    k8s.ovn.org/egressip-owner-ref: egressip-47032
  creationTimestamp: "2022-11-28T10:27:37Z"
  finalizers:
  - cloudprivateipconfig.cloud.network.openshift.io/finalizer
  generation: 1
  name: 10.0.1.138
  resourceVersion: "87523"
  uid: e4604e76-64d8-4735-87a2-eb50d28854cc
spec:
  node: huirwang-1128a-s6j6t-worker-westus-2
status:
  conditions:
  - lastTransitionTime: "2022-11-28T10:33:29Z"
    message: 'Error processing cloud assignment request, err: <nil>'
    observedGeneration: 1
    reason: CloudResponseError
    status: "False"
    type: Assigned
  node: huirwang-1128a-s6j6t-worker-westus-2

oc logs cloud-network-config-controller-6f7b994ddc-vhtbp  -n openshift-cloud-network-config-controller

.......
E1128 10:30:43.590807       1 controller.go:165] error syncing '10.0.1.138': error assigning CloudPrivateIPConfig: "10.0.1.138" to node: "huirwang-1128a-s6j6t-worker-westus-2", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="InvalidRequestFormat" Message="Cannot parse the request." Details=[{"code":"DuplicateResourceName","message":"Resource /subscriptions//resourceGroups//providers/Microsoft.Network/networkInterfaces/ has two child resources with the same name (huirwang-1128a-s6j6t-worker-westus-2_10.0.1.138)."}], requeuing in cloud-private-ip-config workqueue
I1128 10:30:44.051422       1 cloudprivateipconfig_controller.go:271] CloudPrivateIPConfig: "10.0.1.166" will be added to node: "huirwang-1128a-s6j6t-worker-westus-1"
E1128 10:30:44.301259       1 controller.go:165] error syncing '10.0.1.166': error assigning CloudPrivateIPConfig: "10.0.1.166" to node: "huirwang-1128a-s6j6t-worker-westus-1", err: network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="InvalidRequestFormat" Message="Cannot parse the request." Details=[{"code":"DuplicateResourceName","message":"Resource /subscriptions//resourceGroups//providers/Microsoft.Network/networkInterfaces/ has two child resources with the same name (huirwang-1128a-s6j6t-worker-westus-1_10.0.1.166)."}], requeuing in cloud-private-ip-config workqueue
..........

Expected results:

EgressIP can be applied successfully.

Additional info:

Bug OCPBUGS-1410: E2E: intermittent failure is seen on tests for devfile

View the Description View the linked PRs

Clone of https://bugzilla.redhat.com/show_bug.cgi?id=2106803 to backport the e2e fix to 4.11 and 4.10.

Description of problem: E2E: intermittent failure is seen on tests for devfile due to network call to devfile registry

Deploy git workload with devfile from topology page: A-04-TC01

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_console/11726/pull-ci-openshift-console-master-e2e-gcp-console/1547046629775773696/artifacts/e2e-gcp-console/test/artifacts/gui_test_screenshots/cypress/screenshots/e2e/add-flow-ci.feature/Deploy%20git%20workload%20with%20devfile%20from%20topology%20page%20A-04-TC01%20--%20after%20each%20hook%20(failed).png

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_console/11768/pull-ci-openshift-console-master-e2e-gcp-console/1547061730478133248

Version-Release number of selected component (if applicable):

How reproducible: Intermittent

Steps to Reproduce:
1. Run test for add-flow-ci.feature to test Deploy git workload with devfile from topology page: A-04-TC01

Actual results:

Expected results: Show always pass

Additional info:

https://github.com/openshift/console/pull/12020

Bug OCPBUGS-4324: Backport PodNetworkConnectivityCheck for must-gather

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-4250~~. The following is the description of the original issue:
—
Description of problem:

Added a script to collect PodNetworkConnectivityChecks to able to view the overall status of the pod network connectivity.

Current must-gather collects the contents of `openshift-network-diagnostics` but does not collect the PodNetworkConnectivityCheck.

Version-Release number of selected component (if applicable):

4.12, 4.11, 4.10

https://github.com/openshift/must-gather/pull/338

Bug OCPBUGS-4340: oc get dc fails when AllRequestBodies audit-profile is set in apiserver

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-501~~. The following is the description of the original issue:
—
Description of problem:

Version-Release number of selected component (if applicable): 4.10.16

How reproducible: Always

Steps to Reproduce:
1. Edit the apiserver resource and add spec.audit.customRules field

$ oc get apiserver cluster -o yaml
spec:
audit:
customRules:

group: system:authenticated:oauth
profile: AllRequestBodies
group: system:authenticated
profile: AllRequestBodies
profile: Default

2. Allow the kube-apiserver pods to rollout new revision.
3. Once the kube-apiserver pods are in new revision execute $ oc get dc

Actual results:

Error from server (InternalError): an error on the server ("This request caused apiserver to panic. Look in the logs for details.") has prevented the request from succeeding (get deploymentconfigs.apps.openshift.io)

Expected results: The command "oc get dc" should display the deploymentconfig without any error.

Additional info:

https://github.com/openshift/openshift-apiserver/pull/336

Bug OCPBUGS-1: Test Bug

View the Description View the linked PRs

Test description

https://github.com/openshift/driver-toolkit/pull/77

Bug OCPBUGS-2730: systemReserved:ephemeral-storage in KubeletConfig doesn't work as expected

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-2079~~. The following is the description of the original issue:
—
Description of problem:

The setting of systemReserved: ephemeral-storage in KubeletConfig is not working as expected.

Version-Release number of selected component (if applicable):

4.10.z, may exist on other OCP versions as well.

How reproducible:

always

Steps to Reproduce:

1. Create a KubeletConfig on the node:

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: system-reserved-config
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/master: ""
  kubeletConfig:
    systemReserved:
      cpu: 500m
      memory: 500Mi
      ephemeral-storage: 10Gi


2. Check node allocatable storage with command: oc describe node |grep -C 5 ephemeral-storage

Actual results:

The Allocatable:ephemeral-storage on the node is not capacity.ephemeral-storage - systemReserved.ephemeral-storage - eviction-thresholds (10% of the capacity.ephemeral-storage by default)

Expected results:

The Allocatable:ephemeral-storage on the node should be capacity.ephemeral-storage - systemReserved.ephemeral-storage - eviction-thresholds (10% of the capacity.ephemeral-storage by default)

Additional info:

The root cause might be: process argument '--system-reserved=cpu=500m,memory=500Mi' overwrote the setting in /etc/kubernetes/kubelet.conf, one example:

root        6824       1 27 Sep30 ?        1-09:00:24 kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --runtime-cgroups=/system.slice/crio.service --node-labels=node-role.kubernetes.io/master,node.openshift.io/os_id=rhcos --node-ip=192.168.58.47 --minimum-container-ttl-duration=6m0s --cloud-provider= --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --hostname-override= --register-with-taints=node-role.kubernetes.io/master=:NoSchedule --pod-infra-container-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7b6408460148cb73c59677dbc2c261076bc07226c43b0c9192cc70aef5ba62 --system-reserved=cpu=500m,memory=500Mi --v=2 --housekeeping-interval=30s

https://github.com/openshift/machine-config-operator/pull/3381

Bug OCPBUGS-3116: [2117255] Failed to dump flows for flow sync, stderr: "ovs-ofctl: br-ext is not a bridge or a socket"

View the Description View the linked PRs

Description of problem:
-----------------------
On dualstack baremetal IPI cluster next error message is present in ovnkube logs:

oc logs -n openshift-ovn-kubernetes ovnkube-node-rvggh -c ovnkube-node
...

E0810 02:12:46.343460 353971 node_linux.go:593] Failed to dump flows for flow sync, stderr: "ovs-ofctl: br-ext is not a bridge or a socket\n", error: exit status 1
E0810 02:13:16.347603 353971 node_linux.go:593] Failed to dump flows for flow sync, stderr: "ovs-ofctl: br-ext is not a bridge or a socket\n", error: exit status 1
E0810 02:13:46.351108 353971 node_linux.go:593] Failed to dump flows for flow sync, stderr: "ovs-ofctl: br-ext is not a bridge or a socket\n", error: exit status 1
E0810 02:14:16.355047 353971 node_linux.go:593] Failed to dump flows for flow sync, stderr: "ovs-ofctl: br-ext is not a bridge or a socket\n", error: exit status 1
E0810 02:14:46.358950 353971 node_linux.go:593] Failed to dump flows for flow sync, stderr: "ovs-ofctl: br-ext is not a bridge or a socket\n", error: exit status 1
I0810 02:15:13.313945 353971 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Service total 9 items received
E0810 02:15:16.362737 353971 node_linux.go:593] Failed to dump flows for flow sync, stderr: "ovs-ofctl: br-ext is not a bridge or a socket\n", error: exit status 1
E0810 02:15:46.366490 353971 node_linux.go:593] Failed to dump flows for flow sync, stderr: "ovs-ofctl: br-ext is not a bridge or a socket\n", error: exit status 1
E0810 02:16:16.369963 353971 node_linux.go:593] Failed to dump flows for flow sync, stderr: "ovs-ofctl: br-ext is not a bridge or a socket\n", error: exit status 1
I0810 02:16:24.306561 353971 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Endpoints total 560 items received
E0810 02:16:46.373482 353971 node_linux.go:593] Failed to dump flows for flow sync, stderr: "ovs-ofctl: br-ext is not a bridge or a socket\n", error: exit status 1
E0810 02:17:16.377497 353971 node_linux.go:593] Failed to dump flows for flow sync, stderr: "ovs-ofctl: br-ext is not a bridge or a socket\n", error: exit status 1
E0810 02:17:46.380726 353971 node_linux.go:593] Failed to dump flows for flow sync, stderr: "ovs-ofctl: br-ext is not a bridge or a socket\n", error: exit status 1
I0810 02:18:15.325871 353971 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 50 items received
E0810 02:18:16.384732 353971 node_linux.go:593] Failed to dump flows for flow sync, stderr: "ovs-ofctl: br-ext is not a bridge or a socket\n", error: exit status 1
I0810 02:18:38.299738 353971 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Pod total 9 items received
E0810 02:18:46.388162 353971 node_linux.go:593] Failed to dump flows for flow sync, stderr: "ovs-ofctl: br-ext is not a bridge or a socket\n", error: exit status 1
E0810 02:19:16.391669 353971 node_linux.go:593] Failed to dump flows for flow sync, stderr: "ovs-ofctl: br-ext is not a bridge or a socket\n", error: exit status 1

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
~~OCP-4~~.10.26

ovn-2021-21.12.0-58.el8fdp.x86_64
ovn-2021-host-21.12.0-58.el8fdp.x86_64
ovn-2021-central-21.12.0-58.el8fdp.x86_64
ovn-2021-vtep-21.12.0-58.el8fdp.x86_64

How reproducible:
-----------------
so far spotted on 2 different clusters

Steps to Reproduce:
-------------------
1. Deploy dualstack baremetal IPI cluster with OVNKubernetesHybrid network(add next to cluster's config before running cluster install):

defaultNetwork:
type: OVNKubernetes
ovnKubernetesConfig:
hybridOverlayConfig:
hybridClusterNetwork: []

Actual results:
---------------
Error message in logs

Expected results:
-----------------
No error message in logs

Additional info:
----------------
Baremetal dualstack setup with 3 masters and 4 workers, bonding configured for baremetal network on masters and workers

https://github.com/openshift/ovn-kubernetes/pull/1350

Story CONSOLE-2999: Update OCP branding

View the Description View the linked PRs

[Updated story request]

Decision is to always display Red Hat OpenShift logo for OCP instead of conditionally. And also update the OCP login, errors, providers templates. https://openshift.github.io/oauth-templates/

Related note in comments.

[Original request]

If the ACM or the ACS dynamic plugin is enabled and there is not a custom branding set, then the default "Red Hat Openshift" branding should be shown.

This was identified as an issue during the Hybrid Console Scrum on 11/15/20201

PRs associated with this change

https://github.com/openshift/console/pull/10940 [merged]

https://github.com/openshift/oauth-templates/pull/20 [merged]

https://github.com/openshift/cluster-authentication-operator/pull/540 [merged]

https://github.com/openshift/console/pull/10940

Bug OCPBUGS-1419: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/console/pull/12056

Bug OCPBUGS-5790: [vsphere-csi-driver-operator] CSI cloud.conf doesn't list multiple datacenters when specified

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-1805~~. The following is the description of the original issue:
—
The vSphere CSI cloud.conf lists the single datacenter from platform workspace config but in a multi-zone setup (https://github.com/openshift/enhancements/pull/918 ) there may be more than the one datacenter.

This issue is resulting in PVs failing to attach because the virtual machines can't be find in any other datacenter. For example:

0s Warning FailedAttachVolume pod/image-registry-85b5d5db54-m78vp AttachVolume.Attach failed for volume "pvc-ab1a0611-cb3b-418d-bb3b-1e7bbe2a69ed" : rpc error: code = Internal desc = failed to find VirtualMachine for node:"rbost-zonal-ghxp2-worker-3-xm7gw". Error: virtual machine wasn't found

The machine above lives in datacenter-2 but the CSI cloud.conf is only aware of the datacenter IBMCloud.

$ oc get cm vsphere-csi-config -o yaml  -n openshift-cluster-csi-drivers | grep datacenters
    datacenters = "IBMCloud"

https://github.com/openshift/vmware-vsphere-csi-driver-operator/pull/130

Bug OCPBUGS-998: [OpenShiftSDN ] API svc un-reachability on OCP upgrade 4.10->4.11

View the Description View the linked PRs

Description of problem:

During an upgrade from 4.10.0-0.nightly-2022-09-06-081345 -> 4.11.0-0.nightly-2022-09-06-074353

DNS operator stuck in progressing state with following error in dns-defalt pod

2022-09-07T00:47:35.959931289Z [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://172.30.0.1:443/version": dial tcp 172.30.0.1:443: i/o timeout

must-gather: http://shell.lab.bos.redhat.com/~anusaxen/must-gather-136172-050448657.tar.gz

Version-Release number of selected component (if applicable):

4.11.0-0.nightly-2022-09-06-074353

How reproducible:

Intermittent

Steps to Reproduce:

1. Perform upgrade from 4.10->4.11
2.
3.

Actual results:

Upgrade unsuccessful due to API svc unreachability

Expected results:

Upgrade should be successful

Additional info:

$ omg get co | grep -v "True.*False.*False"
NAME                                      VERSION                             AVAILABLE  PROGRESSING  DEGRADED  SINCE
dns                                       4.10.0-0.nightly-2022-09-06-081345  True       True         False     3h4m
network                                   4.11.0-0.nightly-2022-09-06-074353  True       True         False     3h5m

$ omg logs dns-default-5mqz4 -c dns -n openshift-dns
/home/anusaxen/Downloads/must-gather-136208-318613238/must-gather.local.4033693973186012455/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-d995e1608ba83a1f71b5a1c49705aa3cef38618e0674ee256332d1b0e2cb0e23/namespaces/openshift-dns/pods/dns-default-5mqz4/dns/dns/logs/current.log
2022-09-07T00:45:36.355161941Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:36.854403897Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:37.354367197Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:37.861427861Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:38.355095996Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:38.855063626Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:39.355170472Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:39.855296624Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:40.355481413Z [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
2022-09-07T00:45:40.855455283Z [WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API
2022-09-07T00:45:40.856166731Z .:5353
2022-09-07T00:45:40.856166731Z [INFO] plugin/reload: Running configuration SHA512 = 7c3db3182389e270fe3af971fef9e7157a34bde01eb750d58c2d4895a965ecb2e2d00df34594b6a819653088b6efa2590e748ce23f9e16cbb0f44d74a4fc3587
2022-09-07T00:45:40.856166731Z CoreDNS-1.9.2
2022-09-07T00:45:40.856166731Z linux/amd64, go1.18.4, 
2022-09-07T00:46:05.957503195Z [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://172.30.0.1:443/version": dial tcp 172.30.0.1:443: i/o timeout
2022-09-07T00:47:25.444876534Z [INFO] SIGTERM: Shutting down servers then terminating
2022-09-07T00:47:25.444984367Z [INFO] plugin/health: Going into lameduck mode for 20s
2022-09-07T00:47:35.959931289Z [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://172.30.0.1:443/version": dial tcp 172.30.0.1:443: i/o timeout

https://github.com/openshift/sdn/pull/488

Bug OCPBUGS-9986: MTU migration configuration is cleaned up prematurely while in progress

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-7445~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-7207~~. The following is the description of the original issue:
—
At some point in the mtu-migration development a configuration file was generated at /etc/cno/mtu-migration/config which was used as a flag to indicate to configure-ovs that a migration procedure was in progress. When that file was missing, it was assumed the migration procedure was over and configure-ovs did some cleaning on behalf of it.

But that changed and /etc/cno/mtu-migration/config is never set. That causes configure-ovs to remove mtu-migration information when the procedure is still in progress making it to use incorrect MTU values and either causing nodes to be tainted with "ovn.k8s.org/mtu-too-small" blocking the procedure itself or causing network disruption until the procedure is over.

However, this was not a problem for the CI job as it doesn't use the migration procedure as documented for the sake of saving limited time available to run CI jobs. The CI merges two steps of the procedure into one so that there is never a reboot while the procedure is in progress and hiding this issue.

This was probably not detected in QE as well for the same reason as CI.

https://github.com/openshift/machine-config-operator/pull/3607

Bug OCPBUGS-2222: ITP is not supported with DPU mode

View the Description View the linked PRs

Description of problem:

Dummy bug that is needed to track backport of https://github.com/ovn-org/ovn-kubernetes/pull/2975/commits/816e30a1fbb5beb8b20fe3e96906285762dd8eb6 which is already merged in 4.12

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/ovn-kubernetes/pull/1305

Bug OCPBUGS-2919: Update Prometheus Alerts

View the Description View the linked PRs

Our Prometheus alerts are inconsistent with both upstream and sometimes our own vendor folder. Let's do a clean update run before the next release is branched off.

https://github.com/openshift/cluster-etcd-operator/pull/957

Bug OCPBUGS-7510: [release-4.11] fail early on missing node status envs

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-7373~~. The following is the description of the original issue:
—
Originally reported by lance5890 in issue https://github.com/openshift/cluster-etcd-operator/issues/1000

Under some circumstances the static pod machinery fails to populate the node status in time to generate the correct env variables for ETCD_URL_HOST, ETCD_NAME etc. The pods that come up will fail to accept those variables.

This is particularly pronounced in SNO topologies, leading to installation failures.

The fix is to fail fast in the targetconfig/envvar controller to ensure the CEO goes degraded instead of silently failing on the rollout of an invalid static pod.

https://github.com/openshift/cluster-etcd-operator/pull/1006

Bug OCPBUGS-10382: cluster-machine-approver doesn't ignore case for CSR hostnames

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-676~~. The following is the description of the original issue:
—
the machine approver isn't recognizing hostnames that use capital letters as valid even though DNS is case-insensitive

an example of this is in ~~OHSS-14709~~:

I0822 19:04:51.587266       1 controller.go:114] Reconciling CSR: csr-vdtpv
I0822 19:04:51.600941       1 csr_check.go:156] csr-vdtpv: CSR does not appear to be client csr
I0822 19:04:51.603648       1 csr_check.go:542] retrieving serving cert from ip-100-66-119-117.ec2.internal (100.66.119.117:10250)
I0822 19:04:51.604003       1 csr_check.go:181] Failed to retrieve current serving cert: dial tcp 100.66.119.117:10250: connect: connection refused
I0822 19:04:51.604017       1 csr_check.go:201] Falling back to machine-api authorization for ip-100-66-119-117.ec2.internal
E0822 19:04:51.604024       1 csr_check.go:392] csr-vdtpv: DNS name 'ip-100-66-119-117.tech-ace-maint-prd.aws.delta.com' not in machine names: ip-100-66-119-117.ec2.internal ip-100-66-119-117.ec2.internal ip-100-66-119-117.tech-ACE-maint-prd.aws.delta.com
I0822 19:04:51.604033       1 csr_check.go:204] Could not use Machine for serving cert authorization: DNS name 'ip-100-66-119-117.tech-ace-maint-prd.aws.delta.com' not in machine names: ip-100-66-119-117.ec2.internal ip-100-66-119-117.ec2.internal ip-100-66-119-117.tech-ACE-maint-prd.aws.delta.com
I0822 19:04:51.606777       1 controller.go:199] csr-vdtpv: CSR not authorized

This can be worked around by manually approving the CSR

The relevant line in the machine approver appears to be here: https://github.com/openshift/cluster-machine-approver/blob/master/pkg/controller/csr_check.go#L378

https://github.com/openshift/cluster-machine-approver/pull/181

Bug OCPBUGS-3285: OCP 4.10.33 uses a weak 3DES cipher in the VMWare CSI Operator for communication and provides no method to disable it

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-2083~~. The following is the description of the original issue:
—
Description of problem:
Currently we are running VMWare CSI Operator in OpenShift 4.10.33. After running vulnerability scans, the operator was discovered to be running a known weak cipher 3DES. We are attempting to upgrade or modify the operator to customize the ciphers available. We were looking at performing a manual upgrade via Quay.io but can't seem to pull the image and was trying to steer away from performing a custom install from scratch. Looking for any suggestions into mitigated the weak cipher in the kube-rbac-proxy under VMware CSI Operator.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/vmware-vsphere-csi-driver-operator/pull/119

Bug OCPBUGS-697: Dockerfile.okd: include custom OperatorHub manifest

View the Description View the linked PRs

Since 4.11 OCP comes with OperatorHub definition which declares a capability
and enables all catalog sources. For OKD we want to enable just community-operators
as users may not have Red Hat pull secret set.
This commit would ensure that OKD version of marketplace operator gets
its own OperatorHub manifest with a custom set of operator catalogs enabled

https://github.com/operator-framework/operator-marketplace/pull/485

Bug OCPBUGS-1846: Facing issue while configuring egress IP pool in OCP cluster which uses STS

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-1629~~. The following is the description of the original issue:
—
Description of problem:

It is a disconnected cluster on AWS. There is an issue configuring Egress IP where the cluster uses STS. While looking into cloud-network-config-controller pod it is trying to connect to the global sts service "https://sts.amazonaws.com/" rather it should connect to the regional one "https://ec2.ap-southeast-1.amazonaws.com".

Version-Release number of selected component (if applicable):

How reproducible:

Always

Steps to Reproduce:

1. Create a disconected OCP cluster on AWS.

$ oc get netnamespace | grep egress
egress-ip-test                                     2689387    ["172.16.1.24"]

$ oc get hostsubnet
NAME                                              HOST                                              HOST IP        SUBNET          EGRESS CIDRS   EGRESS IPS
ip-172-16-1-151.ap-southeast-1.compute.internal   ip-172-16-1-151.ap-southeast-1.compute.internal   172.16.1.151   10.130.0.0/23                  
ip-172-16-1-53.ap-southeast-1.compute.internal    ip-172-16-1-53.ap-southeast-1.compute.internal    172.16.1.53    10.131.0.0/23                  ["172.16.1.24"]
ip-172-16-2-15.ap-southeast-1.compute.internal    ip-172-16-2-15.ap-southeast-1.compute.internal    172.16.2.15    10.128.0.0/23                  
ip-172-16-2-77.ap-southeast-1.compute.internal    ip-172-16-2-77.ap-southeast-1.compute.internal    172.16.2.77    10.128.2.0/23                  
ip-172-16-3-111.ap-southeast-1.compute.internal   ip-172-16-3-111.ap-southeast-1.compute.internal   172.16.3.111   10.129.0.0/23                  
ip-172-16-3-79.ap-southeast-1.compute.internal    ip-172-16-3-79.ap-southeast-1.compute.internal    172.16.3.79    10.129.2.0/23

$ oc logs sdn-controller-6m5kb -n openshift-sdn I0922 04:09:53.348615       1 vnids.go:105] Allocated netid 2689387 for namespace "egress-ip-test"
E0922 04:24:00.682018       1 egressip.go:254] Ignoring invalid HostSubnet ip-172-16-1-53.ap-southeast-1.compute.internal (host: "ip-172-16-1-53.ap-southeast-1.compute.internal", ip: "172.16.1.53", subnet: "10.131.0.0/23"): related node object "ip-172-16-1-53.ap-southeast-1.compute.internal" has an incomplete annotation "cloud.network.openshift.io/egress-ipconfig", CloudEgressIPConfig: <nil>

 $ oc logs cloud-network-config-controller-5c7556db9f-x78bs -n openshift-cloud-network-config-controller

E0922 04:26:59.468726       1 controller.go:165] error syncing 'ip-172-16-2-77.ap-southeast-1.compute.internal': error retrieving the private IP configuration for node: ip-172-16-2-77.ap-southeast-1.compute.internal, err: error: cannot list ec2 instance for node: ip-172-16-2-77.ap-southeast-1.compute.internal, err: WebIdentityErr: failed to retrieve credentials
caused by: RequestError: send request failed
caused by: Post "https://sts.amazonaws.com/": dial tcp 54.239.29.25:443: i/o timeout, requeuing in node workqueue

$ oc get Infrastructure -o yaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: Infrastructure
  metadata:
    creationTimestamp: "2022-09-22T03:28:15Z"
    generation: 1
    name: cluster
    resourceVersion: "598"
    uid: 994da301-2a96-43b7-b43b-4b7c18d4b716
  spec:
    cloudConfig:
      name: ""
    platformSpec:
      aws:
        serviceEndpoints:
        - name: sts
          url: https://sts.ap-southeast-1.amazonaws.com
        - name: ec2
          url: https://ec2.ap-southeast-1.amazonaws.com
        - name: elasticloadbalancing
          url: https://elasticloadbalancing.ap-southeast-1.amazonaws.com
      type: AWS
  status:
    apiServerInternalURI: https://api-int.openshiftyy.ocpaws.sadiqueonline.com:6443
    apiServerURL: https://api.openshiftyy.ocpaws.sadiqueonline.com:6443
    controlPlaneTopology: HighlyAvailable
    etcdDiscoveryDomain: ""
    infrastructureName: openshiftyy-wfrpf
    infrastructureTopology: HighlyAvailable
    platform: AWS
    platformStatus:
      aws:
        region: ap-southeast-1
        serviceEndpoints:
        - name: ec2
          url: https://ec2.ap-southeast-1.amazonaws.com
        - name: elasticloadbalancing
          url: https://elasticloadbalancing.ap-southeast-1.amazonaws.com
        - name: sts
          url: https://sts.ap-southeast-1.amazonaws.com
      type: AWS
kind: List
metadata:
  resourceVersion: ""

$ oc get secret aws-cloud-credentials -n openshift-machine-api -o json |jq -r .data.credentials |base64 -d 
[default]
sts_regional_endpoints = regional
role_arn = arn:aws:iam::015719942846:role/sputhenp-sts-yy-openshift-machine-api-aws-cloud-credentials
web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
 
[ec2-user@ip-172-17-1-229 ~]$ oc get secret cloud-credential-operator-iam-ro-creds -n openshift-cloud-credential-operator -o json |jq -r .data.credentials |base64 -d 
[default]
sts_regional_endpoints = regional
role_arn = arn:aws:iam::015719942846:role/sputhenp-sts-yy-openshift-cloud-credential-operator-cloud-creden
web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
 
[ec2-user@ip-172-17-1-229 ~]$ oc get secret installer-cloud-credentials -n openshift-image-registry -o json |jq -r .data.credentials |base64 -d 
[default]
sts_regional_endpoints = regional
role_arn = arn:aws:iam::015719942846:role/sputhenp-sts-yy-openshift-image-registry-installer-cloud-credent
web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
 
[ec2-user@ip-172-17-1-229 ~]$ oc get secret cloud-credentials -n openshift-ingress-operator -o json |jq -r .data.credentials |base64 -d 
[default]
sts_regional_endpoints = regional
role_arn = arn:aws:iam::015719942846:role/sputhenp-sts-yy-openshift-ingress-operator-cloud-credentials
web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
 
[ec2-user@ip-172-17-1-229 ~]$ oc get secret cloud-credentials -n openshift-cloud-network-config-controller -o json |jq -r .data.credentials |base64 -d 
[default]
sts_regional_endpoints = regional
role_arn = arn:aws:iam::015719942846:role/sputhenp-sts-yy-openshift-cloud-network-config-controller-cloud-
web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
 
[ec2-user@ip-172-17-1-229 ~]$ oc get secret ebs-cloud-credentials -n openshift-cluster-csi-drivers -o json |jq -r .data.credentials |base64 -d
[default]
sts_regional_endpoints = regional
role_arn = arn:aws:iam::015719942846:role/sputhenp-sts-yy-openshift-cluster-csi-drivers-ebs-cloud-credenti
web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token

Actual results:

Egress IP not configured properly and cloud-network-config-controller trying to connect to global STS service.

Expected results:

Egress IP should get configured and cloud-network-config-controller should connect to regional STS service instead of global.

Additional info:

https://github.com/openshift/cloud-network-config-controller/pull/63

Bug OCPBUGS-4294: Backport to 4.11 specify resources.requests for operator pod

View the Description View the linked PRs

Description of problem:

TO address: 'Static Pod is managed but errored" err="managed container xxx does not have Resource.Requests'

Version-Release number of selected component (if applicable):

4.11

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/cluster-kube-scheduler-operator/pull/449

Bug OCPBUGS-4908: [OCP 4.11] ironic container images have old packages

View the Description View the linked PRs

we need to make sure that the ironic containers use the latest available bugfix versions

https://github.com/openshift/ironic-image/pull/336

Bug OCPBUGS-5882: Upgradeability check is throttled too much and with unnecessary non-determinism

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-5879~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-5505~~. The following is the description of the original issue:
—

Description of problem:

The upgradeability check in CVO is throttled (essentially cached) for a nondeterministic period of time, same as the minimal sync period computed at runtime. The period can be up to 4 minutes, determined at CVO start time as 2minutes * (0..1 + 1). We agreed with Trevor that such throttling is unnecessarily aggressive (the check is not that expensive). It also causes CI flakes, because the matching test only has 3 minutes timeout. Additionally, the non-determinism and longer throttling results makes UX worse by actions done in the cluster may have their observable effect delayed.

Version-Release number of selected component (if applicable):

discovered in 4.10 -> 4.11 upgrade jobs

How reproducible:

The test seems to flake ~10% of 4.10->4.11 Azure jobs (sippy). There does not seem to be that much impact on non-Azure jobs though which is a bit weird.

Steps to Reproduce:

Inspect the CVO log ~~and E2E logs from failing jobs~~ with the provided [^check-cvo.py] helper:

$ ./check-cvo.py cvo.log && echo PASS || echo FAIL

Preferably, inspect CVO logs of clusters that just underwent an upgrade (upgrades makes the original problematic behavior more likely to surface)

Actual results:

$ ./check-cvo.py openshift-cluster-version_cluster-version-operator-5b6966c474-g4kwk_cluster-version-operator.log && echo PASS || echo FAIL
FAIL: Cache hit at 11:59:55.332339 0:03:13.665006 after check at 11:56:41.667333
FAIL: Cache hit at 12:06:22.663215 0:03:13.664964 after check at 12:03:08.998251
FAIL: Cache hit at 12:12:49.997119 0:03:13.665598 after check at 12:09:36.331521
FAIL: Cache hit at 12:19:17.328510 0:03:13.664906 after check at 12:16:03.663604
FAIL: Cache hit at 12:25:44.662290 0:03:13.666759 after check at 12:22:30.995531
Upgradeability checks:           5
Upgradeability check cache hits: 12
FAIL

Note that the bug is probabilistic, so not all unfixed clusters will exhibit the behavior. My guess of the incidence rate is about 30-40%.

Expected result

$ ./check-cvo.py openshift-cluster-version_cluster-version-operator-7b8f85d455-mk9fs_cluster-version-operator.log && echo PASS || echo FAIL
Upgradeability checks:           12
Upgradeability check cache hits: 11
PASS

The actual numbers are not relevant (unless the upgradeabilily check count is zero, which means the test is not conclusive, the script warns about that), lack of failure is.

Additional info:

$ curl --silent https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.11-upgrade-from-stable-4.10-e2e-azure-upgrade/1607602927633960960/artifacts/e2e-azure-upgrade/gather-extra/artifacts/pods/openshift-cluster-version_cluster-version-operator-7b7d4b5bbd-zjqdt_cluster-version-operator.log | grep upgradeable.go
...
I1227 06:50:59.023190       1 upgradeable.go:122] Cluster current version=4.10.46
I1227 06:50:59.042735       1 upgradeable.go:42] Upgradeable conditions were recently checked, will try later.
I1227 06:51:14.024345       1 upgradeable.go:42] Upgradeable conditions were recently checked, will try later.
I1227 06:53:23.080768       1 upgradeable.go:42] Upgradeable conditions were recently checked, will try later.
I1227 06:56:59.366010       1 upgradeable.go:122] Cluster current version=4.11.0-0.ci-2022-12-26-193640

$ curl --silent https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.11-upgrade-from-stable-4.10-e2e-azure-upgrade/1607602927633960960/artifacts/e2e-azure-upgrade/openshift-e2e-test/artifacts/e2e.log | grep 'Kubernetes 1.25 and therefore OpenShift 4.12'
Dec 27 06:51:15.319: INFO: Waiting for Upgradeable to be AdminAckRequired for "Kubernetes 1.25 and therefore OpenShift 4.12 remove several APIs which require admin consideration. Please see the knowledge article https://access.redhat.com/articles/6955381 for details and instructions." ...
Dec 27 06:54:15.413: FAIL: Error while waiting for Upgradeable to complain about AdminAckRequired with message "Kubernetes 1.25 and therefore OpenShift 4.12 remove several APIs which require admin consideration. Please see the knowledge article https://access.redhat.com/articles/6955381 for details and instructions.": timed out waiting for the condition

The test passes. Also, the "Upgradeable conditions were recently checked, will try later." messages in CVO logs should never occur after a deterministic, short amount of time (I propose 1 minute) after upgradeability was checked.

I tested the throttling period in https://github.com/openshift/cluster-version-operator/pull/880. With the period of 15m, the test passrate was 4 of 9. Wiht the period of 1m, the test did not fail at all.

Some context in Slack thread

https://github.com/openshift/cluster-version-operator/pull/885

Bug OCPBUGS-3614: [4.11] [perf/scale] libovsdb builds transaction logs but throws them away

View the Description View the linked PRs

libovsdb builds transaction log messages for every transaction and then throws them away if the log level is not 4 or above. This wastes a bunch of CPU at scale and increases pod ready latency.

https://github.com/openshift/ovn-kubernetes/pull/1374

Bug OCPBUGS-4233: Updating ose-cloud-network-config-controller images to be consistent with ART

View the Description View the linked PRs

Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/cloud-network-config-controller/pull/81

Bug OCPBUGS-4838: [4.11] Pod LSP missing from PortGroup

View the Description View the linked PRs

Description of problem:

For some reason, the LSP of a pod is not properly added to the port group where the ACL of a NetworkPolicy is applied. This results on the networkpolicy not being applied to the pod and communication not possible.

Version-Release number of selected component (if applicable):

4.10

How reproducible:

Always with a concrete pod at customer environment.

Steps to Reproduce:

(not known exactly yet)

Actual results:

LSP not in port group. ACL not applied. Netpol not in effect.

Expected results:

LSP in port group. ACL applied. Netpol in effect.

Additional info:

Details in private comments, as they involve sensitive data.

Deleting the pod does nothing, but it is possible that this has something to do with the pod being recreated with the same name (although the LSPs UUIDs are different in each incarnation).

https://github.com/openshift/ovn-kubernetes/pull/1447

Bug OCPBUGS-5145: [4.11] virtual media provisioning fails when iLO Ironic driver is used

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-5100~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-5068~~. The following is the description of the original issue:
—
Description of problem:

virtual media provisioning fails when iLO Ironic driver is used

Version-Release number of selected component (if applicable):

4.13

How reproducible:

Always

Steps to Reproduce:

1. attempt virtual media provisioning on a node configured with ilo-virtualmedia:// drivers
2.
3.

Actual results:

Provisioning fails with "An auth plugin is required to determine endpoint URL" error

Expected results:

Provisioning succeeds

Additional info:

Relevant log snippet:

3742 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector [None req-e58ac1f2-fac6-4d28-be9e-983fa900a19b - - - - - -] Unable to start managed inspection for node e4445d43-3458-4cee-9cbe-6da1de75      78cd: An auth plugin is required to determine endpoint URL: keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin: An auth plugin is required to determine endpoint URL
 3743 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector Traceback (most recent call last):
 3744 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.9/site-packages/ironic/drivers/modules/inspector.py", line 210, in _start_managed_inspection
 3745 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector     task.driver.boot.prepare_ramdisk(task, ramdisk_params=params)
 3746 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.9/site-packages/ironic_lib/metrics.py", line 59, in wrapped
 3747 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector     result = f(*args, **kwargs)
 3748 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.9/site-packages/ironic/drivers/modules/ilo/boot.py", line 408, in prepare_ramdisk
 3749 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector     iso = image_utils.prepare_deploy_iso(task, ramdisk_params,
 3750 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.9/site-packages/ironic/drivers/modules/image_utils.py", line 624, in prepare_deploy_iso
 3751 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector     return prepare_iso_image(inject_files=inject_files)
 3752 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.9/site-packages/ironic/drivers/modules/image_utils.py", line 537, in _prepare_iso_image
 3753 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector     image_url = img_handler.publish_image(
 3754 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.9/site-packages/ironic/drivers/modules/image_utils.py", line 193, in publish_image
 3755 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector     swift_api = swift.SwiftAPI()
 3756 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.9/site-packages/ironic/common/swift.py", line 66, in __init__
 3757 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector     endpoint = keystone.get_endpoint('swift', session=session)

https://github.com/openshift/ironic-image/pull/342

Bug OCPBUGS-6176: Tracker: Configure ignored namespaces into multus-admission-controller (4.11)

View the Description View the linked PRs

Description of problem: Issue described in following issue: https://github.com/openshift/multus-admission-controller/issues/40

Fixed in: https://github.com/openshift/cluster-network-operator/pull/1515

Version-Release number of selected component (if applicable): OCP 4.10

Official Red Hat tracker. Issue has been merged already.

https://github.com/openshift/multus-admission-controller/pull/56

Bug OCPBUGS-7330: When setting allowedRegistries urls the openshift-samples operator is degraded

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-855~~. The following is the description of the original issue:
—
Description of problem:

When setting the allowedregistries like the example below, the openshift-samples operator is degraded:

oc get image.config.openshift.io/cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Image
metadata:
  annotations:
    release.openshift.io/create-only: "true"
  creationTimestamp: "2020-12-16T15:48:20Z"
  generation: 2
  name: cluster
  resourceVersion: "422284920"
  uid: d406d5a0-c452-4a84-b6b3-763abb51d7a5
spec:
  additionalTrustedCA:
    name: registry-ca
  allowedRegistriesForImport:
  - domainName: quay.io
    insecure: false
  - domainName: registry.redhat.io
    insecure: false
  - domainName: registry.access.redhat.com
    insecure: false
  - domainName: registry.redhat.io/redhat/redhat-operator-index
    insecure: true
  - domainName: registry.redhat.io/redhat/redhat-marketplace-index
    insecure: true
  - domainName: registry.redhat.io/redhat/certified-operator-index
    insecure: true
  - domainName: registry.redhat.io/redhat/community-operator-index
    insecure: true
  registrySources:
    allowedRegistries:
    - quay.io
    - registry.redhat.io
    - registry.rijksapps.nl
    - registry.access.redhat.com
    - registry.redhat.io/redhat/redhat-operator-index
    - registry.redhat.io/redhat/redhat-marketplace-index
    - registry.redhat.io/redhat/certified-operator-index
    - registry.redhat.io/redhat/community-operator-index


oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.10.21   True        False         False      5d13h   
baremetal                                  4.10.21   True        False         False      450d    
cloud-controller-manager                   4.10.21   True        False         False      94d     
cloud-credential                           4.10.21   True        False         False      624d    
cluster-autoscaler                         4.10.21   True        False         False      624d    
config-operator                            4.10.21   True        False         False      624d    
console                                    4.10.21   True        False         False      42d     
csi-snapshot-controller                    4.10.21   True        False         False      31d     
dns                                        4.10.21   True        False         False      217d    
etcd                                       4.10.21   True        False         False      624d    
image-registry                             4.10.21   True        False         False      94d     
ingress                                    4.10.21   True        False         False      94d     
insights                                   4.10.21   True        False         False      104s    
kube-apiserver                             4.10.21   True        False         False      624d    
kube-controller-manager                    4.10.21   True        False         False      624d    
kube-scheduler                             4.10.21   True        False         False      624d    
kube-storage-version-migrator              4.10.21   True        False         False      31d     
machine-api                                4.10.21   True        False         False      624d    
machine-approver                           4.10.21   True        False         False      624d    
machine-config                             4.10.21   True        False         False      17d     
marketplace                                4.10.21   True        False         False      258d    
monitoring                                 4.10.21   True        False         False      161d    
network                                    4.10.21   True        False         False      624d    
node-tuning                                4.10.21   True        False         False      31d     
openshift-apiserver                        4.10.21   True        False         False      42d     
openshift-controller-manager               4.10.21   True        False         False      22d     
openshift-samples                          4.10.21   True        True          True       31d     Samples installation in error at 4.10.21: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "}
operator-lifecycle-manager                 4.10.21   True        False         False      624d    
operator-lifecycle-manager-catalog         4.10.21   True        False         False      624d    
operator-lifecycle-manager-packageserver   4.10.21   True        False         False      31d     
service-ca                                 4.10.21   True        False         False      624d    
storage                                    4.10.21   True        False         False      113d  


After applying the fix as described here(  https://access.redhat.com/solutions/6547281 ) it is resolved:
oc patch configs.samples.operator.openshift.io cluster --type merge --patch '{"spec": {"samplesRegistry": "registry.redhat.io"}}'

But according the the BZ this should be fixed in 4.10.3 https://bugzilla.redhat.com/show_bug.cgi?id=2027745 but the issue is still occur in our 4.10.21 cluster:

oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.21   True        False         31d     Error while reconciling 4.10.21: the cluster operator openshift-samples is degraded

Version-Release number of selected component (if applicable):

How reproducible:

100%

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/cluster-samples-operator/pull/490

Bug OCPBUGS-772: [Original Bug ID: 2095852] Unable to create Network Policies: error: unexpectedly found multiple equivalent ACLs (arp v/s arp||nd) (ns_netpol1 v/s ns_netpol2)

View the Description View the linked PRs

Description of problem:

Intended to backport the corresponding https://bugzilla.redhat.com/show_bug.cgi?id=2095852 which has been fixed already for this version.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

Bug OCPBUGS-4860: [4.11] Network Policy executes duplicate transactions for every pod update

View the Description View the linked PRs

Description of problem:

With every pod update we are executing a mutate operation to add the pod port to the port group or add the pod IP to an address set. This functionally doesn't hurt, since mutate will not add duplicate values to the same set. However, this is bad for performance. For example, with a 730 network policies affecting a pod, and issuing 7 pod updates would result in over 5k transactions.

https://github.com/openshift/ovn-kubernetes/pull/1447

Bug OCPBUGS-5518: Upgrade fails with message Cluster operator console is not available

View the Description View the linked PRs

This bug is a backport clone of [Bugzilla Bug 2089950](https://bugzilla.redhat.com/show_bug.cgi?id=2089950). The following is the description of the original bug:
—
Description of problem: Some upgrades failed during scale testing with messages indicating the console operator is not available. In total 5 out of 2200 clusters failed with this pattern.

These clusters are all configured with the Console operator disabled in order to reduce overall OCP cpu use in the Telecom environment. The following CR is applied:
apiVersion: operator.openshift.io/v1
kind: Console
metadata:
annotations:
include.release.openshift.io/ibm-cloud-managed: "false"
include.release.openshift.io/self-managed-high-availability: "false"
include.release.openshift.io/single-node-developer: "false"
release.openshift.io/create-only: "true"
ran.openshift.io/ztp-deploy-wave: "10"
name: cluster
spec:
logLevel: Normal
managementState: Removed
operatorLogLevel: Normal

From one cluster (sno01175) the ClusterVersion conditions show:

oc get clusterversion version -o jsonpath=' {.status.conditions}
' | jq
[

{ "lastTransitionTime": "2022-05-19T01:44:13Z", "message": "Done applying 4.9.26", "status": "True", "type": "Available" }
,

{ "lastTransitionTime": "2022-05-24T14:57:50Z", "message": "Cluster operator console is degraded", "reason": "ClusterOperatorDegraded", "status": "True", "type": "Failing" }
,

{ "lastTransitionTime": "2022-05-24T13:49:43Z", "message": "Unable to apply 4.10.13: wait has exceeded 40 minutes for these operators: console", "reason": "ClusterOperatorDegraded", "status": "True", "type": "Progressing" }
,

{ "lastTransitionTime": "2022-05-21T02:07:06Z", "status": "True", "type": "RetrievedUpdates" }
,

{ "lastTransitionTime": "2022-05-24T13:53:05Z", "message": "Payload loaded version=\"4.10.13\" image=\"quay.io/openshift-release-dev/ocp-release@sha256:4f516616baed3cf84585e753359f7ef2153ae139c2e80e0191902fbd073c4143\"", "reason": "PayloadLoaded", "status": "True", "type": "ReleaseAccepted" }
,
{ "lastTransitionTime": "2022-05-24T13:57:05Z", "message": "Cluster operator kube-apiserver should not be upgraded between minor versions: KubeletMinorVersionUpgradeable: Kubelet minor version (1.22.5+5c84e52) on node sno01175 will not be supported in the next OpenShift minor version upgrade.", "reason": "KubeletMinorVersion_KubeletMinorVersionUnsupportedNextUpgrade", "status": "False", "type": "Upgradeable" }
]

Another cluster (sno01959) has very similar conditions with slight variation in the Failing and Progressing messages:

{ "lastTransitionTime": "2022-05-24T14:32:42Z", "message": "Cluster operator console is not available", "reason": "ClusterOperatorNotAvailable", "status": "True", "type": "Failing" }

,

{ "lastTransitionTime": "2022-05-24T13:52:04Z", "message": "Unable to apply 4.10.13: the cluster operator console has not yet successfully rolled out", "reason": "ClusterOperatorNotAvailable", "status": "True", "type": "Progressing" }

,

Version-Release number of selected component (if applicable): 4.9.26 upgrade to 4.10.13

How reproducible: 5 out of 2200

Steps to Reproduce:
1. Disable console with managementState: Removed
2. Starting OCP version 4.9.26
3. Initiate upgrade to 4.10.13 via ClusterVersion CR

Actual results: Cluster upgrade is stuck (no longer progressing) for 5+ hours

Expected results: Cluster upgrade completes

Additional info:

https://github.com/openshift/console-operator/pull/715

Bug OCPBUGS-1012: Spike in pod-latency graph observed due to ovnkube-master restarts

View the Description View the linked PRs

Description of problem:

When running node-density (245 pods/node) on a 120 node cluster, we see that there is a huge spike (~22s) in Avg pod-latency. When the spike occurs we see all the ovnkube-master pods go through a restart.

The restart happens because of (ovnkube-master pods)

2022-08-10T04:04:44.494945179Z panic: reflect: call of reflect.Value.Len on ptr Value

Version-Release number of selected component (if applicable):

4.12.0-0.nightly-2022-08-09-114621

How reproducible:

Steps to Reproduce:
1. Run node-density on a 120 node cluster

Actual results:

Spike observed in pod-latency graph ~22s

Expected results:

Steady pod-latency graph ~4s

Additional info:

https://github.com/openshift/ovn-kubernetes/pull/1266

Bug OCPBUGS-1556: Uninstalled operator can't be reinstalled if it included a conversion webhook

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-78~~. The following is the description of the original issue:
—
Copied from an upstream issue: https://github.com/operator-framework/operator-lifecycle-manager/issues/2830

What did you do?

When attempting to reinstall an operator that uses conversion webhooks by

Deleting the operator subscription and any CSVs associated with it
Recreating the operator subscription

The resulting InstallPlan enters a failed state with message similar to

error validating existing CRs against new CRD's schema for "devworkspaces.workspace.devfile.io": error listing resources in GroupVersionResource schema.GroupVersionResource{Group:"workspace.devfile.io", Version:"v1alpha1", Resource:"devworkspaces"}: conversion webhook for workspace.devfile.io/v1alpha2, Kind=DevWorkspace failed: Post "https://devworkspace-controller-manager-service.test-namespace.svc:443/convert?timeout=30s": service "devworkspace-controller-manager-service" not found

When the original CSVs are deleted, the operator's main deployment and service are removed, but CRDs are left in-cluster. However, since the service/CA bundle/deployment that serve the conversion webhook are removed, conversion webhooks are broken at that point. Eventually this impacts garbage collection on the cluster as well.

This can be reproduced by installing the DevWorkspace Operator from the Red Hat catalog. (I can provide yamls/upstream images that reproduce as well, if that's helpful). It may be necessary to create a DevWorkspace in the cluster before deletion, e.g. by oc apply -f https://raw.githubusercontent.com/devfile/devworkspace-operator/main/samples/plain.yaml

What did you expect to see?
Operator is able to be reinstalled without removing CRDs and all instances.

What did you see instead? Under which circumstances?
It's necessary to completely remove the operator including CRDs. For our operator (DevWorkspace), this also makes uninstall especially complicated as finalizers are used (so CRDs cannot be deleted if the controller is removed, and the controller cannot be restored by reinstalling)

Environment

operator-lifecycle-manager version: 4.10.24

Kubernetes version information: Kubernetes Version: v1.23.5+012e945 (OpenShift 4.10.24)

Kubernetes cluster kind: OpenShift

https://github.com/openshift/operator-framework-olm/pull/388

Bug OCPBUGS-1749: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/aws-ebs-csi-driver/pull/209

Bug OCPBUGS-526: Prerelease report bug link should be updated to JIRA instead of Bugzilla

View the Description View the linked PRs

We are in the process of moving our bug tracking to JIRA. We should update the report bug link in the help menu to use JIRA instead of Bugzilla for new bugs. Opening as a medium severity bug since this only impacts prerelease OpenShift versions. For release versions, we have users open customer cases.

https://github.com/openshift/console/pull/11965

Bug OCPBUGS-6879: Branch name should sanitised to match actual github branch name in repository plr list

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-6755~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-3316~~. The following is the description of the original issue:
—
Description of problem:

Branch name in repository pipelineruns list view should match the actual github branch name.

Version-Release number of selected component (if applicable):

4.11.z

How reproducible:

alwaus

Steps to Reproduce:

1. Create a repository
2. Trigger the pipelineruns by push or pull request event on the github

Actual results:

Branch name contains "refs-heads-" prefix in front of the actual branch name eg: "refs-heads-cicd-demo" (cicd-demo is the branch name)

Expected results:

Branch name should be the acutal github branch name. just `cicd-demo`should be shown in the branch column.

Additional info:
Ref: https://coreos.slack.com/archives/CHG0KRB7G/p1667564311865459

https://github.com/openshift/console/pull/12509

Bug OCPBUGS-10594: aws: mismatch between RHCOS and AWS SDK regions

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-10497~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-10213~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-8468~~. The following is the description of the original issue:
—
Description of problem:

RHCOS is being published to new AWS regions (https://github.com/openshift/installer/pull/6861) but aws-sdk-go need to be bumped to recognize those regions

Version-Release number of selected component (if applicable):

master/4.14

How reproducible:

always

Steps to Reproduce:

1. openshift-install create install-config
2. Try to select ap-south-2 as a region
3.

Actual results:

New regions are not found. New regions are: ap-south-2, ap-southeast-4, eu-central-2, eu-south-2, me-central-1.

Expected results:

Installer supports and displays the new regions in the Survey

Additional info:

See https://github.com/openshift/installer/blob/master/pkg/asset/installconfig/aws/regions.go#L13-L23

https://github.com/openshift/installer/pull/7000

Bug OCPBUGS-3507: [4.11.z] Incorrect network configuration in worker node with two interfaces

View the Description View the linked PRs

Description of problem:

Upgrade OCP 4.11 --> 4.12 fails with one 'NotReady,SchedulingDisabled' node and MachineConfigDaemonFailed.

Version-Release number of selected component (if applicable):

Upgrade from OCP 4.11.0-0.nightly-2022-09-19-214532 on top of OSP RHOS-16.2-RHEL-8-20220804.n.1 to 4.12.0-0.nightly-2022-09-20-040107.

Network Type: OVNKubernetes

How reproducible:

Twice out of two attempts.

Steps to Reproduce:

1. Install OCP 4.11.0-0.nightly-2022-09-19-214532 (IPI) on top of OSP RHOS-16.2-RHEL-8-20220804.n.1.
   The cluster is up and running with three workers:
   $ oc get clusterversion
   NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
   version   4.11.0-0.nightly-2022-09-19-214532   True        False         51m     Cluster version is 4.11.0-0.nightly-2022-09-19-214532

2. Run the OC command to upgrade to 4.12.0-0.nightly-2022-09-20-040107:
$ oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release:4.12.0-0.nightly-2022-09-20-040107 --allow-explicit-upgrade --force=true
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Requesting update to release image registry.ci.openshift.org/ocp/release:4.12.0-0.nightly-2022-09-20-040107 

3. The upgrade is not succeeds: [0]
$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-09-19-214532   True        True          17h     Unable to apply 4.12.0-0.nightly-2022-09-20-040107: wait has exceeded 40 minutes for these operators: network

One node degrided to 'NotReady,SchedulingDisabled' status:
$ oc get nodes
NAME                          STATUS                        ROLES    AGE   VERSION
ostest-9vllk-master-0         Ready                         master   19h   v1.24.0+07c9eb7
ostest-9vllk-master-1         Ready                         master   19h   v1.24.0+07c9eb7
ostest-9vllk-master-2         Ready                         master   19h   v1.24.0+07c9eb7
ostest-9vllk-worker-0-4x4pt   NotReady,SchedulingDisabled   worker   18h   v1.24.0+3882f8f
ostest-9vllk-worker-0-h6kcs   Ready                         worker   18h   v1.24.0+3882f8f
ostest-9vllk-worker-0-xhz9b   Ready                         worker   18h   v1.24.0+3882f8f

$ oc get pods -A | grep -v -e Completed -e Running
NAMESPACE                                          NAME                                                         READY   STATUS      RESTARTS       AGE
openshift-openstack-infra                          coredns-ostest-9vllk-worker-0-4x4pt                          0/2     Init:0/1    0              18h
 
$ oc get events
LAST SEEN   TYPE      REASON                                        OBJECT            MESSAGE
7m15s       Warning   OperatorDegraded: MachineConfigDaemonFailed   /machine-config   Unable to apply 4.12.0-0.nightly-2022-09-20-040107: failed to apply machine config daemon manifests: error during waitForDaemonsetRollout: [timed out waiting for the condition, daemonset machine-config-daemon is not ready. status: (desired: 6, updated: 6, ready: 5, unavailable: 1)]
7m15s       Warning   MachineConfigDaemonFailed                     /machine-config   Cluster not available for [{operator 4.11.0-0.nightly-2022-09-19-214532}]: failed to apply machine config daemon manifests: error during waitForDaemonsetRollout: [timed out waiting for the condition, daemonset machine-config-daemon is not ready. status: (desired: 6, updated: 6, ready: 5, unavailable: 1)]

$ oc get co
NAME                                       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.12.0-0.nightly-2022-09-20-040107   True        False         False      18h    
baremetal                                  4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
cloud-controller-manager                   4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
cloud-credential                           4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
cluster-autoscaler                         4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
config-operator                            4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
console                                    4.12.0-0.nightly-2022-09-20-040107   True        False         False      18h    
control-plane-machine-set                  4.12.0-0.nightly-2022-09-20-040107   True        False         False      17h    
csi-snapshot-controller                    4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
dns                                        4.12.0-0.nightly-2022-09-20-040107   True        True          False      19h     DNS "default" reports Progressing=True: "Have 5 available node-resolver pods, want 6."
etcd                                       4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
image-registry                             4.12.0-0.nightly-2022-09-20-040107   True        True          False      18h     Progressing: The registry is ready...
ingress                                    4.12.0-0.nightly-2022-09-20-040107   True        False         False      18h    
insights                                   4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
kube-apiserver                             4.12.0-0.nightly-2022-09-20-040107   True        True          False      18h     NodeInstallerProgressing: 1 nodes are at revision 11; 2 nodes are at revision 13
kube-controller-manager                    4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
kube-scheduler                             4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
kube-storage-version-migrator              4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
machine-api                                4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
machine-approver                           4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
machine-config                             4.11.0-0.nightly-2022-09-19-214532   False       True          True       16h     Cluster not available for [{operator 4.11.0-0.nightly-2022-09-19-214532}]: failed to apply machine config daemon manifests: error during waitForDaemonsetRollout: [timed out waiting for the condition, daemonset machine-config-daemon is not ready. status: (desired: 6, updated: 6, ready: 5, unavailable: 1)]
marketplace                                4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
monitoring                                 4.12.0-0.nightly-2022-09-20-040107   True        False         False      18h    
network                                    4.12.0-0.nightly-2022-09-20-040107   True        True          True       19h     DaemonSet "/openshift-ovn-kubernetes/ovnkube-node" rollout is not making progress - last change 2022-09-20T14:16:13Z...
node-tuning                                4.12.0-0.nightly-2022-09-20-040107   True        False         False      17h    
openshift-apiserver                        4.12.0-0.nightly-2022-09-20-040107   True        False         False      18h    
openshift-controller-manager               4.12.0-0.nightly-2022-09-20-040107   True        False         False      17h    
openshift-samples                          4.12.0-0.nightly-2022-09-20-040107   True        False         False      17h    
operator-lifecycle-manager                 4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
operator-lifecycle-manager-catalog         4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
operator-lifecycle-manager-packageserver   4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
service-ca                                 4.12.0-0.nightly-2022-09-20-040107   True        False         False      19h    
storage                                    4.12.0-0.nightly-2022-09-20-040107   True        True          False      19h     ManilaCSIDriverOperatorCRProgressing: ManilaDriverNodeServiceControllerProgressing: Waiting for DaemonSet to deploy node pods...

[0] http://pastebin.test.redhat.com/1074531

Actual results:

OCP 4.11 --> 4.12 upgrade fails.

Expected results:

OCP 4.11 --> 4.12 upgrade success.

Additional info:

Attached logs of the NotReady node - [^journalctl_ostest-9vllk-worker-0-4x4pt.log.tar.gz]

https://github.com/openshift/machine-config-operator/pull/3422

Bug OCPBUGS-4301: Backport to 4.11 Specify resources.requests for operator pod

View the Description View the linked PRs

Description of problem:

To address: 'Static Pod is managed but errored" err="managed container xxx does not have Resource.Requests'

Version-Release number of selected component (if applicable):

4.11

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

Already merged in https://github.com/openshift/cluster-kube-apiserver-operator/pull/1398

https://github.com/openshift/cluster-kube-apiserver-operator/pull/1412

Bug OCPBUGS-685: panic in etcdcli

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-675~~. The following is the description of the original issue:
—
Description of problem:

A cluster hit a panic in etcd operator in bootstrap:
I0829 14:46:02.736582 1 controller_manager.go:54] StaticPodStateController controller terminated
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1e940ab]

goroutine 2701 [running]:
github.com/openshift/cluster-etcd-operator/pkg/etcdcli.checkSingleMemberHealth({0x29374c0, 0xc00217d920}, 0xc0021fb110)
github.com/openshift/cluster-etcd-operator/pkg/etcdcli/health.go:135 +0x34b
github.com/openshift/cluster-etcd-operator/pkg/etcdcli.getMemberHealth.func1()
github.com/openshift/cluster-etcd-operator/pkg/etcdcli/health.go:58 +0x7f
created by github.com/openshift/cluster-etcd-operator/pkg/etcdcli.getMemberHealth
github.com/openshift/cluster-etcd-operator/pkg/etcdcli/health.go:54 +0x2ac
Version-Release number of selected component (if applicable):

How reproducible:

Pulled up a 4.12 cluster and hit panic during bootstrap

Steps to Reproduce:

1.
2.
3.

Actual results:

panic as above

Expected results:

no panic

Additional info:

https://github.com/openshift/cluster-etcd-operator/pull/924

Bug OCPBUGS-9967: [4.12] Remove ETCD liviness probe.

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-7830~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-7729~~. The following is the description of the original issue:
—
Description of problem:

Etcd's liveliness probe should be removed.

Version-Release number of selected component (if applicable):

4.11

Additional info:

When the Master Hosts hit CPU load this can cause a cascading restart loop for etcd and kube-api due to the etcd liveliness probes failing. Due to this loop load on the masters stays high because the api and controllers restarting over and over again..  

There is no reason for etcd to have a liveliness probe, we removed this probe in 3.11 due issues like this.

https://github.com/openshift/cluster-etcd-operator/pull/1025

Bug OCPBUGS-2451: e2e tests: Installs Red Hat Integration - 3scale operator test is failing due to change of Operator name

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-2181~~. The following is the description of the original issue:
—
Description of problem:

E2E test Installs Red Hat Integration - 3scale operator test is failing due to change of Operator name

CI Search: https://search.ci.openshift.org/?search=Installs+Red+Hat+Integration+-+3scale+operator+in+test+namespace+and+creates+3scale+Backend+Schema+operand+instance&maxAge=24h&context=1&type=bug%2Bissue%2Bjunit&name=pull-ci-openshift-console-master-e2e-gcp-console&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

https://github.com/openshift/console/pull/12175

Bug OCPBUGS-306: Cluster-version operator ClusterOperator checks are unecessarily slow on update

View the Description View the linked PRs

We're seeing a slight uptick in how long upgrades are taking[1][2]. We are not 100% sure of the cause, but it looks like it started with 4.11 rc.7. There's no obvious culprits in the diff[3].

Looking at some of the jobs, we are seeing the gaps between kube-scheduler being updated and then machine-api appear to take longer. Example job run[4] showing 10+ minutes waiting for it.

TRT had a debugging session, and we have two suggestions:

Adding logging around when CVO sees an operator version changed

Instead of a fixed polling interval at 5 minutes (which is what we think CVO is doing), would it be possible to trigger on the CO to know when to look again? We think there could be some substantial savings on upgrade time by doing this.

[1] https://search.ci.openshift.org/graph/metrics?metric=job%3Aduration%3Atotal%3Aseconds&job=periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade&job=periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-aws-sdn-upgrade&job=periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-azure-upgrade&job=periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-gcp-ovn-upgrade&job=periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-gcp-sdn-upgrade
[2] https://sippy.dptools.openshift.org/sippy-ng/tests/4.12/analysis?test=Cluster%20upgrade.%5Bsig-cluster-lifecycle%5D%20cluster%20upgrade%20should%20complete%20in%2075.00%20minutes
[3] https://amd64.ocp.releases.ci.openshift.org/releasestream/4-stable/release/4.11.0-rc.7
[4] https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.12-e2e-azure-sdn-upgrade/1556865989923049472

https://github.com/openshift/cluster-version-operator/pull/826

Bug OCPBUGS-7858: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/ovn-kubernetes/pull/1543

Bug OCPBUGS-528: [4.11] VirtualMediaViaExternalNetwork is broken with virtual media TLS

View the Description View the linked PRs

The relevant code in ironic-image was not updated to support TLS, so it still uses the old port and explicit http://

https://github.com/openshift/ironic-image/pull/295

Bug OCPBUGS-721: Show Git icon in repository link in details page should be based on the git provider

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-451~~. The following is the description of the original issue:
—
Description of problem:

Git icon shown in the repository details page should be based on the git provider.

Version-Release number of selected component (if applicable):
4.11

How reproducible:
Always

Steps to Reproduce:
1. Create a Repository with gitlab repo url
2. Navigate to the detail page.

Actual results:

github icon is displayed for the gitlab url.

Expected results:

gitlab icon should be displayed for the gitlab url.

Additional info:

use `GitLabIcon` and `BitBucketIcon` from patternfly react-icons.

https://github.com/openshift/console/pull/12005

Bug OCPBUGS-1890: noAllowedAddressPairs set to true causes all the ports to be added without the allowed address pairs

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-1765~~. The following is the description of the original issue:
—
Description of problem:

If a customer creates a machine with a networks section like this

networks:
- filter: {}
noAllowedAddressPairs: false
subnets:
- filter: {}
uuid: primary-subnet-uuid
- filter: {}
noAllowedAddressPairs: true
subnets:
- filter: {}
uuid: other-subnet-uuid
primarySubnet: primary-subnet-uuid

Then all the ports are created without the allowed address pairs.

Doing some research in the source code, I have found that:
- For each entry on the networks: section, networks are filtered as per its filter: section[1]
- Then, if the subnets: section of the network entry is not empty, for each of the network IDs found above[2], 2 things are done that are relevant for this situatoin:
- The net ID is saved on a netsWithoutAllowedAddressPairs[3]. That map is later checked while creating any port[4].
- For each subnet entry that matches the network ID, a port is created[5].

So, the problematic behavior happens due to the following:

- Both entries in the networks array have empty filters. This means that both entries selected all the neutron networks.
- This configuration results in one port per subnet as expected because, in the later traversal of the subnets array of each entry[5], it is filtering by subnet and creating a single port as expected.
- However, the entry with "noAllowedAddressPairs: true" is selecting all the neutron networks, so it adds all of them to the netsWithoutAllowedAddressPairs map[3], regardless of the subnets filtering.
- As all the networks are in noAllowedAddressPairs: true array, all the ports created for the VM have their allowed address pairs removed[4].

Why do we consider this behavior undesired?

I understand that, if we create a port for a network that has no allowed pairs, we create all the other ports in the same networks without the pairs. However, it is surprising that a port in a network is removed the allowed address pairs due to a setting in an entry that yielded no port on that network. In other words, one would expect that the same subnet filtering that happens on each network entry in what regards yielding ports for the VM would also work for the noAllowedPairs parameter.

Version-Release number of selected component (if applicable):

4.10.30

How reproducible:

Always

Steps to Reproduce:

1. Create a machineset like in the description
2.
3.

Actual results:

All ports have no address pairs

Expected results:

Only the port on the secondary subnet has no address pairs.

Additional info:

A simple workaround would be to just fill the filter so that a single network is selected for each network entry.

References:
[1] - https://github.com/openshift/cluster-api-provider-openstack/blob/f6b51710d4f395ded401347589447f5f41dd5c4c/pkg/cloud/openstack/clients/machineservice.go#L576
[2] - https://github.com/openshift/cluster-api-provider-openstack/blob/f6b51710d4f395ded401347589447f5f41dd5c4c/pkg/cloud/openstack/clients/machineservice.go#L580
[3] - https://github.com/openshift/cluster-api-provider-openstack/blob/f6b51710d4f395ded401347589447f5f41dd5c4c/pkg/cloud/openstack/clients/machineservice.go#L581-L583
[4] - https://github.com/openshift/cluster-api-provider-openstack/blob/f6b51710d4f395ded401347589447f5f41dd5c4c/pkg/cloud/openstack/clients/machineservice.go#L658-L660
[5] - https://github.com/openshift/cluster-api-provider-openstack/blob/f6b51710d4f395ded401347589447f5f41dd5c4c/pkg/cloud/openstack/clients/machineservice.go#L610-L625

https://github.com/openshift/cluster-api-provider-openstack/pull/243

Bug OCPBUGS-4829: [4.11] [OVN]Sometimes after reboot egress node, egress IP cannot be applied anymore.

View the Description View the linked PRs

Description of problem:

[OVN][OSP] After reboot egress node, egress IP cannot be applied anymore.

Version-Release number of selected component (if applicable):

4.12.0-0.nightly-2022-11-07-181244

How reproducible:

Frequently happened in automation. But didn't reproduce it in manual.

Steps to Reproduce:

1. Label one node as egress node

2.
Config one egressIP object
STEP: Check  one EgressIP assigned in the object.

Nov  8 15:28:23.591: INFO: egressIPStatus: [{"egressIP":"192.168.54.72","node":"huirwang-1108c-pg2mt-worker-0-2fn6q"}]

3.
Reboot the node, wait for the node ready.

Actual results:

EgressIP cannot be applied anymore. Waited more than 1 hour.
 oc get egressip
NAME             EGRESSIPS       ASSIGNED NODE   ASSIGNED EGRESSIPS
egressip-47031   192.168.54.72

Expected results:

The egressIP should be applied correctly.

Additional info:


Some logs
E1108 07:29:41.849149       1 egressip.go:1635] No assignable nodes found for EgressIP: egressip-47031 and requested IPs: [192.168.54.72]
I1108 07:29:41.849288       1 event.go:285] Event(v1.ObjectReference{Kind:"EgressIP", Namespace:"", Name:"egressip-47031", UID:"", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'NoMatchingNodeFound' no assignable nodes for EgressIP: egressip-47031, please tag at least one node with label: k8s.ovn.org/egress-assignable


W1108 07:33:37.401149       1 egressip_healthcheck.go:162] Could not connect to huirwang-1108c-pg2mt-worker-0-2fn6q (10.131.0.2:9107): context deadline exceeded
I1108 07:33:37.401348       1 master.go:1364] Adding or Updating Node "huirwang-1108c-pg2mt-worker-0-2fn6q"
I1108 07:33:37.437465       1 egressip_healthcheck.go:168] Connected to huirwang-1108c-pg2mt-worker-0-2fn6q (10.131.0.2:9107)

After this log, seems like no logs related to "192.168.54.72" happened.

https://github.com/openshift/ovn-kubernetes/pull/1445

Bug OCPBUGS-5245: [release-4.11] OVNK: NAT issue for packets exceeding check_pkt_larger() for NodePort services that route to hostNetworked pods

View the Description View the linked PRs

Description of problem:

Create Loadbalancer type service within the OCP 4.11.x OVNKubernetes cluster to expose the api server endpoint, the service does not response for normal oc request. 
But some of them are working, like "oc whoami", "oc get --raw /api"

Version-Release number of selected component (if applicable):

4.11.8 with OVNKubernetes

How reproducible:

always

Steps to Reproduce:

1. Setup openshift cluster 4.11 on AWS with OVNKubernetes as the default network
2. Create the following service under openshift-kube-apiserver namespace to expose the api
----
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "1800"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  name: test-api
  namespace: openshift-kube-apiserver
spec:
  allocateLoadBalancerNodePorts: true
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  loadBalancerSourceRanges:
  - <my_ip>/32
  ports:
  - nodePort: 31248
    port: 6443
    protocol: TCP
    targetPort: 6443
  selector:
    apiserver: "true"
    app: openshift-kube-apiserver
  sessionAffinity: None
  type: LoadBalancer

3. Setup the DNS resolution for the access
xxx.mydomain.com ---> <elb-auto-generated-dns>

4. Try to access the cluster api via the service above by updating the kubeconfig to use the custom dns name

Actual results:

No response from the server side.

$ time oc get node -v8
I1025 08:29:10.284069  103974 loader.go:375] Config loaded from file:  bmeng.kubeconfig
I1025 08:29:10.294017  103974 round_trippers.go:420] GET https://rh-api.bmeng-ccs-ovn.3o13.s1.devshift.org:6443/api/v1/nodes?limit=500
I1025 08:29:10.294035  103974 round_trippers.go:427] Request Headers:
I1025 08:29:10.294043  103974 round_trippers.go:431]     Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json
I1025 08:29:10.294052  103974 round_trippers.go:431]     User-Agent: oc/openshift (linux/amd64) kubernetes/e40bd2d
I1025 08:29:10.365119  103974 round_trippers.go:446] Response Status: 200 OK in 71 milliseconds
I1025 08:29:10.365142  103974 round_trippers.go:449] Response Headers:
I1025 08:29:10.365148  103974 round_trippers.go:452]     Audit-Id: 83b9d8ae-05a4-4036-bff6-de371d5bec12
I1025 08:29:10.365155  103974 round_trippers.go:452]     Cache-Control: no-cache, private
I1025 08:29:10.365161  103974 round_trippers.go:452]     Content-Type: application/json
I1025 08:29:10.365167  103974 round_trippers.go:452]     X-Kubernetes-Pf-Flowschema-Uid: 2abc2e2d-ada3-4cb8-a86f-235df3a4e214
I1025 08:29:10.365173  103974 round_trippers.go:452]     X-Kubernetes-Pf-Prioritylevel-Uid: 02f7a188-43c7-4827-af58-5ebe861a1891
I1025 08:29:10.365179  103974 round_trippers.go:452]     Date: Tue, 25 Oct 2022 08:29:10 GMT
^C
real    17m4.840s
user    0m0.567s
sys    0m0.163s


However, it has the correct response if using --raw to request, eg:
$ oc get --raw /api/v1  --kubeconfig bmeng.kubeconfig 
{"kind":"APIResourceList","groupVersion":"v1","resources":[{"name":"bindings","singularName":"","namespaced":true,"kind":"Binding","verbs":["create"]},{"name":"componentstatuses","singularName":"","namespaced":false,"kind":"ComponentStatus","verbs":["get","list"],"shortNames":["cs"]},{"name":"configmaps","singularName":"","namespaced":true,"kind":"ConfigMap","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["cm"],"storageVersionHash":"qFsyl6wFWjQ="},{"name":"endpoints","singularName":"","namespaced":true,"kind":"Endpoints","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ep"],"storageVersionHash":"fWeeMqaN/OA="},{"name":"events","singularName":"","namespaced":true,"kind":"Event","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ev"],"storageVersionHash":"r2yiGXH7wu8="},{"name":"limitranges","singularName":"","namespaced":true,"kind":"LimitRange","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["limits"],"storageVersionHash":"EBKMFVe6cwo="},{"name":"namespaces","singularName":"","namespaced":false,"kind":"Namespace","verbs":["create","delete","get","list","patch","update","watch"],"shortNames":["ns"],"storageVersionHash":"Q3oi5N2YM8M="},{"name":"namespaces/finalize","singularName":"","namespaced":false,"kind":"Namespace","verbs":["update"]},{"name":"namespaces/status","singularName":"","namespaced":false,"kind":"Namespace","verbs":["get","patch","update"]},{"name":"nodes","singularName":"","namespaced":false,"kind":"Node","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["no"],"storageVersionHash":"XwShjMxG9Fs="},{"name":"nodes/proxy","singularName":"","namespaced":false,"kind":"NodeProxyOptions","verbs":["create","delete","get","patch","update"]},{"name":"nodes/status","singularName":"","namespaced":false,"kind":"Node","verbs":["get","patch","update"]},{"name":"persistentvolumeclaims","singularName":"","namespaced":true,"kind":"PersistentVolumeClaim","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["pvc"],"storageVersionHash":"QWTyNDq0dC4="},{"name":"persistentvolumeclaims/status","singularName":"","namespaced":true,"kind":"PersistentVolumeClaim","verbs":["get","patch","update"]},{"name":"persistentvolumes","singularName":"","namespaced":false,"kind":"PersistentVolume","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["pv"],"storageVersionHash":"HN/zwEC+JgM="},{"name":"persistentvolumes/status","singularName":"","namespaced":false,"kind":"PersistentVolume","verbs":["get","patch","update"]},{"name":"pods","singularName":"","namespaced":true,"kind":"Pod","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["po"],"categories":["all"],"storageVersionHash":"xPOwRZ+Yhw8="},{"name":"pods/attach","singularName":"","namespaced":true,"kind":"PodAttachOptions","verbs":["create","get"]},{"name":"pods/binding","singularName":"","namespaced":true,"kind":"Binding","verbs":["create"]},{"name":"pods/ephemeralcontainers","singularName":"","namespaced":true,"kind":"Pod","verbs":["get","patch","update"]},{"name":"pods/eviction","singularName":"","namespaced":true,"group":"policy","version":"v1","kind":"Eviction","verbs":["create"]},{"name":"pods/exec","singularName":"","namespaced":true,"kind":"PodExecOptions","verbs":["create","get"]},{"name":"pods/log","singularName":"","namespaced":true,"kind":"Pod","verbs":["get"]},{"name":"pods/portforward","singularName":"","namespaced":true,"kind":"PodPortForwardOptions","verbs":["create","get"]},{"name":"pods/proxy","singularName":"","namespaced":true,"kind":"PodProxyOptions","verbs":["create","delete","get","patch","update"]},{"name":"pods/status","singularName":"","namespaced":true,"kind":"Pod","verbs":["get","patch","update"]},{"name":"podtemplates","singularName":"","namespaced":true,"kind":"PodTemplate","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"storageVersionHash":"LIXB2x4IFpk="},{"name":"replicationcontrollers","singularName":"","namespaced":true,"kind":"ReplicationController","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["rc"],"categories":["all"],"storageVersionHash":"Jond2If31h0="},{"name":"replicationcontrollers/scale","singularName":"","namespaced":true,"group":"autoscaling","version":"v1","kind":"Scale","verbs":["get","patch","update"]},{"name":"replicationcontrollers/status","singularName":"","namespaced":true,"kind":"ReplicationController","verbs":["get","patch","update"]},{"name":"resourcequotas","singularName":"","namespaced":true,"kind":"ResourceQuota","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["quota"],"storageVersionHash":"8uhSgffRX6w="},{"name":"resourcequotas/status","singularName":"","namespaced":true,"kind":"ResourceQuota","verbs":["get","patch","update"]},{"name":"secrets","singularName":"","namespaced":true,"kind":"Secret","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"storageVersionHash":"S6u1pOWzb84="},{"name":"serviceaccounts","singularName":"","namespaced":true,"kind":"ServiceAccount","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["sa"],"storageVersionHash":"pbx9ZvyFpBE="},{"name":"serviceaccounts/token","singularName":"","namespaced":true,"group":"authentication.k8s.io","version":"v1","kind":"TokenRequest","verbs":["create"]},{"name":"services","singularName":"","namespaced":true,"kind":"Service","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["svc"],"categories":["all"],"storageVersionHash":"0/CO1lhkEBI="},{"name":"services/proxy","singularName":"","namespaced":true,"kind":"ServiceProxyOptions","verbs":["create","delete","get","patch","update"]},{"name":"services/status","singularName":"","namespaced":true,"kind":"Service","verbs":["get","patch","update"]}]}

Expected results:

The normal oc request should be working.

Additional info:

There is no such issue for clusters with openshift-sdn with the same OpenShift version and same LoadBalancer service.

We suspected that it might be related to the MTU setting, but this cannot explain why OpenShiftSDN works well.

Another thing might be related is that the OpenShiftSDN is using iptables for service loadbalancing and OVN is dealing that within the OVN services.

Please let me know if any debug log/info is needed.

https://github.com/openshift/ovn-kubernetes/pull/1505

Task OSDOCS-3257: Add content type to oc CLI doc generation

View the Description View the linked PRs

The two modules that are auto generated for the CLI docs need to add ":_content-type: REFERENCE" to the top of the files. Update the doc generation templates to add these.

https://github.com/openshift/oc/pull/1072

Bug OCPBUGS-197: rpm-ostree start timeout when upgrade from 4.11.0-rc.5-aarch64 to 4.11.0-rc.6-aarch64

View the Description View the linked PRs

clone of 4.12 bug https://issues.redhat.com/browse/OCPBUGSM-47294

https://github.com/openshift/machine-config-operator/pull/3292

Bug OCPBUGS-3942: Whereabouts CNI timesout while iterating exclude range [backport 4.11]

View the Description View the linked PRs

Description of problem:

When creating a pod with an additional network that contains a `spec.config.ipam.exclude` range, any address within the excluded range is still iterated while searching for a suitable IP candidate. As a result, pod creation times out when large exclude ranges are used.

Version-Release number of selected component (if applicable):

How reproducible:

with big exclude ranges, 100%

Steps to Reproduce:

1. create network-attachment-definition with a large range:

$ cat <<EOF| oc apply -f -       
apiVersion: k8s.cni.cncf.io/v1                                            
kind: NetworkAttachmentDefinition
metadata:
  name: nad-w-excludes
spec:
  config: |-
    {
      "cniVersion": "0.3.1",
      "name": "macvlan-net",
      "type": "macvlan",
      "master": "ens3",
      "mode": "bridge",
      "ipam": {
         "type": "whereabouts",
         "range": "fd43:01f1:3daa:0baa::/64",
         "exclude": [ "fd43:01f1:3daa:0baa::/100" ],
         "log_file": "/tmp/whereabouts.log",
         "log_level" : "debug"
      }
    }
EOF
2. create a pod with the network attached:

$ cat <<EOF|oc apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: pod-with-exclude-range
  annotations:
    k8s.v1.cni.cncf.io/networks: nad-w-excludes
spec:
  containers:
  - name: pod-1
    image: openshift/hello-openshift
EOF

3. check pod status, event log and whereabouts logs after a while: 

$ oc get pods
NAME                        READY   STATUS              RESTARTS   AGE
pod-with-exclude-range      0/1     ContainerCreating   0          2m23s

$ oc get events
<...>
6m39s       Normal    Scheduled                                    pod/pod-with-exclude-range                   Successfully assigned default/pod-with-exclude-range to <worker-node>
6m37s       Normal    AddedInterface                               pod/pod-with-exclude-range                   Add eth0 [10.129.2.49/23] from openshift-sdn
2m39s       Warning   FailedCreatePodSandBox                       pod/pod-with-exclude-range                   Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceeded

$ oc debug node/<worker-node> - tail /host/tmp/whereabouts.log
Starting pod/<worker-node>-debug ...
To use host binaries, run `chroot /host`
2022-10-27T14:14:50Z [debug] Finished leader election
2022-10-27T14:14:50Z [debug] IPManagement: {fd43:1f1:3daa:baa::1 ffffffffffffffff0000000000000000} , <nil>
2022-10-27T14:14:59Z [debug] Used defaults from parsed flat file config @ /etc/kubernetes/cni/net.d/whereabouts.d/whereabouts.conf
2022-10-27T14:14:59Z [debug] ADD - IPAM configuration successfully read: {Name:macvlan-net Type:whereabouts Routes:[] Datastore:kubernetes Addresses:[] OmitRanges:[fd43:01f1:3daa:0baa::/80] DNS: {Nameservers:[] Domain: Search:[] Options:[]} Range:fd43:1f1:3daa:baa::/64 RangeStart:fd43:1f1:3daa:baa:: RangeEnd:<nil> GatewayStr: EtcdHost: EtcdUsername: EtcdPassword:********* EtcdKeyFile: EtcdCertFile: EtcdCACertFile: LeaderLeaseDuration:1500 LeaderRenewDeadline:1000 LeaderRetryPeriod:500 LogFile:/tmp/whereabouts.log LogLevel:debug OverlappingRanges:true SleepForRace:0 Gateway:<nil> Kubernetes: {KubeConfigPath:/etc/kubernetes/cni/net.d/whereabouts.d/whereabouts.kubeconfig K8sAPIRoot:} ConfigurationPath:PodName:pod-with-exclude-range PodNamespace:default} 
2022-10-27T14:14:59Z [debug] Beginning IPAM for ContainerID: f4ffd0e07d6c1a2b6ffb0fa29910c795258792bb1a1710ff66f6b48fab37af82
2022-10-27T14:14:59Z [debug] Started leader election
2022-10-27T14:14:59Z [debug] OnStartedLeading() called
2022-10-27T14:14:59Z [debug] Elected as leader, do processing
2022-10-27T14:14:59Z [debug] IPManagement - mode: 0 / containerID:f4ffd0e07d6c1a2b6ffb0fa29910c795258792bb1a1710ff66f6b48fab37af82 / podRef: default/pod-with-exclude-range
2022-10-27T14:14:59Z [debug] IterateForAssignment input >> ip: fd43:1f1:3daa:baa:: | ipnet: {fd43:1f1:3daa:baa:: ffffffffffffffff0000000000000000} | first IP: fd43:1f1:3daa:baa::1 | last IP: fd43:1f1:3daa:baa:ffff:ffff:ffff:ffff

Actual results:

Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceeded

Expected results:

additional network gets attached to the pod

Additional info:

https://github.com/openshift/whereabouts-cni/pull/104

Bug OCPBUGS-747: [4.11] Disconnected IPI OCP cluster install on baremetal fails when hostname of master nodes does not include the text "master

View the Description View the linked PRs

Description of problem:

Disconnected IPI OCP 4.10.22 cluster install on baremetal fails when hostname of master nodes does not include "master"

Version-Release number of selected component (if applicable): 4.10.22

How reproducible: Perform disconnected IPI install of OCP 4.10.22 on bare metal with master nodes that do not contain the text "master"

Steps to Reproduce:

Perform disconnected IPI install of OCP 4.10.22 on bare metal with master nodes that do not contain the text "master"

Actual results: master nodes do come up.

Expected results: master nodes should come up despite that the text "master" is not in their hostname.

Additional info:

Disconnected IPI OCP 4.10.22 cluster install on baremetal fails when hostname of master nodes does not include "master"

The code for the cluster-baremetal-operator at the following link:

https://github.com/openshift/cluster-baremetal-operator/blob/49d7b249c5dcef8228f206eff4530a25f03b201f/controllers/provisioning_controller.go#L441

The following condition is concerning:

if strings.Contains(bmh.Name, "master") && len(bmh.Spec.BootMACAddress) > 0

The packages reveal that bmh.Name references the name inside the metadata of the BMH object.

Should a customer have masters with names that do not include the text "master", the above condition can never become true, and so, the following slice is never created :

macs = append(macs, bmh.Spec.BootMACAddress)

Bug OCPBUGS-4163: [release-4.11] The last egressIP in IP capacity cannot be applied correctly

View the Description View the linked PRs

Description of problem:

The last egressIP in IP capacity cannot be applied correctly

Version-Release number of selected component (if applicable):

4.11.0-0.nightly-2022-11-27-164248

How reproducible:

Found this failure in automation case which used to pass

Steps to Reproduce:

In AWS, one worker node is with IP capacity "ipv4:14"
 $oc describe node ip-10-0-129-21.us-west-2.compute.internal | egrep -C 3 egress
Annotations:        cloud.network.openshift.io/egress-ipconfig:
                      [{"interface":"eni-03820ba0eca427fbf","ifaddr":{"ipv4":"10.0.128.0/18"},"capacity":{"ipv4":14,"ipv6":15}}]
                    csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-0cb3ae15bf3cffd9f"}
                    k8s.ovn.org/host-addresses: ["10.0.129.21"]

1. Label above node as egress node
2. Created 15 egressIP objects, each egressIP object with one egressIP.

Actual results:

 13 egressIPs applied correctly , last one is in wrong status.
% oc get egressip
NAME                EGRESSIPS      ASSIGNED NODE                               ASSIGNED EGRESSIPS
egressip-47208-0    10.0.135.200   ip-10-0-129-21.us-west-2.compute.internal   10.0.135.200
egressip-47208-1    10.0.162.178   ip-10-0-129-21.us-west-2.compute.internal   10.0.162.178
egressip-47208-10   10.0.144.46    ip-10-0-129-21.us-west-2.compute.internal   10.0.144.46
egressip-47208-11   10.0.191.91    ip-10-0-129-21.us-west-2.compute.internal   10.0.191.91
egressip-47208-12   10.0.133.215   ip-10-0-129-21.us-west-2.compute.internal   10.0.133.215
egressip-47208-13   10.0.174.207   ip-10-0-129-21.us-west-2.compute.internal   10.0.174.207
egressip-47208-14   10.0.176.224   ip-10-0-129-21.us-west-2.compute.internal   10.0.176.224
egressip-47208-2    10.0.184.114   ip-10-0-129-21.us-west-2.compute.internal   10.0.184.114
egressip-47208-3    10.0.167.224   ip-10-0-129-21.us-west-2.compute.internal   10.0.167.224
egressip-47208-4    10.0.187.148   ip-10-0-129-21.us-west-2.compute.internal   10.0.187.148
egressip-47208-5    10.0.184.109   ip-10-0-129-21.us-west-2.compute.internal   10.0.184.109
egressip-47208-6    10.0.155.208                                               
egressip-47208-7    10.0.134.13    ip-10-0-129-21.us-west-2.compute.internal   10.0.134.13
egressip-47208-8    10.0.142.255   ip-10-0-129-21.us-west-2.compute.internal   10.0.142.255
egressip-47208-9    10.0.170.197

% oc get cloudprivateipconfig
NAME           AGE
10.0.133.215   113s
10.0.134.13    113s
10.0.135.200   113s
10.0.142.255   113s
10.0.144.46    113s
10.0.155.208   113s
10.0.162.178   113s
10.0.167.224   113s
10.0.174.207   113s
10.0.176.224   113s
10.0.184.109   113s
10.0.184.114   113s
10.0.187.148   113s
10.0.191.91    113s

 oc get cloudprivateipconfig 10.0.155.208  -o yaml
apiVersion: cloud.network.openshift.io/v1
kind: CloudPrivateIPConfig
metadata:
  annotations:
    k8s.ovn.org/egressip-owner-ref: egressip-47208-6
  creationTimestamp: "2022-11-28T09:13:47Z"
  finalizers:
  - cloudprivateipconfig.cloud.network.openshift.io/finalizer
  generation: 1
  name: 10.0.155.208
  resourceVersion: "72869"
  uid: 0143a07b-0a30-4de8-bfd7-589ed0c3d7dc
spec:
  node: ip-10-0-129-21.us-west-2.compute.internal
status:
  conditions:
  - lastTransitionTime: "2022-11-28T09:16:51Z"
    message: 'Error processing cloud assignment request, err: <nil>'
    observedGeneration: 1
    reason: CloudResponseError
    status: "False"
    type: Assigned
  node: ip-10-0-129-21.us-west-2.compute.internal

Expected results:

As IP capacity for this node is 14, so here should have 14 egress IP applied successfully. No cloudprivateipconfig IP in error status.

Additional info:

https://github.com/openshift/cloud-network-config-controller/pull/83

Bug OCPBUGS-524: Plugin page error boundary message is not cleared after leaving page

View the Description View the linked PRs

Created attachment 1905034 [details]
Plugin page with error

Steps to reproduce:

1. Install a plugin with a page that has a runtime error. (Demo Plugin -> Dynamic Nav 1 currently has an error for me, but you can reproduce by editing any plugin and introducing an error.)
2. Observe the "something went wrong" error message.
3. Navigate to any other page (e.g. Workloads -> Pods)

Expected result:

The pods page is displayed.

Action result:

The error message persists. There is no way to clear except to refresh the browser.

https://github.com/openshift/console/pull/11968

Bug OCPBUGS-2014: CI: Backend unit tests fails because devfile registry was updated (mock response)

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-1678~~. The following is the description of the original issue:
—
Description of problem:
pkg/devfile/sample_test.go fails after devfile registry was updated (https://github.com/devfile/registry/pull/126)

~~OCPBUGS-1677~~ is about updating our assertion so that the CI job runs successfully again. We might want to backport this as well.

This is about updating the code that the test should use a mock response instead of the latest registry content OR check some specific attributes instead of comparing the full JSON response.

Version-Release number of selected component (if applicable):
4.12

How reproducible:
Always

Steps to Reproduce:
1. Clone openshift/console
2. Run ./test-backend.sh

Actual results:
Unit tests fail

Expected results:
Unit tests should pass again

Additional info:

https://github.com/openshift/console/pull/12137

Bug OCPBUGS-1762: [4.11] provisioning interface on master node not getting ipv4 dhcp ip address from bootstrap dhcp server on OCP IPI BareMetal install

View the Description View the linked PRs

Description of problem:

Provisioning interface on master node not getting ipv4 dhcp ip address from bootstrap dhcp server on OCP 4.10.16 IPI BareMetal install.

Customer is performing an OCP 4.10.16 IPI BareMetal install and bootstrap node provisions just fine, but when master nodes are booted for provisioning, they are not getting an ipv4 address via dhcp. As such, the install is not moving forward at this point.

Version-Release number of selected component (if applicable):

OCP 4.10.16

How reproducible:

Perform OCP 4.10.16 IPI BareMetal install.

Actual results:

provisioning interface comes up (as evidenced by ipv6 address) but is not getting an ipv4 address via dhcp. OCP install / provisioning fails at this point.

Expected results:

provisioning interface successfully received an ipv4 ip address and successfully provisioned master nodes (and subsequently worker nodes as well.)

Additional info:

As a troubleshooting measure, manually adding an ipv4 ip address did allow the coreos image on the bootstrap node to be reached via curl.

Further, the kernel boot line for the first master node was updated for a static ip addresss assignment for further confirmation that the master node would successfully image this way which further confirming that the issue is the provisioning interface not receiving an ipv4 ip address from the dhcp server.

https://github.com/openshift/cluster-baremetal-operator/pull/298

Bug OCPBUGS-2058: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/cloud-provider-vsphere/pull/26

Bug OCPBUGS-5766: [4.11] unit test data race with egress ip tests

View the Description View the linked PRs

Description of problem:

Data race seen in unit tests:
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_ovn-kubernetes/1448/pull-ci-openshift-ovn-kubernetes-release-4.11-unit/1604898712423763968/artifacts/test/build-log.txt

https://github.com/openshift/ovn-kubernetes/pull/1471

Bug OCPBUGS-10215: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

Bug OCPBUGS-4398: CVE-2022-27191 ose-installer-container: golang: crash in a golang.org/x/crypto/ssh server [openshift-4]

View the Description View the linked PRs

Description of problem:

This is a clone of https://bugzilla.redhat.com/show_bug.cgi?id=2074299 for backporting purposes.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/installer/pull/6652

Bug OCPBUGS-5422: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/installer/pull/6743

Bug OCPBUGS-892: Update RHCOS release browser url

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-884~~. The following is the description of the original issue:
—
Description of problem:

Since the decomissioning of the psi cluster, and subsequent move of the rhcos release browser, product builds machine-os-images builds have been failing. See e.g. https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=47565717

Version-Release number of selected component (if applicable):

4.12, 4.11, 4.10.

How reproducible:

Have ART build the image

Steps to Reproduce:

1. Have ART build the image

Actual results:

Build failure

Expected results:

Build succesful

Additional info:

https://github.com/openshift/machine-os-images/pull/22

Bug OCPBUGS-10486: node healthz server is missing in ovnk

View the Description View the linked PRs

Just like kube proxy, ovnk should expose port 10256 on every node, so that cloud LBs can send health checks and know which nodes are available. This is relevant for services with externalTrafficPolicy=Cluster.

https://github.com/openshift/ovn-kubernetes/pull/1594

Bug OCPBUGS-1984: Install Helm chart form doesn't allow the user select a specific version

View the Description View the linked PRs

Description of problem:

When the user installs a helm chart, the dropdown to select a specific version is always disabled. Also for helm charts that can upgraded or downgraded after installation. For example the nodejs helm chart.

Version-Release number of selected component (if applicable):

At least 4.11, maybe all versions, but a backport to 4.11 is fine

How reproducible:

Always

Steps to Reproduce:

1. Switch to developer perspective
2. Navigate to Add > Helm chart
3. Select the Nodejs helm chart
4. Try to select another version
..
When the user installs the not selectable version and edit the helm chart there is another version to selec

Actual results:

The version is not selectable.

Expected results:

The version should be selectable.

Additional info:

https://github.com/openshift/console/pull/12135

Bug OCPBUGS-6689: OLM details page crashes on incomplete ClusterServiceVersion resource

View the Description View the linked PRs

Description of problem:
When creating a incomplete ClusterServiceVersion resource the OLM details page crashes (on 4.11).

apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  name: minimal-csv
  namespace: christoph
spec:
  apiservicedefinitions:
    owned:
      - group: A
        kind: A
        name: A
        version: v1
  customresourcedefinitions:
    owned:
      - kind: B
        name: B
        version: v1
  displayName: My minimal CSV
  install:
    strategy: ''

Version-Release number of selected component (if applicable):
Crashes on 4.8-4.11, work fine from 4.12 onwards.

How reproducible:
Alway

Steps to Reproduce:
1. Apply the ClusterServiceVersion YAML from above
2. Open the Admin perspective > Installed Operator > Operator detail page

Actual results:
Details page crashes on tab A and B.

Expected results:
Page should not crash

Additional info:
Thi is a follow up on https://bugzilla.redhat.com/show_bug.cgi?id=2084287

https://github.com/openshift/console/pull/12478

Bug OCPBUGS-1786: CI: Backend unit tests fails because devfile registry was updated (fix assertion)

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-1677~~. The following is the description of the original issue:
—
Description of problem:
pkg/devfile/sample_test.go fails after devfile registry was updated (https://github.com/devfile/registry/pull/126)

This issue is about updating our assertion so that the CI job runs successfully again. We might want to backport this as well.

~~OCPBUGS-1678~~ is about updating the code that the test should use a mock response instead of the latest registry content OR check some specific attributes instead of comparing the full JSON response.

Version-Release number of selected component (if applicable):
4.12

How reproducible:
Always

Steps to Reproduce:
1. Clone openshift/console
2. Run ./test-backend.sh

Actual results:
Unit tests fail

Expected results:
Unit tests should pass again

Additional info:

https://github.com/openshift/console/pull/12108

Bug OCPBUGS-541: Input values in Instantiate Template are disappeared randomly in the developer console

View the Description View the linked PRs

#Description of problem:

Developer Console > +ADD > Develoeper Catalog > Service > select Types Templates > Initiate Template

Input values in Instantiate Template are disappeared randomly.

#Version-Release number of selected component (if applicable):

Customer ENV

OCP4.10.9
Developer Console
Edge 88x. / Edge 85.0.x / Chrome 97.x /Chrome 88.x
Internet Disconnected OCP cluster

quicklab test ENV

Developer Console
OCP4.10.12
Chrome 100

#How reproducible:

I reproduced this issue in ocp410ovn shared cluster in the quicklab

https://console-openshift-console.apps.sharedocp4upi410ovn.lab.upshift.rdu2.redhat.com

Select Apache HTTP Server > Input name "test" in Application Hostname box
After several seconds, the value has disappeared in the web console.

#Steps to Reproduce:

0. Developer Console > +ADD > Develoeper Catalog > Service > select Types Templates > Initiate Template

1. Input values in the box of template menu.

2. The values are disappeared after several seconds later. (20s~ or randomly)

3. Many users have experienced this issue.

The web browser version of users experiencing this issue.

Customer: Edge 88x. / Edge 85.0.x / Chrome 97.x /Chrome 88.x
My browser: Chrome 102.x

==> the browser version doesn't matter.

#Actual results:

Input values in "Instantiate Template" are disappeared randomly.
Users can't use the Initiate Template feature in the Dev console.

#Expected results:
Input values remain in the web console and users creat the object by the "Instantiate Template"

#Additional info:

See "Application Name" has disappeared in the video I attached.

https://github.com/openshift/console/pull/11982

Bug OCPBUGS-7727: [4.11] Fix disaster recovery test [sig-etcd][Feature:DisasterRecovery][Disruptive] [Feature:EtcdRecovery] Cluster should restore itself after quorum loss [Serial]

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-7633~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-1125~~. The following is the description of the original issue:
—
(originally reported in BZ as https://bugzilla.redhat.com/show_bug.cgi?id=1983200)

test:
[sig-etcd][Feature:DisasterRecovery][Disruptive] [Feature:EtcdRecovery] Cluster should restore itself after quorum loss [Serial]

is failing frequently in CI, see search results:
https://search.ci.openshift.org/?maxAge=168h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job&search=%5C%5Bsig-etcd%5C%5D%5C%5BFeature%3ADisasterRecovery%5C%5D%5C%5BDisruptive%5C%5D+%5C%5BFeature%3AEtcdRecovery%5C%5D+Cluster+should+restore+itself+after+quorum+loss+%5C%5BSerial%5C%5D

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-disruptive-4.8/1413625606435770368
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-disruptive-4.8/1415075413717159936

—

some brief triaging from Thomas Jungblut on:
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-disruptive-4.11/1568747321334697984

it seems the last guard pod doesn't come up, etcd operator installs this properly and the revision installer also does not spout any errors. It just doesn't progress to the latest revision. At first glance doesn't look like an issue with etcd itself, but needs to be taken a closer look at for sure.

https://github.com/openshift/origin/pull/27735

Bug OCPBUGS-8260: Update to 4.13.0-ec.3 stuck on leaked MachineConfig

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-8261~~. The following is the description of the original issue:
—

Description of problem:

An update from 4.13.0-ec.2 to 4.13.0-ec.3 stuck on:

$ oc get clusteroperator machine-config
NAME             VERSION       AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
machine-config   4.13.0-ec.2   True        True          True       30h     Unable to apply 4.13.0-ec.3: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, error pool worker is not ready, retrying. Status: (pool degraded: true total: 105, ready 105, updated: 105, unavailable: 0)]

The worker MachineConfigPool status included:

Unable to find source-code formatter for language: node. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml      type: NodeDegraded
    - lastTransitionTime: "2023-02-16T14:29:21Z"
      message: 'Failed to render configuration for pool worker: Ignoring MC 99-worker-generated-containerruntime
        generated by older version 8276d9c1f574481043d3661a1ace1f36cd8c3b62 (my version:
        c06601510c0917a48912cc2dda095d8414cc5182)'

Version-Release number of selected component (if applicable):

4.13.0-ec.3. The behavior was apparently introduced as part of ~~OCPBUGS-6018~~, which has been backported, so the following update targets are expected to be vulnerable: 4.10.52+, 4.11.26+, 4.12.2+, and 4.13.0-ec.3.

How reproducible:

100%, when updating into a vulnerable release, if you happen to have leaked MachineConfig.

Steps to Reproduce:

1. 4.12.0-ec.1 dropped cleanUpDuplicatedMC. Run a later release, like 4.13.0-ec.2.
2. Create more than one KubeletConfig or ContainerRuntimeConfig targeting the worker pool (or any pool other than master). The number of clusters who have had redundant configuration objects like this is expected to be small.
3. (Optionally?) delete the extra KubeletConfig and ContainerRuntimeConfig.
4. Update to 4.13.0-ec.3.

Actual results:

Update sticks on the machine-config ClusterOperator, as described above.

Expected results:

Update completes without issues.

https://github.com/openshift/machine-config-operator/pull/3577

Bug OCPBUGS-1552: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/ovn-kubernetes/pull/1272

Bug OCPBUGS-1876: [release-4.11] download 'aliyun'

View the Description View the linked PRs

Description of problem:

The alibabacloud client "aliyun" would be used when pre-configuring some resources (e.g. VPC, bastion host, etc.) before launching an OCP cluster with customization.

Version-Release number of selected component (if applicable):

4.11

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/installer/pull/6377

Bug OCPBUGS-4030: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/cluster-monitoring-operator/pull/1821

Bug OCPBUGS-5075: [4.11] SNO not able to bring up Provisioning resource in 4.11.17

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-4696~~. The following is the description of the original issue:
—
Description of problem:

metal3 pod does not come up on SNO when creating Provisioning with provisioningNetwork set to Disabled

The issue is that on SNO, there is no Machine, and no BareMetalHost, it is looking of Machine objects to populate the provisioningMacAddresses field. However, when provisioningNetwork is Disabled, provisioningMacAddresses is not used anyway.

You can work around this issue by populating provisioningMacAddresses with a dummy address, like this:

kind: Provisioning
metadata:
  name: provisioning-configuration
spec:
  provisioningMacAddresses:
  - aa:aa:aa:aa:aa:aa
  provisioningNetwork: Disabled
  watchAllNamespaces: true

Version-Release number of selected component (if applicable):

4.11.17

How reproducible:

Try to bring up Provisioning on SNO in 4.11.17 with provisioningNetwork set to Disabled

apiVersion: metal3.io/v1alpha1
kind: Provisioning
metadata:
  name: provisioning-configuration
spec:
  provisioningNetwork: Disabled
  watchAllNamespaces: true

Steps to Reproduce:

1.
2.
3.

Actual results:

controller/provisioning "msg"="Reconciler error" "error"="machines with cluster-api-machine-role=master not found" "name"="provisioning-configuration" "namespace"="" "reconciler group"="metal3.io" "reconciler kind"="Provisioning"

Expected results:

metal3 pod should be deployed

Additional info:

This issue is a result of this change: https://github.com/openshift/cluster-baremetal-operator/pull/307
See this Slack thread: https://coreos.slack.com/archives/CFP6ST0A3/p1670530729168599

https://github.com/openshift/cluster-baremetal-operator/pull/318

Bug OCPBUGS-5926: NMstate removes egressip in OpenShift cluster with SDN plugin

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-4486~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-95~~. The following is the description of the original issue:
—
In an OpenShift cluster with OpenShiftSDN network plugin with egressIP and NMstate operator configured, there are some conditions when the egressIP is deconfigured from the network interface.

The bug is 100% reproducible.

Steps for reproducing the issue are:

1. Install a cluster with OpenShiftSDN network plugin.

2. Configure egressip for a project.

3. Install NMstate operator.

4. Create a NodeNetworkConfigurationPolicy.

5. Identify on which node the egressIP is present.

6. Restart the nmstate-handler pod running on the identified node.

7. Verify that the egressIP is no more present.

Restarting the sdn pod related to the identified node will reconfigure the egressIP in the node.

This issue has a high impact since any changes triggered for the NMstate operator will prevent application traffic. For example, in the customer environment, the issue is triggered any time a new node is added to the cluster.

The expectation is that NMstate operator should not interfere with SDN configuration.

https://github.com/openshift/sdn/pull/492

Bug OCPBUGS-433: Nutanix platform validations run at `create manifests` stage

View the Description View the linked PRs

This is a clone of ~~OCPBUGSM-47085~~
Version:

$ openshift-install version
4.11.0-rc2

Platform:

Nutanix

On `openshift-installer create manifests` stage a connection to Prism is made (see https://github.com/openshift/installer/blob/master/pkg/asset/installconfig/nutanix/validation.go#L15-L36=)

This make generating manifests separately impossible, which breaks Assisted Installer flow. Instead of storing sensitive user information, Assisted Installer sets fake details in install-config.yaml and asks user to update these after installation has completed.

With validation happening on `openshift-install create manifests` phase installation process can't start with invalid credentials.

Please move this validation to ValidateForProvisioning, similar to vSphere

https://github.com/openshift/installer/pull/6251

Bug OCPBUGS-2745: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/ovn-kubernetes/pull/1334

Bug OCPBUGS-1099: Missing $SEARCH domain in /etc/resolve.conf for OCP v4.9.31 cluster

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-224~~. The following is the description of the original issue:
—
Description of problem:
OCP v4.9.31 cluster didn't have the $search domain in /etc/resolv.conf, which was there in the v4.8.29 OCP cluster. This was observed in all the nodes of the v4.9.31 cluster.
~~~
OpenShift 4.9.31
sh-4.4# cat /etc/resolv.conf

Generated by KNI resolv prepender NM dispatcher script
nameserver 172.xx.xx.xx
nameserver 10.xx.xx.xx
nameserver 10.xx.xx.xx
nameserver 10.xx.xx.xx

OpenShift 4.8.29

Generated by KNI resolv prepender NM dispatcher script
search sepia.lab.iad2.dc.paas.redhat.com
nameserver 172.xx.xx.xx
nameserver 10.xx.xx.xx
nameserver 10.xx.xx.xx
nameserver 10.xx.xx.xx
~~~

ENV: OpenStack IAD2, IPI installation. Connected cluster.

Version-Release number of selected component (if applicable):
OCP v4.9.31

How reproducible:
Always

Steps to Reproduce:
1. Install IPI cluster on OpenStack IAD2 platform having cluster version 4.9.31
2. Debug to any of the node(master/worker)
3. Check and confirm the missing search domain on all nodes of the cluster.

Actual results:
The search domain was missing when checked in `/etc/resolv.conf` file on all nodes of the cluster causing serious issues in the cluster.

Expected results:
The installer should embed the search domain in /etc/resolv.conf file on all nodes of the cluster.

Additional info:

Cu was trying to deploy secure Kerberos on the CoreOS nodes and it failed when the IPA-client install command failed. This is when the customer noticed this unusual behavior. They did not manually update the resolv.conf file to include the $search domain. They instead added the script below to /etc/NetworkManager/dispatcher.d/ and restarted NetworkManager on the node to fix this issue and installation was successful.
~~~
#!/bin/bash

set -eo pipefail

DISPATCHER_FILE="/etc/NetworkManager/dispatcher.d/30-resolv-prepender"
DOMAINS="$(grep -E '\s*DOMAINS=.*iad2.dc.paas.redhat.com' $DISPATCHER_FILE \

grep -oE '[a-z0-9]*.dev.iad2.dc.paas.redhat.com' \

tr '\n' ' ')"

>&2 echo "IT-PaaS: overwriting search domains in /etc/resolv.conf with: $DOMAINS"

sed -e "/^search/d" \
-e "/Generated by/c# Generated by KNI resolv prepender NM dispatcher script \nsearch $DOMAINS" \
/etc/resolv.conf > /etc/resolv.tmp

mv /etc/resolv.tmp /etc/resolv.conf
~~~

Cu confirms that the $search domain was missing since the cluster was freshly installed/ They even confirmed this with a fresh new cluster as well that it was missing.

The fresh cluster was initially installed at v4.9.31 but was updated afterward to v4.9.43 (the latest z-stream) to see if the updates fixed anything but it didn't make any difference. The cluster is currently running v4.9.43 and shows the $search domain missing in the /etc/resolv.conf file on all nodes.

https://github.com/openshift/machine-config-operator/pull/3330

Bug OCPBUGS-3490: Hosted ovnkube-master pods always fail during initial creation

View the Description View the linked PRs

While running a PerfScale test we noticed that the hosted ovnkube-master pods always initially error on deployment. They eventually succeed on retry however.

This is running quay.io/openshift-release-dev/ocp-release:4.11.11-x86_64 for the hosted clusters and the hypershift operator is quay.io/hypershift/hypershift-operator:4.11 on a 4.11.9 management cluster.

An example of the error in the ovnkube-master container:

```

F1102 13:27:51.935600 1 ovnkube.go:133] error when trying to initialize libovsdb SB client: unable to connect to any endpoints: failed to connect to ssl:ovnkube-master-0.ovnkube-master-internal.clusters-perf-pqd-0021.svc.cluster.local:9642: failed to open connection: dial tcp 10.131.8.25:9642: connect: connection refused. failed to connect to ssl:ovnkube-master-1.ovnkube-master-internal.clusters-perf-pqd-0021.svc.cluste

```

https://github.com/openshift/cluster-network-operator/pull/1641

Bug OCPBUGS-509: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/machine-config-operator/pull/3307

Bug OCPBUGS-2046: Remove policy/v1beta1 in 4.11 and later

View the Description View the linked PRs

~~OCPBUGS-1251~~ landed an admin-ack gate in 4.11.z to help admins prepare for Kubernetes 1.25 API removals which are coming in OpenShift 4.12. Poking around in a 4.12.0-ec.2 cluster where APIRemovedInNextReleaseInUse is firing:

$ oc --as system:admin adm must-gather -- /usr/bin/gather_audit_logs
$ zgrep -h v1beta1/poddisruptionbudget must-gather.local.1378724704026451055/quay*/audit_logs/kube-apiserver/*.log.gz | jq -r '.verb + " " + (.user | .username + " " + (.extra["authentication.kubernetes.io/pod-name"] | tostr
ing))' | sort | uniq -c
parse error: Invalid numeric literal at line 29, column 6
     28 watch system:serviceaccount:openshift-machine-api:cluster-autoscaler ["cluster-autoscaler-default-5cf997b8d6-ptgg7"]

Finding the source for that container:

$ oc --as system:admin -n openshift-machine-api get -o json pod cluster-autoscaler-default-5cf997b8d6-ptgg7 | jq -r '.status.containerStatuses[].image'
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f81ab7ce0c851ba5e5169bba717cb54716ce5457cbe89d159c97a5c25fd820ed
$ oc image info quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f81ab7ce0c851ba5e5169bba717cb54716ce5457cbe89d159c97a5c25fd820ed | grep github
             SOURCE_GIT_URL=https://github.com/openshift/kubernetes-autoscaler
             io.openshift.build.commit.url=https://github.com/openshift/kubernetes-autoscaler/commit/1dac0311b9842958ec630273428b74703d51c1c9
             io.openshift.build.source-location=https://github.com/openshift/kubernetes-autoscaler

Poking about in the source:

$ git clone --depth 30 --branch master https://github.com/openshift/kubernetes-autoscaler.git
$ cd kubernetes-autoscaler
$ find . -name vendor
./addon-resizer/vendor
./cluster-autoscaler/vendor
./vertical-pod-autoscaler/e2e/vendor
./vertical-pod-autoscaler/vendor

Lots of vendoring. I haven't checked to see how new the client code is in the various vendor packages. But the main issue seems to be the v1beta1 in:

$ git grep policy cluster-autoscaler/core cluster-autoscaler/utils | grep policy.*v1beta1
cluster-autoscaler/core/scaledown/actuation/actuator_test.go:   policyv1beta1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/core/scaledown/actuation/actuator_test.go:                                   eviction := createAction.GetObject().(*policyv1beta1.Eviction)
cluster-autoscaler/core/scaledown/actuation/drain.go:   policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/core/scaledown/actuation/drain_test.go:      policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/core/scaledown/legacy/legacy.go:     policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/core/scaledown/legacy/wrapper.go:    policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/core/scaledown/scaledown.go: policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/core/static_autoscaler_test.go:      policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/utils/drain/drain.go:        policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/utils/drain/drain_test.go:   policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/utils/kubernetes/listers.go: policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/utils/kubernetes/listers.go: v1policylister "k8s.io/client-go/listers/policy/v1beta1"

The main change from v1beta1 to v1 involves spec.selector; I dunno if that's relevant to the autoscaler use-case or not.

Do we run autoscaler CI? I was poking around a bit, but did not find a 4.12 periodic excercising the autoscaler that might have turned up this alert and issue.

https://github.com/openshift/kubernetes-autoscaler/pull/244

Bug OCPBUGS-8510: [4.11] egress firewall acls are deleted on restart

View the Description View the linked PRs

Description of problem:

During restart egress firewall acls will be deleted and re-created from scratch, meaning that egress firewall rules won't be applied for some time during restart

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/ovn-kubernetes/pull/1560

Bug OCPBUGS-3932: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/cloud-network-config-controller/pull/78

Bug OCPBUGS-5418: Dev Sandbox clusters uses clusterType OSD and there is no way to enforce DEVSANDBOX

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-5185~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-5165~~. The following is the description of the original issue:
—
Currently, the Dev Sandbox clusters sends the clusterType "OSD" instead of "DEVSANDBOX" because the configuration annotations of the console config are automatically overridden by some SyncSets.

Open Dev Sandbox and browser console and inspect window.SERVER_FLAGS.telemetry

https://github.com/openshift/console/pull/12413

Bug OCPBUGS-6002: Add more logs to "oc extract" in mco-first boot service

View the Description View the linked PRs

Description of problem:

In order to understand what is going on with OCPBUGS-5379 we want to add more logs

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/machine-config-operator/pull/3497

Bug OCPBUGS-2616: e2e-gcp-builds is permafailing

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-1237~~. The following is the description of the original issue:
—
job=pull-ci-openshift-origin-master-e2e-gcp-builds=all

This test has started permafailing on e2e-gcp-builds:

[sig-builds][Feature:Builds][Slow] s2i build with environment file in sources Building from a template should create a image from "test-env-build.json" template and run it in a pod [apigroup:build.openshift.io][apigroup:image.openshift.io]

The error in the test says

Sep 13 07:03:30.345: INFO: At 2022-09-13 07:00:21 +0000 UTC - event for build-test-pod: {kubelet ci-op-kg1t2x13-4e3c6-7hrm8-worker-a-66nwd} Pulling: Pulling image "image-registry.openshift-image-registry.svc:5000/e2e-test-build-sti-env-nglnt/test@sha256:262820fd1a94d68442874346f4c4024fdf556631da51cbf37ce69de094f56fe8"
Sep 13 07:03:30.345: INFO: At 2022-09-13 07:00:23 +0000 UTC - event for build-test-pod: {kubelet ci-op-kg1t2x13-4e3c6-7hrm8-worker-a-66nwd} Pulled: Successfully pulled image "image-registry.openshift-image-registry.svc:5000/e2e-test-build-sti-env-nglnt/test@sha256:262820fd1a94d68442874346f4c4024fdf556631da51cbf37ce69de094f56fe8" in 1.763914719s
Sep 13 07:03:30.345: INFO: At 2022-09-13 07:00:23 +0000 UTC - event for build-test-pod: {kubelet ci-op-kg1t2x13-4e3c6-7hrm8-worker-a-66nwd} Created: Created container test
Sep 13 07:03:30.345: INFO: At 2022-09-13 07:00:23 +0000 UTC - event for build-test-pod: {kubelet ci-op-kg1t2x13-4e3c6-7hrm8-worker-a-66nwd} Started: Started container test
Sep 13 07:03:30.345: INFO: At 2022-09-13 07:00:24 +0000 UTC - event for build-test-pod: {kubelet ci-op-kg1t2x13-4e3c6-7hrm8-worker-a-66nwd} Pulled: Container image "image-registry.openshift-image-registry.svc:5000/e2e-test-build-sti-env-nglnt/test@sha256:262820fd1a94d68442874346f4c4024fdf556631da51cbf37ce69de094f56fe8" already present on machine
Sep 13 07:03:30.345: INFO: At 2022-09-13 07:00:25 +0000 UTC - event for build-test-pod: {kubelet ci-op-kg1t2x13-4e3c6-7hrm8-worker-a-66nwd} Unhealthy: Readiness probe failed: Get "http://10.129.2.63:8080/": dial tcp 10.129.2.63:8080: connect: connection refused
Sep 13 07:03:30.345: INFO: At 2022-09-13 07:00:26 +0000 UTC - event for build-test-pod: {kubelet ci-op-kg1t2x13-4e3c6-7hrm8-worker-a-66nwd} BackOff: Back-off restarting failed container

https://github.com/openshift/origin/pull/27480

Bug OCPBUGS-4530: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/installer/pull/6668

Bug OCPBUGS-658: [release-4.11] OVN master trying to deleteLogicalPort for object which is already gone

View the Description View the linked PRs

Description of problem:
This is a clone of https://issues.redhat.com/browse/OCPBUGS-469

Description of problem: Numerous erroreneous logs in OVN master

I0823 18:00:11.163491 1 obj_retry.go:1063] Retry object setup: *v1.Pod openshift-operator-lifecycle-manager/collect-profiles-27687900-hlp6k
I0823 18:00:11.163546 1 obj_retry.go:1096] Removing old object: *v1.Pod openshift-operator-lifecycle-manager/collect-profiles-27687900-hlp6k
I0823 18:00:11.163555 1 pods.go:124] Deleting pod: openshift-operator-lifecycle-manager/collect-profiles-27687900-hlp6k
I0823 18:00:11.163631 1 obj_retry.go:1103] Retry delete failed for *v1.Pod openshift-operator-lifecycle-manager/collect-profiles-27687900-hlp6k, will try again later: deleteLogicalPort failed for pod openshift-operator-lifecycle-manager_collect-profiles-27687900-hlp6k: unable to locate portUUID+nodeName for pod openshift-operator-lifecycle-manager/collect-profiles-27687900-hlp6k: error getting logical port <nil>: object not found
W0823 18:00:41.163633 1 obj_retry.go:1031] Dropping retry entry for *v1.Pod openshift-operator-lifecycle-manager/collect-profiles-27687900-hlp6k: exceeded number of failed attempts

Must-gather: http://shell.lab.bos.redhat.com/~anusaxen/must-gather.local.2234927131259452300/

Version-Release number of selected component (if applicable): 4.12.0-0.nightly-2022-08-23-031342

How reproducible: Always

Steps to Reproduce:
1. Bring up OVN cluster on 4.12
2.
3.

Actual results: deleteLogicalPort failed for already gone object

Expected results: deleteLogicalPort should not keep retrying post object deletion

Additional info:

https://github.com/openshift/ovn-kubernetes/pull/1256

Bug OCPBUGS-6993: Hosted ovnkubernetes pods are not being spread among workers evenly

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-4238~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-3883~~. The following is the description of the original issue:
—
While doing a PerfScale test of we noticed that the ovnkube pods are not being spread out evenly among the available workers. Instead they are all stacking on a few until they fill up the available allocatable ebs volumes (25 in the case of m5 instances that we see here).

An example from partway through our 80 hosted cluster test when there were ~30 hosted clusters created/in progress

There are 24 workers available:

```

$ for i in `oc get nodes ~~l node-role.kubernetes.io/worker=,node-role.kubernetes.io/infra!=,node-role.kubernetes.io/workload!= | egrep -v "NAME" | awk '{ print $1 }'`; do echo $i `oc describe node $i | grep -v openshift~~ | grep ovnkube -c`; done
ip-10-0-129-227.us-west-2.compute.internal 0
ip-10-0-136-22.us-west-2.compute.internal 25
ip-10-0-136-29.us-west-2.compute.internal 0
ip-10-0-147-248.us-west-2.compute.internal 0
ip-10-0-150-147.us-west-2.compute.internal 0
ip-10-0-154-207.us-west-2.compute.internal 0
ip-10-0-156-0.us-west-2.compute.internal 0
ip-10-0-157-1.us-west-2.compute.internal 4
ip-10-0-160-253.us-west-2.compute.internal 0
ip-10-0-161-30.us-west-2.compute.internal 0
ip-10-0-164-98.us-west-2.compute.internal 0
ip-10-0-168-245.us-west-2.compute.internal 0
ip-10-0-170-103.us-west-2.compute.internal 0
ip-10-0-188-169.us-west-2.compute.internal 25
ip-10-0-188-194.us-west-2.compute.internal 0
ip-10-0-191-51.us-west-2.compute.internal 5
ip-10-0-192-10.us-west-2.compute.internal 0
ip-10-0-193-200.us-west-2.compute.internal 0
ip-10-0-193-27.us-west-2.compute.internal 7
ip-10-0-199-1.us-west-2.compute.internal 0
ip-10-0-203-161.us-west-2.compute.internal 0
ip-10-0-204-40.us-west-2.compute.internal 23
ip-10-0-220-164.us-west-2.compute.internal 0
ip-10-0-222-59.us-west-2.compute.internal 0

```

This is running quay.io/openshift-release-dev/ocp-release:4.11.11-x86_64 for the hosted clusters and the hypershift operator is quay.io/hypershift/hypershift-operator:4.11 on a 4.11.9 management cluster

https://github.com/openshift/cluster-network-operator/pull/1702

Bug OCPBUGS-1723: Node-exporter on a bare metal AMD EPYC setup (backport to 4.11)

View the Description View the linked PRs

This is a backport from https://issues.redhat.com/browse/OCPBUGS-1044

Description of problem:

https://github.com/prometheus/node_exporter/issues/2299

The node exporter pod when ran on a bare metal worker using an AMD EPYC CPU crashes and fails to start up and crashes with the following error message.    State:       Waiting
      Reason:    CrashLoopBackOff
    Last State:  Terminated
      Reason:    Error
      Message:   05.145Z caller=node_exporter.go:115 level=info collector=tapestats
ts=2022-09-07T20:25:05.145Z caller=node_exporter.go:115 level=info collector=textfile
ts=2022-09-07T20:25:05.145Z caller=node_exporter.go:115 level=info collector=thermal_zone
ts=2022-09-07T20:25:05.146Z caller=node_exporter.go:115 level=info collector=time
ts=2022-09-07T20:25:05.146Z caller=node_exporter.go:115 level=info collector=timex
ts=2022-09-07T20:25:05.146Z caller=node_exporter.go:115 level=info collector=udp_queues
ts=2022-09-07T20:25:05.146Z caller=node_exporter.go:115 level=info collector=uname
ts=2022-09-07T20:25:05.146Z caller=node_exporter.go:115 level=info collector=vmstat
ts=2022-09-07T20:25:05.146Z caller=node_exporter.go:115 level=info collector=xfs
ts=2022-09-07T20:25:05.146Z caller=node_exporter.go:115 level=info collector=zfs
ts=2022-09-07T20:25:05.146Z caller=node_exporter.go:199 level=info msg="Listening on" address=127.0.0.1:9100
ts=2022-09-07T20:25:05.146Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false
panic: "node_rapl_package-0-die-0_joules_total" is not a valid metric name

 Apparently this is a known issue (See Github link) and was fixed in a later upstream.

Version-Release number of selected component (if applicable):

4.11.0

How reproducible:

Every-time

Steps to Reproduce:

1. Provision a bare metal node using an AMD EPYC CPU
2. Node-exporter pods that try to start on the nodes will crash with error message

Actual results:

Node-exporter pods cannot run on the new nodes

Expected results:

Node exporter pods should be able to start up and run like on any other node

Additional info:

As mentioned above this issue was tracked and fixed in a later upstream of node-exporter

https://github.com/prometheus/node_exporter/issues/2299

Would we be able to get the fixed version pulled for 4.11?

https://github.com/openshift/node_exporter/pull/108

Bug OCPBUGS-2809: Project auth cache sync blocks list handler

View the Description View the linked PRs

An RW mutex was introduced to the project auth cache with https://github.com/openshift/openshift-apiserver/pull/267, taking exclusive access during cache syncs. On clusters with extremely high object counts for namespaces and RBAC, syncs appear to be extremely slow (on the order of several minutes). The project LIST handler acquires the same mutex in shared mode as part of its critical path.

https://github.com/openshift/openshift-apiserver/pull/327

Bug OCPBUGS-378: downloading govc is impacted by github rate limiting

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-262~~. The following is the description of the original issue:
—
github rate limit failures for upi image downloading govc.

https://search.ci.openshift.org/?search=build+error%3A+error+building+at+STEP+%22RUN+curl+-L+-o&maxAge=48h&context=1&type=all&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

https://github.com/openshift/installer/pull/6246

Bug OCPBUGS-2515: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/console/pull/12182

Bug OCPBUGS-7322: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/machine-api-provider-openstack/pull/57

Bug OCPBUGS-4655: [4.11] ovn-kubernetes ovnkube-master containers crashlooping after 4.11.0-0.okd-2022-10-15-073651 update

View the Description View the linked PRs

Description of problem:

The ovn-kubernetes ovnkube-master containers are continuously crashlooping since we updated to 4.11.0-0.okd-2022-10-15-073651.

Log Excerpt:

] [] []  [{kubectl-client-side-apply Update networking.k8s.io/v1 2022-09-12 12:25:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:ingress":{},"f:policyTypes":{}}} }]},Spec:NetworkPolicySpec{PodSelector:{map[] []},Ingress:[]NetworkPolicyIngressRule{NetworkPolicyIngressRule{Ports:[]NetworkPolicyPort{},From:[]NetworkPolicyPeer{NetworkPolicyPeer{PodSelector:&v1.LabelSelector{MatchLabels:map[string]string{access: true,},MatchExpressions:[]LabelSelectorRequirement{},},NamespaceSelector:nil,IPBlock:nil,},},},},Egress:[]NetworkPolicyEgressRule{},PolicyTypes:[Ingress],},} &NetworkPolicy{ObjectMeta:{allow-from-openshift-ingress  compsci-gradcentral  a405f843-c250-40d7-8dd4-a759f764f091 217304038 1 2022-09-22 14:36:38 +0000 UTC <nil> <nil> map[] map[] [] []  [{openshift-apiserver Update networking.k8s.io/v1 2022-09-22 14:36:38 +0000 UTC FieldsV1 {"f:spec":{"f:ingress":{},"f:policyTypes":{}}} }]},Spec:NetworkPolicySpec{PodSelector:{map[] []},Ingress:[]NetworkPolicyIngressRule{NetworkPolicyIngressRule{Ports:[]NetworkPolicyPort{},From:[]NetworkPolicyPeer{NetworkPolicyPeer{PodSelector:nil,NamespaceSelector:&v1.LabelSelector{MatchLabels:map[string]string{policy-group.network.openshift.io/ingress: ,},MatchExpressions:[]LabelSelectorRequirement{},},IPBlock:nil,},},},},Egress:[]NetworkPolicyEgressRule{},PolicyTypes:[Ingress],},}]: cannot clean up egress default deny ACL name: error in transact with ops [{Op:mutate Table:Port_Group Row:map[] Rows:[] Columns:[] Mutations:[{Column:acls Mutator:delete Value:{GoSet:[{GoUUID:60cb946a-46e9-4623-9ba4-3cb35f018ed6}]}}] Timeout:<nil> Where:[where column _uuid == {ccdd01bf-3009-42fb-9672-e1df38190cd7}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:mutate Table:Port_Group Row:map[] Rows:[] Columns:[] Mutations:[{Column:acls Mutator:delete Value:{GoSet:[{GoUUID:60cb946a-46e9-4623-9ba4-3cb35f018ed6}]}}] Timeout:<nil> Where:[where column _uuid == {10bbf229-8c1b-4c62-b36e-4ba0097722db}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:delete Table:ACL Row:map[] Rows:[] Columns:[] Mutations:[] Timeout:<nil> Where:[where column _uuid == {7b55ba0c-150f-4a63-9601-cfde25f29408}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:delete Table:ACL Row:map[] Rows:[] Columns:[] Mutations:[] Timeout:<nil> Where:[where column _uuid == {60cb946a-46e9-4623-9ba4-3cb35f018ed6}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:}] results [{Count:1 Error: Details: UUID:{GoUUID:} Rows:[]} {Count:1 Error: Details: UUID:{GoUUID:} Rows:[]} {Count:1 Error: Details: UUID:{GoUUID:} Rows:[]} {Count:1 Error: Details: UUID:{GoUUID:} Rows:[]} {Count:0 Error:referential integrity violation Details:cannot delete ACL row 7b55ba0c-150f-4a63-9601-cfde25f29408 because of 1 remaining reference(s) UUID:{GoUUID:} Rows:[]}] and errors []: referential integrity violation: cannot delete ACL row 7b55ba0c-150f-4a63-9601-cfde25f29408 because of 1 remaining reference(s)

Additional info:

https://github.com/okd-project/okd/issues/1372

Issue persisted through update to 4.11.0-0.okd-2022-10-28-153352

must-gather: https://nbc9-snips.cloud.duke.edu/snips/must-gather.local.2859117512952590880.zip

https://github.com/openshift/ovn-kubernetes/pull/1444

Bug OCPBUGS-7530: [4.11] Bootimage bump tracker

View the Description View the linked PRs

Tracker issue for bootimage bump in 4.11. This issue should block issues which need a bootimage bump to fix.

The previous bump was ~~OCPBUGS-3362~~.

https://github.com/openshift/installer/pull/6875

Bug OCPBUGS-1326: backport opm serve cache commands

View the Description View the linked PRs

During initial backporting, due to a number of other colliding commits in upstream, the cobra commands facilitating caching did not get downstreamed.

This is to downstream those two lines.

https://github.com/openshift/operator-framework-olm/pull/382

Bug OCPBUGS-3876: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/operator-framework-olm/pull/413

Bug OCPBUGS-4185: attachable-volumes-aws-ebs incorrect for AWS m* instance types

View the Description View the linked PRs

Following the trail
https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/1139
https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/1175
https://github.com/openshift/aws-ebs-csi-driver/pull/206

Looks like the fix should be in 4.12, but it still see it being 39 vs ~24 on an m6i instance type.

It seems that the kubelet applies this capacity to the node in 4.11 and earlier and, thus, unlikely to receive this fix for attachable volumes in the upstream CSI driver. 4.12 behavior is currently unknown but it seems that the kubelet might still be setting this capacity.

The actual issue is that kube scheduler schedules pods that require PVs to nodes where those PVs can not be attached.

https://github.com/openshift/aws-ebs-csi-driver/pull/213

Bug OCPBUGS-8399: Traffic from egress IPs was interrupted after Cluster patch to Openshift 4.10.46

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-7474~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-6714~~. The following is the description of the original issue:
—
Description of problem:

Traffic from egress IPs was interrupted after Cluster patch to Openshift 4.10.46

a customer cluster was patched. It is an Openshift 4.10.46 cluster with SDN.

More description about issue is available in private comment below since it contains customer data.

https://github.com/openshift/sdn/pull/513

Bug OCPBUGS-9921: Image registry panics while deploying OCP in ap-southeast-4 AWS region

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-8491~~. The following is the description of the original issue:
—
Description of problem:

Image registry pods panic while deploying OCP in ap-southeast-4 AWS region

Version-Release number of selected component (if applicable):

4.12.0

How reproducible:

Deploy OCP in AWS ap-southeast-4 region

Steps to Reproduce:

Deploy OCP in AWS ap-southeast-4 region

Actual results:

panic: Invalid region provided: ap-southeast-4

Expected results:

Image registry pods should come up with no errors

Additional info:

Story MON-1913: Expose field in CMO configmap to configure the retention period of Thanos Ruler

View the Description View the linked PRs

Users can't configure the retention period for Thanos Ruler currently and the default value is 24h (from the prometheus operator).

https://github.com/openshift/cluster-monitoring-operator/pull/1651

Task MGMT-9440: Fix single-node serial tests API crash

View the Description View the linked PRs

See these threads https://coreos.slack.com/archives/G01F05P2PTL/p1645982017061749?thread_ts=1645970469.871559&cid=G01F05P2PTL for more information

Bug OCPBUGS-2092: Use floating tags in golang imagestream

View the Description View the linked PRs

As discussed previously by email, customer support case 03211616 requests a means to use the latest patch version of a given X.Y golang via imagestreams, with the blocking issue being the lack of X.Y tags for the go-toolset containers on RHCC.

The latter has now been fixed with the latest version also getting a :1.17 tag, and the imagestream source has been modified accordingly, which will get picked up in 4.12. We can now fix this in 4.11.z by backporting this to the imagestream files bundled in cluster-samples-operator.

/cc Ian Watson Feny Mehta

https://github.com/openshift/cluster-samples-operator/pull/467

Bug OCPBUGS-2113: [4.11] etcd and kube-apiserver pods get restarted due to failed liveness probes while deleting/re-creating pods on SNO

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-1329~~. The following is the description of the original issue:
—
Description of problem:

etcd and kube-apiserver pods get restarted due to failed liveness probes while deleting/re-creating pods on SNO

Version-Release number of selected component (if applicable):

4.10.32

How reproducible:

Not always, after ~10 attempts

Steps to Reproduce:

1. Deploy SNO with Telco DU profile applied
2. Create multiple pods with local storage volumes attached(attaching yaml manifest)
3. Force delete and re-create pods 10 times

Actual results:

etcd and kube-apiserver pods get restarted, making to cluster unavailable for a period of time

Expected results:

etcd and kube-apiserver do not get restarted

Additional info:

Attaching must-gather.

Please let me know if any additional info is required. Thank you!

https://github.com/openshift/cluster-etcd-operator/pull/943

Bug OCPBUGS-4099: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/machine-config-operator/pull/3434

Bug OCPBUGS-6640: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/ovn-kubernetes/pull/1491

Bug OCPBUGS-2938: Duplicate prometheus rules for API SLOs after upgrade

View the Description View the linked PRs

This bug is a backport clone of [Bugzilla Bug 2094362](https://bugzilla.redhat.com/show_bug.cgi?id=2094362). The following is the description of the original bug:
—
Description of problem:
A change [1] was introduced to split the kube-apiserver SLO rules into 2 groups to reduce the load on Prometheus (see bug 2004585).

[1] https://github.com/openshift/cluster-kube-apiserver-operator/commit/4a1751ee86cda37f0d9ea520cac09f91ebc3abe9

Version-Release number of selected component (if applicable):
4.9 (because the change was backported to 4.9.z)

How reproducible:
Always

Steps to Reproduce:
1. Install OCP 4.9
2. Retrieve kube-apiserver-slos*
oc get -n openshift-kube-apiserver prometheusrules kube-apiserver-slos -o yaml
oc get -n openshift-kube-apiserver prometheusrules kube-apiserver-slos-basic -o yaml

Actual results:

The KubeAPIErrorBudgetBurn alert with labels

{long="1h",namespace="openshift-kube-apiserver",severity="critical",short="5m"}

exists both in kube-apiserver-slos and kube-apiserver-slos-basic.

The alerting rules is evaluated twice. The same is true for recording rules like "apiserver_request:burnrate1h" and in this case, it can trigger warning logs in the Prometheus pods:

> level=warn component="rule manager" group=kube-apiserver.rules msg="Error on ingesting out-of-order result from rule evaluation" numDropped=283

Expected results:

I presume that kube-apiserver-slos shouldn't exist since it's been replaced by kube-apiserver-slos-basic and kube-apiserver-slos-extended.

Additional info:

Discovered while investigating bug 2091902

https://github.com/openshift/cluster-kube-apiserver-operator/pull/1397

Bug OCPBUGS-562: [4.11] Bootimage bump tracker

View the Description View the linked PRs

Tracker bug for bootimage bump in 4.11. This bug should block bugs which need a bootimage bump to fix.

https://github.com/openshift/installer/pull/6459

Bug OCPBUGS-1939: [4.11] specify `add_inheritable_capabilities` on upgrades from 4.11

View the Description View the linked PRs

4.12 will have an option in cri-o: add_inheritable_capabilities which will allow a user to opt-out of dropping inheritable capabilities (which comes as a fix for CVE-2022-27652). We should add it by default as a drop-in in 4.11 so clusters that upgrade from it inherit the old behavior

https://github.com/openshift/machine-config-operator/pull/3352

Bug OCPBUGS-6046: Sync stable branch for CPO release-1.24 into release-4.11

View the Description View the linked PRs

Description of problem:

release-4.11 of openshift/cloud-provider-openstack is missing some commits that were backported in upstream project into the release-1.24 branch.
We should import them in our downstream fork.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/cloud-provider-openstack/pull/166

Bug OCPBUGS-6920: Tracker: Configure ignored namespaces into multus-admission-controller (4.11,CNO)

View the Description View the linked PRs

Description of problem: Issue described in following issue: https://github.com/openshift/multus-admission-controller/issues/40

Fixed in: https://github.com/openshift/cluster-network-operator/pull/1515

Version-Release number of selected component (if applicable): OCP 4.10

Official Red Hat tracker. Issue has been merged already.

https://github.com/openshift/cluster-network-operator/pull/1698

Bug OCPBUGS-7319: [4.11] [OVNK] Add support for service session affinity timeout

View the Description View the linked PRs

Description of problem:

For OVNK to become CNCF complaint, we need to support session affinity timeout feature and enable the e2e's on OpenShift side. This bug tracks the efforts to get this into 4.12 OCP.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

Bug OCPBUGS-1497: [4.11.z] oc adm inspect --rotated-pod-logs not working properly for static pods

View the Description View the linked PRs

Description of problem:

The path used by --rotated-pod-logs to gather the rotated pod logs from /var/log/pods node folder via /api/v1/nodes/${NODE}/proxy/logs/${LOG_PATH} is only valid for regular pods but not for static pods.

The main problem is that, while normal pods have their rotated logs at this /var/log/pods/${POD_NAME}_${POD_UID_IN_API}/${CONTAINER_NAME}, static pods have them at /var/log/pods/${POD_NAME}_${CONFIG_HASH}/${CONTAINER_NAME} because the UID cannot be known at the time that the static pod is born (because static pods are created by kubelet before registering them in the kube-apiserver, and UID is assigned by the kube-apiserver).

The visible results of that are:

Spurious errors of not found resources related to the pods.
Rotated pod logs are not gathered even if present.

Version-Release number of selected component (if applicable):

4.10

How reproducible:

Always if there are static pods.

Steps to Reproduce:

1. oc adm inspect --rotated-pod-logs ns/openshift-etcd (or any other project with static pods).

Actual results:

Rotated pods not gathered.

Errors like these

error: errors occurred while gathering data:
    one or more errors occurred while gathering pod-specific data for namespace: openshift-etcd

    [one or more errors occurred while gathering container data for pod etcd-master-0.example.net:

    the server could not find the requested resource, one or more errors occurred while gathering container data for pod etcd-master-1.example.net:

    the server could not find the requested resource, one or more errors occurred while gathering container data for pod etcd-master-2.example.net:

    the server could not find the requested resource]

Expected results:

No errors like the ones above and rotated pod logs to be gathered, if present.

Additional info:

Despite being marked as experimental, this --rotated-pod-logs is used in must-gather, so this issue can be easily reproduced by just running a default must-gather. I focused on bare oc adm inspect reproducers for simplicity.

https://github.com/openshift/oc/pull/1239

Bug OCPBUGS-4446: service account token secret reference

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-1428~~. The following is the description of the original issue:
—
Description of problem:

When using an OperatorGroup attached to a service account, AND if there is a secret present in the namespace, the operator installation will fail with the message:
the service account does not have any API secret sa=testx-ns/testx-sa
This issue seems similar to https://bugzilla.redhat.com/show_bug.cgi?id=2094303 - which was resolved in 4.11.0 - however, the new element now, is that the presence of a secret in the namespace  is causing the issue.
The name of the secret seems significant - suggesting something somewhere is depending on the order that secrets are listed in. For example, If the secret in the namespace is called "asecret", the problem does not occur. If it is called "zsecret", the problem always occurs.

"zsecret" is not a "kubernetes.io/service-account-token". The issue I have raised here relates to Opaque secrets - zsecret is an Opaque secret. The issue may apply to other types of secrets, but specifically my issue is that when there is an opaque secret present in the namespace, the operator install fails as described. I aught to be allowed to have an opaque secret present in the namespace where I am installing the operator.

Version-Release number of selected component (if applicable):

4.11.0 & 4.11.1

How reproducible:

100% reproducible

Steps to Reproduce:

1.Create namespace: oc new-project testx-ns
2. oc apply -f api-secret-issue.yaml

Actual results:

Expected results:

Additional info:

API YAML:

cat api-secret-issue.yaml
apiVersion: v1
kind: Secret
metadata:
name: zsecret
namespace: testx-ns
annotations:
kubernetes.io/service-account.name: testx-sa
type: Opaque
stringData:
mykey: mypass

—

apiVersion: v1
kind: ServiceAccount
metadata:
name: testx-sa
namespace: testx-ns

—

kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
name: testx-og
namespace: testx-ns
spec:
serviceAccountName: "testx-sa"
targetNamespaces:
- testx-ns

—

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: testx-role
namespace: testx-ns
rules:

apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]

—

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: testx-rolebinding
namespace: testx-ns
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: testx-role
subjects:

kind: ServiceAccount
name: testx-sa
namespace: testx-ns

—

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: etcd-operator
namespace: testx-ns
spec:
channel: singlenamespace-alpha
installPlanApproval: Automatic
name: etcd
source: community-operators
sourceNamespace: openshift-marketplace

https://github.com/openshift/operator-framework-olm/pull/416

Bug OCPBUGS-4551: Backport 4.11: Guard Pod Hostnames Too Long and Truncated Down Into Collisions With Other Masters

View the Description View the linked PRs

Discovered in the must gather kubelet_service.log from https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-gcp-sdn-upgrade/1586093220087992320

It appears the guard pod names are too long, and being truncated down to where they will collide with those from the other masters.

From kubelet logs in this run:

❯ grep openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-maste kubelet_service.log
Oct 28 23:58:55.693391 ci-op-3hj6pnwf-4f6ab-lv57z-master-1 kubenswrapper[1657]: E1028 23:58:55.693346    1657 kubelet_pods.go:413] "Hostname for pod was too long, truncated it" podName="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-master-1" hostnameMaxLen=63 truncatedHostname="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-maste"
Oct 28 23:59:03.735726 ci-op-3hj6pnwf-4f6ab-lv57z-master-0 kubenswrapper[1670]: E1028 23:59:03.735671    1670 kubelet_pods.go:413] "Hostname for pod was too long, truncated it" podName="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-master-0" hostnameMaxLen=63 truncatedHostname="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-maste"
Oct 28 23:59:11.168082 ci-op-3hj6pnwf-4f6ab-lv57z-master-2 kubenswrapper[1667]: E1028 23:59:11.168041    1667 kubelet_pods.go:413] "Hostname for pod was too long, truncated it" podName="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-master-2" hostnameMaxLen=63 truncatedHostname="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-maste"

This also looks to be happening for openshift-kube-scheduler-guard, kube-controller-manager-guard, possibly others.

Looks like they should be truncated further to make room for random suffixes in https://github.com/openshift/library-go/blame/bd9b0e19121022561dcd1d9823407cd58b2265d0/pkg/operator/staticpod/controller/guard/guard_controller.go#L97-L98

Unsure of the implications here, it looks a little scary.

Bug OCPBUGS-7789: [4.11.z] Fix kubevirt-console tests

View the Description View the linked PRs

Description of problem:

[4.11.z] Fix kubevirt-console tests

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/console/pull/12325

Bug OCPBUGS-6492: Inconsistent user expeience for creating new projects after persmissions have been revoked

View the Description View the linked PRs

Description of problem:

Remove the self-provisioner role for the system authenticated users as per  https://access.redhat.com/solutions/4040541 to stop users from having the ability to create new projects, but the customer has found this is only partially working. It appears that when you use cluster Web UI Administrator view, the "Create Project" button is not available but switching to the default Developer view default user can create a project

Version-Release number of selected component (if applicable):

How reproducible:

Follow https://access.redhat.com/solutions/1529893

Steps to Reproduce:

1. oc adm policy remove-cluster-role-from-group self-provisioner system:authenticated:oauth
2. log back in as user and switch between admin/Dev view
3. User still has link showing in Dev console

Actual results:

Create new project link still exists

Expected results:

Create new project link should be removed, similar to Admin Console

Additional info:

Although the loink still exists, the user get's a correct permission denied message.

https://github.com/openshift/console/pull/12514

Bug OCPBUGS-4410: [4.13] Improve ironic logging configuration in metal3

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-4311~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-4305~~. The following is the description of the original issue:
—
Description of problem:

Please add an option to DISABLE debug in ironic-api. Presently it is enabled by default and there is no way to disable it or reduce log level

https://github.com/metal3-io/ironic-image/blob/main/ironic-config/ironic.conf.j2#L3

Version-Release number of selected component (if applicable): none

How reproducible: Every time

Steps to Reproduce:

Please check source code here: https://github.com/metal3-io/ironic-image/blob/main/ironic-config/ironic.conf.j2#L3

It is enabled by default and there is no way to disable it or reduce log level

Actual results:

Please check Case: 03371411, the log file grew to 409 GB

Expected results: Need a way to disable debug

Additional info: Case 03371411. A cluster must gather and log file can be found in the case.

https://github.com/openshift/ironic-image/pull/328

Bug OCPBUGS-688: Adding day2 remote worker node requires manually approving CSRs

View the Description View the linked PRs

Description of problem: defined in https://bugzilla.redhat.com/show_bug.cgi?id=2051533

When adding remote worker node using ZTP the agent finishes the installation and is marked as done. 
oc get agent -o wide
NAME                                   CLUSTER   APPROVED   ROLE     STAGE   HOSTNAME                                      REQUESTED HOSTNAME
0277804e-2a7c-4d95-9d0f-e22a190d582a   spoke-0   true       worker   Done    spoke-worker-0-0.spoke-0.qe.lab.redhat.com    spoke-worker-0-0
12efa520-5b99-4474-805d-931e46ad43f7   spoke-0   true       master   Done    spoke-master-0-2.spoke-0.qe.lab.redhat.com    spoke-master-0-2
3b8eec89-f26f-4896-8f71-8a810894c560   spoke-0   true       master   Done    spoke-master-0-0.spoke-0.qe.lab.redhat.com    spoke-master-0-0
3fb3749e-c132-4258-ad1a-08a0445c9022   spoke-0   true       worker   Done    spoke-worker-0-1.spoke-0.qe.lab.redhat.com    spoke-worker-0-1
728559e9-5543-41d9-adb0-e58196f765af   spoke-0   true       master   Done    spoke-master-0-1.spoke-0.qe.lab.redhat.com    spoke-master-0-1
982e1ff6-6e83-4800-b061-8cdfd0b844fb   spoke-0   true       worker   Done    spoke-rwn-0-1.spoke-rwn-0.qe.lab.redhat.com   spoke-rwn-0-1
a76eaa6a-b351-429f-bfa1-e53a70503573   spoke-0   true       worker   Done    spoke-rwn-0-0.spoke-rwn-0.qe.lab.redhat.com   spoke-rwn-0-0



Logging into the spoke cluster the bmh and machine resources are created and the node resource is not:
oc get bmh -n openshift-machine-api
NAME                STATE                    CONSUMER                       ONLINE   ERROR                            AGE
spoke-master-0-0    unmanaged                spoke-0-pxbfh-master-0         true                                      3h32m
spoke-master-0-1    unmanaged                spoke-0-pxbfh-master-1         true                                      3h32m
spoke-master-0-2    unmanaged                spoke-0-pxbfh-master-2         true                                      3h32m
spoke-rwn-0-0-bmh   externally provisioned   spoke-0-spoke-rwn-0-0-bmh      true     provisioned registration error   168m
spoke-rwn-0-1-bmh   externally provisioned   spoke-0-spoke-rwn-0-1-bmh      true     provisioned registration error   168m
spoke-worker-0-0    unmanaged                spoke-0-pxbfh-worker-0-65mrb   true                                      3h32m
spoke-worker-0-1    unmanaged                spoke-0-pxbfh-worker-0-nnmcq   true                                      3h32m     

 oc get machine -n openshift-machine-api
NAME                           PHASE         TYPE   REGION   ZONE   AGE
spoke-0-pxbfh-master-0         Running                              3h33m
spoke-0-pxbfh-master-1         Running                              3h33m
spoke-0-pxbfh-master-2         Running                              3h33m
spoke-0-pxbfh-worker-0-65mrb   Running                              3h19m
spoke-0-pxbfh-worker-0-nnmcq   Running                              3h20m
spoke-0-spoke-rwn-0-0-bmh      Provisioned                          169m
spoke-0-spoke-rwn-0-1-bmh      Provisioned                          169m

Note: bmh is in error state:
Normal  ProvisionedRegistrationError  30m   metal3-baremetal-controller  Host adoption failed: Error while attempting to adopt node 529b3e75-5d04-4486-9296-269081d0ec02: Error validating Redfish virtual media. Some parameters were missing in node's driver_info. Missing are: ['deploy_kernel', 'deploy_ramdisk'].

oc get nodes
NAME                                         STATUS   ROLES    AGE   VERSION
spoke-master-0-0.spoke-0.qe.lab.redhat.com   Ready    master   72m   v1.22.3+2cb6068
spoke-master-0-1.spoke-0.qe.lab.redhat.com   Ready    master   50m   v1.22.3+2cb6068
spoke-master-0-2.spoke-0.qe.lab.redhat.com   Ready    master   72m   v1.22.3+2cb6068
spoke-worker-0-0.spoke-0.qe.lab.redhat.com   Ready    worker   51m   v1.22.3+2cb6068
spoke-worker-0-1.spoke-0.qe.lab.redhat.com   Ready    worker   51m   v1.22.3+2cb6068


node-bootstrapper CSR is created but not auto-approved; periodically another node-strapper csr is created until it is manually approved:

oc get csr | grep Pending
csr-5ll2g                                        9m9s    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Pending
csr-f8vbl                                        8m24s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Pending

Version-Release number of selected component (if applicable):

assisted-service master at revision af0bafb3f7f629932f8c3dc31ccddedfe6984926
ocp version: 4.10.0-rc.1

How reproducible:

1. Install remote worker node using ztp

2. Wait for node resource to be created

Steps to Reproduce:

1. Install remote worker node using ztp

2. Wait for node resource to be created

Actual results:

node-bootstrapper and node CSR are not auto-approved and node resource is not created.  The bmh resource remains in registration error

Expected results:

node-bootstrapper and node CSR should be auto-approved and node resource created.  The bmh resource should not be in registration error

Additional info:

https://github.com/openshift/machine-config-operator/pull/3305

Bug OCPBUGS-6622: The MCO can generate a rendered config with old KubeletConfig contents, blocking upgrades

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-6018~~. The following is the description of the original issue:
—
This is a public clone of OCPBUGS-3821

The MCO can sometimes render a rendered-config in the middle of an upgrade with old MCs, e.g.:

the containerruntimeconfigcontroller creates a new containerruntimeconfig due to the update
the template controller finishes re-creating the base configs
the kubeletconfig errors long enough and doesn't finish until after 2

This will cause the render controller to create a new rendered MC that uses the OLD kubeletconfig-MC, which at best is a double reboot for 1 node, and at worst block the update and break maxUnavailable nodes per pool.

https://github.com/openshift/machine-config-operator/pull/3511

Bug OCPBUGS-749: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/image-registry/pull/343

Bug OCPBUGS-1104: package-server-manager does not migrate packageserver CSV from v0.17.0 to v0.18.3 on OCP 4.8 -> 4.9 upgrade

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-858~~. The following is the description of the original issue:
—
Description of problem:

In OCP 4.9, the package-server-manager was introduced to manage the packageserver CSV. However, when OCP 4.8 in upgraded to 4.9, the packageserver stays stuck in v0.17.0, which is the version in OCP 4.8, and v0.18.3 does not roll out, which is the version in OCP 4.9

Version-Release number of selected component (if applicable):

How reproducible:

Always

Steps to Reproduce:

1. Install OCP 4.8

2. Upgrade to OCP 4.9 

$ oc get clusterversion 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2022-08-31-160214   True        True          50m     Working towards 4.9.47: 619 of 738 done (83% complete)

$ oc get clusterversion 
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.47    True        False         4m26s   Cluster version is 4.9.47

Actual results:

Check packageserver CSV. It's in v0.17.0 

$ oc get csv  NAME            DISPLAY          VERSION   REPLACES   PHASE packageserver   Package Server   0.17.0               Succeeded

Expected results:

packageserver CSV is at 0.18.3

Additional info:

packageserver CSV version in 4.8: https://github.com/openshift/operator-framework-olm/blob/release-4.8/manifests/0000_50_olm_15-packageserver.clusterserviceversion.yaml#L12

packageserver CSV version in 4.9: https://github.com/openshift/operator-framework-olm/blob/release-4.9/pkg/manifests/csv.yaml#L8

https://github.com/openshift/operator-framework-olm/pull/381

Bug OCPBUGS-2554: ingress, authentication and console operator goes to degraded after switching default application router scope

View the Description View the linked PRs

Description of problem:
Switching the spec.endpointPublishingStrategy.loadBalancer.scope of the default ingresscontroller results in a degraded ingress operator. The routes using that endpoint like the console URL become inaccessible.
Degraded operators after scope change:

$ oc get co | grep -v ' True        False         False'
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.11.4    False       False         True       72m     OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.kartrosa.ukld.s1.devshift.org/healthz": EOF
console                                    4.11.4    False       False         False      72m     RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.kartrosa.ukld.s1.devshift.org): Get "https://console-openshift-console.apps.kartrosa.ukld.s1.devshift.org": EOF
ingress                                    4.11.4    True        False         True       65m     The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)

We have noticed that each time this happens the underlying AWS loadbalancer gets recreated which is as expected however the router pods probably do not get notified about the new loadbalancer. The instances in the new loadbalancer become 'outOfService'.

Restarting one of the router pods fixes the issue and brings back a couple of instances under the loadbalancer back to 'InService' which leads to the operators becoming happy again.

Version-Release number of selected component (if applicable):

ingress in 4.11.z however we suspect this issue to also apply to older versions

How reproducible:

Consistently reproducible

Steps to Reproduce:

1. Create a test OCP 4.11 cluster in AWS
2. Switch the spec.endpointPublishingStrategy.loadBalancer.scope of the default ingresscontroller in openshift-ingress-operator to Internal from External (or vice versa)
3. New Loadbalancer is created in AWS for the default router service, however the instances behind are not in service

Actual results:

ingress, authentication and console operators go into a degraded state. Console URL of the cluster is inaccessible

Expected results:

The ingresscontroller scope transition from internal->External (or vice versa) is smooth without any downtime or operators going into degraded state. The console is accessible.

https://github.com/openshift/ovn-kubernetes/pull/1389

Bug OCPBUGS-5472: [vsphere-CSI-Driver-Operator] The storageclass "thin-csi" could not be re-created after deleting

View the Description View the linked PRs

Description of problem:

The storageclass "thin-csi" is created by vsphere-CSI-Driver-Operator, after deleting it manually, it should be re-created immediately.

Version-Release number of selected component (if applicable):

4.11.4

How reproducible:

Always

Steps to Reproduce:

1. Check storageclass in running cluster, thin-csi is present:
$ oc get sc 
NAME             PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
thin (default)   kubernetes.io/vsphere-volume   Delete          Immediate              false                  41m
thin-csi         csi.vsphere.vmware.com         Delete          WaitForFirstConsumer   true                   38m

2. Delete thin-csi storageclass:
$ oc delete sc thin-csi
storageclass.storage.k8s.io "thin-csi" deleted

3. Check storageclass again, thin-csi is not present:
$ oc get sc
NAME             PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
thin (default)   kubernetes.io/vsphere-volume   Delete          Immediate           false                  50m

4. Check vmware-vsphere-csi-driver-operator log:
......
I0909 03:47:42.172866       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1662695014\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1662695014\" (2022-09-09 02:43:34 +0000 UTC to 2023-09-09 02:43:34 +0000 UTC (now=2022-09-09 03:47:42.172853123 +0000 UTC))"I0909 03:49:38.294962       
1 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOFI0909 03:49:38.295468       
1 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOFI0909 03:49:38.295765       
1 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF

5. Only first time creating in vmware-vsphere-csi-driver-operator log:
$ oc -n openshift-cluster-csi-drivers logs vmware-vsphere-csi-driver-operator-7cc6d44b5c-c8czw | grep -i "storageclass"I0909 03:46:31.865926   1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-csi-drivers", Name:"vmware-vsphere-csi-driver-operator", UID:"9e0c3e2d-d403-40a1-bf69-191d7aec202b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'StorageClassCreated' Created StorageClass.storage.k8s.io/thin-csi because it was missing

Actual results:

The storageclass "thin-csi" could not be re-created after deleting

Expected results:

The storageclass "thin-csi" should be re-created after deleting

Additional info:

https://github.com/openshift/vmware-vsphere-csi-driver-operator/pull/127

Bug OCPBUGS-1720: [4.11] ovnkube-trace does wrong validation for outport on shared gateway

View the Description View the linked PRs

Description of problem:

Whenever one runs ovnkube-trace from an in-cluster pod to a pod in the host network that is in different node, the following spurious error appears despite of the underlying ovn-trace being correct:

ovn-trace indicates failure from ingress-canary-7zhxs to router-default-6758fb465c-s66rv - output to "k8s-worker-0.example.redhat.com" not matched

This is caused because as per[1], if the destination pod is in host network, the outport is expected to be of the form "k8s-${NODE_NAME}", which is true only if either in local gateway or if the source pod is in the same node than the destination pod.

This is already fixed in the master branch[2], but we would need this to be backported to previous releases.

Version-Release number of selected component (if applicable):

4.11.4

How reproducible:

Always

Steps to Reproduce:

1. ovnkube-trace from pod in the SDN to pod in host network
2.
3.

Actual results:

Wrong error

Expected results:

No wrong error

Additional info:

References:
[1] - https://github.com/openshift/ovn-kubernetes/blob/release-4.11/go-controller/cmd/ovnkube-trace/ovnkube-trace.go#L771-L777
[2] - https://github.com/openshift/ovn-kubernetes/blob/master/go-controller/cmd/ovnkube-trace/ovnkube-trace.go#L755-L769

https://github.com/openshift/ovn-kubernetes/pull/1436

Bug OCPBUGS-2659: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/ovn-kubernetes/pull/1325

Bug OCPBUGS-2716: Devfile Catalog and Import a Devfile on a fully disconnected cluster should fail directly instead of timeout after 30sec

View the Description View the linked PRs

Description of problem:
Users on a fully-disconnected cluster could not see Devfiles in the developer catalog or import a Devfiles. That's fine.

But the API calls /api/devfile/samples/ and /api/devfile/ takes 30 seconds until they fail with a 504 Gateway timeout error.

If possible they should fail immediately.

Version-Release number of selected component (if applicable):
This might happen since 4.8

Tested this yet only on 4.12.0-0.nightly-2022-09-07-112008

How reproducible:
Always

Steps to Reproduce:

Start a disconnected cluster with a proxy
Open the browser network inspector and filter for /api/devfile
Switch to Developer perspective
1. Navigate to Add > Developer Catalog (All Services) > Devfiles
2. Or Add > Import from Git > and enter https://github.com/devfile-samples/devfile-sample-go-basic.git

Actual results:

Network call fails after 30 seconds
Import doesn't work

Expected results:

Network calls should fail immediately
We doesn't expect that the import will work

Additional info:
The console Pod log contains this error:

E0909 10:28:18.448680 1 devfile-handler.go:74] Failed to parse devfile: failed to populateAndParseDevfile: Get "https://registry.devfile.io/devfiles/go": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

https://github.com/openshift/console/pull/12186

Bug OCPBUGS-7950: Repositories list does not show the running pipelinerun as last pipelinerun

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-6816~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-6799~~. The following is the description of the original issue:
—
Description of problem:
The pipelines -> repositories list view in Dev Console does not show the running pipelineline as the last pipelinerun in the table.

Original BugZilla Link: https://bugzilla.redhat.com/show_bug.cgi?id=2016006
OCPBUGSM: https://issues.redhat.com/browse/OCPBUGSM-36408

https://github.com/openshift/console/pull/12603

Bug OCPBUGS-5694: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/machine-config-operator/pull/3499

Bug OCPBUGS-536: vsphere: installer for vsphere does not have steal clock accounting enabled

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-268~~. The following is the description of the original issue:
—
The linux kernel was updated:
https://lkml.org/lkml/2020/3/20/1030
to include steal

{time,clock}

accounting

This would greatly assist in troubleshooting vSphere performance issues
caused by over-provisioned ESXi hosts.

Bug OCPBUGS-2293: CVO skips reconciling the installed optional resources in the 4.11 to 4.12 upgrade

View the Description View the linked PRs

Description of problem:

In a 4.11 cluster with only openshift-samples enabled, the 4.12 introduced optional COs console and insights are installed. While upgrading to 4.12, CVO considers them to be disabled explicitly and skips reconciling them. So these COs are not upgraded to 4.12.

Installed COs cannot be disabled, so CVO is supposed to implicitly enable them.


$ oc get clusterversion -oyaml
{
  "apiVersion": "config.openshift.io/v1",
     "kind": "ClusterVersion",
     "metadata": {
         "creationTimestamp": "2022-09-30T05:02:31Z",
         "generation": 3,
         "name": "version",
         "resourceVersion": "134808",
         "uid": "bd95473f-ffda-402d-8fe3-74f852a9d6eb"
     },
     "spec": {
         "capabilities": {
             "additionalEnabledCapabilities": [
                 "openshift-samples"
             ],
             "baselineCapabilitySet": "None"
         },
         "channel": "stable-4.11",
         "clusterID": "8eda5167-a730-4b39-be1d-214a80506d34",
         "desiredUpdate": {
             "force": true,
             "image": "registry.ci.openshift.org/ocp/release@sha256:2c8e617830f84ac1ee1bfcc3581010dec4ae5d9cad7a54271574e8d91ef5ecbc",
             "version": ""
         }
     },
     "status": {
         "availableUpdates": null,
         "capabilities": {
             "enabledCapabilities": [
                 "openshift-samples"
             ],
             "knownCapabilities": [
                 "Console",
                 "Insights",
                 "Storage",
                 "baremetal",
                 "marketplace",
                 "openshift-samples"
             ]
         },
         "conditions": [
             {
                 "lastTransitionTime": "2022-09-30T05:02:33Z",
                 "message": "Unable to retrieve available updates: currently reconciling cluster version 4.12.0-0.nightly-2022-09-28-204419 not found in the \"stable-4.11\" channel",
                 "reason": "VersionNotFound",
                 "status": "False",
                 "type": "RetrievedUpdates"
             },
             {
                 "lastTransitionTime": "2022-09-30T05:02:33Z",
                 "message": "Capabilities match configured spec",
                 "reason": "AsExpected",
                 "status": "False",
                 "type": "ImplicitlyEnabledCapabilities"
             },
             {
                 "lastTransitionTime": "2022-09-30T05:02:33Z",
                 "message": "Payload loaded version=\"4.12.0-0.nightly-2022-09-28-204419\" image=\"registry.ci.openshift.org/ocp/release@sha256:2c8e617830f84ac1ee1bfcc3581010dec4ae5d9cad7a54271574e8d91ef5ecbc\" architecture=\"amd64\"",
                 "reason": "PayloadLoaded",
                 "status": "True",
                 "type": "ReleaseAccepted"
             },
             {
                 "lastTransitionTime": "2022-09-30T05:23:18Z",
                 "message": "Done applying 4.12.0-0.nightly-2022-09-28-204419",
                 "status": "True",
                 "type": "Available"
             },
             {
                 "lastTransitionTime": "2022-09-30T07:05:42Z",
                 "status": "False",
                 "type": "Failing"
             },
             {
                 "lastTransitionTime": "2022-09-30T07:41:53Z",
                 "message": "Cluster version is 4.12.0-0.nightly-2022-09-28-204419",
                 "status": "False",
                 "type": "Progressing"
             }
         ],
         "desired": {
             "image": "registry.ci.openshift.org/ocp/release@sha256:2c8e617830f84ac1ee1bfcc3581010dec4ae5d9cad7a54271574e8d91ef5ecbc",
             "version": "4.12.0-0.nightly-2022-09-28-204419"
         },
         "history": [
             {
                 "completionTime": "2022-09-30T07:41:53Z",
                 "image": "registry.ci.openshift.org/ocp/release@sha256:2c8e617830f84ac1ee1bfcc3581010dec4ae5d9cad7a54271574e8d91ef5ecbc",
                 "startedTime": "2022-09-30T06:42:01Z",
                 "state": "Completed",
                 "verified": false,
                 "version": "4.12.0-0.nightly-2022-09-28-204419"
             },
             {
                 "completionTime": "2022-09-30T05:23:18Z",
                 "image": "registry.ci.openshift.org/ocp/release@sha256:5a6f6d1bf5c752c75d7554aa927c06b5ea0880b51909e83387ee4d3bca424631",
                 "startedTime": "2022-09-30T05:02:33Z",
                 "state": "Completed",
                 "verified": false,
                 "version": "4.11.0-0.nightly-2022-09-29-191451"
             }
         ],
         "observedGeneration": 3,
         "versionHash": "CSCJ2fxM_2o="
     }
 }

$ oc get co
 NAME                                       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.12.0-0.nightly-2022-09-28-204419   True        False         False      93m     
cloud-controller-manager                   4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h56m   
cloud-credential                           4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h59m   
cluster-autoscaler                         4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h53m   
config-operator                            4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h54m   
console                                    4.11.0-0.nightly-2022-09-29-191451   True        False         False      3h45m   
control-plane-machine-set                  4.12.0-0.nightly-2022-09-28-204419   True        False         False      117m    
csi-snapshot-controller                    4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h54m   
dns                                        4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h53m   
etcd                                       4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h52m   
image-registry                             4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h46m   
ingress                                    4.12.0-0.nightly-2022-09-28-204419   True        False         False      151m    
insights                                   4.11.0-0.nightly-2022-09-29-191451   True        False         False      3h48m   
kube-apiserver                             4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h50m   
kube-controller-manager                    4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h51m   
kube-scheduler                             4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h51m   
kube-storage-version-migrator              4.12.0-0.nightly-2022-09-28-204419   True        False         False      91m     
machine-api                                4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h50m   
machine-approver                           4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h54m   
machine-config                             4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h52m   
monitoring                                 4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h44m   
network                                    4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h55m   
node-tuning                                4.12.0-0.nightly-2022-09-28-204419   True        False         False      113m    
openshift-apiserver                        4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h48m   
openshift-controller-manager               4.12.0-0.nightly-2022-09-28-204419   True        False         False      113m    
openshift-samples                          4.12.0-0.nightly-2022-09-28-204419   True        False         False      116m    
operator-lifecycle-manager                 4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h54m   
operator-lifecycle-manager-catalog         4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h54m   
operator-lifecycle-manager-packageserver   4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h48m   
service-ca                                 4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h54m   
storage                                    4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h54m

Version-Release number of selected component (if applicable):

4.12.0-0.nightly-2022-09-28-204419

How reproducible:

Always

Steps to Reproduce:

1. Install a 4.11 cluster with only openshift-samples enabled
2. Upgrade to 4.12
3.

Actual results:

The 4.12 introduced optional CO console and insights are not upgraded to 4.12

Expected results:

All the installed COs get upgraded

Additional info:

https://github.com/openshift/cluster-version-operator/pull/854

Bug OCPBUGS-3101: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/etcd/pull/170

Bug OCPBUGS-1349: [OCPonOpenstack] Remove clustername length limitation

View the Description View the linked PRs

Description of problem:

Currently when installing Openshift on the Openstack cluster name length limit is allowed to  14 characters.
Customer wants to know if is it possible to override this validation when installing Openshift on Openstack and create a cluster name that is greater than 14 characters.

Version : OCP 4.8.5 UPI Disconnected 
Environment : Openstack 16 

Issue:
User reports that they are getting error for OCP cluster in Openstack UPI, where the name of the cluster is > 14 characters.

Error events :
~~~
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/usr/local/bin/openshift-install", "create", "manifests", "--dir=/home/gitlab-runner/builds/WK8mkokN/0/CPE/SKS/pipelines/non-prod/ocp4-openstack-build/ocpinstaller/install-upi"], "delta": "0:00:00.311397", "end": "2022-09-03 21:38:41.974608", "msg": "non-zero return code", "rc": 1, "start": "2022-09-03 21:38:41.663211", "stderr": "level=fatal msg=failed to fetch Master Machines: failed to load asset \"Install Config\": invalid \"install-config.yaml\" file: metadata.name: Invalid value: \"sks-osp-inf-cpe-1-cbr1a\": cluster name is too long, please restrict it to 14 characters", "stderr_lines": ["level=fatal msg=failed to fetch Master Machines: failed to load asset \"Install Config\": invalid \"install-config.yaml\" file: metadata.name: Invalid value: \"sks-osp-inf-cpe-1-cbr1a\": cluster name is too long, please restrict it to 14 characters"], "stdout": "", "stdout_lines": []}
~~~

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Users are getting error "cluster name is too long" when clustername contains more than 14 characters for OCP on Openstack

Expected results:

The 14 characters limit should be change for the OCP clustername on Openstack

Additional info:

https://github.com/openshift/installer/pull/6355

Bug OCPBUGS-2199: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/cluster-kube-apiserver-operator/pull/1386

Bug OCPBUGS-2983: [RFE] 4.11 Azure DiskEncryptionSet static validation does not support upper-case letters

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-2895~~. The following is the description of the original issue:
—
Description of problem:

Current validation will not accept Resource Groups or DiskEncryptionSets which have upper-case letters.

Version-Release number of selected component (if applicable):

4.11

How reproducible:

Attempt to create a cluster/machineset using a DiskEncryptionSet with an RG or Name with upper-case letters

Steps to Reproduce:

1. Create cluster with DiskEncryptionSet with upper-case letters in DES name or in Resource Group name

Actual results:

See error message:

encountered error: [controlPlane.platform.azure.defaultMachinePlatform.osDisk.diskEncryptionSet.resourceGroup: Invalid value: \"v4-e2e-V62447568-eastus\": invalid resource group format, compute[0].platform.azure.defaultMachinePlatform.osDisk.diskEncryptionSet.resourceGroup: Invalid value: \"v4-e2e-V62447568-eastus\": invalid resource group format]

Expected results:

Create a cluster/machineset using the existing and valid DiskEncryptionSet

Additional info:

I have submitted a PR for this already, but it needs to be reviewed and backported to 4.11: https://github.com/openshift/installer/pull/6513

https://github.com/openshift/installer/pull/6537

Bug OCPBUGS-1030: Developer catalog fails to load

View the Description View the linked PRs

Description of problem:

Customer is facing issue similar to https://github.com/devfile/api/issues/897

Version-Release number of selected component (if applicable):

OCP 4.10.17

How reproducible:
N/A
Steps to Reproduce:
1.
2.
3.

Actual results:

Expected results:

Additional info:

Tried working around it with ALL_PROXY but it did not help. Note because the console operator reverts changes pretty quickly testing this was a bit of a PITA

https://github.com/openshift/console/pull/12028

Bug OCPBUGS-3908: [4.11] OVN-Kubernetes should not send IPs with leading zeros to OVN

View the Description View the linked PRs

Description of problem:

If you set a services cluster IP to an IP with a leading zero (e.g. 192.168.0.011), ovn-k should normalise this and remove the leading zero before sending it to ovn.

This was seen by me on a CI run executing the k8 test here: test/e2e/network/funny_ips.go +75

you can reproduce using that above test.

Have a read of the text there:

 43 // What are funny IPs:  
 44 // The adjective is because of the curl blog that explains the history and the problem of liberal  
 45 // parsing of IP addresses and the consequences and security risks caused the lack of normalization,
 46 // mainly due to the use of different notations to abuse parsers misalignment to bypass filters.
 47 // xref: https://daniel.haxx.se/blog/2021/04/19/curl-those-funny-ipv4-addresses/   
 48 //     
 49 // Since golang 1.17, IPv4 addresses with leading zeros are rejected by the standard library.
 50 // xref: https://github.com/golang/go/issues/30999
 51 //     
 52 // Because this change on the parsers can cause that previous valid data become invalid, Kubernetes
 53 // forked the old parsers allowing leading zeros on IPv4 address to not break the compatibility.
 54 //     
 55 // Kubernetes interprets leading zeros on IPv4 addresses as decimal, users must not rely on parser
 56 // alignment to not being impacted by the associated security advisory: CVE-2021-29923 golang
 57 // standard library "net" - Improper Input Validation of octal literals in golang 1.16.2 and below
 58 // standard library "net" results in indeterminate SSRF & RFI vulnerabilities. xref:
 59 // https://nvd.nist.gov/vuln/detail/CVE-2021-29923

northd is logging an error about this also:

|socket_util|ERR|172.30.0.011:7180: bad IP address "172.30.0.011" 
...
2022-08-23T14:14:21.968Z|01839|ovn_util|WARN|bad ip address or port for load balancer key 172.30.0.011:7180

Also, I see the error:

E0823 14:14:34.135115    3284 gateway_shared_intf.go:600] Failed to delete conntrack entry for service e2e-funny-ips-8626/funny-ip: failed to delete conntrack entry for service e2e-funny-ips-8626/funny-ip with svcVIP 172.30.0.011, svcPort 7180, protocol TCP: value "<nil>" passed to DeleteConntrack is not an IP address

We should normalise the IPs before sending to OVN-k. I see also theres conntrack error when trying to set this bad IP.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. See above k8 test

Actual results:

Leading zero IP sent to OVN

Expected results:

No leading zero IP sent to OVN

Additional info:

https://github.com/openshift/ovn-kubernetes/pull/1393

Bug OCPBUGS-5459: Topology sidebar actions doesn't show the latest resource data

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-4960~~. The following is the description of the original issue:
—
Description of problem:
This is a follow up on ~~OCPBUGSM-47202~~ (https://bugzilla.redhat.com/show_bug.cgi?id=2110570)

While ~~OCPBUGSM-47202~~ fixes the issue specific for Set Pod Count, many other actions aren't fixed. When the user updates a Deployment with one of this options, and selects the action again, the old values are still shown.

Version-Release number of selected component (if applicable)
4.8-4.12 as well as master with the changes of ~~OCPBUGSM-47202~~

How reproducible:
Always

Steps to Reproduce:

Import a deployment
Select the deployment to open the topology sidebar
Click on actions and one of the 4 options to update the deployment with a modal
1. Edit labels
2. Edit annotatations
3. Edit update strategy
4. Edit resource limits
Click on the action again and check if the data in the modal reflects the changes from step 3

Actual results:
Old data (labels, annotations, etc.) was shown.

Expected results:
Latest data should be shown

Additional info:

https://github.com/openshift/console/pull/12417

Bug OCPBUGS-7043: "Delete dependent objects of this resource" might cause confusions

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-6766~~. The following is the description of the original issue:
—
This is a clone of https://bugzilla.redhat.com/show_bug.cgi?id=2083087 (~~OCPBUGSM-44070~~) to backport this issue.

Description of problem:
"Delete dependent objects of this resource" is a bit of confusing for some users because when creating the Application in Dev console not only the deployment but also IS, route, svc, secret objects will be created as well. When deleting the Application (in fact it is deployment), there is an option called "Delete dependent objects of this resource" and some users might think this means the IS, route, svc and any other objects which are created alongside with the deployment will be deleted as well

Version-Release number of selected component (if applicable):
4.8

How reproducible:
Always

Steps to Reproduce:
1. Create Application in Dev console
2. Delete the deployment
3. Check "Delete dependent objects of this resource"

Actual results:
Only deployment will be deleted and IS, svc, route will not be deleted

Expected results:
We either change the description of this option, or we really delete IS, svc, route and any other objects created under this Application.

Additional info:

https://github.com/openshift/console/pull/12532

Bug OCPBUGS-7069: PipelineRun task status overlaps status text

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-6913~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-186~~. The following is the description of the original issue:
—
Description of problem:
When resizing the browser window, the PipelineRun task status bar would overlap the status text that says "Succeeded" in the screenshot.

Actual results:
Status text is overlapped by the task status bar

Expected results:
Status text breaks to a newline or gets shortened by "..."

https://github.com/openshift/console/pull/12534

Bug OCPBUGS-7079: Bump to kubernetes 1.24.11

View the Description View the linked PRs

This fix contains the following changes coming from updated version of kubernetes up to v1.24.10:

Changelog:
v1.24.11: https://github.com/kubernetes/kubernetes/blob/release-1.24/CHANGELOG/CHANGELOG-1.24.md#changelog-since-v12410
v1.24.10: https://github.com/kubernetes/kubernetes/blob/release-1.24/CHANGELOG/CHANGELOG-1.24.md#changelog-since-v1249
v1.24.9: https://github.com/kubernetes/kubernetes/blob/release-1.24/CHANGELOG/CHANGELOG-1.24.md#changelog-since-v1248
v1.24.8: https://github.com/kubernetes/kubernetes/blob/release-1.24/CHANGELOG/CHANGELOG-1.24.md#changelog-since-v1247
v1.24.7: https://github.com/kubernetes/kubernetes/blob/release-1.24/CHANGELOG/CHANGELOG-1.24.md#changelog-since-v1246

https://github.com/openshift/kubernetes/pull/1500

Bug OCPBUGS-7987: Azure Disk volume is taking time to attach/detach

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-7885~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-7617~~. The following is the description of the original issue:
—
Description of problem:

Azure Disk volume is taking time to attach/detach
Version-Release number of selected component (if applicable):

Openshift ARO 4.10.30
How reproducible:

While performing scaledown and scaleup of statefulset pod takes time to attach and detach volume from nodes.

Reviewed must-gather and test output will share my findings in comments.

Steps to Reproduce:
1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/azure-disk-csi-driver-operator/pull/71

Bug OCPBUGS-393: Setting disableNetworkDiagnostics: true does not persist when network-operator pod gets re-created

View the Description View the linked PRs

Description of problem:

Setting disableNetworkDiagnostics: true does not persist when network-operator pod gets re-created.

Version-Release number of selected component (if applicable):
4.11.0-rc.0

How reproducible:
100%

Steps to Reproduce:

1. $ oc patch network.operator.openshift.io/cluster --patch '{"spec":{"disableNetworkDiagnostics":true}}' --type=merge
network.operator.openshift.io/cluster patched

2. $ oc -n openshift-network-operator delete pods network-operator-9b68954c6-bclx6
pod "network-operator-9b68954c6-bclx6" deleted

3. $ oc get network.operator.openshift.io cluster -o json | jq .spec.disableNetworkDiagnostics
false

Actual results:
disableNetworkDiagnostics set to false, not set to true as configured in step 1

Expected results:
disableNetworkDiagnostics set to true

Additional info:

Attaching must-gather.

https://github.com/openshift/cluster-network-operator/pull/1530

Bug OCPBUGS-4529: [OSP]Enhancement: suppress the misleading error logs for cloud-network-config-controller pod

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-683~~. The following is the description of the original issue:
—
Description of problem:

Version-Release number of selected component (if applicable):

{ 4.12.0-0.nightly-2022-08-21-135326 }

How reproducible:

Steps to Reproduce:

{See https://bugzilla.redhat.com/show_bug.cgi?id=2118563#c5,
The following messages here are "normal" on startup, but it is very misleading with error statement, suggest suppress them or update them to some more clear context that we can know they are in normal process.

E0818 02:18:53.709223       1 controller.go:165] error syncing 'br709bt-b5564-6jgdx-worker-0-c955q': error retrieving the private IP configuration for node: br709bt-b5564-6jgdx-worker-0-c955q, err: cannot parse valid nova server ID from providerId '', requeuing in node workqueue
E0818 02:18:53.715530       1 controller.go:165] error syncing 'br709bt-b5564-6jgdx-worker-0-sl9jn': error retrieving the private IP configuration for node: br709bt-b5564-6jgdx-worker-0-sl9jn, err: cannot parse valid nova server ID from providerId '', requeuing in node workqueue
E0818 02:18:53.735885       1 controller.go:165] error syncing 'br709bt-b5564-6jgdx-worker-0-sl9jn': error retrieving the private IP configuration for node: br709bt-b5564-6jgdx-worker-0-sl9jn, err: cannot parse valid nova server ID from providerId '', requeuing in node workqueue
E0818 02:18:53.775984       1 controller.go:165] error syncing 'br709bt-b5564-6jgdx-worker-0-sl9jn': error retrieving the private IP configuration for node: br709bt-b5564-6jgdx-worker-0-sl9jn, err: cannot parse valid nova server ID from providerId '', requeuing in node workqueue
E0818 02:18:53.790449       1 controller.go:165] error syncing 'br709bt-b5564-6jgdx-worker-0-c955q': error retrieving the private IP configuration for node: br709bt-b5564-6jgdx-worker-0-c955q, err: cannot parse valid nova server ID from providerId '', requeuing in node workqueue
E0818 02:18:53.856911       1 controller.go:165] error syncing 'br709bt-b5564-6jgdx-worker-0-sl9jn': error retrieving the private IP configuration for node: br709bt-b5564-6jgdx-worker-0-sl9jn, err: cannot parse valid nova server ID from providerId '', requeuing in node workqueue
E0818 02:18:53.950782       1 controller.go:165] error syncing 'br709bt-b5564-6jgdx-worker-0-c955q': error retrieving the private IP configuration for node: br709bt-b5564-6jgdx-worker-0-c955q, err: cannot parse valid nova server ID from providerId '', requeuing in node workqueue
E0818 02:18:54.017583       1 controller.go:165] error syncing 'br709bt-b5564-6jgdx-worker-0-sl9jn': error retrieving the private IP configuration for node: br709bt-b5564-6jgdx-worker-0-sl9jn, err: cannot parse valid nova server ID from providerId '', requeuing in node workqueue
E0818 02:18:54.271967       1 controller.go:165] error syncing 'br709bt-b5564-6jgdx-worker-0-c955q': error retrieving the private IP configuration for node: br709bt-b5564-6jgdx-worker-0-c955q, err: cannot parse valid nova server ID from providerId '', requeuing in node workqueue
E0818 02:18:54.338944       1 controller.go:165] error syncing 'br709bt-b5564-6jgdx-worker-0-sl9jn': error retrieving the private IP configuration for node: br709bt-b5564-6jgdx-worker-0-sl9jn, err: cannot parse valid nova server ID from providerId '', requeuing in node workqueue
E0818 02:18:54.916988       1 controller.go:165] error syncing 'br709bt-b5564-6jgdx-worker-0-c955q': error retrieving the private IP configuration for node: br709bt-b5564-6jgdx-worker-0-c955q, err: cannot parse valid nova server ID from providerId '', requeuing in node workqueue
E0818 02:18:54.982211       1 controller.go:165] error syncing 'br709bt-b5564-6jgdx-worker-0-sl9jn': error retrieving the private IP configuration for node: br709bt-b5564-6jgdx-worker-0-sl9jn, err: cannot parse valid nova server ID from providerId '', requeuing in node workqueue}

Actual results:

Expected results:

Additional info:

https://github.com/openshift/cloud-network-config-controller/pull/85

Bug OCPBUGS-2766: 'oc login' should be robust in the face of gather failures

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-2495~~. The following is the description of the original issue:
—
Failures like:

$ oc login --token=...

Logged into "https://api..." as "..." using the token provided.

Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get projects.project.openshift.io)

break login, which tries to gather information before saving the configuration, including a giant project list.

Ideally login would be able to save the successful login credentials, even when the informative gathering had difficulties. And possibly the informative gathering could be made conditional (--quiet or similar?) so expensive gathering could be skipped in use-cases where the context was not needed.

https://github.com/openshift/oc/pull/1274

Bug OCPBUGS-515: [OCPonRHV] CSI provisioned disks are effectively preallocated due to go-ovirt-client setting Provisioned and Initial size of the disk to the same value

View the Description View the linked PRs

When a thin provisioned COW format disk is created on OCP on RHV via CSI driver (a PVC -
https://github.com/openshift/ovirt-csi-driver/blob/master/deploy/example/storage-claim.yaml

with for example requested storage 100 GB), the go-ovirt-client behaviour makes it so that the created disk has virtual size 100 GB and it's actual size is 110 GB.

But this is thin provisioned disk, so the initial size of the disk should be default of the engine and then grow as needed, it shouldn't be this big.

This causes all the disks created this way to be functionally preallocated (since it eats all that space), which is a real waste of space.

How reproducible: 100%

Steps to Reproduce:
1. Create a storage claim (PVC) in Openshift (
https://github.com/openshift/ovirt-csi-driver/blob/master/deploy/example/storage-claim.yaml
) using the default storage class (or any other storage class with thinProvisioning: "true") and with requested storage i.e. 100Gi

$ oc create -f storage-claim.yaml

2. In the RHV web console navigate to Storage -> Disks and check Virtual size and Actual size of the created disk (PVC)

Actual results:
Disk from our example with requested storage 100GB reports virtual size 100GB and actual size 110 GB.

Expected results:
Thin provisioned disks should start with small initial size and then grow as needed, so its actual size should be considerably smaller (the default initial size set by the engine should be 2.5 GB if I'm not mistaken).

Note: The extra 10GB in the actual size are caused by overhead for the qcow2 disk format, which is 10%, and this was tracked here as a separate issue:
https://bugzilla.redhat.com/show_bug.cgi?id=2097139

https://github.com/openshift/ovirt-csi-driver/pull/115

Bug OCPBUGS-5928: [4.11] nodeport not reachable port connection timeout

View the Description View the linked PRs

Description of problem:

NodePort port not accessible

Version-Release number of selected component (if applicable):

OCP 4.8.20

How reproducible:

$oc -n ui-nprd get services -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
docker-registry ClusterIP 10.201.219.240 <none> 5000/TCP 24d app=registry
docker-registry-lb LoadBalancer 10.201.252.253 internal-xxxxxx.xx-xxxx-1.elb.amazonaws.com 5000:30779/TCP 3d22h app=registry
docker-registry-np NodePort 10.201.216.26 <none> 5000:32428/TCP 3d16h app=registry

$oc debug node/ip-xxx.ca-central-1.compute.internal
Starting pod/ip-xxx.ca-central-1computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.81.23.96
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# nc -vz 10.81.23.96 32428
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connection timed out.

In a new-created namespaces the same deployment works:

[RHEL7:> oc project
Using project "test-c1" on server "https://api.xx.xx.xxxx.xx.xx:6443".
[RHEL7:- ~/tmp]> oc port-forward service/docker-registry-np 5000:5000
Forwarding from 127.0.0.1:5000 -> 5000

[1]+ Stopped oc4 port-forward service/docker-registry-np 5000:5000
[RHEL7: ~/tmp]> bg %1
[1]+ oc4 port-forward service/docker-registry-np 5000:5000 &
[RHEL7: ~/tmp]> nc -v localhost 5000
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 127.0.0.1:5000.
Handling connection for 5000

[RHEL7: ~/tmp]> kill %1
[RHEL7: ~/tmp]>
[1]+ Terminated oc4 port-forward service/docker-registry-np 5000:5000
[RHEL7: ~/tmp]> oc get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
docker-registry-np NodePort 10.201.224.174 <none> 5000:31793/TCP 68s

[RHEL7: ~/tmp]> oc get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
registry-75b7c7fd94-rx29j 1/1 Running 0 7m5s 10.201.1.29 ip-xxx.ca-central-1.compute.internal <none> <none>
[RHEL7: ~/tmp]> oc debug node/ip-xxx.ca-central-1.compute.internal
Starting pod/ip-xxxca-central-1computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.81.23.87
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# nc -v 10.81.23.87 31793
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 10.81.23.87:31793.

Actual results:

Working on new created namespace
Not working on already created namespace

Expected results:

Suppose to work on all namespaces.

Additional info:

This cluster get upgrade from 4.7.x to 4.8 and then they manually enable OVN.
The issue was happening on all namespaces but after restarting the ovnkube-master-xxxx pods only the newly created namespaces work.

https://github.com/openshift/ovn-kubernetes/pull/1478

Bug OCPBUGS-1007: CVE-2021-3121 telemeter-container: [1924548] telemeter-container: gogo/protobuf: plugin/unmarshal/unmarshal.go lacks certain index validation [openshift-4]

View the Description View the linked PRs

Backport clone of https://issues.redhat.com/browse/OCPBUGSM-24281

openshift-4 tracking bug for telemeter-container: see the bugs linked in the "Blocks" field of this bug for full details of the security issue(s).

This bug is never intended to be made public, please put any public notes in the blocked bugs.

Impact: Moderate
Public Date: 11-Jan-2021
PM Fix/Wontfix Decision By: 04-May-2021
Resolve Bug By: 11-Jan-2022

In case the dates above are already past, please evaluate this bug in your next prioritization review and make a decision then. Remember to explicitly set CLOSED:WONTFIX if you decide not to fix this bug.

Please see the Security Errata Policy for further details: https://docs.engineering.redhat.com/x/9RBqB

https://github.com/openshift/telemeter/pull/421

Bug OCPBUGS-10772: TestNewAppRun unit test failing

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-10622~~. The following is the description of the original issue:
—
Description of problem:

Unit test failing 

=== RUN   TestNewAppRunAll/app_generation_using_context_dir
    newapp_test.go:907: app generation using context dir: Error mismatch! Expected <nil>, got supplied context directory '2.0/test/rack-test-app' does not exist in 'https://github.com/openshift/sti-ruby'
    --- FAIL: TestNewAppRunAll/app_generation_using_context_dir (0.61s)

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

see for example https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_oc/1376/pull-ci-openshift-oc-master-images/1638172620648091648

Actual results:

unit tests fail

Expected results:

TestNewAppRunAll unit test should pass

Additional info:

https://github.com/openshift/oc/pull/1380

Bug OCPBUGS-7010: [release-4.11] Egress FW ACL rules are invalid in dualstack mode

View the Description View the linked PRs

Description of problem:
Follow-up of: https://issues.redhat.com/browse/SDN-2988

This failure is perma-failing in the e2e-metal-ipi-ovn-dualstack-local-gateway jobs.

Example: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.13-e2e-metal-ipi-ovn-dualstack-local-gateway/1597574181430497280
Search CI: https://search.ci.openshift.org/?search=when+using+openshift+ovn-kubernetes+should+ensure+egressfirewall+is+created&maxAge=336h&context=1&type=junit&name=e2e-metal-ipi-ovn-dualstack-local-gateway&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job
Sippy: https://sippy.dptools.openshift.org/sippy-ng/jobs/4.13/analysis?filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22equals%22%2C%22value%22%3A%22periodic-ci-openshift-release-master-nightly-4.13-e2e-metal-ipi-ovn-dualstack-local-gateway%22%7D%5D%7D

Version-Release number of selected component (if applicable):

4.12,4.13

How reproducible:

Every time

Steps to Reproduce:

1. Setup dualstack KinD cluster
2. Create egress fw policy with spec
Spec:
  Egress:
    To:
      Cidr Selector:  0.0.0.0/0
    Type:             Deny
3. create a pod and ping to 1.1.1.1

Actual results:

Egress policy does not block flows to external IP

Expected results:

Egress policy blocks flows to external IP

Additional info:

It seems mixing ip4 and ip6 operands in ACL matchs doesnt work

https://github.com/openshift/ovn-kubernetes/pull/1507

Bug OCPBUGS-8298: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/baremetal-operator/pull/256

Bug OCPBUGS-1991: kube-controller-manager resource quota controller needs to stop watching all events

View the Description View the linked PRs

This bug is a backport clone of [Bugzilla Bug 2118318](https://bugzilla.redhat.com/show_bug.cgi?id=2118318). The following is the description of the original bug:
—
+++ This bug was initially created as a clone of Bug #2117569 +++

Description of problem:

The _{garbage collector} resource quota controller must ignore ALL events; otherwise, if a rogue controller or a workload causes unbound event creation, performance will degrade as it has to process the events.

Fix: https://github.com/kubernetes/kubernetes/pull/110939

This bug is to track fix in master (4.12) and also allow to backport to 4.11.1

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:

Expected results:

Additional info:

— Additional comment from Michal Fojtik on 2022-08-11 10:52:28 UTC —

I'm using FastFix here as we need to backport this to 4.11.1 to avoid support churn for busy clusters or clusters doing upgrades.

— Additional comment from ART BZ Bot on 2022-08-11 15:13:32 UTC —

Elliott changed bug status from MODIFIED to ON_QA.
This bug is expected to ship in the next 4.12 release.

— Additional comment from zhou ying on 2022-08-12 03:03:34 UTC —

checked the payload commit id , the payload 4.12.0-0.nightly-2022-08-11-191750 has container the fixed pr .

oc adm release info registry.ci.openshift.org/ocp/release:4.12.0-0.nightly-2022-08-11-191750 --commit-urls |grep hyperkube
Warning: the default reading order of registry auth file will be changed from "${HOME}/.docker/config.json" to podman registry config locations in the future version. "${HOME}/.docker/config.json" is deprecated, but can still be used for storing credentials as a fallback. See https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md for the order of podman registry config locations.
hyperkube https://github.com/openshift/kubernetes/commit/da80cd038ee5c3b45ba36d4b48b42eb8a74439a3

commit da80cd038ee5c3b45ba36d4b48b42eb8a74439a3 (HEAD -> master, origin/release-4.13, origin/release-4.12, origin/master, origin/HEAD)
Merge: a9d6306a701 055b96e614a
Author: OpenShift Merge Robot <openshift-merge-robot@users.noreply.github.com>
Date: Thu Aug 11 15:13:05 2022 +0000

Merge pull request #1338 from benluddy/openshift-pick-110888

Bug 2117569: UPSTREAM: 110888: feat: fix a bug thaat not all event be ignored by gc controller

https://github.com/openshift/kubernetes/pull/1379

Bug OCPBUGS-2160: Track changes of serviceAccountIssuer in operator status

View the Description View the linked PRs

Description of problem:

This issue exists to drive the backport process of https://github.com/openshift/api/pull/1313

According to the Kubernetes documentation, starting from Kubernetes 1.22, the service-account-issuer flag can be specified multiple times. The first value is then used to generate new tokens and other values are accepted. Using this field can prevent cluster disruptions and allows for smoother reconfiguration of this field.

see: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection

The status field will allow us to keep track of "used" service account issuers and also expire/prune them.

this is a replacement for: #1309

xref: https://issues.redhat.com/browse/AUTH-309

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/cluster-kube-apiserver-operator/pull/1394

Bug OCPBUGS-2901: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/coredns/pull/79

Bug OCPBUGS-1847: Image registry panics while deploying OCP in me-central-1 AWS region

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-1717~~. The following is the description of the original issue:
—
Description of problem:

Image registry pods panic while deploying OCP in me-central-1 AWS region

Version-Release number of selected component (if applicable):

4.11.2

How reproducible:

Deploy OCP in AWS me-central-1 region

Steps to Reproduce:

Deploy OCP in AWS me-central-1 region

Actual results:

panic: Invalid region provided: me-central-1

Expected results:

Image registry pods should come up with no errors

Additional info:

Bug OCPBUGS-3089: Update CNCC owners file

View the Description View the linked PRs

Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/cloud-network-config-controller/pull/71

Bug OCPBUGS-7404: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/vmware-vsphere-csi-driver/pull/58

Bug OCPBUGS-1354: [4.11] ETCD Operator goes degraded when a second internal node ip is added

View the Description View the linked PRs

This was originally reported in BZ as https://bugzilla.redhat.com/show_bug.cgi?id=2046335

—

Description of problem:

The issue reported here https://bugzilla.redhat.com/show_bug.cgi?id=1954121 still occur (tested on OCP 4.8.11, the CU also verified that the issue can happen even with OpenShift 4.7.30, 4.8.17 and 4.9.11)

How reproducible:

Attach a NIC to a master node will trigger the issue

Steps to Reproduce:
1. Deploy an OCP cluster (I've tested it IPI on AWS)
2. Attach a second NIC to a running master node (in my case "ip-10-0-178-163.eu-central-1.compute.internal")

Actual results:

~~~
$ oc get node ip-10-0-178-163.eu-central-1.compute.internal -o json | jq ".status.addresses"
[

{ "address": "10.0.178.163", "type": "InternalIP" }

,

{ "address": "10.0.187.247", "type": "InternalIP" }

,

{ "address": "ip-10-0-178-163.eu-central-1.compute.internal", "type": "Hostname" }

,

{ "address": "ip-10-0-178-163.eu-central-1.compute.internal", "type": "InternalDNS" }

]

$ oc get co etcd
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
etcd 4.8.11 True False True 31h

$ oc get co etcd -o json | jq ".status.conditions[0]"

{ "lastTransitionTime": "2022-01-26T15:47:42Z", "message": "EtcdCertSignerControllerDegraded: [x509: certificate is valid for 10.0.178.163, not 10.0.187.247, x509: certificate is valid for ::1, 10.0.178.163, 127.0.0.1, ::1, not 10.0.187.247]", "reason": "EtcdCertSignerController_Error", "status": "True", "type": "Degraded" }

~~~

Expected results:

To have the certificate valid also for the second IP (the newly created one "10.0.187.247")

Additional info:

Deleting the following secrets seems to solve the issue:
~~~
$ oc get secret ~~n openshift-etcd | grep kubernetes.io/tls | grep ^etcd~~
etcd-client kubernetes.io/tls 2 61s
etcd-peer-ip-10-0-132-49.eu-central-1.compute.internal kubernetes.io/tls 2 61s
etcd-peer-ip-10-0-178-163.eu-central-1.compute.internal kubernetes.io/tls 2 61s
etcd-peer-ip-10-0-202-187.eu-central-1.compute.internal kubernetes.io/tls 2 60s
etcd-serving-ip-10-0-132-49.eu-central-1.compute.internal kubernetes.io/tls 2 60s
etcd-serving-ip-10-0-178-163.eu-central-1.compute.internal kubernetes.io/tls 2 59s
etcd-serving-ip-10-0-202-187.eu-central-1.compute.internal kubernetes.io/tls 2 59s
etcd-serving-metrics-ip-10-0-132-49.eu-central-1.compute.internal kubernetes.io/tls 2 58s
etcd-serving-metrics-ip-10-0-178-163.eu-central-1.compute.internal kubernetes.io/tls 2 59s
etcd-serving-metrics-ip-10-0-202-187.eu-central-1.compute.internal kubernetes.io/tls 2 58s

$ oc get secret ~~n openshift-etcd | grep kubernetes.io/tls | grep ^etcd~~ | awk '

{print $1}

' | xargs -I {} oc delete secret {} -n openshift-etcd
secret "etcd-client" deleted
secret "etcd-peer-ip-10-0-132-49.eu-central-1.compute.internal" deleted
secret "etcd-peer-ip-10-0-178-163.eu-central-1.compute.internal" deleted
secret "etcd-peer-ip-10-0-202-187.eu-central-1.compute.internal" deleted
secret "etcd-serving-ip-10-0-132-49.eu-central-1.compute.internal" deleted
secret "etcd-serving-ip-10-0-178-163.eu-central-1.compute.internal" deleted
secret "etcd-serving-ip-10-0-202-187.eu-central-1.compute.internal" deleted
secret "etcd-serving-metrics-ip-10-0-132-49.eu-central-1.compute.internal" deleted
secret "etcd-serving-metrics-ip-10-0-178-163.eu-central-1.compute.internal" deleted
secret "etcd-serving-metrics-ip-10-0-202-187.eu-central-1.compute.internal" deleted

$ oc get co etcd -o json | jq ".status.conditions[0]"

{ "lastTransitionTime": "2022-01-26T15:52:21Z", "message": "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found", "reason": "AsExpected", "status": "False", "type": "Degraded" }

~~~

https://github.com/openshift/cluster-etcd-operator/pull/931

Bug OCPBUGS-2501: [RFE] Add new Azure instance types to the official "tested/supported" list

View the Description View the linked PRs

We have created a fix in 4.12 that fetches instance type information from Azure API instead of updating the lists. We feel that backporting that fix is too risky, but agreed to update the list in older versions.

Description of problem:

Add the following instance types to azure_instance_types list[1]:

Standard_D8s_v5
Standard_E8s_v5
Standard_E16s_v5

Version-Release number of selected component (if applicable):
OCP 4.8

Steps to Reproduce:
1. Migrate worker/infra nodes to above mentioned (missing) v5 instance types
2. "Failed to set autoscaling from zero annotations, instance type unknown"

Actual results:

"Failed to set autoscaling from zero annotations, instance type unknown"
New v5 instance types not officially tested/supported

Expected results:
The new instance types are available in the azure_instance_types list[1] and no errors/warnings are observed after migrating:

Standard_D8s_v5
Standard_E8s_v5
Standard_E16s_v5

Additional info:

The related v4 instance types are already available[1] - I suspect adding the mentioned v5 instance types is a minor update:

Standard_D8s_v4
Standard_E8s_v4
Standard_E16s_v4

1) azure_instance_types.go
https://github.com/openshift/cluster-api-provider-azure/blob/release-4.8/pkg/cloud/azure/actuators/machineset/azure_instance_types.go

https://github.com/openshift/machine-api-provider-azure/pull/37

Bug OCPBUGS-4475: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/ovn-kubernetes/pull/1417

Bug OCPBUGS-673: crio occasionally fails to start during deployment

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-669~~. The following is the description of the original issue:
—
Description of problem:

This is an OCP clone of https://bugzilla.redhat.com/show_bug.cgi?id=2099794

In summary, NetworkManager reports the network as being up before the ipv6 address of the primary interface is ready and crio fails to bind to it.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/baremetal-runtimecfg/pull/192

Bug OCPBUGS-3882: MCO does not sync kubeAPIServerServingCAData to controllerconfig if there are not ready nodes

View the Description View the linked PRs

This bug is a backport clone of [Bugzilla Bug 2034883](https://bugzilla.redhat.com/show_bug.cgi?id=2034883). The following is the description of the original bug:
—
Description of problem:

Situation (starting point):

There is an ongoing change to the machine-config-daemon daemonset being applied by the machine-config-operator pod. It is waiting for the daemonset to roll out.
There are some nodes not ready, so daemonset rollout never ends and waiting on that ends in timeout error.

Problem:

Machine-config-operator pods stops trying to reconcile stuff whenever it finds timeout error in waiting for the machine-config-daemon rollout
This implies that the `spec.kubeAPIServerServingCAData` field of controllerconfig/machine-config-controller object is not updated when the kube-apiserver-operator updates kube-apiserver-to-kubelet-client-ca configmap.
Without that field updated, a kube-apiserver-to-kubelet-client-ca change is never rolled out to the nodes.
That ultimately leads to cluster-wide unavailability of "oc logs", "oc rsh" etc. commands when the kube-apiserver-operator starts using a client cert signed by the new kube-apiserver-to-kubelet-client-ca to access kubelet ports.

Version-Release number of MCO (Machine Config Operator) (if applicable):

4.7.21

Platform (AWS, VSphere, Metal, etc.): (not relevant)

Are you certain that the root cause of the issue being reported is the MCO (Machine Config Operator)?
(Y/N/Not sure): Y

How reproducible:

Always if the said conditions are met.

Steps to Reproduce:
1. Have some nodes not ready
2. Force a change that requires machine-config-daemon daemonset rollout (I think that changing proxy settings would work for this)
3. Wait until a new kube-apiserver-to-kubelet-client-ca is rolled out by kube-apiserver-operator

Actual results:

New kube-apiserver-to-kubelet-client-ca not forwarded to controllerconfig, kube-apiserver-to-kubelet-client-ca not deployed on nodes

Expected results:

kube-apiserver-to-kubelet-client-ca forwarded to controllerconfig, kube-apiserver-to-kubelet-client-ca deployed to nodes.

Additional info:

In comments

https://github.com/openshift/machine-config-operator/pull/3421

Story OADP-22: Send Telemetry metrics on OADP

View the Description View the linked PRs

We will want to establish some basic metrics we can report back to Telemetry.

Let's consider:

Operator installs
Backups, created, success, error
Restores created, success, error

Below is some background info from MTC when we added Telemetry support that may help

See: https://github.com/konveyor/metrics-queries/blob/master/README.md

Design/Development info:

OpenShift Monitoring Integration Guide

Sending metrics via telemetry

Monitoring integration with OLM operators

https://www.openshift.com/blog/observability-superpower-correlation

Source Code:

https://github.com/konveyor/mig-controller/blob/master/pkg/controller/migmigration/metrics.go

https://github.com/openshift/cluster-monitoring-operator/pull/1536

Bug OCPBUGS-826: co/kube-controller-manager degraded: GarbageCollectorDegraded: error fetching rules: Get "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules": dial tcp 172.30.153.28:9091: connect: cannot assign requested address

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-212~~. The following is the description of the original issue:
—
Description of problem:

oc --context build02 get clusterversion
NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.0-ec.1   True        False         45h     Error while reconciling 4.12.0-ec.1: the cluster operator kube-controller-manager is degraded

oc --context build02 get co kube-controller-manager
NAME                      VERSION       AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
kube-controller-manager   4.12.0-ec.1   True        False         True       2y87d   GarbageCollectorDegraded: error fetching rules: Get "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules": dial tcp 172.30.153.28:9091: connect: cannot assign requested address

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:

Expected results:

Additional info:

build02 is a build farm cluster in CI production.
I can provide credentials to access the cluster if needed.

https://github.com/openshift/cluster-kube-controller-manager-operator/pull/654

Bug OCPBUGS-10605: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/csi-driver-manila-operator/pull/177

Bug OCPBUGS-4579: Default to floating automaticRestart for new GCP instances

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-4504~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-1557~~. The following is the description of the original issue:
—
Seen in an instance created recently by a 4.12.0-ec.2 GCP provider:

  "scheduling": {
    "automaticRestart": false,
    "onHostMaintenance": "MIGRATE",
    "preemptible": false,
    "provisioningModel": "STANDARD"
  },

From GCP's docs, they may stop instances on hardware failures and other causes, and we'd need automaticRestart: true to auto-recover from that. Also from GCP docs, the default for automaticRestart is true. And on the Go provider side, we doc:

If omitted, the platform chooses a default, which is subject to change over time, currently that default is "Always".

But the implementing code does not actually float the setting. Seems like a regression here, which is part of 4.10:

$ git clone https://github.com/openshift/machine-api-provider-gcp.git
$ cd machine-api-provider-gcp
$ git log --oneline origin/release-4.10 | grep 'migrate to openshift/api'
44f0f958 migrate to openshift/api

But that's not where the 4.9 and earlier code is located:

$ git branch -a | grep origin/release
  remotes/origin/release-4.10
  remotes/origin/release-4.11
  remotes/origin/release-4.12
  remotes/origin/release-4.13

Hunting for 4.9 code:

$ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.9.48-x86_64 | grep gcp
  gcp-machine-controllers                        https://github.com/openshift/cluster-api-provider-gcp                       c955c03b2d05e3b8eb0d39d5b4927128e6d1c6c6
  gcp-pd-csi-driver                              https://github.com/openshift/gcp-pd-csi-driver                              48d49f7f9ef96a7a42a789e3304ead53f266f475
  gcp-pd-csi-driver-operator                     https://github.com/openshift/gcp-pd-csi-driver-operator                     d8a891de5ae9cf552d7d012ebe61c2abd395386e

So looking there:

$ git clone https://github.com/openshift/cluster-api-provider-gcp.git
$ cd cluster-api-provider-gcp
$ git log --oneline | grep 'migrate to openshift/api'
...no hits...
$ git grep -i automaticRestart origin/release-4.9  | grep -v '"description"\|compute-gen.go'
origin/release-4.9:vendor/google.golang.org/api/compute/v1/compute-api.json:        "automaticRestart": {

Not actually clear to me how that code is structured. So 4.10 and later GCP machine-API providers are impacted, and I'm unclear on 4.9 and earlier.

https://github.com/openshift/machine-api-provider-gcp/pull/30

Bug OCPBUGS-8257: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/machine-api-operator/pull/1123

Bug OCPBUGS-2011: [4.11] ironic clear_job_queue and reset_idrac pending issues

View the Description View the linked PRs

Description of problem:

There is a bug affecting verify steps functionality on iDRAC hardware in OpenShift 4.11 and 4.10. Original bug report has been made against 4.10:

https://issues.redhat.com/browse/OCPBUGS-1740

While I am not aware of this issue being reported against 4.11, due to the fact that the fix is only present in 4.12 codebase, 4.11 versions will also be affected by this issue.

This bug is created to meet automation requirements for backporting the fixes from 4.12 version to 4.11 (and then to 4.11 in the bug quoted above).

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/ironic-image/pull/302

Bug OCPBUGS-2105: intra namespace allow network policy doesn't work after applying ingress&egress deny all network policy

View the Description View the linked PRs

Description of problem:

intra namespace allow network policy doesn't work after applying ingress&egress deny all network policy

Version-Release number of selected component (if applicable):

OpenShift 4.10.12

How reproducible:

Always

Steps to Reproduce:
1. Define deny all network policy for egress an ingress in a namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

2. Define the following network policy to allow the traffic between the pods in the namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-intra-namespace-001
spec:
  egress:
  - to:
    - podSelector: {}
  ingress:
  - from:
    - podSelector: {}
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

3. Test the connectivity between two pods from the namespace.

Actual results:

The connectivity is not allowed

Expected results:

The connectivity should be allowed between pods from the same namespace.

Additional info:

After performing a test and analyzing SDN flows for the namespace:

sh-4.4# ovs-ofctl dump-flows -O OpenFlow13 br0 | grep --color 0x964376 
 cookie=0x0, duration=99375.342s, table=20, n_packets=14, n_bytes=588, priority=100,arp,in_port=21,arp_spa=10.128.2.20,arp_sha=00:00:0a:80:02:14/00:00:ff:ff:ff:ff actions=load:0x964376->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=1681.845s, table=20, n_packets=11, n_bytes=462, priority=100,arp,in_port=24,arp_spa=10.128.2.23,arp_sha=00:00:0a:80:02:17/00:00:ff:ff:ff:ff actions=load:0x964376->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=99375.342s, table=20, n_packets=135610, n_bytes=759239814, priority=100,ip,in_port=21,nw_src=10.128.2.20 actions=load:0x964376->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=1681.845s, table=20, n_packets=2006, n_bytes=12684967, priority=100,ip,in_port=24,nw_src=10.128.2.23 actions=load:0x964376->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=99375.342s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.128.2.20 actions=load:0x964376->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=1681.845s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.128.2.23 actions=load:0x964376->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=975.129s, table=27, n_packets=0, n_bytes=0, priority=150,reg0=0x964376,reg1=0x964376 actions=goto_table:30
 cookie=0x0, duration=99375.342s, table=70, n_packets=145260, n_bytes=11722173, priority=100,ip,nw_dst=10.128.2.20 actions=load:0x964376->NXM_NX_REG1[],load:0x15->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=1681.845s, table=70, n_packets=2336, n_bytes=191079, priority=100,ip,nw_dst=10.128.2.23 actions=load:0x964376->NXM_NX_REG1[],load:0x18->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=975.129s, table=80, n_packets=0, n_bytes=0, priority=150,reg0=0x964376,reg1=0x964376 actions=output:NXM_NX_REG2[]

We see that the following rule doesn't match because `reg1` hasn't been defined:

 cookie=0x0, duration=975.129s, table=27, n_packets=0, n_bytes=0, priority=150,reg0=0x964376,reg1=0x964376 actions=goto_table:30

https://github.com/openshift/sdn/pull/464

Bug OCPBUGS-5099: [4.11] coreos-installer output not available in the logs

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-5067~~. The following is the description of the original issue:
—
Description of problem:

Since coreos-installer writes to stdout, its logs are not available for us.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/ironic-agent-image/pull/67

Bug OCPBUGS-5876: wal: max entry size limit exceeded

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-5761~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-5458~~. The following is the description of the original issue:
—
reported in https://coreos.slack.com/archives/C027U68LP/p1673010878672479

Description of problem:

Hey guys, I have a openshift cluster that was upgraded to version 4.9.58 from version 4.8. After the upgrade was done, the etcd pod on master1 isn't coming up and is crashlooping. and it gives the following error: {"level":"fatal","ts":"2023-01-06T12:12:58.709Z","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"wal: max entry size limit exceeded, recBytes: 13279, fileSize(313430016) - offset(313418480) - padBytes(1) = entryLimit(11535)","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\t/remote-source/cachito-gomod-with-deps/app/server/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\t/remote-source/cachito-gomod-with-deps/app/server/etcdmain/main.go:40\nmain.main\n\t/remote-source/cachito-gomod-with-deps/app/server/main.go:32\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:225"}

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/etcd/pull/178

Bug OCPBUGS-4528: Disconnected Openshift cluster on AWS having problem with manual egress IP assignment

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-1417~~. The following is the description of the original issue:
—
Description of problem:

Egress IP is not being assigned to primary interface of node as per hostsubnet definition. The issue being observed at an Openshift cluster hosted on Disconnected AWS environment.  Following steps were performed at AWS end:

- Disconnected VPC was created and installation of Openshift was done as per documentation.
- Elastic IP could not be used as it is a disconnected environment. Customer identified a free IP from same subnet as the node and modified interface of the node to add a secondary IP.

It seems cloud.network.openshift.io/egress-ipconfig annotation is need on the node to attach IP to primary interface but its missing. From SDN POD log on the same node I  could see its complaining about 'an incomplete annotation "cloud.network.openshift.io/egress-ipconfig"'. Will share more details over comments.

Version-Release number of selected component (if applicable):

Openshift 4.10.28

How reproducible:

Always

Steps to Reproduce:

1. Create a disconnected environment on AWS
2. find a free IP from subnet where a worker node is hosted and add that as secondary  IP to NIC of that node.
3. Configure hostsubnet and netnamespace on Openshift cluster

Actual results:

- Eress IP is not being attached to primary interface of node for which hostsubnet has been configured

Expected results:

- Egress IP should get configured without any issue.

Additional info:

https://github.com/openshift/cloud-network-config-controller/pull/84

Bug OCPBUGS-759: AWS tagging limit hit issue when trying to add more than 10 tags

View the Description View the linked PRs

Description of problem:

AWS tagging - when applying user defined tags you cannot add more than 10

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/installer/pull/6303

Bug OCPBUGS-1974: ProjectHelmChartRepository display name (spec.name) is not used in Helm Charts catalog

View the Description View the linked PRs

Description of problem:

When creating a ProjectHelmChartRepository (with or without the form) and setting a display name (as `spec.name`), this value is not used in the developer catalog / Helm Charts catalog filter sidebar.

It shows (and watches) the display names of `HelmChartRepository` resources.

Version-Release number of selected component (if applicable):

4.11

How reproducible:

Always

Steps to Reproduce:

1. Switch to Developer Perspective
2. Navigate to Add > "Helm Chart repositories"
3. Enter "ibm-charts" as "Chart repository name"
4. Enter URL https://raw.githubusercontent.com/IBM/charts/master/repo/community/index.yaml as URL)
5. Press on create
6. Open the YAML editor and change the `spec.name` attribute to "IBM Charts"
7. Save the change
8. Navigate to Add > "Helm Chart"

Actual results:

The filter navigation on the left side shows "Chart Repositories" "Ibm Chart". A camel case version of the resource name.

Expected results:

It should show the "spec.name" "IBM Charts" if defined and fallback to the current implementation if the optional spec.name is not defined.

Additional info:

There is a bug discussing that the display name could not be entered directly, https://bugzilla.redhat.com/show_bug.cgi?id=2106366. This bug here is only about the catalog output.

https://github.com/openshift/console/pull/12130

Bug OCPBUGS-1070: Update ODC owners

View the Description View the linked PRs

Description of problem: Backport owners for 4.11 (only)

https://github.com/openshift/console/pull/11936

Bug OCPBUGS-4405: Azure UPI installation failed to scale up worker nodes using machinests

View the Description View the linked PRs

Description of problem:
Cannot scale up worker node have deploying OCP 4.11.1 cluster via UPI on Azure

5h2m Warning FailedCreate machine/pokus-2knkh-worker-northeurope1-f6kc4 InvalidConfiguration: failed to reconcile machine "pokus-2knkh-worker-northeurope1-f6kc4": failed to create vm pokus-2knkh-worker-northeurope1-f6kc4: failure sending request for machine pokus-2knkh-worker-northeurope1-f6kc4: cannot create vm: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=404 - Original Error: Code="NotFound" Message="The Image '/subscriptions/e639e479-2737-4b3d-b338-f1928f6429a1/resourceGroups/mlpipe-2163-azpln-rg/providers/Microsoft.Compute/images/pokus-2knkh-gen2' cannot be found in 'northeurope' region."

Customer would like to have the installer create machineset from the inital installation, therefore Kubernetes manifest files that define the worker machines were not removed during the installation.

Highlights:
Can I please let help verifying if these are the correct steps to have the initial installation created and manage the worker machines?Is there an explanation on how changing the image to -gen2 in [concat(parameters('baseName'),'-gen2')] from the 02_storage.json template can resolve the problem?
Version-Release number of selected component (if applicable):

Environment:
OCP 4.11.1 UPI install on Azure using ARM
VM size:
bootstrap: Standard_D4s_v3
master: Standard_D4s_v3

How reproducible:
Always

Steps to Reproduce:
Following the step described in the document: Installing a cluster on Azure using ARM templates .

In the install-config.yaml, worker replicas was set to 0

compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform: {}
  replicas: 3   
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3

After creating the manifests described in this step: Creating the Kubernetes manifest and Ignition config files only control plane machines manifests were removed, worker machines manifests remain untouchedAfter three masters and three worker nodes were created by ARM templates, additional worker were added using machine sets via command

oc scale --replicas=1 machineset cluster-g7rzv-worker-francecentral1 -n openshift-machine-api`

Actual results:
No addition node visible from `oc get nodes` and the following error occur:

5h2m Warning FailedCreate machine/pokus-2knkh-worker-northeurope1-f6kc4 InvalidConfiguration: failed to reconcile machine "pokus-2knkh-worker-northeurope1-f6kc4": failed to create vm pokus-2knkh-worker-northeurope1-f6kc4: failure sending request for machine pokus-2knkh-worker-northeurope1-f6kc4: cannot create vm: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=404 - Original Error: Code="NotFound" Message="The Image '/subscriptions/e639e479-2737-4b3d-b338-f1928f6429a1/resourceGroups/mlpipe-2163-azpln-rg/providers/Microsoft.Compute/images/pokus-2knkh-gen2' cannot be found in 'northeurope' region."

The customer found out that this can be resolved if changing the -image to -gen2 in [concat(parameters('baseName'),'-gen2')] from the 02_storage.json template

Expected results:
The installer should be able to create and manage machineset

Additional info:
SFDC case #03304526

Slack discussion, might due to MAO not able to support UPI in Azure Thread1, Thread2

https://github.com/openshift/installer/pull/6681

Bug OCPBUGS-5781: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/cluster-cloud-controller-manager-operator/pull/216

Bug OCPBUGS-6683: [4.11]Improve Pod Admission failure for restricted-v2 denials that pass with restricted

View the Description View the linked PRs

This is a clone of BZ https://bugzilla.redhat.com/show_bug.cgi?id=2117374 which fixed 4.12

Description of problem:
OCP 4.11 introduced the `restricted-v2` SecurityContextConstraint as the default binding for all authenticated users. When a Pod that would have normally be admitted successfully using the `restricted` SCC, but cannot be admitted using a `restricted-v2` SCC, it's not clear to the customer that the problem is related to the default SCC changing from restricted to restricted-v2.

Suggestion is to generate an additional message for this specific use case that makes it clear to customers that they have a workload that is not compatible with restricted-v2.

Example message:
Error creating: pods "nginx-ingress-controller-75bffcfdf8-" is forbidden: the pod fails to validate against the default `restricted-v2` security context constraint, but would validate successfully against the `restricted` security context constraint.

The goal of such a message is to give the customer a breadcrumb that the `restricted-v2` SCC is activated and set as the default, which they may have not been aware of. This would give them a way to find alternatives, such as changing the global default, fixing the application or assigning the `restricted` SCC to the Namespace or ServiceAccount.

Version-Release number of selected component (if applicable):
4.11

How reproducible:
Always

Steps to Reproduce:
1. Install OCP 4.11.0
2. Create a deployment with a podtemplate with .spec.containers[*].securityContext.allowPrivigeEscalation = true in a new namespace

Actual results:
The pod will fail to be admitted with an error such as:

Warning FailedCreate 27m (x25 over 76m) replicaset-controller Error creating: pods "nginx-ingress-controller-75bffcfdf8-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, spec.containers[0].securityContext.allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "noobaa": Forbidden: not usable by user or serviceaccount, provider "noobaa-endpoint": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "rook-ceph": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "rook-ceph-csi": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]

Expected results:
Add an additional or replacement error message when there is no SCC assignment other than `restricted-v2` AND `restricted` would have worked:

Error creating: pods "nginx-ingress-controller-75bffcfdf8-" is forbidden: the pod fails to validate against the default `restricted-v2` security context constraint, but would validate successfully against the `restricted` security context constraint.

Additional info:

https://github.com/openshift/kubernetes/pull/1459

Bug OCPBUGS-10853: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/sdn/pull/522

Bug OCPBUGS-3674: metal3 pod crashloops on OKD in BareMetal IPI or assisted-installer bare metal installations

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-3111~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-2992~~. The following is the description of the original issue:
—
Description of problem:

The metal3-ironic container image in OKD fails during steps in configure-ironic.sh that look for additional Oslo configuration entries as environment variables to configure the Ironic instance. The mechanism by which it fails in OKD but not OpenShift is that the image for OpenShift happens to have unrelated variables set which match the regex, because it is based on the builder image, but the OKD image is based only on a stream8 image without these unrelated OS_ prefixed variables set.

The metal3 pod created in response to even a provisioningNetwork: Disabled Provisioning object will therefore crashloop indefinitely.

Version-Release number of selected component (if applicable):

4.11

How reproducible:

Always

Steps to Reproduce:

1. Deploy OKD to a bare metal cluster using the assisted-service, with the OKD ConfigMap applied to podman play kube, as in :https://github.com/openshift/assisted-service/tree/master/deploy/podman#okd-configuration
2. Observe the state of the metal3 pod in the openshift-machine-api namespace.

Actual results:

The metal3-ironic container repeatedly exits with nonzero, with the logs ending here:

++ export IRONIC_URL_HOST=10.1.1.21
++ IRONIC_URL_HOST=10.1.1.21
++ export IRONIC_BASE_URL=https://10.1.1.21:6385
++ IRONIC_BASE_URL=https://10.1.1.21:6385
++ export IRONIC_INSPECTOR_BASE_URL=https://10.1.1.21:5050
++ IRONIC_INSPECTOR_BASE_URL=https://10.1.1.21:5050
++ '[' '!' -z '' ']'
++ '[' -f /etc/ironic/ironic.conf ']'
++ cp /etc/ironic/ironic.conf /etc/ironic/ironic.conf_orig
++ tee /etc/ironic/ironic.extra
# Options set from Environment variables
++ echo '# Options set from Environment variables'
++ env
++ grep '^OS_'
++ tee -a /etc/ironic/ironic.extra

Expected results:

The metal3-ironic container starts and the metal3 pod is reported as ready.

Additional info:

This is the PR that introduced pipefail to the downstream ironic-image, which is not yet accepted in the upstream:
https://github.com/openshift/ironic-image/pull/267/files#diff-ab2b20df06f98d48f232d90f0b7aa464704257224862780635ec45b0ce8a26d4R3

This is the line that's failing:
https://github.com/openshift/ironic-image/blob/4838a077d849070563b70761957178055d5d4517/scripts/configure-ironic.sh#L57

This is the image base that OpenShift uses for ironic-image (before rewriting in ci-operator):
https://github.com/openshift/ironic-image/blob/4838a077d849070563b70761957178055d5d4517/Dockerfile.ocp#L9

Here is where the relevant environment variables are set in the builder images for OCP:
https://github.com/openshift/builder/blob/973602e0e576d7eccef4fc5810ba511405cd3064/hack/lib/build/version.sh#L87

Here is the final FROM line in the OKD image build (just stream8):
https://github.com/openshift/ironic-image/blob/4838a077d849070563b70761957178055d5d4517/Dockerfile.okd#L9

This results in the following differences between the two images:
$ podman run --rm -it --entrypoint bash quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:519ac06836d972047f311de5e57914cf842716e22a1d916a771f02499e0f235c -c 'env | grep ^OS_'
OS_GIT_MINOR=11
OS_GIT_TREE_STATE=clean
OS_GIT_COMMIT=97530a7
OS_GIT_VERSION=4.11.0-202210061001.p0.g97530a7.assembly.stream-97530a7
OS_GIT_MAJOR=4
OS_GIT_PATCH=0
$ podman run --rm -it --entrypoint bash quay.io/openshift/okd-content@sha256:6b8401f8d84c4838cf0e7c598b126fdd920b6391c07c9409b1f2f17be6d6d5cb -c 'env | grep ^OS_'

Here is what the OS_ prefixed variables should be used for:
https://github.com/metal3-io/ironic-image/blob/807a120b4ce5e1675a79ebf3ee0bb817cfb1f010/README.md?plain=1#L36
https://opendev.org/openstack/oslo.config/src/commit/84478d83f87e9993625044de5cd8b4a18dfcaf5d/oslo_config/sources/_environment.py

It's worth noting that ironic.extra is not consumed anywhere, and is simply being used here to save off the variables that Oslo _might_ be consuming (it won't consume the variables that are present in the OCP builder image, though they do get caught by this regex).

With pipefail set, grep returns non-zero when it fails to find an environment variable that matches the regex, as in the case of the OKD ironic-image builds.

https://github.com/openshift/ironic-image/pull/318

Bug OCPBUGS-1242: Add PowerVS region mon01 to installer

View the Description View the linked PRs

Description of problem:

The 4.11 version of openshift-installer does not support the mon01 zone

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/installer/pull/6341

Bug OCPBUGS-4607: Implement LIST call chunking in openshift-sdn

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-4422~~. The following is the description of the original issue:
—
This bug is a backport clone of [Bugzilla Bug 2050230](https://bugzilla.redhat.com/show_bug.cgi?id=2050230). The following is the description of the original bug:
—
Description of problem:
In a large cluster, sdn daemonset can DoS the kube-apiserver with un-paginated LIST calls on high count resources.

Version-Release number of selected component (if applicable):

How reproducible:
NA

Steps to Reproduce:
NA

Actual results:
Kube API Server and Openshift API Server in one of the cluster keeps restarting, without proper exception. The cluster is not accessible.

Expected results:
Kube API Server and Openshift API Server should be stable.

Additional info:

https://github.com/openshift/sdn/pull/479

Bug OCPBUGS-2050: [release-4.11] DNS operator does not reconcile the openshift-dns namespace

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-1549~~. The following is the description of the original issue:
—
Description of problem:

The cluster-dns-operator does not reconcile the openshift-dns namespace, which has been exposed as an issue in 4.12 due to the requirement for the namespace to have pod-security labels.

If a cluster has been incrementally updated from a version less than or equal to 4.9, the openshift-dns namespace will most likely not contain the required pod-security labels since the namespace was statically created when the cluster was installed with old namespace configuration.

Version-Release number of selected component (if applicable):

4.12

How reproducible:

Always if cluster originally installed with v4.9 or less

Steps to Reproduce:

1. Install v4.9
2. Upgrade to v4.12 (incrementally if required for upgrade path)
3. openshift-dns namespace will be missing pod-security labels

Actual results:

"oc get ns openshift-dns -o yaml" will show missing pod-security labels: 

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    openshift.io/node-selector: ""
    openshift.io/sa.scc.mcs: s0:c15,c0
    openshift.io/sa.scc.supplemental-groups: 1000210000/10000
    openshift.io/sa.scc.uid-range: 1000210000/10000
  creationTimestamp: "2020-05-21T19:36:15Z"
  labels:
    kubernetes.io/metadata.name: openshift-dns
    olm.operatorgroup.uid/3d42c0c1-01cd-4c55-bf88-864f041c7e7a: ""
    openshift.io/cluster-monitoring: "true"
    openshift.io/run-level: "0"
  name: openshift-dns
  resourceVersion: "3127555382"
  uid: 0fb4571e-952f-4bea-bc45-461beec54369
spec:
  finalizers:
  - kubernetes

Expected results:

pod-security labels should exist:
 
 labels:
    kubernetes.io/metadata.name: openshift-dns
    olm.operatorgroup.uid/3d42c0c1-01cd-4c55-bf88-864f041c7e7a: ""
    openshift.io/cluster-monitoring: "true"
    openshift.io/run-level: "0"
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/warn: privileged

Additional info:

Issue found in CI during upgrade

https://coreos.slack.com/archives/C03G7REB4JV/p1663676443155839

https://github.com/openshift/cluster-dns-operator/pull/346

Bug OCPBUGS-2091: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/machine-config-operator/pull/3346

Bug OCPBUGS-6813: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/ovn-kubernetes/pull/1498

Bug OCPBUGS-1710: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/vmware-vsphere-csi-driver-operator/pull/111

Bug OCPBUGS-185: [OCP web console] Search result doesn't clear when user clears name filter in one-shot for any resources

View the Description View the linked PRs

Description of problem:

Users search a resource (for example, Pod) with Name filter applied and input a text to the filter field then the search results filtered accordingly.

Once the results are shown, when the user clear the value in one-shot (i.e. select whole filter text from the field and clear it using delete or backspace key) from the field,

then the search result doesn't clear accordingly and the previous result stays on the page.

Version-Release number of selected components (if applicable):

4.11.0-0.nightly-2022-08-16-194731 & works fine with OCP 4.12 latest version.

How reproducible:

Always

Steps to Reproduce:

Login to OCP web console.
Go to the Search section.
Select the resource filter and choose a resource (for example , Pod)
Change the filter option to "Name" and input a text( for example, installer) to the filter field.
Once the results are shown, select the whole filter text ( i.e. installer).
Clear the text using the Delete or Backspace key.
View the behavior on search results

Actual results:

Search result doesn't clear when user clears name filter in one-shot for any resources.

Expected results:

Search results should clear when the user clears name filters in one-shot for any resources.

Additional info:

Reproduced in both chrome[103.0.5060.114 (Official Build) (64-bit)] and firefox[91.11.0esr (64-bit)] browsers.

Attached screen share for the same issue. SearchIssues.mp4

https://github.com/openshift/console/pull/11975

Bug OCPBUGS-3174: Pod logs: Long lines are corrupted when using timestamps=true

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-3117~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-3084~~. The following is the description of the original issue:
—
Upstream Issue: https://github.com/kubernetes/kubernetes/issues/77603

Long log lines get corrupted when using '--timestamps' by the Kubelet.

The root cause is that the buffer reads up to a new line. If the line is greater than 4096 bytes and '--timestamps' is turrned on the kubelet will write the timestamp and the partial log line. We will need to refactor the ReadLogs function to allow for a partial line read.

https://github.com/kubernetes/kubernetes/blob/f892ab1bd7fd97f1fcc2e296e85fdb8e3e8fb82d/pkg/kubelet/kuberuntime/logs/logs.go#L325

apiVersion: v1
kind: Pod
metadata:
  name: logs
spec:
  restartPolicy: Never
  containers:
  - name: logs
    image: fedora
    args:
    - bash
    - -c
    - 'for i in `seq 1 10000000`; do echo -n $i; done'

kubectl logs logs --timestamps

https://github.com/openshift/kubernetes/pull/1408

Bug OCPBUGS-6989: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/console/pull/12524

Bug OCPBUGS-10425: Newly provisioned machines unable to join cluster

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-10372~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-10220~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-7559~~. The following is the description of the original issue:
—
Description of problem:

When attempting to add nodes to a long-lived 4.12.3 cluster, net new nodes are not able to join the cluster. They are provisioned in the cloud provider (AWS), but never actually join as a node.

Version-Release number of selected component (if applicable):

4.12.3

How reproducible:

Consistent

Steps to Reproduce:

1. On a long lived cluster, add a new machineset

Actual results:

Machines reach "Provisioned" but don't join the cluster

Expected results:

Machines join cluster as nodes

Additional info:

https://github.com/openshift/machine-config-operator/pull/3615

Bug OCPBUGS-1955: [4.11] Dual stack cluster fails on installation when multi-path routing entries exist

View the Description View the linked PRs

Description of problem:

We observed that a dual stack cluster deployed with AI gui only fails.
This cluster is dhcp for ipv4, RA/RS autoconfiguration for ipv6.

It fails with error in the onvkube container

```
I0906 07:45:43.044090   87450 gateway_init.go:261] Initializing Gateway Functionality
I0906 07:45:43.046398   87450 gateway_localnet.go:152] Node local addresses initialized to: map[10.131.31.214:{10.131.31.208 fffffff0} 10.255.0.2:{10.255.0.0 fffffe00} 127.0.0.1:{127.0.0.0 ff000000} 2001:1b74:480:613a:f6e9:d4ff:fef1:6f26:{2001:1b74:480:613a:: ffffffffffffffff0000000000000000} ::1:{::1 ffffffffffffffffffffffffffffffff} fd01:0:0:1::2:{fd01:0:0:1:: ffffffffffffffff0000000000000000} fe80::8ce9:b4ff:fe1a:1208:{fe80:: ffffffffffffffff0000000000000000} fe80::c8ef:ecff:fee3:64c7:{fe80:: ffffffffffffffff0000000000000000} fe80::f6e9:d4ff:fef1:6f26:{fe80:: ffffffffffffffff0000000000000000}]
I0906 07:45:43.047759   87450 helper_linux.go:71] Provided gateway interface "br-ex", found as index: 7
I0906 07:45:43.048045   87450 helper_linux.go:97] Found default gateway interface br-ex 10.131.31.209
I0906 07:45:43.048152   87450 helper_linux.go:71] Provided gateway interface "br-ex", found as index: 7
F0906 07:45:43.048318   87450 ovnkube.go:133] failed to get default gateway interface
```

on the node we observed that there is multi-path entry during

```
default proto ra metric 48 pref medium
        nexthop via fe80::e2f6:2d01:ab14:ec71 dev br-ex weight 1
        nexthop via fe80::e2f6:2d01:ab11:c271 dev br-ex weight 1
```

I manually remove one of the entries (`ip route delete`) and then delete the ovnkube-node pod. Then the installation continues, container works.

Every time there is multiple entry, if the onvkube-node starts, it fails.

Version-Release number of selected component (if applicable):

4.10.30

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

There might a side issue: the interface of the node upon boot takes time to get the ipv6 autoconfiguration, no RS packets seemed to be sent out (observed zero on all routers).

https://github.com/openshift/ovn-kubernetes/pull/1296

Bug OCPBUGS-6907: Image registry Operator does not use Proxy when connecting to openstack

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-6517~~. The following is the description of the original issue:
—
Description of problem:

When the cluster is configured with Proxy the swift client in the image registry operator is not using the proxy to authenticate with OpenStack, so it's unable to reach the OpenStack API. This issue became evident since recently the support was added to not fallback to cinder in case swift is available[1].

[1]https://github.com/openshift/cluster-image-registry-operator/pull/819

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1. Deploy a cluster with proxy and restricted installation
2. 
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/cluster-image-registry-operator/pull/836

Bug OCPBUGS-4408: 4.11: When adding nodes, the overlapped node-subnet can be allocated.

View the Description View the linked PRs

Description of problem:

When adding new nodes to the existing cluster, the newly allocated node-subnet can be overlapped with the existing node.

Version-Release number of selected component (if applicable):

openshift 4.10.30

How reproducible:

It's quite hard to reproduce but  there is a possibility it can happen any time.

Steps to Reproduce:

1. Create a OVN dual-stack cluster
2. add nodes to the existing cluster
3. check the allocated node subnet

Actual results:

Some newly added nodes have the same node-subnet and ovn-k8s-mp0 IP as some existing nodes.

Expected results:

Should have duplicated node-subnet and ovn-k8s-mp0 IP

Additional info:

Additional info can be found at the case 03329155 and the must-gather attached(comment #1) 

% omg logs ovnkube-master-v8crc -n openshift-ovn-kubernetes -c ovnkube-master | grep '2022-09-30T06:42:50.857'
2022-09-30T06:42:50.857031565Z W0930 06:42:50.857020       1 master.go:1422] Did not find any logical switches with other-config
2022-09-30T06:42:50.857112441Z I0930 06:42:50.857099       1 master.go:1003] Allocated Subnets [10.131.0.0/23 fd02:0:0:4::/64] on Node worker01.ss1.samsung.local
2022-09-30T06:42:50.857122455Z I0930 06:42:50.857105       1 master.go:1003] Allocated Subnets [10.129.4.0/23 fd02:0:0:a::/64] on Node oam04.ss1.samsung.local
2022-09-30T06:42:50.857130289Z I0930 06:42:50.857122       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.131.0.0/23","fd02:0:0:4::/64"]}] on node worker01.ss1.samsung.local
2022-09-30T06:42:50.857140773Z I0930 06:42:50.857132       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.129.4.0/23","fd02:0:0:a::/64"]}] on node oam04.ss1.samsung.local
2022-09-30T06:42:50.857166726Z I0930 06:42:50.857156       1 master.go:1003] Allocated Subnets [10.128.2.0/23 fd02:0:0:5::/64] on Node oam01.ss1.samsung.local
2022-09-30T06:42:50.857176132Z I0930 06:42:50.857157       1 master.go:1003] Allocated Subnets [10.131.0.0/23 fd02:0:0:4::/64] on Node rhel01.ss1.samsung.local
2022-09-30T06:42:50.857176132Z I0930 06:42:50.857167       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.128.2.0/23","fd02:0:0:5::/64"]}] on node oam01.ss1.samsung.local
2022-09-30T06:42:50.857185257Z I0930 06:42:50.857157       1 master.go:1003] Allocated Subnets [10.128.6.0/23 fd02:0:0:d::/64] on Node call03.ss1.samsung.local
2022-09-30T06:42:50.857192996Z I0930 06:42:50.857183       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.131.0.0/23","fd02:0:0:4::/64"]}] on node rhel01.ss1.samsung.local
2022-09-30T06:42:50.857200017Z I0930 06:42:50.857190       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.128.6.0/23","fd02:0:0:d::/64"]}] on node call03.ss1.samsung.local
2022-09-30T06:42:50.857282717Z I0930 06:42:50.857258       1 master.go:1003] Allocated Subnets [10.130.2.0/23 fd02:0:0:7::/64] on Node call01.ss1.samsung.local
2022-09-30T06:42:50.857304886Z I0930 06:42:50.857293       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.130.2.0/23","fd02:0:0:7::/64"]}] on node call01.ss1.samsung.local
2022-09-30T06:42:50.857338896Z I0930 06:42:50.857314       1 master.go:1003] Allocated Subnets [10.128.4.0/23 fd02:0:0:9::/64] on Node f501.ss1.samsung.local
2022-09-30T06:42:50.857349485Z I0930 06:42:50.857329       1 master.go:1003] Allocated Subnets [10.131.2.0/23 fd02:0:0:8::/64] on Node call02.ss1.samsung.local
2022-09-30T06:42:50.857371344Z I0930 06:42:50.857354       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.128.4.0/23","fd02:0:0:9::/64"]}] on node f501.ss1.samsung.local
2022-09-30T06:42:50.857371344Z I0930 06:42:50.857361       1 kube.go:99] Setting annotations map[k8s.ovn.org/node-subnets:{"default":["10.131.2.0/23","fd02:0:0:8::/64"]}] on node call02.ss1.samsung.local

https://github.com/openshift/ovn-kubernetes/pull/1415

Bug OCPBUGS-2140: member loses rights after some other user login in openid / group sync

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-533~~. The following is the description of the original issue:
—
Description of problem:

customer is using Azure AD as openid provider and groups synchronization from the provider.

The scenario is the following:

1)

user A login.
groups are created with the membership.
User A is member of a group with admin rights and it's cluster-admin

2)

user B login:
groups are updated with membership
UserB is also member of the group with admin rights and it's cluster admin

3)

user A login:
groups are identical as in the former step.
user A has no administration rights.

The groups memberships are the same in step 2 and 3.
The cluster role bindings of the groups have never changed.

the only way to have user A again the admin rights is to delete the membership from the group and have user A login again.

I have not managed to reproduce this using RH SSO. Neither Azure AD.

But my configuration is not exactly the same yet.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/oauth-apiserver/pull/84

Bug OCPBUGS-689: [2112237] [ Cluster storage Operator 4.x(10/11) ] DefaultStorageClassController report fake message "No default StorageClass for this platform" on Alicloud, IBM

View the Description View the linked PRs

Backport to 4.11.z

https://github.com/openshift/cluster-storage-operator/pull/313

Bug OCPBUGS-3830: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/ovn-kubernetes/pull/1383

Bug OCPBUGS-4720: GCP: missing me-west1 region

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-3021~~. The following is the description of the original issue:
—
Description of problem:

me-west1 is not listed in the survey

Steps to Reproduce:

1. run survey (openshift-install create install-config without an install config file)
2. go through prompts until regions
3.

Actual results:

me-west1 region is missing

Expected results:

me-west1 region is listed (and install succeeds in the region)

https://github.com/openshift/installer/pull/6690

Bug OCPBUGS-7127: Editing Pipeline in the ocp console to get information error

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-5016~~. The following is the description of the original issue:
—
Description of problem:

When editing any pipeline in the openshift console, the correct content cannot be obtained (the obtained information is the initial information).

Version-Release number of selected component (if applicable):

How reproducible:

100%

Steps to Reproduce:

Developer -> Pipeline -> select pipeline -> Details -> Actions -> Edit Pipeline -> YAML view -> Cancel ->  Actions -> Edit Pipeline -> YAML view

Actual results:

displayed content is incorrect.

Expected results:

Get the content of the current pipeline, not the "pipeline create" content.

Additional info:

If cancel or save in the "Pipeline Builder" interface after "Edit Pipeline", can get the expected content.
～
Developer -> Pipeline -> select pipeline -> Details -> Actions -> Edit Pipeline -> Pipeline builder -> Cancel ->  Actions -> Edit Pipeline -> YAML view :Display resource content normally
～

https://github.com/openshift/console/pull/12540

Bug OCPBUGS-737: machineconfig service is failed to start because Podman storage gets corrupted

View the Description View the linked PRs

Description of problem:

During ocp multinode spoke cluster creation agent provisioning is stuck on "configuring" because machineConfig service is crashing on the node.
After restarting the service still fails with

Can't read link "/var/lib/containers/storage/overlay/l/V2OP2CCVMKSOHK2XICC546DUCG" because it does not exist. A storage corruption might have occurred, attempting to recreate the missing symlinks. It might be best wipe the storage to avoid further errors due to storage corruption.

Version-Release number of selected component (if applicable):

Podman 4.0.2 +

How reproducible:

sometimes

Steps to Reproduce:

1. deploy multinode spoke (ipxe + boot order )
2.
3.

Actual results:

4 agents in done state and 1 is in "configuring"

Expected results:

all agents are in "done" state

Additional info:

issue mentioned in https://github.com/containers/podman/issues/14003

Fix: https://github.com/containers/storage/issues/1136

https://github.com/openshift/machine-config-operator/pull/3319

Bug OCPBUGS-4640: Prometheus continuously restarts due to slow WAL replay

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-4489~~. The following is the description of the original issue:
—
This is a clone of issue ~~OCPBUGS-4168~~. The following is the description of the original issue:
—
Description of problem:

Prometheus continuously restarts due to slow WAL replay

Version-Release number of selected component (if applicable):

openshift - 4.11.13

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

https://github.com/openshift/cluster-monitoring-operator/pull/1845

Bug OCPBUGS-947: [4.11] Rebase openshift/etcd 4.11 onto 3.5.5

View the Description View the linked PRs

Changelog between 3.5.5 and 3.5.4:
https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md#v355-tbd

Changelog between 3.5.3 and 3.5.4:
https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md#v354-2022-04-24

https://github.com/openshift/etcd/pull/155

Bug OCPBUGS-2447: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/console/pull/12174

Bug OCPBUGS-2640: prometheus-k8s-0 ends in CrashLoopBackOff with evel=error err="opening storage failed: /prometheus/chunks_head/000002: invalid magic number 0" on SNO after hard reboot tests

View the Description View the linked PRs

Description of problem:

prometheus-k8s-0 ends in CrashLoopBackOff with evel=error err="opening storage failed: /prometheus/chunks_head/000002: invalid magic number 0" on SNO after hard reboot tests

Version-Release number of selected component (if applicable):

4.11.6

How reproducible:

Not always, after ~10 attempts

Steps to Reproduce:

1. Deploy SNO with Telco DU profile applied
2. Hard reboot node via out of band interface
3. oc -n openshift-monitoring get pods prometheus-k8s-0

Actual results:

NAME               READY   STATUS             RESTARTS          AGE
prometheus-k8s-0   5/6     CrashLoopBackOff   125 (4m57s ago)   5h28m

Expected results:

Running

Additional info:

Attaching must-gather.

The pod recovers successfully after deleting/re-creating.


[kni@registry.kni-qe-0 ~]$ oc -n openshift-monitoring logs prometheus-k8s-0
ts=2022-09-26T14:54:01.919Z caller=main.go:552 level=info msg="Starting Prometheus Server" mode=server version="(version=2.36.2, branch=rhaos-4.11-rhel-8, revision=0d81ba04ce410df37ca2c0b1ec619e1bc02e19ef)"
ts=2022-09-26T14:54:01.919Z caller=main.go:557 level=info build_context="(go=go1.18.4, user=root@371541f17026, date=20220916-14:15:37)"
ts=2022-09-26T14:54:01.919Z caller=main.go:558 level=info host_details="(Linux 4.18.0-372.26.1.rt7.183.el8_6.x86_64 #1 SMP PREEMPT_RT Sat Aug 27 22:04:33 EDT 2022 x86_64 prometheus-k8s-0 (none))"
ts=2022-09-26T14:54:01.919Z caller=main.go:559 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2022-09-26T14:54:01.919Z caller=main.go:560 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-09-26T14:54:01.921Z caller=web.go:553 level=info component=web msg="Start listening for connections" address=127.0.0.1:9090
ts=2022-09-26T14:54:01.922Z caller=main.go:989 level=info msg="Starting TSDB ..."
ts=2022-09-26T14:54:01.924Z caller=tls_config.go:231 level=info component=web msg="TLS is disabled." http2=false
ts=2022-09-26T14:54:01.926Z caller=main.go:848 level=info msg="Stopping scrape discovery manager..."
ts=2022-09-26T14:54:01.926Z caller=main.go:862 level=info msg="Stopping notify discovery manager..."
ts=2022-09-26T14:54:01.926Z caller=manager.go:951 level=info component="rule manager" msg="Stopping rule manager..."
ts=2022-09-26T14:54:01.926Z caller=manager.go:961 level=info component="rule manager" msg="Rule manager stopped"
ts=2022-09-26T14:54:01.926Z caller=main.go:899 level=info msg="Stopping scrape manager..."
ts=2022-09-26T14:54:01.926Z caller=main.go:858 level=info msg="Notify discovery manager stopped"
ts=2022-09-26T14:54:01.926Z caller=main.go:891 level=info msg="Scrape manager stopped"
ts=2022-09-26T14:54:01.926Z caller=notifier.go:599 level=info component=notifier msg="Stopping notification manager..."
ts=2022-09-26T14:54:01.926Z caller=main.go:844 level=info msg="Scrape discovery manager stopped"
ts=2022-09-26T14:54:01.926Z caller=manager.go:937 level=info component="rule manager" msg="Starting rule manager..."
ts=2022-09-26T14:54:01.926Z caller=main.go:1120 level=info msg="Notifier manager stopped"
ts=2022-09-26T14:54:01.926Z caller=main.go:1129 level=error err="opening storage failed: /prometheus/chunks_head/000002: invalid magic number 0"

https://github.com/openshift/prometheus/pull/143

Bug OCPBUGS-1739: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

View the Description View the linked PRs

The details of this Jira Card are restricted (Red Hat Employee and Contractors only)

https://github.com/openshift/ovn-kubernetes/pull/1290

Bug OCPBUGS-455: [vsphere] update install-config description for diskType

View the Description View the linked PRs

This bug was initially created as a copy of
Bug #2096605
I am copying this bug because: the parent bug solved the validation aspect of diskType but now the description of diskType in
https://github.com/openshift/installer/blob/master/data/data/install.openshift.io_installconfigs.yaml#L2914-L2923
needs to be updated.

Version: 4.11.0-0.nightly-2022-06-06-201913

Platform: vSphere IPI

What happened?
1. If user inputs an invalid value for platform.vsphere.diskType in install-config.yaml file, there is no validation checking for diskType and doesn't exit with error, but continues the installation, which is not the same behavior as in 4.10.

After all vms are provisioned, I checked that the disk provision type is thick.

2. If user doesn't set platform.vsphere.diskType in install-config.yaml file, the default disk provision type is thick, but not the vSphere default storage policy. On VMC, the default policy is thin, so maybe the description of diskType should also need to be updated.

$ ./openshift-install explain installconfig.platform.vsphere.diskType
KIND: InstallConfig
VERSION: v1

RESOURCE: <string>
Valid Values: "","thin","thick","eagerZeroedThick"
DiskType is the name of the disk provisioning type, valid values are thin, thick, and eagerZeroedThick. When not specified, it will be set according to the default storage policy of vsphere.

What did you expect to happen?
validation for diskType

How to reproduce it (as minimally and precisely as possible)?
set diskType to invalid value in install-config.yaml and install the cluster

https://github.com/openshift/installer/pull/6256

Bug OCPBUGS-4786: Topology gets stuck loading

View the Description View the linked PRs

This is a clone of issue ~~OCPBUGS-3235~~. The following is the description of the original issue:
—

Description of problem:

Frequently we see the loading state of the topology view, even when there aren't many resources in the project.

Including an example

Prerequisites (if any, like setup, operators/versions):

Steps to Reproduce

load topology
if it loads successfully, keep trying until it fails to load

Actual results:

topology will sometimes hang with the loading indicator showing indefinitely

Expected results:

topology should load consistently without fail

Reproducibility (Always/Intermittent/Only Once):

intermittent

Build Details:

4.9

Additional info:

https://github.com/openshift/console/pull/12311

4.11.36

Changes from 4.10.67

Complete Features

Epic Goal

Why is this important?

Scenarios

Acceptance Criteria

Dependencies (internal and external)

Previous Work (Optional):

Open questions::

Done Checklist

Epic Goal

Why is this important?

Scenarios

Acceptance Criteria

Dependencies (internal and external)

Previous Work (Optional):

Open questions::

Done Checklist

Problem Alignment

The Problem

High-Level Approach

Goal & Success

Solution Alignment

Key Capabilities

Key Flows

Epic Goal

Why is this important?

Scenarios

Acceptance Criteria

Dependencies (internal and external)

Previous Work (Optional):

Open questions:

Done Checklist

Epic Goal

Why is this important?

Scenarios

Acceptance Criteria

Dependencies (internal and external)

Previous Work (Optional):

Open questions:

Done Checklist

Epic Goal

Why is this important?

Scenarios

Acceptance Criteria

Dependencies (internal and external)

Previous Work

Open Questions

Description

Acceptance Criteria

Summary (PM+lead)

Motivation (PM+lead)

Goals (lead)

Non-Goals (lead)

Deliverables

Proposal (lead)

User Stories (PM)

Dependencies (internal and external, lead)

Previous Work (lead)

Open questions (lead)

Done Checklist

🏆 What

💖 Why

🗒 Notes

Feature Overview.

Goals

Requirements

(Optional) Use Cases

Questions to answer…

Out of Scope

Background, and strategic fit

Assumptions

Customer Considerations

Documentation Considerations

Epic Goal

Why is this important?

Scenarios

Acceptance Criteria

Dependencies (internal and external)