openshift-docs/modules/ossm-rn-fixed-issues.adoc

////
Module included in the following assemblies:
* service_mesh/v2x/servicemesh-release-notes.adoc
////
:_mod-docs-content-type: REFERENCE
[id="ossm-rn-fixed-issues_{context}"]
= Fixed issues

////
Provide the following info for each issue if possible:
*Consequence* - What user action or situation would make this problem appear (If you have the foo option enabled and did x)? What did the customer experience as a result of the issue? What was the symptom?
*Cause* - Why did this happen?
*Fix* - What did we change to fix the problem?
*Result* - How has the behavior changed as a result? Try to avoid “It is fixed” or “The issue is resolved” or “The error no longer presents”.
////

The following issues have been resolved in previous releases:

[id="ossm-rn-fixed-issues-ossm_{context}"]
== {SMProductShortName} fixed issues
//The explanations of these issues have been reviewed/approved in previous releases.
* https://issues.redhat.com/browse/OSSM-6177[OSSM-6177] Previously, when validation messages were enabled in the `ServiceMeshControlPlane` (SMCP), the `istiod` crashed continuously unless `GatewayAPI` support was enabled. Now, when validation messages are enabled but `GatewayAPI` support is not, the `istiod` does not continuously crash.

* https://issues.redhat.com/browse/OSSM-6163[OSSM-6163] Resolves the following issues:
+
** Previously, an unstable Prometheus image was included in the {SMProductShortName} control plane (SMCP) v2.5, and users were not able to access the Prometheus dashboard. Now, in the {SMProductShortName} operator 2.5.1, the Prometheus image has been updated.
+
** Previously, in the {SMProductShortName} control plane (SMCP), a Grafana data source was not able to set Basic authentication password automatically and users were not able to view metrics from Prometheus in Grafana mesh dashboards. Now, a Grafana data source password is configured under the `secureJsonData` field. Metrics are displayed correctly in dashboards.

* https://issues.redhat.com/browse/OSSM-6148[OSSM-6148] Previously, the {SMPlugin} did not respond when the user clicked any option in the menu of any node on the **Traffic Graph** page. Now, the plugin responds to the selected option in the menu by redirecting to the corresponding details page.

* https://issues.redhat.com/browse/OSSM-6099[OSSM-6099] Previously, the {SMPlugin} failed to load correctly in an IPv6 cluster. Now, the OSSMC plugin configuration has been modified to ensure proper loading in an IPv6 cluster.

* https://issues.redhat.com/browse/OSSM-5960[OSSM-5960] Previously, the {SMPlugin} did not display notification messages such as backend errors or Istio validations. Now, these notifications are displayed correctly at the top of the plugin page.

* https://issues.redhat.com/browse/OSSM-5959[OSSM-5959] Previously, the {SMPlugin} did not display TLS and Istio certification information in the **Overview** page. Now, this information is displayed correctly.

* https://issues.redhat.com/browse/OSSM-5902[OSSM-5902] Previously, the {SMPlugin} redirected to a "Not Found Page" error when the user clicked the **Istio config** health symbol on the **Overview** page. Now, the plugin redirects to the correct **Istio config** details page.

* https://issues.redhat.com/browse/OSSM-5541[OSSM-5541] Previously, an Istio operator pod might keep waiting for the leader lease in some restart conditions. Now, the leader election implementation has been enhanced to avoid this issue.

* https://issues.redhat.com/browse/OSSM-1397[OSSM-1397] Previously, if you removed the `maistra.io/member-of` label from a namespace, the {SMProductShortName} Operator did not automatically reapply the label to the namespace. As a result, sidecar injection did not work in the namespace.
+
The Operator would reapply the label to the namespace when you made changes to the `ServiceMeshMember` object, which triggered the reconciliation of this member object.
+
Now, any change to the namespace also triggers the member object reconciliation.

* https://issues.redhat.com/browse/OSSM-3647[OSSM-3647] Previously, in the {SMProductShortName} control plane (SMCP) v2.2 (Istio 1.12), WasmPlugins were applied only to inbound listeners. Since SMCP v2.3 (Istio 1.14), WasmPlugins have been applied to inbound and outbound listeners by default, which introduced regression for users of the 3scale WasmPlugin. Now, the environment variable `APPLY_WASM_PLUGINS_TO_INBOUND_ONLY` is added, which allows safe migration from SMCP v2.2 to v2.3 and v2.4.
+
The following setting should be added to the SMCP config:
+
[source, yaml]
----
spec:
  runtime:
    components:
      pilot:
        container:
          env:
            APPLY_WASM_PLUGINS_TO_INBOUND_ONLY: "true"

----
+
To ensure safe migration, perform the following steps:
+
--
. Set `APPLY_WASM_PLUGINS_TO_INBOUND_ONLY` in SMCP v2.2.
. Upgrade to 2.4.
. Set `spec.match[].mode: SERVER` in WasmPlugins.
. Remove the previously-added environment variable.
--

* https://issues.redhat.com/browse/OSSM-4851[OSSM-4851] Previously, an error occurred in the operator deploying new pods in a namespace scoped inside the mesh when `runAsGroup`, `runAsUser`, or `fsGroup` parameters were `nil`. Now, a yaml validation has been added to avoid the `nil` value.

* https://issues.redhat.com/browse/OSSM-3771[OSSM-3771] Previously, OpenShift routes could not be disabled for additional ingress gateways defined in a Service Mesh Control Plane (SMCP). Now, a `routeConfig` block can be added to each `additionalIngress` gateway so the creation of OpenShift routes can be enabled or disabled for each gateway.

* https://issues.redhat.com/browse/OSSM-4197[OSSM-4197] Previously, if you deployed a v2.2 or v2.1 of the 'ServiceMeshControlPlane' resource, the `/etc/cni/multus/net.d/` directory was not created. As a result, the `istio-cni` pod failed to become ready, and the `istio-cni` pods log contained the following message:
+
[source,terminal]
----
$ error   Installer exits with open /host/etc/cni/multus/net.d/v2-2-istio-cni.kubeconfig.tmp.841118073: no such file or directory
----
+
Now, if you deploy a v2.2 or v2.1 of the 'ServiceMeshControlPlane' resource, the `/etc/cni/multus/net.d/` directory is created, and the `istio-cni` pod becomes ready.

* https://issues.redhat.com/browse/OSSM-3993[OSSM-3993] Previously, Kiali only supported OpenShift OAuth via a proxy on the standard HTTPS port of `443`. Now, Kiali supports OpenShift OAuth over a non-standard HTTPS port. To enable the port, you must set the `spec.server.web_port` field to the proxy's non-standard HTTPS port in the Kiali CR.

* https://issues.redhat.com/browse/OSSM-3936[OSSM-3936] Previously, the values for the `injection_label_rev` and `injection_label_name` attributes were hardcoded. This prevented custom configurations from taking effect in the Kiali Custom Resource Definition (CRD). Now, the attribute values are not hardcoded. You can customize the values for the `injection_label_rev` and `injection_label_name` attributes in the `spec.istio_labels` specification.

* https://issues.redhat.com/browse/OSSM-3644[OSSM-3644] Previously, the federation egress-gateway received the wrong update of network gateway endpoints, causing extra endpoint entries. Now, the federation-egress gateway has been updated on the server side so it receives the correct network gateway endpoints.

* https://issues.redhat.com/browse/OSSM-3595[OSSM-3595] Previously, the `istio-cni` plugin sometimes failed on {op-system-base} because SELinux did not allow the utility `iptables-restore` to open files in the `/tmp` directory. Now, SELinux passes `iptables-restore` via `stdin` input stream instead of via a file.

* https://issues.redhat.com/browse/OSSM-3586[OSSM-3586] Previously, Istio proxies were slow to start when {gcp-first} metadata servers were not available. When you upgrade to Istio 1.14.6, Istio proxies start as expected on {gcp-short}, even if metadata servers are not available.

* https://issues.redhat.com/browse/OSSM-3025[OSSM-3025] Istiod sometimes fails to become ready. Sometimes, when a mesh contained many member namespaces, the Istiod pod did not become ready due to a deadlock within Istiod. The deadlock is now resolved and the pod now starts as expected.

* https://issues.redhat.com/browse/OSSM-2493[OSSM-2493] Default `nodeSelector` and `tolerations` in SMCP not passed to Kiali. The `nodeSelector` and `tolerations` you add to `SMCP.spec.runtime.defaults` are now passed to the Kiali resource.

* https://issues.redhat.com/browse/OSSM-2492[OSSM-2492] Default tolerations in SMCP not passed to Jaeger. The `nodeSelector` and `tolerations` you add to `SMCP.spec.runtime.defaults` are now passed to the Jaeger resource.

* https://issues.redhat.com/browse/OSSM-2374[OSSM-2374] If you deleted one of the `ServiceMeshMember` resources, then the Service Mesh operator deleted the `ServiceMeshMemberRoll`. While this is expected behavior when you delete the last `ServiceMeshMember`, the operator should not delete the `ServiceMeshMemberRoll` if it contains any members in addition to the one that was deleted. This issue is now fixed and the operator only deletes the ServiceMeshMemberRoll when the last `ServiceMeshMember` resource is deleted.

* https://issues.redhat.com/browse/OSSM-2373[OSSM-2373] Error trying to get OAuth metadata when logging in. To fetch the cluster version, the `system:anonymous` account is used. With the cluster's default bundled ClusterRoles and ClusterRoleBinding, the anonymous account can fetch the version correctly. If the `system:anonymous` account loses its privileges to fetch the cluster version, OpenShift authentication becomes unusable.
+
This is fixed by using the Kiali SA to fetch the cluster version. This also allows for improved security on the cluster.

* https://issues.redhat.com/browse/OSSM-2371[OSSM-2371] Despite Kiali being configured as "view-only," a user can change the proxy logging level via the Workload details' Logs tab's kebab menu. This issue has been fixed so the options under "Set Proxy Log Level" are disabled when Kiali is configured as "view-only."

* https://issues.redhat.com/browse/OSSM-2344[OSSM-2344] Restarting Istiod causes Kiali to flood CRI-O with port-forward requests. This issue occurred when Kiali could not connect to Istiod and Kiali simultaneously issued a large number of requests to istiod. Kiali now limits the number of requests it sends to istiod.

* https://issues.redhat.com/browse/OSSM-2335[OSSM-2335] Dragging the mouse pointer over the Traces scatterchart plot sometimes caused the Kiali console to stop responding due to concurrent backend requests.

* https://issues.redhat.com/browse/OSSM-2221[OSSM-2221] Previously, gateway injection in the `ServiceMeshControlPlane` namespace was not possible because the `ignore-namespace` label was applied to the namespace by default.
+
When creating a v2.4 control plane, the namespace no longer has the `ignore-namespace` label applied, and gateway injection is possible.
+
In the following example, the `oc label` command removes the `ignore-namespace` label from a namespace in an existing deployment:
+
[source,terminal]
----
$ oc label namespace istio-system maistra.io/ignore-namespace-
----
+
where:
+
--
istio_system :: Specified the name of the `ServiceMeshControlPlane` namespace.
--

* https://issues.redhat.com/browse/OSSM-2053[OSSM-2053] Using {SMProductName} Operator 2.2 or 2.3, during SMCP reconciliation, the SMMR controller removed the member namespaces from `SMMR.status.configuredMembers`. This caused the services in the member namespaces to become unavailable for a few moments.
+
Using {SMProductName} Operator 2.2 or 2.3, the SMMR controller no longer removes the namespaces from `SMMR.status.configuredMembers`. Instead, the controller adds the namespaces to `SMMR.status.pendingMembers` to indicate that they are not up-to-date. During reconciliation, as each namespace synchronizes with the SMCP, the namespace is automatically removed from `SMMR.status.pendingMembers`.

* https://issues.redhat.com/browse/OSSM-1962[OSSM-1962] Use `EndpointSlices` in federation controller. The federation controller now uses `EndpointSlices`, which improves scalability and performance in large deployments. The PILOT_USE_ENDPOINT_SLICE flag is enabled by default. Disabling the flag prevents use of federation deployments.

* https://issues.redhat.com/browse/OSSM-1668[OSSM-1668] A new field `spec.security.jwksResolverCA` was added to the Version 2.1 `SMCP` but was missing in the 2.2.0 and 2.2.1 releases. When upgrading from an Operator version where this field was present to an Operator version that was missing this field, the `.spec.security.jwksResolverCA` field was not available in the `SMCP`.

* https://issues.redhat.com/browse/OSSM-1325[OSSM-1325] istiod pod crashes and displays the following error message: `fatal error: concurrent map iteration and map write`.

* https://issues.redhat.com/browse/OSSM-1211[OSSM-1211]
Configuring Federated service meshes for failover does not work as expected.
+
The Istiod pilot log displays the following error: `envoy connection [C289] TLS error: 337047686:SSL routines:tls_process_server_certificate:certificate verify failed`

* https://issues.redhat.com/browse/OSSM-1099[OSSM-1099]
The Kiali console displayed the message `Sorry, there was a problem. Try a refresh or navigate to a different page.`

* https://issues.redhat.com/browse/OSSM-1074[OSSM-1074]
Pod annotations defined in SMCP are not injected in the pods.

* https://issues.redhat.com/browse/OSSM-999[OSSM-999]
Kiali retention did not work as expected. Calendar times were greyed out in the dashboard graph.

* link:https://issues.redhat.com/browse/OSSM-797[OSSM-797] Kiali Operator pod generates `CreateContainerConfigError` while installing or updating the operator.

* https://issues.redhat.com/browse/OSSM-722[OSSM-722]
Namespace starting with `kube` is hidden from Kiali.

* link:https://issues.redhat.com/browse/OSSM-569[OSSM-569] There is no CPU memory limit for the Prometheus `istio-proxy` container. The Prometheus `istio-proxy` sidecar now uses the resource limits defined in `spec.proxy.runtime.container`.

* link:https://issues.redhat.com/browse/OSSM-535[OSSM-535] Support validationMessages in SMCP. The `ValidationMessages` field in the Service Mesh Control Plane can now be set to `True`. This writes a log for the status of the resources, which can be helpful when troubleshooting problems.

* link:https://issues.redhat.com/browse/OSSM-449[OSSM-449] VirtualService and Service causes an error "Only unique values for domains are permitted. Duplicate entry of domain."

* link:https://issues.redhat.com/browse/OSSM-419[OSSM-419] Namespaces with similar names will all show in Kiali namespace list, even though namespaces may not be defined in Service Mesh Member Role.

* link:https://issues.redhat.com/browse/OSSM-296[OSSM-296] When adding health configuration to the Kiali custom resource (CR) is it not being replicated to the Kiali configmap.

* link:https://issues.redhat.com/browse/OSSM-291[OSSM-291] In the Kiali console, on the Applications, Services, and Workloads pages, the "Remove Label from Filters" function is not working.

* link:https://issues.redhat.com/browse/OSSM-289[OSSM-289] In the Kiali console, on the Service Details pages for the 'istio-ingressgateway' and 'jaeger-query' services there are no Traces being displayed. The traces exist in Jaeger.

* link:https://issues.redhat.com/browse/OSSM-287[OSSM-287] In the Kiali console there are no traces being displayed on the Graph Service.

* link:https://issues.redhat.com/browse/OSSM-285[OSSM-285] When trying to access the Kiali console, receive the following error message "Error trying to get OAuth Metadata".
+
Workaround: Restart the Kiali pod.

* link:https://issues.redhat.com/browse/MAISTRA-2735[MAISTRA-2735] The resources that the Service Mesh Operator deletes when reconciling the SMCP changed in {SMProductName} version 2.1. Previously, the Operator deleted a resource with the following labels:

** `maistra.io/owner`
** `app.kubernetes.io/version`

+
Now, the Operator ignores resources that does not also include the `app.kubernetes.io/managed-by=maistra-istio-operator` label. If you create your own resources, you should not add the `app.kubernetes.io/managed-by=maistra-istio-operator` label to them.


* link:https://issues.jboss.org/browse/MAISTRA-2687[MAISTRA-2687] {SMProductName} 2.1 federation gateway does not send the full certificate chain when using external certificates. The {SMProductShortName} federation egress gateway only sends the client certificate. Because the federation ingress gateway only knows about the root certificate, it cannot verify the client certificate unless you add the root certificate to the federation import `ConfigMap`.

* link:https://issues.redhat.com/browse/MAISTRA-2635[MAISTRA-2635] Replace deprecated Kubernetes API. To remain compatible with {product-title} 4.8, the `apiextensions.k8s.io/v1beta1` API was deprecated as of {SMProductName} 2.0.8.

* link:https://issues.redhat.com/browse/MAISTRA-2631[MAISTRA-2631] The WASM feature is not working because podman is failing due to nsenter binary not being present. {SMProductName} generates the following error message: `Error: error configuring CNI network plugin exec: "nsenter": executable file not found in $PATH`. The container image now contains nsenter and WASM works as expected.

* link:https://issues.redhat.com/browse/MAISTRA-2534[MAISTRA-2534] When istiod attempted to fetch the JWKS for an issuer specified in a JWT rule, the issuer service responded with a 502.  This prevented the proxy container from becoming ready and caused deployments to hang. The fix for the link:https://github.com/istio/istio/issues/24629[community bug] has been included in the  {SMProductShortName} 2.0.7 release.

* link:https://issues.jboss.org/browse/MAISTRA-2411[MAISTRA-2411] When the Operator creates a new ingress gateway using `spec.gateways.additionaIngress` in the `ServiceMeshControlPlane`, Operator is not creating a `NetworkPolicy` for the additional ingress gateway like it does for the default istio-ingressgateway. This is causing a 503 response from the route of the new gateway.
+
Workaround: Manually create the `NetworkPolicy` in the `istio-system` namespace.

* link:https://issues.redhat.com/browse/MAISTRA-2401[MAISTRA-2401] CVE-2021-3586 servicemesh-operator: NetworkPolicy resources incorrectly specified ports for ingress resources. The NetworkPolicy resources installed for {SMProductName} did not properly specify which ports could be accessed. This allowed access to all ports on these resources from any pod. Network policies applied to the following resources are affected:

** Galley
** Grafana
** Istiod
** Jaeger
** Kiali
** Prometheus
** Sidecar injector

* link:https://issues.redhat.com/browse/MAISTRA-2378[MAISTRA-2378] When the cluster is configured to use OpenShift SDN with `ovs-multitenant` and the mesh contains a large number of namespaces (200+), the {product-title} networking plugin is unable to configure the namespaces quickly. {SMProductShortName} times out causing namespaces to be continuously dropped from the service mesh and then reenlisted.

* link:https://issues.redhat.com/browse/MAISTRA-2370[MAISTRA-2370] Handle tombstones in listerInformer. The updated cache codebase was not handling tombstones when translating the events from the namespace caches to the aggregated cache, leading to a panic in the go routine.

* link:https://issues.redhat.com/browse/MAISTRA-2117[MAISTRA-2117] Add optional `ConfigMap` mount to operator. The CSV now contains an optional `ConfigMap` volume mount, which mounts the `smcp-templates` `ConfigMap` if it exists. If the `smcp-templates` `ConfigMap` does not exist, the mounted directory is empty. When you create the `ConfigMap`, the directory is populated with the entries from the `ConfigMap` and can be referenced in `SMCP.spec.profiles`. No restart of the Service Mesh operator is required.
+
Customers using the 2.0 operator with a modified CSV to mount the smcp-templates ConfigMap can upgrade to {SMProductName} 2.1. After upgrading, you can continue using an existing ConfigMap, and the profiles it contains, without editing the CSV. Customers that previously used ConfigMap with a different name will either have to rename the ConfigMap or update the CSV after upgrading.

* link:https://issues.redhat.com/browse/MAISTRA-2010[MAISTRA-2010] AuthorizationPolicy does not support `request.regex.headers` field. The `validatingwebhook` rejects any AuthorizationPolicy with the field, and even if you disable that, Pilot tries to validate it using the same code, and it does not work.

* link:https://issues.jboss.org/browse/MAISTRA-1979[MAISTRA-1979] _Migration to 2.0_ The conversion webhook drops the following important fields when converting `SMCP.status` from v2 to v1:

** conditions
** components
** observedGeneration
** annotations
+
Upgrading the operator to 2.0 might break client tools that read the SMCP status using the maistra.io/v1 version of the resource.
+
This also causes the READY and STATUS columns to be empty when you run `oc get servicemeshcontrolplanes.v1.maistra.io`.

ifndef::openshift-rosa-hcp,openshift-rosa,openshift-dedicated[]
* link:https://issues.jboss.org/browse/MAISTRA-1947[MAISTRA-1947] _Technology Preview_ Updates to ServiceMeshExtensions are not applied.
+
Workaround: Remove and recreate the `ServiceMeshExtensions`.
endif::openshift-rosa-hcp,openshift-rosa,openshift-dedicated[]

* link:https://issues.redhat.com/browse/MAISTRA-1983[MAISTRA-1983] _Migration to 2.0_ Upgrading to 2.0.0 with an existing invalid `ServiceMeshControlPlane` cannot easily be repaired. The invalid items in the `ServiceMeshControlPlane` resource caused an unrecoverable error. The fix makes the errors recoverable. You can delete the invalid resource and replace it with a new one or edit the resource to fix the errors. For more information about editing your resource, see [Configuring the Red Hat OpenShift Service Mesh installation].

* link:https://issues.redhat.com/browse/MAISTRA-1502[MAISTRA-1502] As a result of CVEs fixes in version 1.0.10, the Istio dashboards are not available from the *Home Dashboard* menu in Grafana. To access the Istio dashboards, click the *Dashboard* menu in the navigation panel and select the *Manage* tab.

* link:https://issues.redhat.com/browse/MAISTRA-1399[MAISTRA-1399] {SMProductName} no longer prevents you from installing unsupported CNI protocols. The supported network configurations has not changed.

* link:https://issues.jboss.org/browse/MAISTRA-1089[MAISTRA-1089] _Migration to 2.0_ Gateways created in a non-control plane namespace are automatically deleted. After removing the gateway definition from the SMCP spec, you need to manually delete these resources.

* link:https://issues.jboss.org/browse/MAISTRA-858[MAISTRA-858] The following Envoy log messages describing link:https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated[deprecated options and configurations associated with Istio 1.1.x] are expected:
+
** [2019-06-03 07:03:28.943][19][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.api.v2.listener.Filter.config'. This configuration will be removed from Envoy soon.
** [2019-08-12 22:12:59.001][13][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option 'envoy.api.v2.Listener.use_original_dst' from file lds.proto. This configuration will be removed from Envoy soon.

* link:https://issues.jboss.org/browse/MAISTRA-806[MAISTRA-806] Evicted Istio Operator Pod causes mesh and CNI not to deploy.
+
Workaround: If the `istio-operator` pod is evicted while deploying the control pane, delete the evicted `istio-operator` pod.

* link:https://issues.jboss.org/browse/MAISTRA-681[MAISTRA-681] When the {SMProductShortName} control plane has many namespaces, it can lead to performance issues.

* link:https://issues.jboss.org/browse/MAISTRA-193[MAISTRA-193] Unexpected console info messages are visible when health checking is enabled for citadel.

* link:https://bugzilla.redhat.com/show_bug.cgi?id=1821432[Bugzilla 1821432] The toggle controls in {product-title} Custom Resource details page does not update the CR correctly. UI Toggle controls in the {SMProductShortName} Control Plane (SMCP) Overview page in the {product-title} web console sometimes updates the wrong field in the resource. To update a SMCP, edit the YAML content directly or update the resource from the command line instead of clicking the toggle controls.