If you are utilizing vSphere with Tanzu in conjunction with NSX, you can implement multiple T0 Gateways to further segregate your network traffic. To do this, you must set one T0 Gateway as the default during installation. Later, you can configure additional T0 Gateways and selectively override the default T0 Gateway on a per-vSphere-Namespace-basis.
In a really big environment i found our that if you try do deploy a Tanzu Worker Cluster in an additional Segment, other than the one your Supervisor Cluster reside, they never get starting to work. Your will see that Tanzu start to install but never get any infos back.
Difference between NAT and No-NAT
During both, the installation process and vSphere Namespace creation, you have the option to enable or disable Network Address Translation (NAT). Enabling NAT entails two important considerations:
Specifying an Egress-Network, which must be routed. A dedicated Egress IP is assigned to each vSphere Namespace. When objects such as K8s Nodes or Pods within the vSphere Namespace initiate communication with external entities (DNS, ActiveDirectory, NTP,…), they are source NAT’ed to the dedicated Egress IP (one dedicated IP per vSphere Namespace).
Avoiding the need for Namespace Network Routing: Since the communication is source NAT’ed, there is no requirement to route the entire Namespace Network (which contains the K8s Nodes) outside of NSX. To prevent such routing from happening, a RouteMap is installed on the T0 Gateway (this is done automatically).
The Problem
If you now create a vSphere Namespace on a T0 Gateway, other than the default, and try to create a K8s Cluster, only the first Controlplane VM will be created and it will never finish. Thus, no further nodes will be created and the cluster never finishes either.
In most cases, enabling NAT for your supervisor cluster is the preferred approach. Routing a large network such as the Namespace network is a rather rare scenario. However, when creating a K8s cluster (kind cluster or TanzuKubernetesCluster) specific controllers such as tkg-controller or capv need to establish connections to the K8s cluster. These controllers operate as Pods running on the SupervisorClusterControlePlaneNodes. And they utilize a vNIC connected to NSX within one of the Namespace Networks. When network routing is not enabled (due to NAT being disabled), these controllers are unable to establish connections to the K8s clusters residing on another T0 Gateway.
The “to-easy-fix”
Based on our current understanding, one might suggest disabling NAT on the Supervisor during the installation process. While this approach may seem obvious, it is worth noting that in vSphere 8U1 (haven’t tested other version), this is not working. If NAT is disabled during installation, the wizard fails to validate the provided input and instead generates a generic error message:
Invalid field ‘workloads’ in structure ‘com.vmware.vcenter.namespace_management.supervisor.enable_on_compute_cluster_spec’
The real fix (maybe?)
Another approach is, to remove/modify the RouteMap. This comes with another challenge. Since this RouteMapwas created during installation, it is owned by a NSX superuser, hence it can’t be deleted from the GUI.
Instead you have to delete it via API. The first step is to get the current RouteMapconfiguration as JSON
❯ curl –insecure -u admin:‘Password123!’ -X GET “https://nsxt-1.vraccoon.lab/policy/api/v1/infra/tier-0s/T0-Tanzu-1/route-maps”
Line 5-6 has the prefix list from earlier listed. Whatever networks are listed there, will be denied from beeing advertised. Thus, we will remove this prefixlist from this RouteMap. To do so, I’ll save the output, remove the lines and reupload it.
❯ curl –insecure -u admin:‘Password123!’ -X GET “https://nsxt-1.vraccoon.lab/policy/api/v1/infra/tier-0s/T0-Tanzu-1/route-maps/rm_deny_t1_subnets” > rm_deny_t1_subnets.json
NOTE 1: Overwriting the automagically installed RouteMap is probably not supported. Also, if I’d really do this in production, I’d probably modify the prefixlist to only propagate the NSX based SupervisorControlPlane Segment.
It might now take a few minutes, but the cluster provisioning will continue eventually.
To check connectivity, you can try to ping the corresponding segment’s gateway IP. The K8s nodes themself do not reply on icmp requests. But you could do port scans e.g. with netcat.
NOTE 2: There is a prefixlist on your T0 Gateway called like “pl-domain-<clusterMOBID>:<supervisor-uuid>-deny-t1-subnets”. Creating a vSphere Namespace with NAT enabled, adds an entry (containing the Namespace Network) to that prefixlist. This prefixlist is then used as matching criteria on a route-map “rm-deny-t1-subnets” to prevent these networks from being advertised. For some reasons, these entries will always be added to the default T0 Gateway prefixlist (set up during Supervisor installation). Which means, that Namespace Networks from vSphere Namespaces on other T0 Gateways (by using “Override Supervisor network settings“) will always be advertised, unless you create a route map manually. This is probably a bug.
The original Post was from a good friend, vracoon, and can be found here.
