openshift/installer
1
0
Fork 0
mirror of https://github.com/openshift/installer.git synced 2026-02-05 15:47:14 +01:00
Code Issues Releases Activity
Files
main
installer/docs/user/aws/limits.md

95 lines
5.3 KiB
Markdown
Raw Permalink Normal View History

Adding user doc/guide for AWS account and installation
2019-01-09 12:13:17 -05:00
# Limits
You can find a comprehensive list of the default AWS service limits published here:
[AWS Service Limits][service-limits]
Below, we'll identify OpenShift cluster needs and how those impact some of those limits.
## S3
There is a default limit of 100 S3 buckets per account. The installation creates a bucket temporarily. Also, the
registry component creates a permanent bucket. This will limit the number of clusters per account to 99 initially. To
support additional clusters, you must open a support case with AWS.
## VPC
Each cluster creates its own VPC. The default limit of VPCs per region is 5 and will allow 5 clusters. To have more
than 5 clusters, you will need to increase this limit.
updating docs to include deails about VPC gateway endpoints
2019-02-22 18:17:26 -05:00
Each cluster also creates a VPC Gateway Endpoint for a stable connection to S3. The default limit of VPC Gateway
Endpoints per region is 20 and will allow 20 clusters. To have more than 20 clusters, you will need to increase this
limit.
Adding user doc/guide for AWS account and installation
2019-01-09 12:13:17 -05:00
## Elastic Network Interfaces (ENI)
data/aws/vpc: Only create subnet infrastucture for zones with Machine(Set)s This commit updates our Terraform variables to include the worker subnets, and then switches on that (and the master zones) in Terraform to avoid creating subnet infrastructure (NAT gateways, routes, etc.) in zones that have no Machine(Set)s. This helps address limit issues in high-zone regions like us-east-1, as seen in the limits.md change. Note that without a reduction in our default MachineSet creation, the installer defaults will still not work on us-east-1 without a limit bump. The drawback is that users are now on the hook to provision their own subnets in other zones if they decide that they want to grow into a new zone as a day-2 Machine(Set) operation. For now, they'll have to provide their own infrastructure for that, and our user-provided-infrastructure docs should give them sufficient grounding to do so. It's possible that in the future the machine-API or other infrastructure operator could dynamically provision subnets in zones that were not populated at install-time, but I can't hazard a guess as to how likely that will be. The HCL functions for combining the zone lists are documented in [1,2]. [1]: https://www.terraform.io/docs/configuration-0-11/interpolation.html#concat-list1-list2- [2]: https://www.terraform.io/docs/configuration-0-11/interpolation.html#distinct-list-
2019-03-27 14:53:34 -07:00
The default installation creates 21 + the number of availability zones of ENIs (e.g. 21 + 3 = 24 ENIs for a three-zone cluster).
Adding user doc/guide for AWS account and installation
2019-01-09 12:13:17 -05:00
The default limit per region is 350. Additional ENIs are created for additional machines and elastic load balancers
created by cluster usage and deployed workloads. A service limit increase here may be required to satisfy the needs of
additional clusters and deployed workloads.
## Elastic IP (EIP)
data/aws/vpc: Only create subnet infrastucture for zones with Machine(Set)s This commit updates our Terraform variables to include the worker subnets, and then switches on that (and the master zones) in Terraform to avoid creating subnet infrastructure (NAT gateways, routes, etc.) in zones that have no Machine(Set)s. This helps address limit issues in high-zone regions like us-east-1, as seen in the limits.md change. Note that without a reduction in our default MachineSet creation, the installer defaults will still not work on us-east-1 without a limit bump. The drawback is that users are now on the hook to provision their own subnets in other zones if they decide that they want to grow into a new zone as a day-2 Machine(Set) operation. For now, they'll have to provide their own infrastructure for that, and our user-provided-infrastructure docs should give them sufficient grounding to do so. It's possible that in the future the machine-API or other infrastructure operator could dynamically provision subnets in zones that were not populated at install-time, but I can't hazard a guess as to how likely that will be. The HCL functions for combining the zone lists are documented in [1,2]. [1]: https://www.terraform.io/docs/configuration-0-11/interpolation.html#concat-list1-list2- [2]: https://www.terraform.io/docs/configuration-0-11/interpolation.html#distinct-list-
2019-03-27 14:53:34 -07:00
By default, the installer distributes control-plane and compute machines across [all availability zones within a region][availability-zones] to provision the cluster in a highly available configuration.
Please see [this map][az-map] for a current region map with availability zone count.
We recommend selecting regions with 3 or more availability zones.
You can [provide an install-config](../overview.md#multiple-invocations) to [configure](customization.md) the installer to use specific zones to override that default.
Adding user doc/guide for AWS account and installation
2019-01-09 12:13:17 -05:00
data/aws/vpc: Only create subnet infrastucture for zones with Machine(Set)s This commit updates our Terraform variables to include the worker subnets, and then switches on that (and the master zones) in Terraform to avoid creating subnet infrastructure (NAT gateways, routes, etc.) in zones that have no Machine(Set)s. This helps address limit issues in high-zone regions like us-east-1, as seen in the limits.md change. Note that without a reduction in our default MachineSet creation, the installer defaults will still not work on us-east-1 without a limit bump. The drawback is that users are now on the hook to provision their own subnets in other zones if they decide that they want to grow into a new zone as a day-2 Machine(Set) operation. For now, they'll have to provide their own infrastructure for that, and our user-provided-infrastructure docs should give them sufficient grounding to do so. It's possible that in the future the machine-API or other infrastructure operator could dynamically provision subnets in zones that were not populated at install-time, but I can't hazard a guess as to how likely that will be. The HCL functions for combining the zone lists are documented in [1,2]. [1]: https://www.terraform.io/docs/configuration-0-11/interpolation.html#concat-list1-list2- [2]: https://www.terraform.io/docs/configuration-0-11/interpolation.html#distinct-list-
2019-03-27 14:53:34 -07:00
The installer creates a public and private subnet for each configured availability zone.
In each private subnet, a separate [NAT Gateway][nat-gateways] is created and requires a separate [EC2-VPC Elastic IP (EIP)][elastic-ip].
The default limit of 5 is sufficient for a single cluster, unless you have configured your cluster to use more than five zones.
For multiple clusters, a higher limit will likely be required (and will certainly be required to support more than five clusters, even if they are each single-zone clusters).
Adding user doc/guide for AWS account and installation
2019-01-09 12:13:17 -05:00
data/aws/vpc: Only create subnet infrastucture for zones with Machine(Set)s This commit updates our Terraform variables to include the worker subnets, and then switches on that (and the master zones) in Terraform to avoid creating subnet infrastructure (NAT gateways, routes, etc.) in zones that have no Machine(Set)s. This helps address limit issues in high-zone regions like us-east-1, as seen in the limits.md change. Note that without a reduction in our default MachineSet creation, the installer defaults will still not work on us-east-1 without a limit bump. The drawback is that users are now on the hook to provision their own subnets in other zones if they decide that they want to grow into a new zone as a day-2 Machine(Set) operation. For now, they'll have to provide their own infrastructure for that, and our user-provided-infrastructure docs should give them sufficient grounding to do so. It's possible that in the future the machine-API or other infrastructure operator could dynamically provision subnets in zones that were not populated at install-time, but I can't hazard a guess as to how likely that will be. The HCL functions for combining the zone lists are documented in [1,2]. [1]: https://www.terraform.io/docs/configuration-0-11/interpolation.html#concat-list1-list2- [2]: https://www.terraform.io/docs/configuration-0-11/interpolation.html#distinct-list-
2019-03-27 14:53:34 -07:00
### Example: Using North Virginia (us-east-1)
docs: correct spelling in docs Correct spelling to improve the readability of the documentation
2021-08-22 12:11:02 +03:00
North Virginia (us-east-1) has six availability zones, so a higher limit is required unless you configure your cluster to use fewer zones.
data/aws/vpc: Only create subnet infrastucture for zones with Machine(Set)s This commit updates our Terraform variables to include the worker subnets, and then switches on that (and the master zones) in Terraform to avoid creating subnet infrastructure (NAT gateways, routes, etc.) in zones that have no Machine(Set)s. This helps address limit issues in high-zone regions like us-east-1, as seen in the limits.md change. Note that without a reduction in our default MachineSet creation, the installer defaults will still not work on us-east-1 without a limit bump. The drawback is that users are now on the hook to provision their own subnets in other zones if they decide that they want to grow into a new zone as a day-2 Machine(Set) operation. For now, they'll have to provide their own infrastructure for that, and our user-provided-infrastructure docs should give them sufficient grounding to do so. It's possible that in the future the machine-API or other infrastructure operator could dynamically provision subnets in zones that were not populated at install-time, but I can't hazard a guess as to how likely that will be. The HCL functions for combining the zone lists are documented in [1,2]. [1]: https://www.terraform.io/docs/configuration-0-11/interpolation.html#concat-list1-list2- [2]: https://www.terraform.io/docs/configuration-0-11/interpolation.html#distinct-list-
2019-03-27 14:53:34 -07:00
To support the default, all-zone installation, please submit a limit increase for VPC Elastic IPs similar to the following in the support dashboard (to create more than one cluster, a higher limit will be necessary):
Adding user doc/guide for AWS account and installation
2019-01-09 12:13:17 -05:00
![Increase Elastic IP limit in AWS](images/support_increase_elastic_ip.png)
## NAT Gateway
The default limit for NAT Gateways is 5 per availability zone. This is sufficient for up to 5 clusters in a dedicated
account. If you intend to create more than 5 clusters, you will need to request an increase to this limit.
## VPC Gateway
The default limit of VPC Gateways (for S3 access) is 20. Each cluster will create a single S3 gateway endpoint within
the new VPC. If you intend to create more than 20 clusters, you will need to request an increase to this limit.
## Security Groups
Removing text counts of explicit numbered list in the AWS doc.
2019-02-26 17:51:12 -05:00
Each cluster creates distinct security groups. The default limit of 2,500 for new accounts allows for many clusters
to be created. The security groups which exist after the default install are:
Removing unused/deprecated security groups and ports. Updated AWS user doc. Summary: - Console security group unused, replaced along the way by Router ELB - API security group unused, no longer in use after switch to NLB - Ports 80/443 not required, console using service port and Router ELB - Master port 22623 only targeted from inside the VPC - API server only on 6443 (6444 & 6445 carryover from Tectonic) - SSH port only targeted from inside the VPC with move to private - Heapster (Port 4194) no longer used - Master port 10255 (insecure) deprecated and removed in OpenShift - Renamed 10250 to secure, because it is the secure kubelet port
2019-02-23 17:31:36 -05:00
1. VPC default
1. Master
1. Worker
1. Router/Ingress
Adding user doc/guide for AWS account and installation
2019-01-09 12:13:17 -05:00
aws: add new vCPU limit to docs AWS recently changed the instance limit to a vCPU limit Fixes #2736
2019-12-07 03:37:02 +01:00
## vCPU Limits
Adding user doc/guide for AWS account and installation
2019-01-09 12:13:17 -05:00
aws: Double the master instance size We were seeing frequent long requests from etcd. After increasing CPU (2 -> 4 cores) those pauses dropped significantly. Increase the limit until the rebase lands and we can deliver the CPU perf improvements to the control plane. Make a set of changes to connect machine set size to the values passed as input. Update the docs.
2019-01-15 13:24:17 -05:00
By default, a cluster will create:
aws: add new vCPU limit to docs AWS recently changed the instance limit to a vCPU limit Fixes #2736
2019-12-07 03:37:02 +01:00
* One m4.large bootstrap machine (2 vCPUs but removed after install)
Default AWS instance type to 'm5'
2020-07-07 15:06:08 -04:00
* Three m5.xlarge master nodes (4 vCPUs each).
* Three m5.large worker nodes (2 vCPUs each).
aws: Double the master instance size We were seeing frequent long requests from etcd. After increasing CPU (2 -> 4 cores) those pauses dropped significantly. Increase the limit until the rebase lands and we can deliver the CPU perf improvements to the control plane. Make a set of changes to connect machine set size to the values passed as input. Update the docs.
2019-01-15 13:24:17 -05:00
aws: add new vCPU limit to docs AWS recently changed the instance limit to a vCPU limit Fixes #2736
2019-12-07 03:37:02 +01:00
Currently, these vCPU counts are not within a new account's default limit. The default limit is 1 but for all these instances you will need 20. To increase the limit you have to [contact the AWS support](https://console.aws.amazon.com/support/cases?#/create?issueType=service-limit-increase&limitType=ec2-instances).
aws: Double the master instance size We were seeing frequent long requests from etcd. After increasing CPU (2 -> 4 cores) those pauses dropped significantly. Increase the limit until the rebase lands and we can deliver the CPU perf improvements to the control plane. Make a set of changes to connect machine set size to the values passed as input. Update the docs.
2019-01-15 13:24:17 -05:00
If you intend to start with a higher number of workers, enable autoscaling and large workloads
aws: add new vCPU limit to docs AWS recently changed the instance limit to a vCPU limit Fixes #2736
2019-12-07 03:37:02 +01:00
or a different instance type, please ensure you have the necessary remaining vCPU count within the vCPU
limit to satisfy the need. To calculate the vCPU limit you can use the limits calculator in the EC2 console (EC2 -> Limits -> Limits calculator).
Adding user doc/guide for AWS account and installation
2019-01-09 12:13:17 -05:00
## Elastic Load Balancing (ELB/NLB)
By default, each cluster will create 2 network load balancers for the master API server (1 internal, 1 external) and a
single classic elastic load balancer for the router. Additional Kubernetes LoadBalancer Service objects will create
docs/user/aws/limits: Add a trailing period to the LB paragraph Fixing an omission from b383cd75 (Adding user doc/guide for AWS account and installation, 2019-01-09, #1030). While I'm doing pedantic things at the end of the file, also alphabetize the link defintions.
2019-01-17 22:01:23 -08:00
additional [load balancers][load-balancing].
Adding user doc/guide for AWS account and installation
2019-01-09 12:13:17 -05:00
[availability-zones]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html
[az-map]: https://aws.amazon.com/about-aws/global-infrastructure/
docs/user/aws/limits: Add a trailing period to the LB paragraph Fixing an omission from b383cd75 (Adding user doc/guide for AWS account and installation, 2019-01-09, #1030). While I'm doing pedantic things at the end of the file, also alphabetize the link defintions.
2019-01-17 22:01:23 -08:00
[elastic-ip]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html
[load-balancing]: https://aws.amazon.com/elasticloadbalancing/
[nat-gateways]: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html
[service-limits]: https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html
Copy Permalink