openshift/installer
1
0
Fork 0
mirror of https://github.com/openshift/installer.git synced 2026-02-05 15:47:14 +01:00
Code Issues Releases Activity

Adding initial user doc/guide & materials for UPI AWS installation

Browse Source 
Moved the encrypted-AMI section from "Create Cluster" to "Running
Cluster", because it has more value there folks inspecting their
account and what has happened after the fact.  Since we call out the
running instances (by count with a picture) they may wonder "where did
this unique AMI come from it is running"?  It goes along with some of
the other explanations we chase with as well.  The Create Cluster
section just has the whole IPI output, but no explanation of all
wizardry under the covers, calling this particular wrinkle out there
seemed a bit awkward to me.
This commit is contained in:
Stephen Cuppett
2019-03-12 20:56:09 -04:00
committed by W. Trevor King
parent 710f9d3e41
commit 39a926a918
10 changed files with 2059 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/user/aws/README.md
View File

@@ -4,7 +4,7 @@ This document is a guide for preparing a new AWS account for use with OpenShift.
will help prepare an account to create a single cluster and provide insight for adjustments which may be
needed for additional clusters.
Follow along with the steps and links below to configure your AWS cluster:
Follow along with the steps and links below to configure your AWS account and provision an OpenShift cluster:
1. [Route53](route53.md)
2. [Limits](limits.md)

BIN
docs/user/aws/images/install_upi_vpc.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 91 KiB

24
docs/user/aws/install.md
View File

@@ -1,9 +1,15 @@
# Cluster Installation
At this point, you are ready to perform the OpenShift installation outlined [here][cloud-install] and begin at
Step 3: Download the Installer.
At this point, you are ready to perform the OpenShift installation. You have two choices for installing your cluster on
AWS, Installer-Provided Infrastructure (IPI) or User-Provided Infrastructure (UPI). See below for an example of an
IPI install.
## Example
To see a guided example of a UPI install, please see [Install: User-Provided Infrastructure](install_upi.md)
## Example: Installer-Provided Infrastructure (IPI)
The steps for performing an IPI-based install are outlined [here][cloud-install]. Following this guide you may begin at
Step 3: Download the Installer.
### Create Configuration
@@ -33,11 +39,6 @@ INFO Access the OpenShift web-console here: https://console-openshift-console.ap
INFO Login to the console with user: kubeadmin, password: XXXX
```
This creates an encrypted AMI for the bootstrap and control-plane machines.
The encrypted AMI is [copied][encrypted-copy] from the AMI configured in the control-plane machine-API provider spec, which is RHCOS by default.
The encryption uses the default EBS key for your target account and region (`aws kms describe-key --key-id alias/aws/ebs`).
The encrypted AMI is deregistered by `destroy cluster`.
### Running Cluster
In Route53, there will be a new, private hosted zone (for internal lookups):
@@ -48,6 +49,13 @@ In EC2, there will be 6 running instances:
![EC2 instances after install](images/install_nodes.png)
The installation creates an encrypted AMI for the bootstrap and control-plane machines.
The encrypted AMI is [copied][encrypted-copy] from the AMI configured in the control-plane machine-API provider spec,
which is RHCOS by default.
The encryption uses the default EBS key for your target account and region
(`aws kms describe-key --key-id alias/aws/ebs`).
The encrypted AMI is deregistered by `destroy cluster`.
The relationship of the EC2 instances, elastic load balancers (ELBs) and Route53 hosted zones is as depicted:
![Architecture relationship of ELBs and instances](images/install_nodes_elbs.png)

277
docs/user/aws/install_upi.md Normal file
View File

@@ -0,0 +1,277 @@
# Install: User Provided Infrastructure (UPI)
The steps for performing a UPI-based install are outlined here. Several [CloudFormation][cloudformation] templates are
provided to assist in completing these steps or to help model your own. You are also free to create the required
resources through other methods; the CloudFormation templates are just an example.
## Create Ignition Configs
The machines will be started manually. Therefore, it is required to generate the bootstrap and machine Ignition configs
and store them for later steps.
```console
$ openshift-install-linux-amd64 create ignition-configs
? SSH Public Key /home/user_id/.ssh/id_rsa.pub
? Platform aws
? Region us-east-1
? Base Domain example.com
? Cluster Name openshift
? Pull Secret [? for help]
```
After running the command, several files will be available in the directory.
```console
$ tree
.
├── auth
│   └── kubeconfig
├── bootstrap.ign
├── master.ign
├── metadata.json
└── worker.ign
```
## Create/Identify the VPC to be Used
You may create a VPC with various desirable characteristics for your situation (VPN, route tables, etc.). The
VPC configuration and a CloudFormation template is provided [here](../../../upi/aws/cloudformation/01_vpc.yaml).
A created VPC via the template or manually should approximate a setup similar to this:
![Example VPC Architecture Diagram](images/install_upi_vpc.png)
## Create DNS entries and Load Balancers for Control Plane Components
The DNS and load balancer configuration within a CloudFormation template is provided
[here](../../../upi/aws/cloudformation/02_cluster_infra.yaml). It uses a public hosted zone and creates a private hosted
zone similar to the IPI installation method. It also creates load balancers and listeners the same way as the IPI
installation method. This template can be run multiple times within a single VPC and in combination with the VPC
template provided above.
### Optional: Manually Create Load Balancer Configuration
It is needed to create a TCP load balancer for ports 6443 (the Kubernetes API and its extensions) and 22623 (Ignition
configurations for new machines). The targets will be the master nodes. Port 6443 must be accessible to both clients
external to the cluster and nodes within the cluster. Port 22623 must be accessible to nodes within the cluster.
### Optional: Manually Create Route53 Hosted Zones & Records
For the cluster name identified earlier in [Create Ignition Configs](#create-ignition-configs), you must create a DNS
entry which resolves to your created load balancer. The entry `api.$clustername.$domain` should point to the load balancer.
## Create Security Groups and IAM Roles
The security group and IAM configuration within a CloudFormation template is provided
[here](../../../upi/aws/cloudformation/03_cluster_security.yaml). Run this template to get the minimal and permanent
set of security groups and IAM roles needed for an operational cluster. It can also be inspected for the current
set of required rules to facilitate manual creation.
## Launch Temporary Bootstrap Resource
The bootstrap launch and other necessary, temporary security group plus IAM configuration and a CloudFormation
template is provided [here](../../../upi/aws/cloudformation/04_cluster_bootstrap.yaml). Upload your generated `bootstrap.ign`
file to an S3 bucket in your account and run this template to get a bootstrap node along with a predictable clean up of
the resources when complete. It can also be inspected for the set of required attributes via manual creation.
## Launch Permanent Master Nodes
The master launch and other necessary DNS entries for etcd are provided within a CloudFormation
template [here](../../../upi/aws/cloudformation/05_cluster_master_nodes.yaml). Run this template to get three master
nodes. It can also be inspected for the set of required attributes needed for manual creation of the nodes, DNS entries
and load balancer configuration.
## Monitor for `bootstrap-complete` and Initialization
```console
$ bin/openshift-install user-provided-infrastructure bootstrap-complete
INFO Waiting up to 30m0s for the Kubernetes API at https://api.test.example.com:6443...
INFO API v1.12.4+c53f462 up
INFO Waiting up to 30m0s for the bootstrap-complete event...
```
## Destroy Bootstrap Resources
At this point, you should delete the bootstrap resources. If using the CloudFormation template, you would [delete the
stack][delete-stack] created for the bootstrap to clean up all the temporary resources.
## Cleanup Machine API Resources
By querying the Machine API, you'll notice the cluster is attempting to reconcile the predefined
Machine and MachineSet definitions. We will begin to correct that here. In this step, we delete
the pre-defined master nodes. Our masters are not controlled by the Machine API.
### Example: Deleting Master Machine Definitions
```console
$ export KUBECONFIG=auth/kubeconfig
$ oc get machines --namespace openshift-machine-api
NAME INSTANCE STATE TYPE REGION ZONE AGE
test-tkh7l-master-0 m4.xlarge us-east-2 us-east-2a 9m22s
test-tkh7l-master-1 m4.xlarge us-east-2 us-east-2b 9m22s
test-tkh7l-master-2 m4.xlarge us-east-2 us-east-2c 9m21s
test-tkh7l-worker-us-east-2a-qjcxq m4.large us-east-2 us-east-2a 8m6s
test-tkh7l-worker-us-east-2b-nq8zs m4.large us-east-2 us-east-2b 8m6s
test-tkh7l-worker-us-east-2c-ww6c6 m4.large us-east-2 us-east-2c 8m7s
$ oc delete machine --namespace openshift-machine-api test-tkh7l-master-0
machine.machine.openshift.io "test-tkh7l-master-0" deleted
$ oc delete machine --namespace openshift-machine-api test-tkh7l-master-1
machine.machine.openshift.io "test-tkh7l-master-1" deleted
$ oc delete machine --namespace openshift-machine-api test-tkh7l-master-2
machine.machine.openshift.io "test-tkh7l-master-2" deleted
```
## Launch Additional Worker Nodes
To launch workers, you are able to launch individual EC2 instances discretely or by automated processes outside the
cluster (e.g. Auto Scaling Groups). However, you are also able to take advantage of the built in cluster scaling mechanisms
and the machine API in OCP.
### Option 1: Dynamic Compute using Machine API
By default, MachineSets are created and will have failed to launch. We can correct the desired subnet filter,
target security group, RHEL CoreOS AMI and EC2 instance profile.
```console
$ oc get machinesets --namespace openshift-machine-api
NAME DESIRED CURRENT READY AVAILABLE AGE
test-tkh7l-worker-us-east-2a 1 1 11m
test-tkh7l-worker-us-east-2b 1 1 11m
test-tkh7l-worker-us-east-2c 1 1 11m
```
```console
$ oc get machineset --namespace openshift-machine-api test-tkh7l-worker-us-east-2a -o yaml
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
creationTimestamp: 2019-03-14T14:03:03Z
generation: 1
labels:
machine.openshift.io/cluster-api-cluster: test-tkh7l
machine.openshift.io/cluster-api-machine-role: worker
machine.openshift.io/cluster-api-machine-type: worker
name: test-tkh7l-worker-us-east-2a
namespace: openshift-machine-api
resourceVersion: "2350"
selfLink: /apis/machine.openshift.io/v1beta1/namespaces/openshift-machine-api/machinesets/test-tkh7l-worker-us-east-2a
uid: e2a6c8a6-4661-11e9-a9b0-0296069fd3a2
spec:
replicas: 1
selector:
matchLabels:
machine.openshift.io/cluster-api-cluster: test-tkh7l
machine.openshift.io/cluster-api-machineset: test-tkh7l-worker-us-east-2a
template:
metadata:
creationTimestamp: null
labels:
machine.openshift.io/cluster-api-cluster: test-tkh7l
machine.openshift.io/cluster-api-machine-role: worker
machine.openshift.io/cluster-api-machine-type: worker
machine.openshift.io/cluster-api-machineset: test-tkh7l-worker-us-east-2a
spec:
metadata:
creationTimestamp: null
providerSpec:
value:
ami:
id: ami-0eecbb884c8b35b1e
apiVersion: awsproviderconfig.openshift.io/v1beta1
blockDevices:
- ebs:
iops: 0
volumeSize: 120
volumeType: gp2
credentialsSecret:
name: aws-cloud-credentials
deviceIndex: 0
iamInstanceProfile:
id: test-tkh7l-worker-profile
instanceType: m4.large
kind: AWSMachineProviderConfig
metadata:
creationTimestamp: null
placement:
availabilityZone: us-east-2a
region: us-east-2
publicIp: null
securityGroups:
- filters:
- name: tag:Name
values:
- test-tkh7l-worker-sg
subnet:
filters:
- name: tag:Name
values:
- test-tkh7l-private-us-east-2a
tags:
- name: kubernetes.io/cluster/test-tkh7l
value: owned
userDataSecret:
name: worker-user-data
versions:
kubelet: ""
status:
fullyLabeledReplicas: 1
observedGeneration: 1
replicas: 1
```
At this point, you'd edit the YAML to update the relevant values to match your UPI installation.
```console
$ oc edit machineset --namespace openshift-machine-api test-tkh7l-worker-us-east-2a
machineset.machine.openshift.io/test-tkh7l-worker-us-east-2a edited
```
Once the Machine API has a chance to reconcile and begin launching hosts with the correct attributes, you
should start to see new output in your EC2 console and oc commands.
```console
$ oc get machines --namespace openshift-machine-api
NAME INSTANCE STATE TYPE REGION ZONE AGE
test-tkh7l-worker-us-east-2a-hxlqn i-0e7f3a52b2919471e pending m4.4xlarge us-east-2 us-east-2a 3s
```
### Option 2: Manually Launching Worker Instances
The worker launch is provided within a CloudFormation template [here](../../../upi/aws/cloudformation/06_cluster_worker_node.yaml).
You can launch a CloudFormation stack to manage each individual worker. A similar launch configuration could be used by
outside automation or AWS auto scaling groups.
## Configure Router for UPI DNS
TODO: Identify changes needed to Router or Ingress for DNS `*.apps` registration or LoadBalancer creation.
## Monitor for Cluster Completion
```console
$ bin/openshift-install user-provided-infrastructure finish
INFO Waiting up to 30m0s for the cluster to initialize...
```
Also, you can observe the running state of your cluster pods:
```console
$ oc get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-member-ip-10-0-3-111.us-east-2.compute.internal 1/1 Running 0 35m
kube-system etcd-member-ip-10-0-3-239.us-east-2.compute.internal 1/1 Running 0 37m
kube-system etcd-member-ip-10-0-3-24.us-east-2.compute.internal 1/1 Running 0 35m
openshift-apiserver-operator openshift-apiserver-operator-6d6674f4f4-h7t2t 1/1 Running 1 37m
openshift-apiserver apiserver-fm48r 1/1 Running 0 30m
openshift-apiserver apiserver-fxkvv 1/1 Running 0 29m
openshift-apiserver apiserver-q85nm 1/1 Running 0 29m
...
openshift-service-ca-operator openshift-service-ca-operator-66ff6dc6cd-9r257 1/1 Running 0 37m
openshift-service-ca apiservice-cabundle-injector-695b6bcbc-cl5hm 1/1 Running 0 35m
openshift-service-ca configmap-cabundle-injector-8498544d7-25qn6 1/1 Running 0 35m
openshift-service-ca service-serving-cert-signer-6445fc9c6-wqdqn 1/1 Running 0 35m
openshift-service-catalog-apiserver-operator openshift-service-catalog-apiserver-operator-549f44668b-b5q2w 1/1 Running 0 32m
openshift-service-catalog-controller-manager-operator openshift-service-catalog-controller-manager-operator-b78cr2lnm 1/1 Running 0 31m
```
[cloudformation]: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html
[delete-stack]: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-delete-stack.html

437
upi/aws/cloudformation/01_vpc.yaml Normal file
View File

@@ -0,0 +1,437 @@
AWSTemplateFormatVersion: 2010-09-09
Description: Template for Best Practice VPC with 1-3 AZs
Parameters:
VpcCidr:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-4]))$
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16-24
Default: 10.0.0.0/16
Description: CIDR block for VPC
Type: String
AvailabilityZoneCount:
ConstraintDescription: "The number of availability zones (Min: 1, Max: 3)"
MinValue: 1
MaxValue: 3
Default: 1
Description: "How many AZs to create VPC subnets for (Min: 1, Max: 3)"
Type: Number
SubnetBits:
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/19-27
MinValue: 5
MaxValue: 13
Default: 12
Description: "Size of each subnet to create within the availability zones. (Min: 5 = /27, Max: 13 = /19)"
Type: Number
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Network Configuration"
Parameters:
- VpcCidr
- SubnetBits
- Label:
default: "Availability Zones"
Parameters:
- AvailabilityZoneCount
ParameterLabels:
AvailabilityZoneCount:
default: "Availability Zone Count"
VpcCidr:
default: "VPC CIDR"
SubnetBits:
default: "Bits Per Subnet"
Conditions:
DoAz3: !Equals [3, !Ref AvailabilityZoneCount]
DoAz2: !Or [!Equals [2, !Ref AvailabilityZoneCount], Condition: DoAz3]
Resources:
VPC:
Type: "AWS::EC2::VPC"
Properties:
EnableDnsSupport: "true"
EnableDnsHostnames: "true"
CidrBlock: !Ref VpcCidr
Tags:
- Key: Application
Value: !Ref "AWS::StackName"
- Key: Network
Value: Public
- Key: Name
Value: !Ref "AWS::StackName"
PublicSubnet:
Type: "AWS::EC2::Subnet"
Properties:
VpcId: !Ref VPC
CidrBlock: !Select [0, !Cidr [!Ref VpcCidr, 6, !Ref SubnetBits]]
AvailabilityZone: !Select
- 0
- Fn::GetAZs: !Ref "AWS::Region"
Tags:
- Key: Application
Value: !Ref "AWS::StackName"
- Key: Network
Value: Public
PublicSubnet2:
Type: "AWS::EC2::Subnet"
Condition: DoAz2
Properties:
VpcId: !Ref VPC
CidrBlock: !Select [1, !Cidr [!Ref VpcCidr, 6, !Ref SubnetBits]]
AvailabilityZone: !Select
- 1
- Fn::GetAZs: !Ref "AWS::Region"
Tags:
- Key: Application
Value: !Ref "AWS::StackName"
- Key: Network
Value: Public
PublicSubnet3:
Type: "AWS::EC2::Subnet"
Condition: DoAz3
Properties:
VpcId: !Ref VPC
CidrBlock: !Select [2, !Cidr [!Ref VpcCidr, 6, !Ref SubnetBits]]
AvailabilityZone: !Select
- 2
- Fn::GetAZs: !Ref "AWS::Region"
Tags:
- Key: Application
Value: !Ref "AWS::StackName"
- Key: Network
Value: Public
InternetGateway:
Type: "AWS::EC2::InternetGateway"
Properties:
Tags:
- Key: Application
Value: !Ref "AWS::StackName"
- Key: Network
Value: Public
GatewayToInternet:
Type: "AWS::EC2::VPCGatewayAttachment"
Properties:
VpcId: !Ref VPC
InternetGatewayId: !Ref InternetGateway
PublicRouteTable:
Type: "AWS::EC2::RouteTable"
Properties:
VpcId: !Ref VPC
Tags:
- Key: Application
Value: !Ref "AWS::StackName"
- Key: Network
Value: Public
PublicRoute:
Type: "AWS::EC2::Route"
DependsOn: GatewayToInternet
Properties:
RouteTableId: !Ref PublicRouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref InternetGateway
PublicSubnetRouteTableAssociation:
Type: "AWS::EC2::SubnetRouteTableAssociation"
Properties:
SubnetId: !Ref PublicSubnet
RouteTableId: !Ref PublicRouteTable
PublicSubnetRouteTableAssociation2:
Type: "AWS::EC2::SubnetRouteTableAssociation"
Condition: DoAz2
Properties:
SubnetId: !Ref PublicSubnet2
RouteTableId: !Ref PublicRouteTable
PublicSubnetRouteTableAssociation3:
Condition: DoAz3
Type: "AWS::EC2::SubnetRouteTableAssociation"
Properties:
SubnetId: !Ref PublicSubnet3
RouteTableId: !Ref PublicRouteTable
PublicNetworkAcl:
Type: "AWS::EC2::NetworkAcl"
Properties:
VpcId: !Ref VPC
Tags:
- Key: Application
Value: !Ref "AWS::StackName"
- Key: Network
Value: Public
InboundHTTPPublicNetworkAclEntry:
Type: "AWS::EC2::NetworkAclEntry"
Properties:
NetworkAclId: !Ref PublicNetworkAcl
RuleNumber: "100"
Protocol: "6"
RuleAction: allow
Egress: "false"
CidrBlock: 0.0.0.0/0
PortRange:
From: "80"
To: "80"
InboundHTTPSPublicNetworkAclEntry:
Type: "AWS::EC2::NetworkAclEntry"
Properties:
NetworkAclId: !Ref PublicNetworkAcl
RuleNumber: "101"
Protocol: "6"
RuleAction: allow
Egress: "false"
CidrBlock: 0.0.0.0/0
PortRange:
From: "443"
To: "443"
InboundSSHPublicNetworkAclEntry:
Type: "AWS::EC2::NetworkAclEntry"
Properties:
NetworkAclId: !Ref PublicNetworkAcl
RuleNumber: "102"
Protocol: "6"
RuleAction: allow
Egress: "false"
CidrBlock: 0.0.0.0/0
PortRange:
From: "22"
To: "22"
InboundEphemeralPublicNetworkAclEntry:
Type: "AWS::EC2::NetworkAclEntry"
Properties:
NetworkAclId: !Ref PublicNetworkAcl
RuleNumber: "103"
Protocol: "6"
RuleAction: allow
Egress: "false"
CidrBlock: 0.0.0.0/0
PortRange:
From: "1024"
To: "65535"
OutboundPublicNetworkAclEntry:
Type: "AWS::EC2::NetworkAclEntry"
Properties:
NetworkAclId: !Ref PublicNetworkAcl
RuleNumber: "100"
Protocol: "6"
RuleAction: allow
Egress: "true"
CidrBlock: 0.0.0.0/0
PortRange:
From: "0"
To: "65535"
PublicSubnetNetworkAclAssociation:
Type: "AWS::EC2::SubnetNetworkAclAssociation"
Properties:
SubnetId: !Ref PublicSubnet
NetworkAclId: !Ref PublicNetworkAcl
PublicSubnetNetworkAclAssociation2:
Type: "AWS::EC2::SubnetNetworkAclAssociation"
Condition: DoAz2
Properties:
SubnetId: !Ref PublicSubnet2
NetworkAclId: !Ref PublicNetworkAcl
PublicSubnetNetworkAclAssociation3:
Type: "AWS::EC2::SubnetNetworkAclAssociation"
Condition: DoAz3
Properties:
SubnetId: !Ref PublicSubnet3
NetworkAclId: !Ref PublicNetworkAcl
PrivateSubnet:
Type: "AWS::EC2::Subnet"
Properties:
VpcId: !Ref VPC
CidrBlock: !Select [3, !Cidr [!Ref VpcCidr, 6, !Ref SubnetBits]]
AvailabilityZone: !Select
- 0
- Fn::GetAZs: !Ref "AWS::Region"
Tags:
- Key: Application
Value: !Ref "AWS::StackName"
- Key: Network
Value: Private
PrivateRouteTable:
Type: "AWS::EC2::RouteTable"
Properties:
VpcId: !Ref VPC
Tags:
- Key: Application
Value: !Ref "AWS::StackName"
- Key: Network
Value: Private
PrivateSubnetRouteTableAssociation:
Type: "AWS::EC2::SubnetRouteTableAssociation"
Properties:
SubnetId: !Ref PrivateSubnet
RouteTableId: !Ref PrivateRouteTable
NAT:
DependsOn:
- GatewayToInternet
Type: "AWS::EC2::NatGateway"
Properties:
AllocationId:
"Fn::GetAtt":
- EIP
- AllocationId
SubnetId: !Ref PublicSubnet
EIP:
Type: "AWS::EC2::EIP"
Properties:
Domain: vpc
Route:
Type: "AWS::EC2::Route"
Properties:
RouteTableId:
Ref: PrivateRouteTable
DestinationCidrBlock: 0.0.0.0/0
NatGatewayId:
Ref: NAT
PrivateSubnet2:
Type: "AWS::EC2::Subnet"
Condition: DoAz2
Properties:
VpcId: !Ref VPC
CidrBlock: !Select [4, !Cidr [!Ref VpcCidr, 6, !Ref SubnetBits]]
AvailabilityZone: !Select
- 1
- Fn::GetAZs: !Ref "AWS::Region"
Tags:
- Key: Application
Value: !Ref "AWS::StackName"
- Key: Network
Value: Private
PrivateRouteTable2:
Type: "AWS::EC2::RouteTable"
Condition: DoAz2
Properties:
VpcId: !Ref VPC
Tags:
- Key: Application
Value: !Ref "AWS::StackName"
- Key: Network
Value: Private
PrivateSubnetRouteTableAssociation2:
Type: "AWS::EC2::SubnetRouteTableAssociation"
Condition: DoAz2
Properties:
SubnetId: !Ref PrivateSubnet2
RouteTableId: !Ref PrivateRouteTable2
NAT2:
DependsOn:
- GatewayToInternet
Type: "AWS::EC2::NatGateway"
Condition: DoAz2
Properties:
AllocationId:
"Fn::GetAtt":
- EIP2
- AllocationId
SubnetId: !Ref PublicSubnet2
EIP2:
Type: "AWS::EC2::EIP"
Condition: DoAz2
Properties:
Domain: vpc
Route2:
Type: "AWS::EC2::Route"
Condition: DoAz2
Properties:
RouteTableId:
Ref: PrivateRouteTable2
DestinationCidrBlock: 0.0.0.0/0
NatGatewayId:
Ref: NAT2
PrivateSubnet3:
Type: "AWS::EC2::Subnet"
Condition: DoAz3
Properties:
VpcId: !Ref VPC
CidrBlock: !Select [5, !Cidr [!Ref VpcCidr, 6, !Ref SubnetBits]]
AvailabilityZone: !Select
- 2
- Fn::GetAZs: !Ref "AWS::Region"
Tags:
- Key: Application
Value: !Ref "AWS::StackName"
- Key: Network
Value: Private
PrivateRouteTable3:
Type: "AWS::EC2::RouteTable"
Condition: DoAz3
Properties:
VpcId: !Ref VPC
Tags:
- Key: Application
Value: !Ref "AWS::StackName"
- Key: Network
Value: Private
PrivateSubnetRouteTableAssociation3:
Type: "AWS::EC2::SubnetRouteTableAssociation"
Condition: DoAz3
Properties:
SubnetId: !Ref PrivateSubnet3
RouteTableId: !Ref PrivateRouteTable3
NAT3:
DependsOn:
- GatewayToInternet
Type: "AWS::EC2::NatGateway"
Condition: DoAz3
Properties:
AllocationId:
"Fn::GetAtt":
- EIP3
- AllocationId
SubnetId: !Ref PublicSubnet3
EIP3:
Type: "AWS::EC2::EIP"
Condition: DoAz3
Properties:
Domain: vpc
Route3:
Type: "AWS::EC2::Route"
Condition: DoAz3
Properties:
RouteTableId:
Ref: PrivateRouteTable3
DestinationCidrBlock: 0.0.0.0/0
NatGatewayId:
Ref: NAT3
S3Endpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal: '*'
Action:
- '*'
Resource:
- '*'
RouteTableIds:
- !Ref PublicRouteTable
- !Ref PrivateRouteTable
- !If [DoAz2, !Ref PrivateRouteTable2, !Ref "AWS::NoValue"]
- !If [DoAz3, !Ref PrivateRouteTable3, !Ref "AWS::NoValue"]
ServiceName: !Join
- ''
- - com.amazonaws.
- !Ref 'AWS::Region'
- .s3
VpcId: !Ref VPC
Outputs:
VpcId:
Description: ID of the newly created VPC
Value: !Ref VPC
PublicSubnetIds:
Description: Subnet IDs of the public subnets
Value:
!Join [
",",
[!Ref PublicSubnet, !If [DoAz2, !Ref PublicSubnet2, !Ref "AWS::NoValue"], !If [DoAz3, !Ref PublicSubnet3, !Ref "AWS::NoValue"]]
]
PrivateSubnetIds:
Description: Subnet IDs of the private subnets
Value:
!Join [
",",
[!Ref PrivateSubnet, !If [DoAz2, !Ref PrivateSubnet2, !Ref "AWS::NoValue"], !If [DoAz3, !Ref PrivateSubnet3, !Ref "AWS::NoValue"]]
]

283
upi/aws/cloudformation/02_cluster_infra.yaml Normal file
View File

@@ -0,0 +1,283 @@
AWSTemplateFormatVersion: 2010-09-09
Description: Template for Openshift Cluster UPI Network Elements (Route53 & LBs)
Parameters:
ClusterName:
AllowedPattern: ^([a-zA-Z][a-zA-Z0-9]{0,31})$
ConstraintDescription: Cluster name must be alphanumeric, start with a letter and a maximum of 32 characters
Description: A short, representative cluster name to use for hostnames, etc.
Type: String
HostedZoneId:
Description: The Route53 public zone ID to register the targets with (e.g Z21IXYZABCZ2A4)
Type: String
HostedZoneName:
Description: The Route53 zone to register the targets with (No trailing dot - e.g. mycorp.com)
Type: String
Default: "example.com"
PublicSubnets:
Description: The internet-facing subnets
Type: List<AWS::EC2::Subnet::Id>
PrivateSubnets:
Description: The internal subnets
Type: List<AWS::EC2::Subnet::Id>
VpcId:
Description: The VPC created resources will belong.
Type: AWS::EC2::VPC::Id
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Cluster Information"
Parameters:
- ClusterName
- Label:
default: "Network Configuration"
Parameters:
- VpcId
- PublicSubnets
- PrivateSubnets
- Label:
default: "DNS"
Parameters:
- HostedZoneName
- HostedZoneId
ParameterLabels:
ClusterName:
default: "Cluster Name"
VpcId:
default: "VPC ID"
PublicSubnets:
default: "Public Subnets"
PrivateSubnets:
default: "Private Subnets"
HostedZoneName:
default: "Public Hosted Zone Name"
HostedZoneId:
default: "Public Hosted Zone ID"
Resources:
ExtApiElb:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
Name: !Join ["-", ["api-ext", !Ref ClusterName, !Ref "AWS::Region"]]
IpAddressType: ipv4
Subnets: !Ref PublicSubnets
Type: network
IntApiElb:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
Name: !Join ["-", ["api-int", !Ref ClusterName, !Ref "AWS::Region"]]
Scheme: internal
IpAddressType: ipv4
Subnets: !Ref PrivateSubnets
Type: network
IntDns:
Type: "AWS::Route53::HostedZone"
Properties:
HostedZoneConfig:
Comment: "Managed by CloudFormation"
Name: !Join [".", [!Ref ClusterName, !Ref HostedZoneName]]
VPCs:
- VPCId: !Ref VpcId
VPCRegion: !Ref "AWS::Region"
ExternalApiServerRecord:
Type: AWS::Route53::RecordSetGroup
Properties:
Comment: Alias record for the API server
HostedZoneId: !Ref HostedZoneId
RecordSets:
- Name:
!Join [
".",
["api", !Ref ClusterName, !Join ["", [!Ref HostedZoneName, "."]]],
]
Type: A
AliasTarget:
HostedZoneId: !GetAtt ExtApiElb.CanonicalHostedZoneID
DNSName: !GetAtt ExtApiElb.DNSName
InternalApiServerRecord:
Type: AWS::Route53::RecordSetGroup
Properties:
Comment: Alias record for the API server
HostedZoneId: !Ref IntDns
RecordSets:
- Name:
!Join [
".",
["api", !Ref ClusterName, !Join ["", [!Ref HostedZoneName, "."]]],
]
Type: A
AliasTarget:
HostedZoneId: !GetAtt IntApiElb.CanonicalHostedZoneID
DNSName: !GetAtt IntApiElb.DNSName
ExternalApiListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- Type: forward
TargetGroupArn:
Ref: ExternalApiTargetGroup
LoadBalancerArn:
Ref: ExtApiElb
Port: 6443
Protocol: TCP
ExternalApiTargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: ExternalApiTargetGroup
Port: 6443
Protocol: TCP
TargetType: ip
VpcId:
Ref: VpcId
TargetGroupAttributes:
- Key: deregistration_delay.timeout_seconds
Value: 60
InternalApiListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- Type: forward
TargetGroupArn:
Ref: InternalApiTargetGroup
LoadBalancerArn:
Ref: IntApiElb
Port: 6443
Protocol: TCP
InternalApiTargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: InternalApiTargetGroup
Port: 6443
Protocol: TCP
TargetType: ip
VpcId:
Ref: VpcId
TargetGroupAttributes:
- Key: deregistration_delay.timeout_seconds
Value: 60
InternalServiceInternalListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- Type: forward
TargetGroupArn:
Ref: InternalServiceTargetGroup
LoadBalancerArn:
Ref: IntApiElb
Port: 22623
Protocol: TCP
InternalServiceTargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: InternalServiceTargetGroup
Port: 22623
Protocol: TCP
TargetType: ip
VpcId:
Ref: VpcId
TargetGroupAttributes:
- Key: deregistration_delay.timeout_seconds
Value: 60
RegisterTargetLambdaIamRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join ["-", [!Ref ClusterName, "nlb", "lambda", "role"]]
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
- "lambda.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
- PolicyName: !Join ["-", [!Ref ClusterName, "master", "policy"]]
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
[
"elasticloadbalancing:RegisterTargets",
"elasticloadbalancing:DeregisterTargets",
]
Resource: !Ref InternalApiTargetGroup
- Effect: "Allow"
Action:
[
"elasticloadbalancing:RegisterTargets",
"elasticloadbalancing:DeregisterTargets",
]
Resource: !Ref InternalServiceTargetGroup
- Effect: "Allow"
Action:
[
"elasticloadbalancing:RegisterTargets",
"elasticloadbalancing:DeregisterTargets",
]
Resource: !Ref ExternalApiTargetGroup
RegisterNlbIpTargets:
Type: "AWS::Lambda::Function"
Properties:
Handler: "index.handler"
Role:
Fn::GetAtt:
- "RegisterTargetLambdaIamRole"
- "Arn"
Code:
ZipFile: |
import json
import boto3
import cfnresponse
def handler(event, context):
elb = boto3.client('elbv2')
if event['RequestType'] == 'Delete':
elb.deregister_targets(TargetGroupArn=event['ResourceProperties']['TargetArn'],Targets=[{'Id': event['ResourceProperties']['TargetIp']}])
elif event['RequestType'] == 'Create':
elb.register_targets(TargetGroupArn=event['ResourceProperties']['TargetArn'],Targets=[{'Id': event['ResourceProperties']['TargetIp']}])
responseData = {}
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, event['ResourceProperties']['TargetArn']+event['ResourceProperties']['TargetIp'])
Runtime: "python3.7"
Timeout: 120
Outputs:
PrivateHostedZoneId:
Description: Hosted zone ID for the private DNS - needed for private records
Value: !Ref IntDns
ExternalApiLoadBalancerName:
Description: Full name of the External API load balancer created.
Value: !GetAtt ExtApiElb.LoadBalancerFullName
InternalApiLoadBalancerName:
Description: Full name of the Internal API load balancer created.
Value: !GetAtt IntApiElb.LoadBalancerFullName
ApiServerDnsName:
Description: Full hostname of the API server - Needed for ignition configs
Value: !Join [".", ["api", !Ref ClusterName, !Ref HostedZoneName]]
RegisterNlbIpTargetsLambda:
Description: Lambda ARN useful to help register/deregister IP targets for these load balancers
Value: !GetAtt RegisterNlbIpTargets.Arn
ExternalApiTargetGroupArn:
Description: ARN of External API target group
Value: !Ref ExternalApiTargetGroup
InternalApiTargetGroupArn:
Description: ARN of Internal API target group
Value: !Ref InternalApiTargetGroup
InternalServiceTargetGroupArn:
Description: ARN of internal service target group
Value: !Ref InternalServiceTargetGroup

341
upi/aws/cloudformation/03_cluster_security.yaml Normal file
View File

@@ -0,0 +1,341 @@
AWSTemplateFormatVersion: 2010-09-09
Description: Template for Openshift Cluster UPI Security Elements (Security Groups & IAM)
Parameters:
ClusterName:
AllowedPattern: ^([a-zA-Z][a-zA-Z0-9]{0,31})$
ConstraintDescription: Cluster name must be alphanumeric, start with a letter and a maximum of 32 characters
Description: A short, representative cluster name to use for hostnames, etc.
Type: String
VpcCidr:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-4]))$
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16-24
Default: 10.0.0.0/16
Description: CIDR block for VPC
Type: String
VpcId:
Description: The VPC created resources will belong.
Type: AWS::EC2::VPC::Id
PrivateSubnets:
Description: The internal subnets
Type: List<AWS::EC2::Subnet::Id>
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Cluster Information"
Parameters:
- ClusterName
- Label:
default: "Network Configuration"
Parameters:
- VpcId
- VpcCidr
- PrivateSubnets
ParameterLabels:
ClusterName:
default: "Cluster Name"
VpcId:
default: "VPC ID"
VpcCidr:
default: "VPC CIDR"
PrivateSubnets:
default: "Private Subnets"
Resources:
MasterSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Join ["-", ["master-sg", !Ref ClusterName]]
GroupDescription: Cluster Master Security Group
SecurityGroupIngress:
- IpProtocol: icmp
FromPort: 0
ToPort: 0
CidrIp: !Ref VpcCidr
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: !Ref VpcCidr
- IpProtocol: tcp
ToPort: 6443
FromPort: 6443
CidrIp: !Ref VpcCidr
- IpProtocol: tcp
FromPort: 22623
ToPort: 22623
CidrIp: !Ref VpcCidr
VpcId: !Ref VpcId
WorkerSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Join ["-", ["worker-sg", !Ref ClusterName]]
GroupDescription: Cluster Worker Security Group
SecurityGroupIngress:
- IpProtocol: icmp
FromPort: 0
ToPort: 0
CidrIp: !Ref VpcCidr
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: !Ref VpcCidr
VpcId: !Ref VpcId
MasterIngressEtcd:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt MasterSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
Description: etcd
FromPort: 2379
ToPort: 2380
IpProtocol: tcp
MasterIngressVxlan:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt MasterSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
Description: Vxlan packets
FromPort: 4789
ToPort: 4789
IpProtocol: udp
MasterIngressWorkerVxlan:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt MasterSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
Description: Vxlan packets
FromPort: 4789
ToPort: 4789
IpProtocol: udp
MasterIngressInternal:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt MasterSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
Description: Internal cluster communication
FromPort: 9000
ToPort: 9999
IpProtocol: tcp
MasterIngressWorkerInternal:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt MasterSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
Description: Internal cluster communication
FromPort: 9000
ToPort: 9999
IpProtocol: tcp
MasterIngressKube:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt MasterSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
Description: Kubernetes kubelet, scheduler and controller manager
FromPort: 10250
ToPort: 10252
IpProtocol: tcp
MasterIngressWorkerKube:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt MasterSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
Description: Kubernetes kubelet, scheduler and controller manager
FromPort: 10250
ToPort: 10252
IpProtocol: tcp
MasterIngressIngressServices:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt MasterSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
Description: Kubernetes ingress services
FromPort: 30000
ToPort: 32767
IpProtocol: tcp
MasterIngressWorkerIngressServices:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt MasterSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
Description: Kubernetes ingress services
FromPort: 30000
ToPort: 32767
IpProtocol: tcp
WorkerIngressVxlan:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt WorkerSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
Description: Vxlan packets
FromPort: 4789
ToPort: 4789
IpProtocol: udp
WorkerIngressWorkerVxlan:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt WorkerSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
Description: Vxlan packets
FromPort: 4789
ToPort: 4789
IpProtocol: udp
WorkerIngressInternal:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt WorkerSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
Description: Internal cluster communication
FromPort: 9000
ToPort: 9999
IpProtocol: tcp
WorkerIngressWorkerInternal:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt WorkerSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
Description: Internal cluster communication
FromPort: 9000
ToPort: 9999
IpProtocol: tcp
WorkerIngressKube:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt WorkerSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
Description: Kubernetes secure kubelet port
FromPort: 10250
ToPort: 10250
IpProtocol: tcp
WorkerIngressWorkerKube:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt WorkerSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
Description: Internal Kubernetes communication
FromPort: 10250
ToPort: 10250
IpProtocol: tcp
WorkerIngressIngressServices:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt WorkerSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
Description: Kubernetes ingress services
FromPort: 30000
ToPort: 32767
IpProtocol: tcp
WorkerIngressWorkerIngressServices:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt WorkerSecurityGroup.GroupId
SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
Description: Kubernetes ingress services
FromPort: 30000
ToPort: 32767
IpProtocol: tcp
MasterIamRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join ["-", [!Ref ClusterName, "master", "role"]]
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
- "ec2.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
- PolicyName: !Join ["-", [!Ref ClusterName, "master", "policy"]]
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action: "ec2:*"
Resource: "*"
- Effect: "Allow"
Action: "elasticloadbalancing:*"
Resource: "*"
- Effect: "Allow"
Action: "iam:PassRole"
Resource: "*"
- Effect: "Allow"
Action: "s3:GetObject"
Resource: "*"
MasterInstanceProfile:
Type: "AWS::IAM::InstanceProfile"
Properties:
Path: "/"
Roles:
- Ref: "MasterIamRole"
WorkerIamRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join ["-", [!Ref ClusterName, "worker", "role"]]
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
- "ec2.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
- PolicyName: !Join ["-", [!Ref ClusterName, "worker", "policy"]]
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action: "ec2:Describe*"
Resource: "*"
WorkerInstanceProfile:
Type: "AWS::IAM::InstanceProfile"
Properties:
Path: "/"
Roles:
- Ref: "WorkerIamRole"
Outputs:
MasterSecurityGroupId:
Description: Master Security Group ID
Value: !GetAtt MasterSecurityGroup.GroupId
WorkerSecurityGroupId:
Description: Worker Security Group ID
Value: !GetAtt WorkerSecurityGroup.GroupId
MasterInstanceProfile:
Description: Master IAM Instance Profile
Value: !Ref MasterInstanceProfile
WorkerInstanceProfile:
Description: Worker IAM Instance Profile
Value: !Ref WorkerInstanceProfile

216
upi/aws/cloudformation/04_cluster_bootstrap.yaml Normal file
View File

@@ -0,0 +1,216 @@
AWSTemplateFormatVersion: 2010-09-09
Description: Template for Openshift Cluster UPI Bootstrap (EC2 Instance, Security Groups and IAM)
Parameters:
ClusterName:
AllowedPattern: ^([a-zA-Z][a-zA-Z0-9]{0,31})$
ConstraintDescription: Cluster name must be alphanumeric, start with a letter and a maximum of 32 characters
Description: A short, representative cluster name to use for hostnames, etc.
Type: String
RhcosAmi:
Description: Current RHEL CoreOS AMI to use for boostrap
Type: AWS::EC2::Image::Id
AllowedBootstrapSshCidr:
AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|1[0-9]|2[0-9]|3[0-2]))$
ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/0-32
Default: 0.0.0.0/0
Description: CIDR block to allow SSH access to the bootstrap node
Type: String
PublicSubnet:
Description: The public subnet to launch the bootstrap node into
Type: AWS::EC2::Subnet::Id
MasterSecurityGroupId:
Description: The master security group ID (for registering temporary rules)
Type: AWS::EC2::SecurityGroup::Id
VpcId:
Description: The VPC created resources will belong.
Type: AWS::EC2::VPC::Id
BootstrapIgnitionLocation:
Default: s3://my-s3-bucket/bootstrap.ign
Description: Location to fetch bootstrap ignition from. (Recommend to use the autocreated cf-templates bucket.)
Type: String
AutoRegisterELB:
Default: "yes"
AllowedValues:
- "yes"
- "no"
Description: Do you want to invoke NLB registration (requires Lambda ARN parameter to be supplied)?
Type: String
RegisterNlbIpTargetsLambdaArn:
Description: ARN for NLB IP target registration lambda
Type: String
ExternalApiTargetGroupArn:
Description: ARN for external API load balancer target group
Type: String
InternalApiTargetGroupArn:
Description: ARN for internal API load balancer target group
Type: String
InternalServiceTargetGroupArn:
Description: ARN for internal service load balancer target group
Type: String
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Cluster Information"
Parameters:
- ClusterName
- Label:
default: "Host Information"
Parameters:
- RhcosAmi
- BootstrapIgnitionLocation
- MasterSecurityGroupId
- Label:
default: "Network Configuration"
Parameters:
- VpcId
- AllowedBootstrapSshCidr
- PublicSubnet
- Label:
default: "Load Balancer Automation"
Parameters:
- AutoRegisterELB
- RegisterNlbIpTargetsLambdaArn
- ExternalApiTargetGroupArn
- InternalApiTargetGroupArn
- InternalServiceTargetGroupArn
ParameterLabels:
ClusterName:
default: "Cluster Name"
VpcId:
default: "VPC ID"
AllowedBootstrapSshCidr:
default: "Allowed SSH Source"
PublicSubnet:
default: "Public Subnet"
RhcosAmi:
default: "RHEL CoreOS AMI ID"
BootstrapIgnitionLocation:
default: "Bootstrap Ignition Source"
MasterSecurityGroupId:
default: "Master Security Group ID"
AutoRegisterELB:
default: "Use Provided ELB Automation"
Conditions:
DoRegistration: !Equals ["yes", !Ref AutoRegisterELB]
Resources:
BootstrapIamRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join ["-", [!Ref ClusterName, "bootstrap", "role"]]
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
- "ec2.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
- PolicyName: !Join ["-", [!Ref ClusterName, "bootstrap", "policy"]]
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action: "ec2:Describe*"
Resource: "*"
- Effect: "Allow"
Action: "ec2:AttachVolume"
Resource: "*"
- Effect: "Allow"
Action: "ec2:DetachVolume"
Resource: "*"
- Effect: "Allow"
Action: "s3:GetObject"
Resource: "*"
BootstrapInstanceProfile:
Type: "AWS::IAM::InstanceProfile"
Properties:
Path: "/"
Roles:
- Ref: "BootstrapIamRole"
BootstrapSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Join ["-", ["bootstrap-sg", !Ref ClusterName]]
GroupDescription: Cluster Bootstrap Security Group
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: !Ref AllowedBootstrapSshCidr
- IpProtocol: tcp
ToPort: 19531
FromPort: 19531
CidrIp: 0.0.0.0/0
VpcId: !Ref VpcId
BootstrapInstance:
Type: AWS::EC2::Instance
Properties:
ImageId: !Ref RhcosAmi
IamInstanceProfile: !Ref BootstrapInstanceProfile
InstanceType: "i3.large"
NetworkInterfaces:
- AssociatePublicIpAddress: "true"
DeviceIndex: "0"
GroupSet:
- !Ref "BootstrapSecurityGroup"
- !Ref "MasterSecurityGroupId"
SubnetId: !Ref "PublicSubnet"
UserData:
Fn::Base64: !Sub
- '{"ignition":{"config":{"replace":{"source":"${S3Loc}","verification":{}}},"timeouts":{},"version":"2.1.0"},"networkd":{},"passwd":{},"storage":{},"systemd":{}}'
- {
S3Loc: !Ref BootstrapIgnitionLocation
}
Tags:
- Key: "Name"
Value: !Join ["-", [!Ref ClusterName, "bootstrap"]]
- Key: !Join ["", ["kubernetes.io/cluster/", !Ref ClusterName]]
Value: "Owned"
RegisterBootstrapApiTarget:
Condition: DoRegistration
Type: Custom::NLBRegister
Properties:
ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
TargetArn: !Ref ExternalApiTargetGroupArn
TargetIp: !GetAtt BootstrapInstance.PrivateIp
RegisterBootstrapInternalApiTarget:
Condition: DoRegistration
Type: Custom::NLBRegister
Properties:
ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
TargetArn: !Ref InternalApiTargetGroupArn
TargetIp: !GetAtt BootstrapInstance.PrivateIp
RegisterBootstrapInternalServiceTarget:
Condition: DoRegistration
Type: Custom::NLBRegister
Properties:
ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
TargetArn: !Ref InternalServiceTargetGroupArn
TargetIp: !GetAtt BootstrapInstance.PrivateIp
Outputs:
BootstrapInstanceId:
Description: Bootstrap Instance ID
Value: !Ref BootstrapInstance
BootstrapPublicIp:
Description: The bootstrap node public IP address
Value: !GetAtt BootstrapInstance.PublicIp
BootstrapPrivateIp:
Description: The bootstrap node private IP address
Value: !GetAtt BootstrapInstance.PrivateIp

368
upi/aws/cloudformation/05_cluster_master_nodes.yaml Normal file
View File

@@ -0,0 +1,368 @@
AWSTemplateFormatVersion: 2010-09-09
Description: Template for Openshift Cluster UPI Node Launch (EC2 master instances)
Parameters:
ClusterName:
AllowedPattern: ^([a-zA-Z][a-zA-Z0-9]{0,31})$
ConstraintDescription: Cluster name must be alphanumeric, start with a letter and a maximum of 32 characters
Description: A short, representative cluster name to use for hostnames, etc.
Type: String
RhcosAmi:
Description: Current RHEL CoreOS AMI to use for boostrap
Type: AWS::EC2::Image::Id
AutoRegisterDNS:
Default: "yes"
AllowedValues:
- "yes"
- "no"
Description: Do you want to invoke DNS etcd registration (requires Hosted Zone info provided)?
Type: String
PrivateHostedZoneId:
Description: The Route53 private zone ID to register the etcd targets with (e.g Z21IXYZABCZ2A4)
Type: String
PrivateHostedZoneName:
Description: The Route53 zone to register the targets with (No trailing dot - e.g. cluster.mycorp.com)
Type: String
Master0Subnet:
Description: The subnets (recommend private) to launch the master nodes into
Type: AWS::EC2::Subnet::Id
Master1Subnet:
Description: The subnets (recommend private) to launch the master nodes into
Type: AWS::EC2::Subnet::Id
Master2Subnet:
Description: The subnets (recommend private) to launch the master nodes into
Type: AWS::EC2::Subnet::Id
MasterSecurityGroupId:
Description: The master security group ID to associate with master nodes.
Type: AWS::EC2::SecurityGroup::Id
IgnitionLocation:
Default: https://api.$CLUSTER_NAME.$DOMAIN:22623/config/master
Description: Location to fetch bootstrap ignition from. (Recommend to use the autocreated ignition config location.)
Type: String
CertificateAuthorities:
Default: data:text/plain;charset=utf-8;base64,ABC...xYz==
Description: Base64 encoded certificate authority string to use.
Type: String
MasterInstanceProfileName:
Description: IAM profile to associate with master nodes.
Type: String
MasterInstanceType:
Default: m4.xlarge
Type: String
AllowedValues:
- "m4.xlarge"
- "m4.2xlarge"
- "m4.4xlarge"
- "m4.8xlarge"
- "m4.10xlarge"
- "m4.16xlarge"
- "c4.2xlarge"
- "c4.4xlarge"
- "c4.8xlarge"
- "r4.xlarge"
- "r4.2xlarge"
- "r4.4xlarge"
- "r4.8xlarge"
- "r4.16xlarge"
AutoRegisterELB:
Default: "yes"
AllowedValues:
- "yes"
- "no"
Description: Do you want to invoke NLB registration (requires Lambda ARN parameter to be supplied)?
Type: String
RegisterNlbIpTargetsLambdaArn:
Description: ARN for NLB IP target registration lambda (from cluster_infra_upi.yaml; otherwise select "no" for AutoRegisterELB)
Type: String
ExternalApiTargetGroupArn:
Description: ARN for external API load balancer target group (from cluster_infra_upi.yaml; otherwise select "no" for AutoRegisterELB)
Type: String
InternalApiTargetGroupArn:
Description: ARN for internal API load balancer target group (from cluster_infra_upi.yaml; otherwise select "no" for AutoRegisterELB)
Type: String
InternalServiceTargetGroupArn:
Description: ARN for internal service load balancer target group (from cluster_infra_upi.yaml; otherwise select "no" for AutoRegisterELB)
Type: String
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Cluster Information"
Parameters:
- ClusterName
- Label:
default: "Host Information"
Parameters:
- MasterInstanceType
- RhcosAmi
- IgnitionLocation
- CertificateAuthorities
- MasterSecurityGroupId
- MasterInstanceProfileName
- Label:
default: "Network Configuration"
Parameters:
- VpcId
- AllowedBootstrapSshCidr
- Master0Subnet
- Master1Subnet
- Master2Subnet
- Label:
default: "DNS"
Parameters:
- AutoRegisterDNS
- PrivateHostedZoneName
- PrivateHostedZoneId
- Label:
default: "Load Balancer Automation"
Parameters:
- AutoRegisterELB
- RegisterNlbIpTargetsLambdaArn
- ExternalApiTargetGroupArn
- InternalApiTargetGroupArn
- InternalServiceTargetGroupArn
ParameterLabels:
ClusterName:
default: "Cluster Name"
VpcId:
default: "VPC ID"
Master0Subnet:
default: "Master-0 Subnet"
Master1Subnet:
default: "Master-1 Subnet"
Master2Subnet:
default: "Master-2 Subnet"
MasterInstanceType:
default: "Master Instance Type"
MasterInstanceProfileName:
default: "Master Instance Profile Name"
RhcosAmi:
default: "RHEL CoreOS AMI ID"
BootstrapIgnitionLocation:
default: "Master Ignition Source"
CertificateAuthorities:
default: "Ignition CA String"
MasterSecurityGroupId:
default: "Master Security Group ID"
AutoRegisterDNS:
default: "Use Provided DNS Automation"
AutoRegisterELB:
default: "Use Provided ELB Automation"
PrivateHostedZoneName:
default: "Private Hosted Zone Name"
PrivateHostedZoneId:
default: "Private Hosted Zone ID"
Conditions:
DoRegistration: !Equals ["yes", !Ref AutoRegisterELB]
DoDns: !Equals ["yes", !Ref AutoRegisterDNS]
Resources:
Master0:
Type: AWS::EC2::Instance
Properties:
ImageId: !Ref RhcosAmi
IamInstanceProfile: !Ref MasterInstanceProfileName
InstanceType: !Ref MasterInstanceType
NetworkInterfaces:
- AssociatePublicIpAddress: "false"
DeviceIndex: "0"
GroupSet:
- !Ref "MasterSecurityGroupId"
SubnetId: !Ref "Master0Subnet"
UserData:
Fn::Base64: !Sub
- '{"ignition":{"config":{"append":[{"source":"${SOURCE}","verification":{}}]},"security":{"tls":{"certificateAuthorities":[{"source":"${CA_BUNDLE}","verification":{}}]}},"timeouts":{},"version":"2.2.0"},"networkd":{},"passwd":{},"storage":{},"systemd":{}}'
- {
SOURCE: !Ref IgnitionLocation,
CA_BUNDLE: !Ref CertificateAuthorities,
}
Tags:
- Key: "Name"
Value: !Join ["-", [!Ref ClusterName, "master", "0"]]
- Key: !Join ["", ["kubernetes.io/cluster/", !Ref ClusterName]]
Value: "owned"
- Key: "clusterid"
Value: !Ref ClusterName
RegisterMaster0:
Condition: DoRegistration
Type: Custom::NLBRegister
Properties:
ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
TargetArn: !Ref ExternalApiTargetGroupArn
TargetIp: !GetAtt Master0.PrivateIp
RegisterMaster0InternalApiTarget:
Condition: DoRegistration
Type: Custom::NLBRegister
Properties:
ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
TargetArn: !Ref InternalApiTargetGroupArn
TargetIp: !GetAtt Master0.PrivateIp
RegisterMaster0InternalServiceTarget:
Condition: DoRegistration
Type: Custom::NLBRegister
Properties:
ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
TargetArn: !Ref InternalServiceTargetGroupArn
TargetIp: !GetAtt Master0.PrivateIp
Master1:
Type: AWS::EC2::Instance
Properties:
ImageId: !Ref RhcosAmi
IamInstanceProfile: !Ref MasterInstanceProfileName
InstanceType: !Ref MasterInstanceType
NetworkInterfaces:
- AssociatePublicIpAddress: "false"
DeviceIndex: "0"
GroupSet:
- !Ref "MasterSecurityGroupId"
SubnetId: !Ref "Master1Subnet"
UserData:
Fn::Base64: !Sub
- '{"ignition":{"config":{"append":[{"source":"${SOURCE}","verification":{}}]},"security":{"tls":{"certificateAuthorities":[{"source":"${CA_BUNDLE}","verification":{}}]}},"timeouts":{},"version":"2.2.0"},"networkd":{},"passwd":{},"storage":{},"systemd":{}}'
- {
SOURCE: !Ref IgnitionLocation,
CA_BUNDLE: !Ref CertificateAuthorities,
}
Tags:
- Key: "Name"
Value: !Join ["-", [!Ref ClusterName, "master", "1"]]
- Key: !Join ["", ["kubernetes.io/cluster/", !Ref ClusterName]]
Value: "owned"
- Key: "clusterid"
Value: !Ref ClusterName
RegisterMaster1:
Condition: DoRegistration
Type: Custom::NLBRegister
Properties:
ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
TargetArn: !Ref ExternalApiTargetGroupArn
TargetIp: !GetAtt Master1.PrivateIp
RegisterMaster1InternalApiTarget:
Condition: DoRegistration
Type: Custom::NLBRegister
Properties:
ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
TargetArn: !Ref InternalApiTargetGroupArn
TargetIp: !GetAtt Master1.PrivateIp
RegisterMaster1InternalServiceTarget:
Condition: DoRegistration
Type: Custom::NLBRegister
Properties:
ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
TargetArn: !Ref InternalServiceTargetGroupArn
TargetIp: !GetAtt Master1.PrivateIp
Master2:
Type: AWS::EC2::Instance
Properties:
ImageId: !Ref RhcosAmi
IamInstanceProfile: !Ref MasterInstanceProfileName
InstanceType: !Ref MasterInstanceType
NetworkInterfaces:
- AssociatePublicIpAddress: "false"
DeviceIndex: "0"
GroupSet:
- !Ref "MasterSecurityGroupId"
SubnetId: !Ref "Master2Subnet"
UserData:
Fn::Base64: !Sub
- '{"ignition":{"config":{"append":[{"source":"${SOURCE}","verification":{}}]},"security":{"tls":{"certificateAuthorities":[{"source":"${CA_BUNDLE}","verification":{}}]}},"timeouts":{},"version":"2.2.0"},"networkd":{},"passwd":{},"storage":{},"systemd":{}}'
- {
SOURCE: !Ref IgnitionLocation,
CA_BUNDLE: !Ref CertificateAuthorities,
}
Tags:
- Key: "Name"
Value: !Join ["-", [!Ref ClusterName, "master", "2"]]
- Key: !Join ["", ["kubernetes.io/cluster/", !Ref ClusterName]]
Value: "owned"
- Key: "clusterid"
Value: !Ref ClusterName
RegisterMaster2:
Condition: DoRegistration
Type: Custom::NLBRegister
Properties:
ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
TargetArn: !Ref ExternalApiTargetGroupArn
TargetIp: !GetAtt Master2.PrivateIp
RegisterMaster2InternalApiTarget:
Condition: DoRegistration
Type: Custom::NLBRegister
Properties:
ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
TargetArn: !Ref InternalApiTargetGroupArn
TargetIp: !GetAtt Master2.PrivateIp
RegisterMaster2InternalServiceTarget:
Condition: DoRegistration
Type: Custom::NLBRegister
Properties:
ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
TargetArn: !Ref InternalServiceTargetGroupArn
TargetIp: !GetAtt Master2.PrivateIp
EtcdSrvRecords:
Condition: DoDns
Type: AWS::Route53::RecordSet
Properties:
HostedZoneId: !Ref PrivateHostedZoneId
Name: !Join [".", ["_etcd-server-ssl._tcp", !Ref PrivateHostedZoneName]]
ResourceRecords:
- !Join [
" ",
["0 10 2380", !Join [".", ["etcd-0", !Ref PrivateHostedZoneName]]],
]
- !Join [
" ",
["0 10 2380", !Join [".", ["etcd-1", !Ref PrivateHostedZoneName]]],
]
- !Join [
" ",
["0 10 2380", !Join [".", ["etcd-2", !Ref PrivateHostedZoneName]]],
]
TTL: 60
Type: SRV
Etcd0Record:
Condition: DoDns
Type: AWS::Route53::RecordSet
Properties:
HostedZoneId: !Ref PrivateHostedZoneId
Name: !Join [".", ["etcd-0", !Ref PrivateHostedZoneName]]
ResourceRecords:
- !GetAtt Master0.PrivateIp
TTL: 60
Type: A
Etcd1Record:
Condition: DoDns
Type: AWS::Route53::RecordSet
Properties:
HostedZoneId: !Ref PrivateHostedZoneId
Name: !Join [".", ["etcd-1", !Ref PrivateHostedZoneName]]
ResourceRecords:
- !GetAtt Master1.PrivateIp
TTL: 60
Type: A
Etcd2Record:
Condition: DoDns
Type: AWS::Route53::RecordSet
Properties:
HostedZoneId: !Ref PrivateHostedZoneId
Name: !Join [".", ["etcd-2", !Ref PrivateHostedZoneName]]
ResourceRecords:
- !GetAtt Master2.PrivateIp
TTL: 60
Type: A

120
upi/aws/cloudformation/06_cluster_worker_node.yaml Normal file
View File

@@ -0,0 +1,120 @@
AWSTemplateFormatVersion: 2010-09-09
Description: Template for Openshift Cluster UPI Node Launch (EC2 worker instance)
Parameters:
ClusterName:
AllowedPattern: ^([a-zA-Z][a-zA-Z0-9]{0,31})$
ConstraintDescription: Cluster name must be alphanumeric, start with a letter and a maximum of 32 characters
Description: A short, representative cluster name to use for hostnames, etc.
Type: String
RhcosAmi:
Description: Current RHEL CoreOS AMI to use for boostrap
Type: AWS::EC2::Image::Id
WorkerSubnet:
Description: The subnets (recommend private) to launch the master nodes into
Type: AWS::EC2::Subnet::Id
WorkerSecurityGroupId:
Description: The master security group ID to associate with master nodes.
Type: AWS::EC2::SecurityGroup::Id
IgnitionLocation:
Default: https://api.$CLUSTER_NAME.$DOMAIN:22623/config/worker
Description: Location to fetch bootstrap ignition from. (Recommend to use the autocreated ignition config location.)
Type: String
CertificateAuthorities:
Default: data:text/plain;charset=utf-8;base64,ABC...xYz==
Description: Base64 encoded certificate authority string to use.
Type: String
WorkerInstanceProfileName:
Description: IAM profile to associate with master nodes.
Type: String
WorkerInstanceType:
Default: m4.large
Type: String
AllowedValues:
- "m4.large"
- "m4.xlarge"
- "m4.2xlarge"
- "m4.4xlarge"
- "m4.8xlarge"
- "m4.10xlarge"
- "m4.16xlarge"
- "c4.large"
- "c4.xlarge"
- "c4.2xlarge"
- "c4.4xlarge"
- "c4.8xlarge"
- "r4.large"
- "r4.xlarge"
- "r4.2xlarge"
- "r4.4xlarge"
- "r4.8xlarge"
- "r4.16xlarge"
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Cluster Information"
Parameters:
- ClusterName
- Label:
default: "Host Information"
Parameters:
- WorkerInstanceType
- RhcosAmi
- IgnitionLocation
- CertificateAuthorities
- WorkerSecurityGroupId
- WorkerInstanceProfileName
- Label:
default: "Network Configuration"
Parameters:
- VpcId
- WorkerSubnet
ParameterLabels:
ClusterName:
default: "Cluster Name"
VpcId:
default: "VPC ID"
WorkerSubnet:
default: "Worker Subnet"
WorkerInstanceType:
default: "Worker Instance Type"
WorkerInstanceProfileName:
default: "Worker Instance Profile Name"
RhcosAmi:
default: "RHEL CoreOS AMI ID"
BootstrapIgnitionLocation:
default: "Worker Ignition Source"
CertificateAuthorities:
default: "Ignition CA String"
WorkerSecurityGroupId:
default: "Worker Security Group ID"
Resources:
Worker0:
Type: AWS::EC2::Instance
Properties:
ImageId: !Ref RhcosAmi
IamInstanceProfile: !Ref WorkerInstanceProfileName
InstanceType: !Ref WorkerInstanceType
NetworkInterfaces:
- AssociatePublicIpAddress: "false"
DeviceIndex: "0"
GroupSet:
- !Ref "WorkerSecurityGroupId"
SubnetId: !Ref "WorkerSubnet"
UserData:
Fn::Base64: !Sub
- '{"ignition":{"config":{"append":[{"source":"${SOURCE}","verification":{}}]},"security":{"tls":{"certificateAuthorities":[{"source":"${CA_BUNDLE}","verification":{}}]}},"timeouts":{},"version":"2.2.0"},"networkd":{},"passwd":{},"storage":{},"systemd":{}}'
- {
SOURCE: !Ref IgnitionLocation,
CA_BUNDLE: !Ref CertificateAuthorities,
}
Tags:
- Key: "Name"
Value: !Join ["-", [!Ref ClusterName, "worker"]]
- Key: !Join ["", ["kubernetes.io/cluster/", !Ref ClusterName]]
Value: "owned"
- Key: "clusterid"
Value: !Ref ClusterName