Download Vapor-Network-Conventions-v0-12

Vapor Network Conventions, Version 0.6 (Draft)
The Vapor network fabric is a shared academic resource operated by OIT and governed
by representatives of the Vapor partners. It provides an isolated, high-speed network
fabric for experimentation and operation of various cloud-computing platforms by
various academic and business units. The policies listed herein are intended to be as open
as possible while preserving the goals of the Vapor project, which are to provide a
framework for sharing resources, autonomous action, reliability, compartmentalization,
and low maintenance for cloud-based research and instruction.
Conventions:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
Public networks in Vapor will pass through firewalls on pace-rtr.
Only networks that originate on pace-rtr or are private to Vapor may be
carried across the Vapor network core.
Vapor networks may not be be carried on any non-Vapor switches.
A research-level SLA applies to the Vapor network fabric. This is 5x8xNBD
(i.e. the hardware contract SLA) response or best effort.
Self-service and CLI access can be provided to the owners of the edge
switches, but not the core switches.
Pace-rtr and the core 9508s are shared resources that are used by other
production and research services. On these devices, Vapor usage must be
designed to avoid resource and feature conflicts on these devices.
A convention on which VLAN numbers to use so they don’t conflict with
campus VLANS. [Probably a block in the 2000-3999 range, excluding those
networks used on pace-rtr]
Packet delivery is not guaranteed on the Vapor IP network, just as it isn’t
guaranteed across the Internet. Network applications on the Vapor network
should be hardened against network transport interruptions; they should
recover or restart themselves automatically.
Campus networks may be locally trunked to Vapor switches (but not carried
across the Vapor network core; see #2
A typical switch pair layout is described in this document; conforming as
closely as possible to this template is requested.
The Vapor network border configuration is described later in this document. It
provides dynamic NAT/PAT for hosts routing from the Vapor L3 spine and
can be configured to provide static NAT mapping for addresses in this
network. Additional border access policies are open for discussion.
Public addresses
There are two ways of providing public addresses for Vapor networks (so far)
1.
2.
1
Local attachment of a public network to a Vapor edge switch and connection
of a NIC (or trunk) from a server. Then the server can be given a public
address on that network.
Using static NAT through a firewall on pace-rtr.gatech.edu.
9/30/2015
Vapor Network Conventions, Version 0.6 (Draft)
a. Use static NAT on the vapor-1470-fw firewall to map a public address on
one of the assigned public networks (143.215.11.0/24 so far) to a private
address on one of the Vapor private networks. This step currently has to
be done by a firewall administrator.
b. You may obtain access to your assigned addresses in GTIPAM and
fw.noc.gatech.edu. But you might need to ask for this to be configured.
c. On fw.noc.gatech.edu, add the private host address to the vapor-1470-fw
firewall. Link the private address to the assigned public address (e.g. link
172.31.x.y to 143.215.11.z) to document it and so you don’t have to
remember whether the firewall rules require the inside or outside address
to be specified. Then enter firewall rules for the assigned public address
(143.215.11.z in this case) to expose ports on the interface to which the
private address is assigned (i.e. 172.31.x.y).
d. There are not going to be a massive number of available IPv4 public
addresses for assignment, so it is going to be necessary to conserve the
public addresses. For banks of management addresses, it may be
necessary to configure a system on the Vapor networks as a pass-through
system with a single public address on one side and access to an entire
network of public management addresses on the other.
Things we need on the Vapor fabric
1.
2.
3.
4.
A network for managing the switch fabric (143.215.12.0/24, VLAN 1212)
A transport network (one VLAN with an IP address interface on each router,
running OSPF) for carrying internal layer-3 traffic and over which VXLAN
can be tunneled (172.31.254.0/24, VLAN 2300)
A network for managing the cloud node hardware: mgt modules and possibly
management interfaces for the cpu (using private networks for now; should
we change this?)
Some really-private VLANs that are switch-pair-local to accommodate RHEL
OSP7 requirements for a private provisioning network gatewayed by the
provisioning server.
Commentary:
The 143.215.12.0/24, VLAN 1212 will be used as a switch management network.
Implement a routed backbone amongst the 9300s on Vapor. I've got the L3 licenses and
the IP ranges reserved.
- The 2300 VLAN (172.31.254.0/24) is trunked to each of the 9300s and each
9300 has a physical address and a virtual VRRP address for a pair in a VRF named
"vapor". (PACE will be a single switch and not be a VPC pair yet.)
- The routing switches will be connected by authenticated OSPF over the 2300
VLAN. The routing switches will keep their routing tables using static routes until the
9Ks support dynamic routing protocols across the VPC peer-link. I don't expect to turn on
2
9/30/2015
Vapor Network Conventions, Version 0.6 (Draft)
routing on the 9508s -- they will just pass the 2300 VLAN and other Vapor VLANs at
layer-2 as necessary.
Network allocations:
Other VLANs in the 2300+ range will be assigned to subnets in the 172.31.1-253.0/24
space, with a router (.1) interfaces on the nearest Vapor 9300 cluster. Each cluster (or
cluster cell) will get at least one of these networks.
Addresses for VLAN 2300 interfaces per switch
172.31.254.11 ebb-9k-11-rtr.vapor.gatech.edu
172.31.254.12 ebb-9k-12-rtr.vapor.gatech.edu
172.31.254.13 french-9k-13-rtr.vapor.gatech.edu
172.31.254.14 french-9k-14-rtr.vapor.gatech.edu
172.31.254.15 pace-9k-15-rtr.vapor.gatech.edu
172.31.254.17 isye-9k-17-rtr.vapor.gatech.edu
172.31.254.18 isye-9k-18-rtr.vapor.gatech.edu
172.31.254.19 ccb-9k-19-rtr.vapor.gatech.edu
172.31.254.20 ccb-9k-20-rtr.vapor.gatech.edu
172.31.254.21 rnoc-9k-21-rtr.vapor.gatech.edu
172.31.254.22 rnoc-9k-22-rtr.vapor.gatech.edu
VRRP virtual address for VLAN2300 on each switch pair
172.31.254.31 ebb-9k-rtr.vapor.gatech.edu, VRRP group 31
172.31.254.33 french-9k-rtr.vapor.gatech.edu, VRRP group 33
172.31.254.35 pace-9k-rtr.vapor.gatech.edu, VRRP group 35
172.31.254.37 isye-9k-rtr.vapor.gatech.edu, VRRP group 37
172.31.254.39 ccb-9k-rtr.vapor.gatech.edu, VRRP group 39
172.31.254.41 rnoc-9k-rtr.vapor.gatech.edu, VRRP group 41
VLANs and IPs allocated to each data center
French 2316-2323
172.31.16-23.0/24
EBB 2332-2339
172.31.32-39.0/24
PACE 2348-2355
172.31.48-55.0/24
ISYE 2364-2371
172.31.64-71.0/24
COC 2380-2387
172.31.80-87.0/24
RNOC 2396-2403
172.31.96-103.0/24
Management addresses allocated in 143.215.12.0/24
143.215.12.9 vapor-management-1212-fw-ha.gatech.edu
143.215.12.10 vapor-management-1212-fw.gatech.edu
143.215.12.11 ebb-9k-12-11.vapor.gatech.edu
143.215.12.12 ebb-9k-12-12.vapor.gatech.edu
143.215.12.13 french-9k-12-13.vapor.gatech.edu
143.215.12.14 french-9k-12-14.vapor.gatech.edu
(Note 15-16 addresses are available for PACE if needed.)
3
9/30/2015
Vapor Network Conventions, Version 0.6 (Draft)
143.215.12.17 isye-9k-12-17.vapor.gatech.edu
143.215.12.18 isye-9k-12-18.vapor.gatech.edu
143.215.12.19 ccb-9k-12-19.vapor.gatech.edu
143.215.12.20 ccb-9k-12-20.vapor.gatech.edu
143.215.12.21 tsrb-9k-12-21.vapor.gatech.edu
143.215.12.22 tsrb-9k-12-22.vapor.gatech.edu
Switch-pair-local VLANs:
2000 – 2029 (reusable on each switch-pair)
- If a provisioning network needs to be trunked to other data centers, then use
one of the VLANs assigned to one of the data centers.
- These VLANs are not to be trunked outside of switch pairs.
Vapor (Federated Academic Cloud) Typical Switch Pair
pace-rtr
Traffic
Layer 3
Layer 2
Physical interconnects
100G
40G
2 x 40G
10G
2 x 10G
1G or 10G
PACE 9508
BCDC
PACE 9508
Rich
VLAN 1212: 143.215.12.0/24 - switch management
VLAN 2300: 172.31.254.0/24 - Vapor L3 spine
Other Vapor public VLANS from pace-rtr
eth2/8
eth2/8
eth2/5
N93128TX
eth2/6
mgmt0
eth2/7
eth2/5
peer-link
peer-keepalive
eth2/6
N93128TX
mgmt0
eth2/7
• Routed on L3 spine or not
• DNS available
• Local use for management or
tenant networks
• Split among data centers
• In the range:
VLAN 2316: 172.31.16.0/24 ....
....
VLAN 2553: 172.31.253.0/24
Local
Campus
VLANs
4
9/30/2015
Vapor Network Conventions, Version 0.6 (Draft)
Vapor (Federated Academic Cloud) Border Configuration
Campus
Core
Routers
Campus
Core
143.215.250.129/29
3470
143.215.250.133/29
vapor-1470-fw
NAT/PAT assigns
143.215.11.240/29
to Vapor networks;
rest of
143.215.11.0/24
reserved for static
NAT
pace-948-fw
NAT/PAT assigns
public addresses to
PACE networks
143.215.250.132/29 1470
Vapor
VRF
pace-rtr
172.31.254.1/16
pace-rtr
(Selected
route
leakage)
default
VRF
2300
PACE 9508 Rich
PACE 9508 BCDC
VLAN 2300
172.31.0.0/16
Other
PACE
subnets
Existing
Vapor (Federated Academic Cloud) Network Roadmap
pace-rtr
Traffic
PACE 9508
Rich
Layer 3
Layer 2
Physical interconnects
100G
40G
2 x 40G
10G
2 x 10G
PACE 9508
BCDC
GT
Enterprise
Directory
OpenStack
Reference
Cluster
Rich
1G or 10G
Vapor
Community
Cloud BCDC
93128TX
CoE/CoS
French
2x93128TX
.13/.14
College of
Computing
2x93128TX
.19/.20
5
PACE
OpenStack
Testbed - Rich
93128TX
ISyE
Groseclose
2x93128TX
.17/.18
CoE/CoS EEB
2x93128TX
.11/.12
RNOC
OpenStack
Cloud
2x9396PX
.21/.22
RNOC
Software
Defined
Infrastructure
(SDI) Testbed
7004
OIT VMware
OpenStack
(future)
9/30/2015
Vapor Network Conventions, Version 0.6 (Draft)
What it takes to add a Nexus 9300 switch into the Vapor environment:
Apply these lines to the configuration along with an Enterprise Services license, making
edits where directed.
feature vrrp
feature ospf
vrf context vapor
ip route 0.0.0.0/0 172.31.254.1 100
ip route 172.31.16.0/21 172.31.254.33 210
ip route 172.31.32.0/21 172.31.254.31 210
ip route 172.31.48.0/21 172.31.254.33 210
ip route 172.31.64.0/21 172.31.254.37 210
ip route 172.31.80.0/21 172.31.254.39 210
ip route 172.31.96.0/21 172.31.254.41 210
! Routing instance — no dynamic routing yet – it will come with planned NX-OS update
! router ospf Vapor
! vrf vapor
! router-id 172.31.254.XX
! area 0 authentication message-digest
! Uplink VLANs — change to correct uplink port-channel
int port-channel 30
switchport trunk allowed vlan add 2300
! Spine VLAN — change 2 addresses to spine addresses for switch; change priority to 100 for second
switch
vlan 2300
vrf member vapor
ip address 172.31.254.16/24
no ip redirect
mtu 9216
ip router ospf Vapor area 172.31.0.0
ip ospf message-digest-key 1 md5 0 [redacted]
no shutdown
vrrp 35
priority 100
address 172.31.254.35
no shutdown
! Tenant networks — Template: change VLANs, 2 addresses, vrrp, and priority 100 on backup
vlan 2348
int vlan 2348
vrf member vapor
ip address 172.31.48.2/24
no ip redirect
mtu 9216
6
9/30/2015
Vapor Network Conventions, Version 0.6 (Draft)
ip router ospf Vapor area 172.31.0.0
no shut
vrrp 1
priority 120
address 172.31.48.1
no shut
7
9/30/2015