VXLAN EVPN Fabric で仮想ネットワークの設定を Anaible を使って自動化する

VXLAN EVPN の環境で Overlay network の設定を Ansible で自動化してみました。VXLAN Fabric では、L2 ネットワークを 1つ追加するだけでも全ての Leaf スイッチに対して設定を行う必要があります。この設定を自動化する ansible playbook を書いてみました。

構成と作成する仮想ネットワークは次のようになっています。

f:id:naoki029:20190908091009p:plain

スイッチは仮想 OS の NX-OS 9000v 9.3(1) です。

Ansible

Playbook

- name: Provisioning VXLAN Overlay Network
  hosts: leaf
  gather_facts: no
  connection: network_cli
  vars:
    asn: "65000"
  tasks:
    - name: Crete vlans
      nxos_vlan:
        vlan_id: "{{ item.vlan_id }}"
        name: "{{ item.name }} "
        mapped_vni: "{{ item.vni_id }}"
      with_items:
        - "{{ vlans }}"
    - name: Create VRF
      nxos_vrf:
        name: "{{ vrf.name }}"
        vni: "{{ vrf.vni_id }}"
        rd: auto
    - name: Manage VRF AF
      nxos_vrf_af:
        afi: ipv4
        vrf: "{{ vrf.name }}"
        route_target_both_auto_evpn: True
    - name: Create SVI interface
      nxos_interface:
        name: "Vlan{{item.vlan_id}}"
      with_items:
        - "{{ vlans }}"
    - name: Associate VRF to SVI interface
      nxos_vrf_interface:
        vrf: "{{ vrf.name }}"
        interface: "Vlan{{item.vlan_id}}"
      with_items:
        - "{{ vlans }}"
    - name: Set IP address to SVI interface
      when: item.address is defined
      nxos_l3_interface:
        name: "Vlan{{item.vlan_id}}"
        ipv4: "{{item.address }}"
      with_items:
        - "{{ vlans }}"
    - name: Enable anycast gateway
      when: item.address is defined
      nxos_interface:
        fabric_forwarding_anycast_gateway: True
        name: "Vlan{{item.vlan_id}}"
      with_items:
        - "{{ vlans }}"
    - name: Enable ip forwarding
      when: item.address is not defined
      nxos_interface:
        ip_forward: enable
        name: "Vlan{{item.vlan_id}}"
      with_items:
        - "{{ vlans }}"
    - name: Manage BGP address family
      nxos_bgp_af:
        vrf: "{{vrf.name}}"
        asn: "{{asn}}"
        afi: ipv4
        safi: unicast
    - name: Set l2 vni to nve
      when: item.address is defined
      nxos_vxlan_vtep_vni:
        ingress_replication: bgp
        interface: nve1
        vni: "{{item.vni_id}}"
      with_items:
        - "{{ vlans }}"
    - name: Set l3 vni to nve
      when: item.address is not defined
      nxos_vxlan_vtep_vni:
        interface: nve1
        vni: "{{item.vni_id}}"
        assoc_vrf: True
      with_items:
        - "{{ vlans }}"

Inventory

[leaf]
172.16.0.101
172.16.0.102
172.16.0.103
[leaf:vars]
ansible_ssh_user = admin
ansible_connection = local

Variables

vlans:
  - vlan_id: 100
    name: VNI10000
    vni_id: 10000
    address: 192.168.0.254/24
    mask: 24
  - vlan_id: 101
    name: VNI10001
    vni_id: 10001
    address: 192.168.1.254/24
  - vlan_id: 900
    name: VNI90000_vrf_blue
    vni_id: 90000
vrf:
  name: blue
  vni_id: 90000

気づいたこと

nxos_vlan は2回実行しても、ステータスが changed になる。

nxos_interface で ip forward、fabric forwarding anycast gateway を設定した後に、nxos_vrf_interface で Interface に VRF を割り当てると設定が消える。VRF に紐づく L3 の設定なので当たり前ですが、公式ドキュメントの説明に物理インターフェースの管理と記載があるので、注意が必要であると思いました。

Manages physical attributes of interfaces of NX-OS switches.

結果

[root@localhost ansible]# ansible-playbook -i inventories/hosts provision_overlay_nw.yaml  -e @group_vars/vrf_blue.yaml  --timeout=300

PLAY [Provisioning VXLAN Overlay Network] *****************************************************************************************************************************

TASK [Crete vlans] ****************************************************************************************************************************************************
changed: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})

TASK [Create VRF] *****************************************************************************************************************************************************
changed: [172.16.0.103]
changed: [172.16.0.102]
changed: [172.16.0.101]

TASK [Manage VRF AF] **************************************************************************************************************************************************
changed: [172.16.0.103]
changed: [172.16.0.102]
changed: [172.16.0.101]

TASK [Create SVI interface] *******************************************************************************************************************************************
changed: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})

TASK [Associate VRF to SVI interface] *********************************************************************************************************************************
changed: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})

TASK [Set IP address to SVI interface] ********************************************************************************************************************************
changed: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})

TASK [Enable anycast gateway] *****************************************************************************************************************************************
changed: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})

TASK [Enable ip forwarding] *******************************************************************************************************************************************
skipping: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
skipping: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
skipping: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
skipping: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})

TASK [Manage BGP address family] **************************************************************************************************************************************
changed: [172.16.0.103]
changed: [172.16.0.101]
changed: [172.16.0.102]

TASK [Set l2 vni to nve] **********************************************************************************************************************************************
changed: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
changed: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})

TASK [Set l3 vni to nve] **********************************************************************************************************************************************
skipping: [172.16.0.101] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
skipping: [172.16.0.102] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
skipping: [172.16.0.102] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.101] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
skipping: [172.16.0.103] => (item={u'vni_id': 10000, u'mask': 24, u'name': u'VNI10000', u'vlan_id': 100, u'address': u'192.168.0.254/24'})
skipping: [172.16.0.103] => (item={u'vni_id': 10001, u'name': u'VNI10001', u'vlan_id': 101, u'address': u'192.168.1.254/24'})
changed: [172.16.0.102] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.101] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})
changed: [172.16.0.103] => (item={u'vni_id': 90000, u'name': u'VNI90000_vrf_blue', u'vlan_id': 900})

PLAY RECAP ************************************************************************************************************************************************************
172.16.0.101               : ok=11   changed=11   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
172.16.0.102               : ok=11   changed=11   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
172.16.0.103               : ok=11   changed=11   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

設定された config

config の差分を確認してみます。

[root@localhost ~]# diff -y -W 80  before/Leaf1-running-config after/Leaf1-running-config
version 9.3(1) Bios:version             version 9.3(1) Bios:version
hostname Leaf1                          hostname Leaf1
--- snip ---
fabric forwarding anycast-gateway-mac   fabric forwarding anycast-gateway-mac
vlan 1                                | vlan 1,100-101,900
                                      | vlan 100
                                      >   name VNI10000
                                      >   vn-segment 10000
                                      > vlan 101
                                      >   name VNI10001
                                      >   vn-segment 10001
                                      > vlan 900
                                      >   name VNI90000_vrf_blue
                                      >   vn-segment 90000
                                      >
                                      > vrf context blue
                                      >   vni 90000
                                      >   rd auto
                                      >   address-family ipv4 unicast
                                      >     route-target both auto evpn
--- snip ---

interface Vlan1                         interface Vlan1

                                      > interface Vlan100
                                      >   no shutdown
                                      >   vrf member blue
                                      >   ip address 192.168.0.254/24
                                      >   fabric forwarding mode anycast-gate
                                      >
                                      > interface Vlan101
                                      >   no shutdown
                                      >   vrf member blue
                                      >   ip address 192.168.1.254/24
                                      >   fabric forwarding mode anycast-gate
                                      >
                                      > interface Vlan900
                                      >   no shutdown
                                      >   vrf member blue
                                      >   ip forward
                                      >
interface nve1                          interface nve1
  no shutdown                             no shutdown
  host-reachability protocol bgp          host-reachability protocol bgp
  source-interface loopback1              source-interface loopback1
                                      >   member vni 10000
                                      >     ingress-replication protocol bgp
                                      >   member vni 10001
                                      >     ingress-replication protocol bgp
                                      >   member vni 90000 associate-vrf

router bgp 65000                        router bgp 65000
  address-family ipv4 unicast             address-family ipv4 unicast
  address-family l2vpn evpn               address-family l2vpn evpn
  neighbor 1.1.1.1                        neighbor 1.1.1.1
    remote-as 65000                         remote-as 65000
    update-source loopback0                 update-source loopback0
    address-family ipv4 unicast             address-family ipv4 unicast
    address-family l2vpn evpn               address-family l2vpn evpn
      send-community extended                 send-community extended
  neighbor 1.1.1.2                        neighbor 1.1.1.2
    remote-as 65000                         remote-as 65000
    update-source loopback0                 update-source loopback0
    address-family ipv4 unicast             address-family ipv4 unicast
    address-family l2vpn evpn               address-family l2vpn evpn
      send-community extended                 send-community extended
                                      >   vrf blue
                                      >     address-family ipv4 unicast

Leaf2、Leaf3 も同様の変更があることを確認しました。

通信確認

L2 と L3 で通信できることを確認できました。

PC-1> ping 192.168.0.2
84 bytes from 192.168.0.2 icmp_seq=1 ttl=64 time=101.646 ms
84 bytes from 192.168.0.2 icmp_seq=2 ttl=64 time=40.088 ms
84 bytes from 192.168.0.2 icmp_seq=3 ttl=64 time=29.891 ms
84 bytes from 192.168.0.2 icmp_seq=4 ttl=64 time=22.204 ms
^C
PC-1> ping 192.168.1.3
84 bytes from 192.168.1.3 icmp_seq=1 ttl=63 time=47.751 ms
84 bytes from 192.168.1.3 icmp_seq=2 ttl=63 time=38.885 ms
84 bytes from 192.168.1.3 icmp_seq=3 ttl=63 time=32.026 ms
84 bytes from 192.168.1.3 icmp_seq=4 ttl=63 time=53.960 ms
^C
PC-1> 

Leaf1が別のLeafに接続されている端末の情報を、BGP で受け取っていることがわかります。

Leaf1# show bgp l2vpn evpn vrf blue
Route Distinguisher: 1.1.1.3:7    (L3VNI 90000)
*>i[2]:[0]:[0]:[48]:[0000.aaaa.0001]:[32]:[192.168.0.254]/272
                      2.2.2.4                           100          0 i
*>i[2]:[0]:[0]:[48]:[0000.aaaa.0001]:[32]:[192.168.1.254]/272
                      2.2.2.4                           100          0 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6801]:[32]:[192.168.1.3]/272
                      2.2.2.5                           100          0 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6802]:[32]:[192.168.0.2]/272
                      2.2.2.4                           100          0 i

まとめ

VXLAN EVPN Fabric の環境に仮想ネットワークを作成する ansible playbook を作成しました。VXLAN Fabric の設定変更は複数のスイッチに対して同時に設定する必要があるので、Ansible を利用する効果が高いと思います。