今年の個人的な目標として、流行のデータセンターネットワーク技術を学びたいと思っています(現状、仕事ではほぼ必要としていません)
EVE-NG上でvJunos-SwitchとvJunos-Routerを使って(EVPN/VXLANによる)IP-Closなネットワークを作ってみました。
今回は設定について書いていきますので、詳しい説明は省略します。
説明が欲しい方は、IP-Closについてはココ、EVPN/VXLANについてはココやココをご覧ください。
今回は初めてというものあり、spineが1台、leafが2台のシンプルな構成とします(冗長構成はおいおい検証していきたい…)
ホストの代わりでルーター(vJunos-Router)を置いています。EVE-NGでLinux置く方法がよくわからん
疎通確認であれば、ルーターでも代替になるということで…。
ゴールとしては「vRouter-1とvRouter-2間でL2VPN(同一NWとして)でつながる」ことです。
spine、leafはvJunos-Switch(Junos 23.2R1.14)、vRouterはvJunos-Router(Junos 23.2R1.15)を使用します。
せっかくなので、物理リンク間は/31なサブネットで、Underlay/Overlayネットワークともに4Byte ASを使います。
1.Underlayネットワーク設定
まず、お互いのLoopbackアドレスを広報するためのUnderlayネットワークを構築します。以下の構成で設定していきます。
//spine-1 set interfaces ge-0/0/0 unit 0 family inet address 10.1.1.0/31 set interfaces ge-0/0/1 unit 0 family inet address 10.1.1.2/31 set interfaces lo0 unit 0 family inet address 172.16.0.1/32 set routing-options router-id 172.16.0.1 set routing-options autonomous-system 4200000001 set protocols bgp group UNDERLAY type external set protocols bgp group UNDERLAY family inet unicast set protocols bgp group UNDERLAY export EXPORT_BGP_UNDERLAY set protocols bgp group UNDERLAY neighbor 10.1.1.1 peer-as 4200001001 set protocols bgp group UNDERLAY neighbor 10.1.1.3 peer-as 4200001002 set policy-options policy-statement EXPORT_BGP_UNDERLAY term 0011 from route-filter 172.16.0.0/24 prefix-length-range /32-/32 set policy-options policy-statement EXPORT_BGP_UNDERLAY term 0011 then accept set policy-options policy-statement EXPORT_BGP_UNDERLAY term 9999 then reject
//leaf-1 set interfaces ge-0/0/0 unit 0 family inet address 10.1.1.1/31 set interfaces lo0 unit 0 family inet address 172.16.0.11/32 set routing-options router-id 172.16.0.11 set routing-options autonomous-system 4200001001 set protocols bgp group UNDERLAY type external set protocols bgp group UNDERLAY family inet unicast set protocols bgp group UNDERLAY export EXPORT_BGP_UNDERLAY set protocols bgp group UNDERLAY neighbor 10.1.1.0 peer-as 4200000001 set policy-options policy-statement EXPORT_BGP_UNDERLAY term 0011 from protocol direct set policy-options policy-statement EXPORT_BGP_UNDERLAY term 0011 from route-filter 172.16.0.11/32 exact set policy-options policy-statement EXPORT_BGP_UNDERLAY term 0011 then accept set policy-options policy-statement EXPORT_BGP_UNDERLAY term 9999 then reject
//leaf-2 set interfaces ge-0/0/0 unit 0 family inet address 10.1.1.3/31 set interfaces lo0 unit 0 family inet address 172.16.0.12/32 set routing-options router-id 172.16.0.12 set routing-options autonomous-system 4200001002 set protocols bgp group UNDERLAY type external set protocols bgp group UNDERLAY family inet unicast set protocols bgp group UNDERLAY export EXPORT_BGP_UNDERLAY set protocols bgp group UNDERLAY neighbor 10.1.1.2 peer-as 4200000001 set policy-options policy-statement EXPORT_BGP_UNDERLAY term 0011 from protocol direct set policy-options policy-statement EXPORT_BGP_UNDERLAY term 0011 from route-filter 172.16.0.12/32 exact set policy-options policy-statement EXPORT_BGP_UNDERLAY term 0011 then accept set policy-options policy-statement EXPORT_BGP_UNDERLAY term 9999 then reject
お互いのLoopbackアドレスが学習できてるか確認します。
//spine-1
lab@spine-1# run show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 1 Peers: 2 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
2 2 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
10.1.1.1 4200001001 15 14 0 0 5:16 Establ
inet.0: 1/1/1/0
10.1.1.3 4200001002 7 6 0 0 1:42 Establ
inet.0: 1/1/1/0
lab@spine-1# run show route protocol bgp
inet.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
Limit/Threshold: 1048576/1048576 destinations
+ = Active Route, - = Last Active, * = Both
172.16.0.11/32 *[BGP/170] 00:08:35, localpref 100
AS path: 4200001001 I, validation-state: unverified
> to 10.1.1.1 via ge-0/0/0.0
172.16.0.12/32 *[BGP/170] 00:05:02, localpref 100
AS path: 4200001002 I, validation-state: unverified
> to 10.1.1.3 via ge-0/0/1.0
//leaf-1
lab@leaf-1# run show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 1 Peers: 1 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
2 2 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
10.1.1.0 4200000001 18 16 0 0 6:17 Establ
inet.0: 2/2/2/0
lab@leaf-1# run show route protocol bgp
inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
Limit/Threshold: 1048576/1048576 destinations
+ = Active Route, - = Last Active, * = Both
172.16.0.1/32 *[BGP/170] 00:08:08, localpref 100
AS path: 4200000001 I, validation-state: unverified
> to 10.1.1.0 via ge-0/0/0.0
172.16.0.12/32 *[BGP/170] 00:04:47, localpref 100
AS path: 4200000001 4200001002 I, validation-state: unverified
> to 10.1.1.0 via ge-0/0/0.0
//leaf-2
lab@leaf-2# run show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 1 Peers: 1 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
2 2 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
10.1.1.2 4200000001 13 10 0 0 3:43 Establ
inet.0: 2/2/2/0
lab@leaf-2# run show route protocol bgp
inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
Limit/Threshold: 1048576/1048576 destinations
+ = Active Route, - = Last Active, * = Both
172.16.0.1/32 *[BGP/170] 00:05:53, localpref 100
AS path: 4200000001 I, validation-state: unverified
> to 10.1.1.2 via ge-0/0/0.0
172.16.0.11/32 *[BGP/170] 00:05:53, localpref 100
AS path: 4200000001 4200001001 I, validation-state: unverified
> to 10.1.1.2 via ge-0/0/0.0
2.Overlayネットワーク設定
続いてOverlayネットワークを構築します。構成図は以下の通りです。
MP-BGPにてEVPN NLRIをやりとりできるようにします。
configのポイントしては、(eBGPにてLoopbackアドレスでPeeringするので)Multihopの設定が必要な点でしょうか。
spineについては、さらにno-nexthop-changeオプションが必要となります。
(今回、spine - leaf間でASNを分けているので、このオプションも入れないと正しくVTEPを検出できません)
//spine-1 set protocols bgp group OVERLAY type external set protocols bgp group OVERLAY multihop no-nexthop-change set protocols bgp group OVERLAY local-address 172.16.0.1 set protocols bgp group OVERLAY family evpn signaling set protocols bgp group OVERLAY neighbor 172.16.0.11 peer-as 4200001001 set protocols bgp group OVERLAY neighbor 172.16.0.12 peer-as 4200001002
//leaf-1 set protocols bgp group OVERLAY type external set protocols bgp group OVERLAY multihop set protocols bgp group OVERLAY local-address 172.16.0.11 set protocols bgp group OVERLAY family evpn signaling set protocols bgp group OVERLAY peer-as 4200000001 set protocols bgp group OVERLAY neighbor 172.16.0.1
//leaf-2 set protocols bgp group OVERLAY type external set protocols bgp group OVERLAY multihop set protocols bgp group OVERLAY local-address 172.16.0.12 set protocols bgp group OVERLAY family evpn signaling set protocols bgp group OVERLAY peer-as 4200000001 set protocols bgp group OVERLAY neighbor 172.16.0.1
OverlayなMP-BGPでもPeeringできてるか確認します。
//spine-1
lab@spine-1# run show bgp summary group OVERLAY
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 2 Peers: 4 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
2 2 0 0 0 0
bgp.evpn.0
0 0 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.0.11 4200001001 10 8 0 0 3:16 Establ
bgp.evpn.0: 0/0/0/0
172.16.0.12 4200001002 10 8 0 0 3:04 Establ
bgp.evpn.0: 0/0/0/0
//leaf-1
lab@leaf-1# run show bgp summary group OVERLAY
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 2 Peers: 2 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
2 2 0 0 0 0
bgp.evpn.0
0 0 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.0.1 4200000001 11 10 0 0 3:43 Establ
bgp.evpn.0: 0/0/0/0
//leaf-2
lab@leaf-2# run show bgp summary group OVERLAY
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 2 Peers: 2 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
2 2 0 0 0 0
bgp.evpn.0
0 0 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.0.1 4200000001 11 10 0 0 3:55 Establ
bgp.evpn.0: 0/0/0/0
3.VXLAN設定
最後にVXLAN周りの設定を行います。構成や用いるパラメーターは以下の通りです。
VLANとVNIのマッピングやVTEPで用いるインターフェイスの定義などをします。
RDやroute-target(vrf-target)なんかも設定していきます。この辺はL2VPNやL3VPNの設定と一緒ですね。
あとついでにvRouterの設定もします。
//leaf-1 set vlans VLAN0012 vlan-id 12 set vlans VLAN0012 vxlan vni 10012 set interfaces ge-0/0/6 unit 0 family ethernet-switching interface-mode trunk set interfaces ge-0/0/6 unit 0 family ethernet-switching vlan members all set protocols evpn encapsulation vxlan set protocols evpn extended-vni-list all set switch-options vtep-source-interface lo0.0 set switch-options route-distinguisher 172.16.0.11:1 set switch-options vrf-target target:539:539
//leaf-2 set vlans VLAN0012 vlan-id 12 set vlans VLAN0012 vxlan vni 10012 set interfaces ge-0/0/6 unit 0 family ethernet-switching interface-mode trunk set interfaces ge-0/0/6 unit 0 family ethernet-switching vlan members all set protocols evpn encapsulation vxlan set protocols evpn extended-vni-list all set switch-options vtep-source-interface lo0.0 set switch-options route-distinguisher 172.16.0.12:1 set switch-options vrf-target target:539:539
//vRouter-1 set interfaces ge-0/0/0 vlan-tagging set interfaces ge-0/0/0 unit 0 vlan-id 12 set interfaces ge-0/0/0 unit 0 family inet address 12.12.12.1/24
//vRouter-2 set interfaces ge-0/0/0 vlan-tagging set interfaces ge-0/0/0 unit 0 vlan-id 12 set interfaces ge-0/0/0 unit 0 family inet address 12.12.12.2/24
設定は以上となります。
4.動作確認
確認作業をしていきます。
この確認作業からホスト(vRouter)のMACアドレスが非常に重要になりますのでメモしておきます。
| ホスト | MACアドレス |
| vRouter-1 ge-0/0/0 | 50:01:00:0a:00:01 |
| vRouter-2 ge-0/0/0 | 50:01:00:0b:00:01 |
まず論より証拠…ではないですが、vRouter間でPingを飛ばしてみます。
//vRouter-1 lab@vRouter-1# run ping 12.12.12.2 PING 12.12.12.2 (12.12.12.2): 56 data bytes 64 bytes from 12.12.12.2: icmp_seq=0 ttl=64 time=3.128 ms 64 bytes from 12.12.12.2: icmp_seq=1 ttl=64 time=3.438 ms 64 bytes from 12.12.12.2: icmp_seq=2 ttl=64 time=3.651 ms 64 bytes from 12.12.12.2: icmp_seq=3 ttl=64 time=3.874 ms 64 bytes from 12.12.12.2: icmp_seq=4 ttl=64 time=3.782 ms //vRouter-2 lab@vRouter-2# run show arp MAC Address Address Name Interface Flags 50:01:00:0a:00:01 12.12.12.1 12.12.12.1 ge-0/0/0.0 none 02:00:00:00:00:10 128.0.0.16 fpc0 em1.0 none Total entries: 2
Pingが飛び、vRouter-2もvRouter-1のARP解決ができてるので、EVPNが正しく動作してそうです。
もう少し詳しく見ていきます。まずleaf-1にて確認してきます。
//leaf-1
lab@leaf-1# run show ethernet-switching vxlan-tunnel-end-point remote
Logical System Name Id SVTEP-IP IFL L3-Idx SVTEP-Mode ELP-SVTEP-IP
<default> 0 172.16.0.11 lo0.0 0
RVTEP-IP L2-RTT IFL-Idx Interface NH-Id RVTEP-Mode ELP-IP Flags
172.16.0.12 default-switch 345 vtep.32769 586 RNVE
VNID MC-Group-IP
10012 0.0.0.0
lab@leaf-1# run show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC,
B - Blocked MAC)
Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
Vlan MAC MAC GBP Logical SVLBNH/ Active
name address flags tag interface VENH Index source
VLAN0012 50:01:00:0a:00:01 D ge-0/0/6.0
VLAN0012 50:01:00:0b:00:01 DR vtep.32769 172.16.0.12「show ethernet-switching vxlan-tunnel-end-point remote」コマンドで(leaf-2の)VTEPが正しく検出できてるのが確認できますね。
また「show ethernet-switching table」コマンドでVTEP経由でvRouter-2 ge-0/0/0のMACアドレスを学習できてるのが確認できます。
さらにleaf-1のshow routeの出力を見ていきます。
//leaf-1
lab@spine-1# run show route table bgp.evpn.0 | no-more
bgp.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
2:172.16.0.11:1::10012::50:01:00:0a:00:01/304 MAC/IP
*[BGP/170] 00:02:10, localpref 100, from 172.16.0.11
AS path: 4200001001 I, validation-state: unverified
> to 10.1.1.1 via ge-0/0/0.0, Push 625
2:172.16.0.12:1::10012::50:01:00:0b:00:01/304 MAC/IP
*[BGP/170] 00:02:13, localpref 100, from 172.16.0.12
AS path: 4200001002 I, validation-state: unverified
> to 10.1.1.3 via ge-0/0/1.0, Push 625
2:172.16.0.11:1::10012::50:01:00:0a:00:01::12.12.12.1/304 MAC/IP
*[BGP/170] 00:02:10, localpref 100, from 172.16.0.11
AS path: 4200001001 I, validation-state: unverified
> to 10.1.1.1 via ge-0/0/0.0, Push 625
2:172.16.0.12:1::10012::50:01:00:0b:00:01::12.12.12.2/304 MAC/IP
*[BGP/170] 00:02:13, localpref 100, from 172.16.0.12
AS path: 4200001002 I, validation-state: unverified
> to 10.1.1.3 via ge-0/0/1.0, Push 625
3:172.16.0.11:1::10012::172.16.0.11/248 IM
*[BGP/170] 00:12:29, localpref 100, from 172.16.0.11
AS path: 4200001001 I, validation-state: unverified
> to 10.1.1.1 via ge-0/0/0.0
3:172.16.0.12:1::10012::172.16.0.12/248 IM
*[BGP/170] 00:12:12, localpref 100, from 172.16.0.12
AS path: 4200001002 I, validation-state: unverified
> to 10.1.1.3 via ge-0/0/1.0Routing Table entryに「2:172.16.0.12:1::10012::50:01:00:0b:00:01::12.12.12.2/304」があるので、vRouter-2のMACアドレス情報が広報されているのが確認できます。
ちなみにJunosのRouting Table entryの出力フォーマットは以下となっています。
[Route-Type]:[RD]::[ESI]:[VNI]::[MAC address info]::[optional-IP info]/xxx //今回、ESIは設定してない(All Zero)なので上記のshow route結果では省略されている。
leaf-2側でもshow routeの出力確認します。
lab@leaf-2# run show route table bgp.evpn.0 | no-more
bgp.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
2:172.16.0.11:1::10012::50:01:00:0a:00:01/304 MAC/IP
*[BGP/170] 00:03:09, localpref 100, from 172.16.0.1
AS path: 4200000001 4200001001 I, validation-state: unverified
> to 10.1.1.2 via ge-0/0/0.0, Push 625
2:172.16.0.12:1::10012::50:01:00:0b:00:01/304 MAC/IP
*[EVPN/170] 00:03:10
Indirect
2:172.16.0.11:1::10012::50:01:00:0a:00:01::12.12.12.1/304 MAC/IP
*[BGP/170] 00:03:09, localpref 100, from 172.16.0.1
AS path: 4200000001 4200001001 I, validation-state: unverified
> to 10.1.1.2 via ge-0/0/0.0, Push 625
2:172.16.0.12:1::10012::50:01:00:0b:00:01::12.12.12.2/304 MAC/IP
*[EVPN/170] 00:03:10
Indirect
3:172.16.0.11:1::10012::172.16.0.11/248 IM
*[BGP/170] 02:05:19, localpref 100, from 172.16.0.1
AS path: 4200000001 4200001001 I, validation-state: unverified
> to 10.1.1.2 via ge-0/0/0.0
3:172.16.0.12:1::10012::172.16.0.12/248 IM
*[EVPN/170] 02:05:26
IndirectvRouter-1のMAC/IP情報の「2:172.16.0.11:1::10012::50:01:00:0a:00:01::12.12.12.1/304」があるので、ちゃんと学習できてることが確認できます。
routeの詳細情報やspine-1でも確認したいところですが、尺の都合で省略します。
まあleaf間で各ホストのMACアドレスが学習できてるなら大丈夫でしょう…。
おまけでspine-leaf間のパケットキャプチャーを見てみます。
まずPing飛ばした際のARPパケットから。
ARP ReqがVTEP(172.16.0.12)宛のUnicastで飛んでますね。
続いてICMPパケットをみます。
こちらも同様にVTEP宛のUnicastでカプセル化されてます。
非常に長くなりましたが、シンプルな構成でホスト間の疎通ができました！！
今後はシリーズ化して冗長構成やService Interfacesを検証したいと思います！！！
5.参考資料
書籍「Deploying Juniper Data Centers with EVPN VXLAN」
EVPN-Presentation.pptxwww.slideshare.net
