Do you have a need to transport Layer2 VLANs across the internet? This technology is widely used in Data Center Interconnection (DCI) schemes.
This walk through will show you how to establish Layer2 connectivity between two (or more) sites while maintaining security across the public Internet.
VXLAN Tunneling via an IPSEC/GRE Tunnel
Lab Diagram
For this lab, we will be using our EVE-NG emulation platform.
Lab Details
- VyOS Router (tested with v1.1.8 and v1.2.0) – The configurations below are specifically for v1.2.0
- Basic LAN Bridge/Switch
- vPC
Note: The “Internet” router in the middle is a simple VyOS router with two interfaces configured to act as a routed path between the two sites, emulating an Internet Services Provider. ETH0 is configured with address 198.51.100.1/30 and ETH1 is configured with address 198.51.100.21/30
Instructions
There are three parts to this setup. IPSEC Tunnel setup, GRE Tunnel setup and finally VXLAN Tunnel setup. We will walk you through each configuration separately.
First, we need to setup the IPSEC tunnel between the two routers. To do this, we first need to build the IPSEC encryption methods. For current best practice encryption methods, we are going to establish an AES256/SHA256 encrypted tunnel using a pre-shared key for authentication. You will need to apply this configuration to both routers.
## Phase 1 - IPSEC IKEv2 ## set vpn ipsec ike-group aes256-sha256 dead-peer-detection action 'restart' set vpn ipsec ike-group aes256-sha256 dead-peer-detection interval '30' set vpn ipsec ike-group aes256-sha256 dead-peer-detection timeout '120' set vpn ipsec ike-group aes256-sha256 ikev2-reauth 'no' set vpn ipsec ike-group aes256-sha256 key-exchange 'ikev2' set vpn ipsec ike-group aes256-sha256 lifetime '3600' set vpn ipsec ike-group aes256-sha256 proposal 1 dh-group '19' set vpn ipsec ike-group aes256-sha256 proposal 1 encryption 'aes256' set vpn ipsec ike-group aes256-sha256 proposal 1 hash 'sha256' ## Phase 2 - ESP Tunnel ## set vpn ipsec esp-group aes256-sha256 compression 'disable' set vpn ipsec esp-group aes256-sha256 lifetime '28800' set vpn ipsec esp-group aes256-sha256 mode 'tunnel' set vpn ipsec esp-group aes256-sha256 pfs 'dh-group19' set vpn ipsec esp-group aes256-sha256 proposal 1 encryption 'aes256' set vpn ipsec esp-group aes256-sha256 proposal 1 hash 'sha256'
We are now going to setup the IPSEC tunnel endpoints. This is were we blend in the IPSEC with the GRE tunnel.
Please make sure to generate your own Pre-Shared key for a production installation.
VYOS-R1
set vpn ipsec site-to-site peer 198.51.100.22 authentication mode 'pre-shared-secret' set vpn ipsec site-to-site peer 198.51.100.22 authentication pre-shared-secret '3mqBSixtMiDy4ngWUKt0uScGH9f2vPoy' set vpn ipsec site-to-site peer 198.51.100.22 connection-type 'initiate' set vpn ipsec site-to-site peer 198.51.100.22 ike-group 'aes256-sha256' set vpn ipsec site-to-site peer 198.51.100.22 ikev2-reauth 'inherit' set vpn ipsec site-to-site peer 198.51.100.22 local-address '198.51.100.2' set vpn ipsec site-to-site peer 198.51.100.22 tunnel 1 allow-nat-networks 'disable' set vpn ipsec site-to-site peer 198.51.100.22 tunnel 1 allow-public-networks 'disable' set vpn ipsec site-to-site peer 198.51.100.22 tunnel 1 esp-group 'aes256-sha256' set vpn ipsec site-to-site peer 198.51.100.22 tunnel 1 local prefix '10.10.0.1/32' set vpn ipsec site-to-site peer 198.51.100.22 tunnel 1 protocol 'gre' set vpn ipsec site-to-site peer 198.51.100.22 tunnel 1 remote prefix '10.10.0.2/32' set vpn ipsec ipsec-interfaces interface 'eth0'
VYOS-R2
set vpn ipsec site-to-site peer 198.51.100.2 authentication mode 'pre-shared-secret' set vpn ipsec site-to-site peer 198.51.100.2 authentication pre-shared-secret '3mqBSixtMiDy4ngWUKt0uScGH9f2vPoy' set vpn ipsec site-to-site peer 198.51.100.2 connection-type 'initiate' set vpn ipsec site-to-site peer 198.51.100.2 ike-group 'aes256-sha256' set vpn ipsec site-to-site peer 198.51.100.2 ikev2-reauth 'inherit' set vpn ipsec site-to-site peer 198.51.100.2 local-address '198.51.100.22' set vpn ipsec site-to-site peer 198.51.100.2 tunnel 1 allow-nat-networks 'disable' set vpn ipsec site-to-site peer 198.51.100.2 tunnel 1 allow-public-networks 'disable' set vpn ipsec site-to-site peer 198.51.100.2 tunnel 1 esp-group 'aes256-sha256' set vpn ipsec site-to-site peer 198.51.100.2 tunnel 1 local prefix '10.10.0.2/32' set vpn ipsec site-to-site peer 198.51.100.2 tunnel 1 protocol 'gre' set vpn ipsec site-to-site peer 198.51.100.2 tunnel 1 remote prefix '10.10.0.1/32' set vpn ipsec ipsec-interfaces interface 'eth0'
There are three distinct settings in this IPSEC tunnel that are different from other IPSEC configurations. First, this specific tunnel only encrypts the GRE protocol. Second, it is using “dummy” interface IPs for the local and remote prefix settings. This is done so that it forces the GRE tunnel to use the IPSEC tunnel for the transit path.
Next, we are going to setup the GRE tunnels. The GRE tunnels are configured to use “dummy” interfaces so that the GRE traffic is forced into the IPSEC tunnel for encryption.
VYOS-R1
set interfaces dummy dum0 address '10.10.0.1/32' set interfaces tunnel tun0 address '10.0.0.1/30' set interfaces tunnel tun0 encapsulation 'gre' set interfaces tunnel tun0 local-ip '10.10.0.1' set interfaces tunnel tun0 multicast 'disable' set interfaces tunnel tun0 remote-ip '10.10.0.2'
VYOS-R2
set interfaces dummy dum0 address '10.10.0.2/32' set interfaces tunnel tun0 address '10.0.0.2/30' set interfaces tunnel tun0 encapsulation 'gre' set interfaces tunnel tun0 local-ip '10.10.0.2' set interfaces tunnel tun0 multicast 'disable' set interfaces tunnel tun0 remote-ip '10.10.0.1'
Once you commit these changes, you should have a GRE/IPSEC tunnel up and running. You can verify this by doing an ICMP ping to each endpoint.
vyos-r1# ping 10.0.0.2 -c 5 PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data. 64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=2.59 ms 64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=1.89 ms 64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=1.17 ms 64 bytes from 10.0.0.2: icmp_seq=4 ttl=64 time=1.13 ms 64 bytes from 10.0.0.2: icmp_seq=5 ttl=64 time=1.51 ms --- 10.0.0.2 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4005ms rtt min/avg/max/mdev = 1.139/1.663/2.599/0.544 ms vyos-r2# ping 10.0.0.1 -c 5 PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data. 64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=1.98 ms 64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=1.40 ms 64 bytes from 10.0.0.1: icmp_seq=3 ttl=64 time=1.18 ms 64 bytes from 10.0.0.1: icmp_seq=4 ttl=64 time=1.67 ms 64 bytes from 10.0.0.1: icmp_seq=5 ttl=64 time=1.48 ms --- 10.0.0.1 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4006ms rtt min/avg/max/mdev = 1.182/1.546/1.981/0.270 ms
We are now ready to create our VXLAN tunnel, which will be configured to transport inside the GRE tunnel. The below configuration should be applied to both routers.
set interfaces bridge br0 set interfaces ethernet eth1 bridge-group bridge 'br0' set interfaces vxlan vxlan0 bridge-group bridge 'br0' set interfaces vxlan vxlan0 group '239.0.0.241' set interfaces vxlan vxlan0 link 'tun0' set interfaces vxlan vxlan0 remote-port '4789' set interfaces vxlan vxlan0 vni '32000'
What this has done, is created a bridge interface that connects the Layer2 interface (ETH1) and the Layer2 interface (VXLAN0) together. We then configured the VXLAN interface to use the ‘tun0’ interface as the transport link for communication. The VNI ID is a user selected ID between the range of 0 – 16777214. We have also instructed the VXLAN interface to use multicast as the communication method.
Once you commit the changes, your VXLAN interface will be online and you should be able to configure identical IP subnets on both sites.
vpc-1> show ip NAME : vpc-1[1] IP/MASK : 192.168.10.10/24 GATEWAY : 0.0.0.0 DNS : MAC : 00:50:79:66:68:01 vpc-2> show ip NAME : vpc-2[1] IP/MASK : 192.168.10.20/24 GATEWAY : 0.0.0.0 DNS : MAC : 00:50:79:66:68:02
vpc-1> ping 192.168.10.20 84 bytes from 192.168.10.20 icmp_seq=1 ttl=64 time=3.376 ms 84 bytes from 192.168.10.20 icmp_seq=2 ttl=64 time=2.884 ms 84 bytes from 192.168.10.20 icmp_seq=3 ttl=64 time=2.556 ms 84 bytes from 192.168.10.20 icmp_seq=4 ttl=64 time=2.158 ms 84 bytes from 192.168.10.20 icmp_seq=5 ttl=64 time=1.941 ms vpc-2> ping 192.168.10.10 84 bytes from 192.168.10.10 icmp_seq=1 ttl=64 time=2.722 ms 84 bytes from 192.168.10.10 icmp_seq=2 ttl=64 time=2.262 ms 84 bytes from 192.168.10.10 icmp_seq=3 ttl=64 time=1.805 ms 84 bytes from 192.168.10.10 icmp_seq=4 ttl=64 time=1.897 ms 84 bytes from 192.168.10.10 icmp_seq=5 ttl=64 time=2.551 ms
From each router, you can also show a MAC address table of each bridge to help validate connectivity on the VXLAN tunnel.
vyos-r1# run show bridge br0 macs port no mac addr is local? ageing timer 1 00:50:79:66:68:01 no 115.96 2 00:50:79:66:68:02 no 115.96 1 50:01:00:03:00:01 yes 0.00 1 50:01:00:03:00:01 yes 0.00 2 fa:f9:e4:5a:d4:cc yes 0.00 2 fa:f9:e4:5a:d4:cc yes 0.00 vyos-r2# run show bridge br0 macs port no mac addr is local? ageing timer 2 00:50:79:66:68:01 no 85.98 1 00:50:79:66:68:02 no 85.98 1 50:01:00:06:00:01 yes 0.00 1 50:01:00:06:00:01 yes 0.00 2 fe:43:87:fd:27:e6 yes 0.00 2 fe:43:87:fd:27:e6 yes 0.00
IPSEC VPN Tunnel Validation
vyos-r1# run show vpn ipsec sa
Connection State Uptime Bytes In/Out Packets In/Out Remote address Remote ID Proposal
--------------------------- ------- -------- -------------- ---------------- ---------------- ----------- -------------------------------------
peer-198.51.100.22-tunnel-1 up 14m30s 1K/1K 13/13 198.51.100.22 N/A AES_CBC_256/HMAC_SHA2_256_128/ECP_521
vyos-r2# run show vpn ipsec sa
Connection State Uptime Bytes In/Out Packets In/Out Remote address Remote ID Proposal
-------------------------- ------- -------- -------------- ---------------- ---------------- ----------- -------------------------------------
peer-198.51.100.2-tunnel-1 up 15m2s 1K/1K 13/13 198.51.100.2 N/A AES_CBC_256/HMAC_SHA2_256_128/ECP_521
/////
vyos-r1# run show vpn ipsec state
src 198.51.100.2 dst 198.51.100.22
proto esp spi 0xc6d0e4ce reqid 3 mode tunnel
replay-window 0 flag af-unspec
auth-trunc hmac(sha256) 0x6fabc09d2d9479c9a40977951eb937e078b391dda747026584ef1f30f01c09ec 128
enc cbc(aes) 0xbd2c8a96b8238552bd0ca256275f97edcc7e6d218a04b2ce2e9fbf9ef36d2546
anti-replay context: seq 0x0, oseq 0xd, bitmap 0x00000000
src 198.51.100.22 dst 198.51.100.2
proto esp spi 0xcb53a035 reqid 3 mode tunnel
replay-window 32 flag af-unspec
auth-trunc hmac(sha256) 0x4e9d1fb82fa6429f8b7af518b00c9c08aeb87a07c9e9212ab3ca4022f2a18c8a 128
enc cbc(aes) 0xbfedf7ef3567160ce3f4b2c1f4907505c60ebd6b5a77b5be493c0913d6c9c3fd
anti-replay context: seq 0xd, oseq 0x0, bitmap 0x00001fff
vyos-r2# run show vpn ipsec state
src 198.51.100.22 dst 198.51.100.2
proto esp spi 0xcb53a035 reqid 3 mode tunnel
replay-window 0 flag af-unspec
auth-trunc hmac(sha256) 0x4e9d1fb82fa6429f8b7af518b00c9c08aeb87a07c9e9212ab3ca4022f2a18c8a 128
enc cbc(aes) 0xbfedf7ef3567160ce3f4b2c1f4907505c60ebd6b5a77b5be493c0913d6c9c3fd
anti-replay context: seq 0x0, oseq 0xd, bitmap 0x00000000
src 198.51.100.2 dst 198.51.100.22
proto esp spi 0xc6d0e4ce reqid 3 mode tunnel
replay-window 32 flag af-unspec
auth-trunc hmac(sha256) 0x6fabc09d2d9479c9a40977951eb937e078b391dda747026584ef1f30f01c09ec 128
enc cbc(aes) 0xbd2c8a96b8238552bd0ca256275f97edcc7e6d218a04b2ce2e9fbf9ef36d2546
anti-replay context: seq 0xd, oseq 0x0, bitmap 0x00001fff
Traffic Capture (Encrypted)
Traffic Capture (Decrypted)
I have also attached the PCAP of this lab traffic so that you can review it as well.
vyos_vxlan_tunnel_eve-lab_traffic_collection
You now have a secure Layer2 inter-connection between your two data centers using VXLAN.