As an Azure MVP, I spent a lot of time teaching and giving webinars on the topic of moving to the cloud. A few months ago, I started to experience issues with my internet service provider (ISP), where I would lose connection, and this was very painful during large webinars. It was embarrassing professionally, so when I moved, I decided to do a major upgrade. I wanted to move to more professional grade equipment with multiple internet providers, so that if one ISP failed, I would not lose connection.

Requirements

Like any project, I first documented my requirements:

• Professional-grade gear, this is how I make my living so no more cutting corners
• Two connections to the internet from separate ISPs: Primary 1Gbps fiber Secondary at least 400Mbps Cable modem
• Automatic failover of the connections to the internet, if one goes down it should failover immediately to the other
• VLANs to isolate my work traffic from the family traffic. All streaming services should go out of the secondary cable. 100% IPTV with no traditional cable boxes.

Architecture

Here is a basic view of the design with the intention that all business traffic will go out of my primary connection to Frontier FiOS and Spectrum as the secondary connection for all IPTV and family traffic. The Unifi USG PRO 4 was selected as the firewall, along with the CloudKey and WAPs. I elected to use the QNAP QGD-1600P to act as my PoE managed switch along with a NAS with 4TB of SSD drives for my ESXi Lab.

Setup of the Unifi Configuration

In order to get this configuration to work I’ve configured the Unifi Network with the following configurations:

Networks>Local Networks

• Two LAN Configurations have been created:
  • DELTAV-LAN: Purpose Corporate, Port LAN1 SUBNET 192.168.1.0/24. DELTAV-LAN was the default configuration and is using the untagged traffic, so there is no VLAN.
  • DELTAV-STREAM: Purpose Corporate, Port LAN2 SUBNET 192.168.2.0/24 VLAN 2. This configured with the firewall configured as the DHCP server with a scope to assign Ips to the AppleTVs, Sonos and family devices.

Internet>WAN Networks

• The two WAN configurations are connected with Frontier in WAN1 and Spectrum in WAN2.
  • Frontier is configured with the default configuration and with custom DNS settings. To the Frontier DNS server and Google.
  • Spectrum is configured as Failover Only along with custom DNS settings.  Notice that it is not part of the VLAN. This should not be configured as the routing inside of the Unifi will allow the traffic to pass from the deltavstream network and VLAN2 over this connection.

WI-FI>Wi-Fi Networks

• Two networks are configured, deltavnet and deltavsteam.
  • Deltavnet is configured using the defaults.
  • Deltavstream is configured using the default but has been assigned to VLAN2.

With these configurations I was now able to join the deltavsteam wifi network, get an IP address and connect to the internet.

When configuring my AppleTVs I selected this network and was able to get to the internet, but with FiOS as the primary connection when using the Spectrum app I am not able to use all of the channels and even get a warning message that “Connect to Spectrum Internet to use your subscription”.

Test failover & Spectrum connection

To test the failover and the cable modem connection I simply unplug the ethernet cable from WAN1 which goes to my FiOS ONT. In only about 10 second the USG PRO 4 failovers over to the Spectrum connection. Now the AppleTV connection to Spectrum using their IP address and goes over the Cable modem connection allowing me to see all my channels and stream IPTV.  Once I plug the FiOS connection back in the Spectrum connection fails back and now the Apple TV can no longer connection to my subscription.

Policy based routing

To make use of my Apple TVs I need a way to have the deltavstream network only send packets over the WAN2 interface meaning to Spectrum. Remember we had configured the network to use WAN2 as “Failover Only”. In order to do this, I need to setup a “policy-based route”, which will forward all traffic from the 192.168.2.0/24 network over WAN2.

UI has a pretty comprehensive article that explains how to do this with a bunch of options (very confusing). In my case, I referenced the section: Routing Traffic Out of WAN2 Based on the Source Network.

The article gives an example that must be completed from the command line using SSH to connect directly to the USG PRO 4. Once this configuration is completed you will be able to see if the configuration works, but this will only be in the “running config”, meaning that it is running in memory. The next time you provision the USG PRO 4 your change will be lost. In order to save the configuration, you must create a config.gateway.json file using your configuration (more on that later).

Create & test policy-based route

To create the route, I first connected to the USG PRO using SSH. ssh deltadan@192.168.1.1

Once connected I ran the show interfaces command to see my interfaces and the IP addresses. I can now see the IP address for my secondary link is 70.XX.53.85. This is the address that Spectrum gave to me.

Using an IP CIDR calculator, I can find the default gateway for that network. It is important for us to find the “router” on the ISP side for the rule as that is how we will forward the traffic using the route

So, now that we have the router IP address, we need to create the route table and then create a LOAD_BALANCE rule to forward the traffic over that interface. I used the following commands in the SSH session to enter these into the running config.

configure
set protocols static table 5 route 0.0.0.0/0 next-hop 70.125.52.1
set firewall modify LOAD_BALANCE rule 2640 action modify
set firewall modify LOAD_BALANCE rule 2640 modify table 5
set firewall modify LOAD_BALANCE rule 2640 source address 192.168.2.0/24
set firewall modify LOAD_BALANCE rule 2640 protocol all
commit;exit

Immediately my Apple TV was able to use all the Spectrum Channels! Both of my connections were still online, but now the policy-based route was in place!

Save the configuration to the Unifi Controller

With the change in place and my IPTV networks now connected via the WAN2 Connection, it is now time for the difficult challenge to have this be a permanent configuration on my USG PRO. In order to do this, I had to create a JSON file with the configurations that I did at the command line before.

NOTE: This difficult and advanced configuration explained in this article. https://help.ui.com/hc/en-us/articles/215458888

I used the following command to dump the configuration:

mca-ctrl -t dump-cfg

Next, I copied the output of the command to Visual Studio code and saved it as a file called config.startup.json. I then searched the file for the configurations I had made with the static table 5 and the rule 2640 that I created.

I created a new file config.gateway.json, and put in a pair of curly braces to start and end the configuration. Next, I pasted the two configurations between them making sure to include the namespace for where the configuration lived in the config that I copied from the firewall.

• The last step was to check my code using this JSON formatter page to ensure my code was at least formatted correctly: .

• With my JSON formatted properly locally and then connected to the Unifi Controller using WINSCP. I FTP’d the file to the /srv/unifi/data/sites/default directory (your site name could be different).

With the config.gateway.json file now in place I connected to the controller with a browser and navigated to the devices page. On the settings I then under Manage Device clicked Provision to Force provision the device. After about a minute the configuration was saved and now the policy based route is a permanent part of my configuration.

If you want to see the configuration command and the config.gateway.json file check out my GitHub repo for this blog:

https://github.com/deltadan/usg-pro-policy-based-route

Happy steaming!

@deltadan