As an Azure MVP, I spent a lot of time teaching and giving webinars on the topic of moving to the cloud. A few months ago, I started to experience issues with my internet service provider (ISP), where I would lose connection, and this was very painful during large webinars. It was embarrassing professionally, so when I moved, I decided to do a major upgrade. I wanted to move to more professional grade equipment with multiple internet providers, so that if one ISP failed, I would not lose connection.
Like any project, I first documented my requirements:
- Professional-grade gear, this is how I make my living so no more cutting corners
- Two connections to the internet from separate ISPs: Primary 1Gbps fiber Secondary at least 400Mbps Cable modem
- Automatic failover of the connections to the internet, if one goes down it should failover immediately to the other
- VLANs to isolate my work traffic from the family traffic. All streaming services should go out of the secondary cable. 100% IPTV with no traditional cable boxes.
Here is a basic view of the design with the intention that all business traffic will go out of my primary connection to Frontier FiOS and Spectrum as the secondary connection for all IPTV and family traffic. The Unifi USG PRO 4 was selected as the firewall, along with the CloudKey and WAPs. I elected to use the QNAP QGD-1600P to act as my PoE managed switch along with a NAS with 4TB of SSD drives for my ESXi Lab.
Setup of the Unifi Configuration
In order to get this configuration to work I’ve configured the Unifi Network with the following configurations:
- Two LAN Configurations have been created:
- DELTAV-LAN: Purpose Corporate, Port LAN1 SUBNET 192.168.1.0/24. DELTAV-LAN was the default configuration and is using the untagged traffic, so there is no VLAN.
- DELTAV-STREAM: Purpose Corporate, Port LAN2 SUBNET 192.168.2.0/24 VLAN 2. This configured with the firewall configured as the DHCP server with a scope to assign Ips to the AppleTVs, Sonos and family devices.
- The two WAN configurations are connected with Frontier in WAN1 and Spectrum in WAN2.
- Frontier is configured with the default configuration and with custom DNS settings. To the Frontier DNS server and Google.
- Spectrum is configured as Failover Only along with custom DNS settings. Notice that it is not part of the VLAN. This should not be configured as the routing inside of the Unifi will allow the traffic to pass from the deltavstream network and VLAN2 over this connection.
- Two networks are configured, deltavnet and deltavsteam.
- Deltavnet is configured using the defaults.
- Deltavstream is configured using the default but has been assigned to VLAN2.
With these configurations I was now able to join the deltavsteam wifi network, get an IP address and connect to the internet.
When configuring my AppleTVs I selected this network and was able to get to the internet, but with FiOS as the primary connection when using the Spectrum app I am not able to use all of the channels and even get a warning message that “Connect to Spectrum Internet to use your subscription”.
Test failover & Spectrum connection
To test the failover and the cable modem connection I simply unplug the ethernet cable from WAN1 which goes to my FiOS ONT. In only about 10 second the USG PRO 4 failovers over to the Spectrum connection. Now the AppleTV connection to Spectrum using their IP address and goes over the Cable modem connection allowing me to see all my channels and stream IPTV. Once I plug the FiOS connection back in the Spectrum connection fails back and now the Apple TV can no longer connection to my subscription.
Policy based routing
To make use of my Apple TVs I need a way to have the deltavstream network only send packets over the WAN2 interface meaning to Spectrum. Remember we had configured the network to use WAN2 as “Failover Only”. In order to do this, I need to setup a “policy-based route”, which will forward all traffic from the 192.168.2.0/24 network over WAN2.
UI has a pretty comprehensive article that explains how to do this with a bunch of options (very confusing). In my case, I referenced the section: Routing Traffic Out of WAN2 Based on the Source Network.
The article gives an example that must be completed from the command line using SSH to connect directly to the USG PRO 4. Once this configuration is completed you will be able to see if the configuration works, but this will only be in the “running config”, meaning that it is running in memory. The next time you provision the USG PRO 4 your change will be lost. In order to save the configuration, you must create a config.gateway.json file using your configuration (more on that later).
Create & test policy-based route
To create the route, I first connected to the USG PRO using SSH. ssh firstname.lastname@example.org
Once connected I ran the show interfaces command to see my interfaces and the IP addresses. I can now see the IP address for my secondary link is 70.XX.53.85. This is the address that Spectrum gave to me.
Using an IP CIDR calculator, I can find the default gateway for that network. It is important for us to find the “router” on the ISP side for the rule as that is how we will forward the traffic using the route
So, now that we have the router IP address, we need to create the route table and then create a LOAD_BALANCE rule to forward the traffic over that interface. I used the following commands in the SSH session to enter these into the running config.
configure set protocols static table 5 route 0.0.0.0/0 next-hop 184.108.40.206 set firewall modify LOAD_BALANCE rule 2640 action modify set firewall modify LOAD_BALANCE rule 2640 modify table 5 set firewall modify LOAD_BALANCE rule 2640 source address 192.168.2.0/24 set firewall modify LOAD_BALANCE rule 2640 protocol all commit;exit
Immediately my Apple TV was able to use all the Spectrum Channels! Both of my connections were still online, but now the policy-based route was in place!
Save the configuration to the Unifi Controller
With the change in place and my IPTV networks now connected via the WAN2 Connection, it is now time for the difficult challenge to have this be a permanent configuration on my USG PRO. In order to do this, I had to create a JSON file with the configurations that I did at the command line before.
NOTE: This difficult and advanced configuration explained in this article. https://help.ui.com/hc/en-us/articles/215458888
I used the following command to dump the configuration:
mca-ctrl -t dump-cfg
Next, I copied the output of the command to Visual Studio code and saved it as a file called config.startup.json. I then searched the file for the configurations I had made with the static table 5 and the rule 2640 that I created.
I created a new file config.gateway.json, and put in a pair of curly braces to start and end the configuration. Next, I pasted the two configurations between them making sure to include the namespace for where the configuration lived in the config that I copied from the firewall.
- The last step was to check my code using this JSON formatter page to ensure my code was at least formatted correctly: .
- With my JSON formatted properly locally and then connected to the Unifi Controller using WINSCP. I FTP’d the file to the /srv/unifi/data/sites/default directory (your site name could be different).
With the config.gateway.json file now in place I connected to the controller with a browser and navigated to the devices page. On the settings I then under Manage Device clicked Provision to Force provision the device. After about a minute the configuration was saved and now the policy based route is a permanent part of my configuration.
If you want to see the configuration command and the config.gateway.json file check out my GitHub repo for this blog:
Thx for the great article.
Why you didn’t choose a UniFi Switch?
I wanted to have both a NAS and PoE Switch as one device. And the QNAP has more PoE power budget for my network as I have many devices.
Amazing solution Dan. I am sure this will suit to many in current lock-down situation.
Hi Dan, Great article. Perhaps we can help make this easier on you and get you onboarded using a 128Technology that offers tunnel-free routing and SD-WAN in your home lab as well as in Azure? We are happy to help fellow mvps!
Great work, Dan! This is exactly what I’ve been wanted to piece together, I just ordered my secondary internet circuit today. Just by looking at your background I’m guessing you have since stitched together a way to monitor throughput on each link. Care to share?
Very helpful article as I read the Unifi guide and was very lost. I have a similar situation but hoping to only route it via cable if the destination address matches something that would require the cable network. Namely to solve for times when I am trying to do a stream on my phone that requires me to be on my cable “home network” and not on the fiber network. At other times I would prefer to be on the fiber for the faster upload speeds. Any ideas?
Really helpful guide, thanks. One thing I’d love to see though is an explanation of these magic numbers/values. Why table ‘5’? Where is ‘LOAD_BALANCE’ defined? Is that default or completely new? Why rule 2640, not 2639? Etc.
Funny enough, I have the exact same configuration within my home/office!
This policy-based setup caused a problem with me when configuring Manual site-to-site VPNs between the office USG and a remote UDM (UniFi Dream Machine). With you being Azure MVP I was wondering whether you ran into the same problem…
Like I said before, my configuration is exactly the same – different subnets, but the same VLAN setup and I also had home and office devices going over different ISPs, but failing over to the other if there was a fault. This setup worked for years!
My problems started when I wanted to setup VPN between my business VLAN (on my USG) and one of the VLANs on a remote UDM. After days of retries I found that the VPN was always unreliable. The VPN between the two locations would always connect, but 2-way traffic between the USG and remote UDM would almost never work. If it did, the success was short lived and 2-way communication only lasted a few minutes. After an hour or less, no network traffic was being permitted over the VPN (even though the VPN connection never dropped).
A UniFi consultant that I worked with pointed to the policy-based routing configured in the config.gateway.json being the culprit, and suggested unplugging the secondary internet connection. When we did that, viola! – the VPN came up and the VPN has been stable ever since. The problem, however, is that I miss my previous setup!
Have you tried setting up any IPsec and/or OpenVPN vpns between two location using your policy-based configuration? If not, could you try and share whether you ran into the same problems, and how you overcame them?