How to setup multiple gateways for multiple Ethernets?

If you would like to use multiple gateways, you should use iproute package and define the gateway in two tables in conjunction with each interface.

Here is an example how to configure it :

Let's assume eth0 has gateway 10.1XX.69.1 and eth3's gateway is 10.1XX.66.1.

So, you should execute following steps :

echo 1 ISP1 >> /etc/iproute2/rt_tables
echo 2 ISP2 >> /etc/iproute2/rt_tables

The next step is to have some routing rules and routes:

For the ISP1 table:

ip route add default via 10.1XX.69.1 dev eth0 table ISP1
ip rule add from 10.1XX.69.0/24 table ISP1


For the ISP2 table:

ip route add default via 10.1XX.66.1 dev eth3 table ISP2
ip rule add from 10.1XX.66.0/24 table ISP2

------------------

In order to make above rules persistent, you should put them in rc.local file like :

cat /etc/rc.local

ip route add default via 10.1XX.69.1 dev eth0 table ISP1
ip rule add from 10.1XX.69.0/24 table ISP1
ip route add default via 10.1XX.66.1 dev eth3 table ISP2
ip rule add from 10.1XX.66.0/24 table ISP2

Note : You should replace the IP addresses in above commands.