You will need at least the following…

1. A pair of stacked switches that support creating an LACP bonded port across the stack on 2 different nodes. This gives you the best of all worlds being able to provide redundancy and increase your bandwidth.
2. Or alternatively, 2 ports on the same or on different unstacked switches. This is the bare minimum you can do to mitigate link failure. Note this setup has no polling mechanism so if the physical ethernet link stays up but is not operational because of device switching failure, or a failure on the another port on the device that provides the uplink, then this wont help you.

On your server you will need 2 (or more) network cards and some “simple” setup

Install the packages that you will need in case you don’t have them already.

• apt-get install ifenslave vlan bridge-utils

The example sets up the following

• eth0 and eth1 bonded together into bond0
• create 2 bridges br8 and br9
• create 2 vlans bond0.8 and bond0.9
• place them in each bridge respecitvely
• add IP details on br9
• br8 has no L3 config on it and in this specific case is used by KVM to bridge virtual machines into as they come online

For option 1 edit your /etc/network/interfaces to look something like this

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo bond0 bond0.8 bond0.9 br8 br9
iface lo inet loopback

iface bond0 inet manual
 bond-slaves eth0 eth1
 bond-mode 802.3ad
 mond-miimon 100
 bond-use-carrier 1
 bond-lacp-rate 1
 bond-min-links 1
 # send traffic over the available links based on src/dst MAC address
 bond-xmit-hash-policy layer2
 mtu 1600

iface bond0.8 inet manual
iface bond0.9 inet manual

iface br8 inet manual
 bridge_stp off
 bridge_ports bond0.8

iface br9 inet static
 address 192.168.0.2
 netmask 255.255.255.0
 gateway 192.168.0.1
 bridge_ports bond0.9
 bridge_stp off

For option 2 edit your /etc/network/interfaces to look something like this (only the bond0) config changes

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo bond0 bond0.8 bond0.9 br8 br9
iface lo inet loopback

iface bond0 inet manual
 slaves eth0 eth1
 bond_mode active-backup
 bond_miimon 100
 bond_downdelay 200
 bond_updelay 200

iface bond0.8 inet manual
iface bond0.9 inet manual

iface br8 inet manual
 bridge_stp off
 bridge_ports bond0.8

iface br9 inet static
 address 192.168.0.2
 netmask 255.255.255.0
 gateway 192.168.0.1
 bridge_ports bond0.9
 bridge_stp off

Most use cases probably will not require bridging or VLAN but I thought it best to provide examples of the entire feature set, you can always reduce to what you need.