Some tips to configure host to host ip over infiniband

Infiniband allows different server to communicate high speed between each other. When you don’t use a switch to control your network you need to run opensm service on one of the host to manage the network.

# apt-get install opensm
# systemctl enable opensm
# systemctl start opensm

Some modules are required to run infiniband and ip over ib. Add them into /etc/modules

mlx4_ib
ib_ipoib
rdma_ucm
ib_umad

Normally you get interface like this one:

# ip a
ibp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 256
    link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:00:1e:67:03:00:58:08:39 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

You can setup an IP address

# ip a add 192.168.10.1/24 dev ibp1s0