Important
|
This overview is valid only when using QEMU backend! |
Which networking type to use is controlled by the NICTYPE
variable. If unset or empty NICTYPE
defaults to user, ie qemu
user networking which requires no further configuration.
With Qemu user networking each jobs gets it’s own isolated network with TCP and UDP routed to the outside. DHCP is provided by qemu. The MAC address of the machine can be controlled with the NICMAC variable. If not set, it’s 52:54:00:12:34:56.
os-autoinst can connect qemu to TAP devices of the host system to leverage advanced network setups provided by the host by setting NICTYPE=tap.
The TAP device to use can be configured with the TAPDEV variable. If not set defined, ist’s automatically set to "tap" + ($worker_instance - 1), i.e. worker1 uses tap0, worker 2 uses tap1 and so on.
For multiple networks per job (see NETWORKS variable), the following numbering scheme is used:
worker1: tap0 tap64 tap128 ...
worker2: tap1 tap65 tap129 ...
worker3: tap2 tap66 tap130 ...
...
MAC address of virtual NIC is controlled by NICMAC variable or automatically computed from $worker_id if not set.
In TAP mode the system administrator is expected to configure the network, required internet access etc on the host manually.
Virtual Distributed Ethernet provides a software switch that runs in user space. It allows to connect several qemu instances without affecting the system’s network configuration.
The openQA workers need a vde_switch instance running. The workers reconfigure the switch as needed by the job.
To start with a basic configuration like qemu user mode networking, create a machine with the following settings:
-
VDE_SOCKETDIR=/run/openqa
-
NICTYPE=vde
-
NICVLAN=0
Start switch and user mode networking:
systemctl start openqa-vde_switch
systemctl start openqa-slirpvde
With this setting all jobs on the same host would be in the same network share the same SLIRP instance though.
The section provides one of the ways for setting up openQA environment to run tests that require network connection between several machines (e.g. client — server tests).
The example of the configuration is applicable for openSUSE and will use Open vSwitch for virtual switch, SuSEfirewall2 or firewalld for NAT and wicked as network manager.
Note
|
Other way to setup the environment with iptables and firewalld is described on Fedora wiki. |
Set Up Open vSwitch
Compared to VDE setup, Open vSwitch is a little bit more complicated to configure, but provides more robust and scalable network.
-
Install and Run Open vSwitch:
zypper in openvswitch systemctl start openvswitch.service systemctl enable openvswitch.service
-
Install and configure os-autoinst-openvswitch.service:
Note
|
os-autoinst-openvswitch.service is a support service that sets vlan number of Open vSwitch ports based on NICVLAN variable - this separates the groups of tests from each other. NICVLAN variable is dynamically assigned by OpenQA scheduler. |
zypper in os-autoinst-openvswitch
systemctl start os-autoinst-openvswitch
systemctl enable os-autoinst-openvswitch
os-autoinst-openvswitch.service uses br0 bridge by default. As it might be used by KVM, configure br1 instead.
# /etc/sysconfig/os-autoinst-openvswitch
OS_AUTOINST_USE_BRIDGE=br1
-
Create virtual bridge br1:
ovs-vsctl add-br br1
Configure Virtual Interfaces
-
Add tap interface for every multi-machine worker:
Note
|
Create as many interfaces as needed for a test. The instructions are provided for three interfaces tap0, tap1, tap2 to be used by worker@1, worker@2, worker@3 workers. TAP interfaces have to be owned by the _openqa-worker user for openQA to be able to access them. |
To create tap interfaces automatically on startup, add appropriate configuration files to the
/etc/sysconfig/network/
directory. Files have to be named as ifcfg-tap<N>
, replacing <N>
with the number for the interface, such as 0
, 1
, 2
(e.g. ifcfg-tap0
, ifcfg-tap1
).
# /etc/sysconfig/network/ifcfg-tap0
BOOTPROTO='none'
IPADDR=''
NETMASK=''
PREFIXLEN=''
STARTMODE='auto'
TUNNEL='tap'
TUNNEL_SET_GROUP='nogroup'
TUNNEL_SET_OWNER='_openqa-worker'
-
Add bridge config with all tap devices that should be connected to it:
The file have to be located in /etc/sysconfig/network/
directory. File name is
ifcfg-br<N>
, where <N>
is the id of the bridge (e.g. 1
).
# /etc/sysconfig/network/ifcfg-br1
BOOTPROTO='static'
IPADDR='10.0.2.2/15'
STARTMODE='auto'
OVS_BRIDGE='yes'
OVS_BRIDGE_PORT_DEVICE_1='tap0'
OVS_BRIDGE_PORT_DEVICE_2='tap1'
OVS_BRIDGE_PORT_DEVICE_3='tap2'
Configure NAT with SuSEfirewall
The IP 10.0.2.2 can be also served as a gateway to access outside network. For this, a NAT between br1 and eth0 must be configured with SuSEfirewall or iptables.
# /etc/sysconfig/SuSEfirewall2
FW_DEV_INT="br1"
FW_ROUTE="yes"
FW_MASQUERADE="yes"
Start SuSEfirewall2 and allow to run on startup:
systemctl start SuSEfirewall2
systemctl enable SuSEfirewall2
Configure NAT with firewalld
To configure NAT with firewalld assign the bridge interface to the internal zone and the interface with access to the network to the external zone:
firewall-cmd --permanent --zone=external --add-interface=eth0
firewall-cmd --permanent --zone=internal --add-interface=br1
Reload firewall configuration using firewall-cmd --reload
command.
To enable masquerade one can use the following command:
firewall-cmd --permanent --zone=external --add-masquerade
ip_forward is enabled automatically if masquerading is enabled:
cat /proc/sys/net/ipv4/ip_forward
1
In case interface is in trusted network it is possible to accept connections by default by changing zone target:
firewall-cmd --permanent --zone=external --set-target=ACCEPT
Alternatively, you can assign interface to the trusted
zone.
If you do not have firewalld service running, you can use firewall-cmd-offline
command for the configuration. Enable service to run on startup:
systemctl start firewalld
systemctl enable firewalld
Also, the firewall-config
GUI tool for firewalld can be used for configuration.
Configure OpenQA Workers
-
Allow workers to run multi-machine jobs:
# /etc/openqa/workers.ini
[global]
WORKER_CLASS = qemu_x86_64,tap
-
Enable workers to be started on system boot:
systemctl enable openqa-worker@1
systemctl enable openqa-worker@2
systemctl enable openqa-worker@3
Grant CAP_NET_ADMIN Capabilities to QEMU
In order to use TAP device which doesn’t exist on the system, it is required to set CAP_NET_ADMIN capability on qemu binary file:
zypper in libcap-progs
setcap CAP_NET_ADMIN=ep /usr/bin/qemu-system-x86_64
Configure Network Manager
-
Check the configuration for the eth0 interface:
Important
|
Ensure, that eth0 interface is configured in /etc/sysconfig/network/ifcfg-eth0 .
Otherwise, wicked will not be able to bring up the interface on start and host will loose network
connection.
|
# /etc/sysconfig/network/ifcfg-eth0
BOOTPROTO='dhcp'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR=''
MTU=''
NAME=''
NETMASK=''
REMOTE_IPADDR=''
STARTMODE='auto'
DHCLIENT_SET_DEFAULT_ROUTE='yes'
-
Start wicked as network service:
Check the network service currently being used:
systemctl show -p Id network.service
If the result is different from Id=wicked.service
(e.g. NetworkManager.service
),
stop the network service:
systemctl stop network.service
systemctl disable network.service
Then switch to wicked:
systemctl enable --force wicked
systemctl start wicked
-
Bring up br1 interface:
wicked ifup br1
-
REBOOT
Boot sequence with wicked < 0.6.23:
-
wicked - creates tap devices
-
openvswitch - creates the bridge br1, adds tap devices to it
-
wicked handles br1 as hotplugged device, assignd the IP 10.0.2.2 to it, updates SuSEfirewall
-
os-autoinst-openvswitch - installs openflow rules, handles vlan assignment
Boot sequence with wicked 0.6.23 and newer:
-
openvswitch
-
wicked - creates the bridge br1 and tap devices, add tap devices to the bridge,
-
SuSEfirewall
-
os-autoinst-openvswitch - installs openflow rules, handles vlan assignment
The configuration and operation can be checked by the following commands:
ovs-vsctl show # shows the bridge br1, the tap devices are assigned to it
ovs-ofctl dump-flows br1 # shows the rules installed by os-autoinst-openvswitch in table=0
-
packets from tapX to br1 create additional rules in table=1
-
packets from br1 to tapX increase packet counts in table=1
-
empty output indicates a problem with os-autoinst-openvswitch service
-
zero packet count or missing rules in table=1 indicate problem with tap devices
iptables -L -v
As long as the SUT has access to external network, there should be nonzero packet count in the forward chain between br1 and external interface.
By default all multi-machine workers have to be on single physical machine. You can join multiple physical machines and its ovs bridges together by GRE tunnel.
If the workers with TAP capability are spread across multiple hosts, the network must be connected. See Open vSwitch documentation for details.
Create gre_tunnel_preup script (change remote_ip value correspondingly on both hosts)
# /etc/wicked/scripts/gre_tunnel_preup.sh
#!/bin/sh
action="$1"
bridge="$2"
ovs-vsctl --may-exist add-port $bridge gre1 -- set interface gre1 type=gre options:remote_ip=<IP address of other host>
And call it by PRE_UP_SCRIPT="wicked:gre_tunnel_preup.sh" entry
# /etc/sysconfig/network/ifcfg-br1
<..>
PRE_UP_SCRIPT="wicked:gre_tunnel_preup.sh"
Allow GRE in firewall
# /etc/sysconfig/SuSEfirewall2
FW_SERVICES_EXT_IP="GRE"
FW_SERVICES_EXT_TCP="1723"
Note
|
When using GRE tunnels keep in mind that VMs inside the ovs bridges have to use MTU=1458 for their physical interfaces (eth0, eth1). If you are using support_server/setup.pm the MTU will be set automatically to that value on support_server itself and it does MTU advertisement for DHCP clients as well. |