Table of Contents
LXC Linux Containers
Installation
Standard install of Ubuntu 14.04, additional packages installed are:- lxc lxctl lxc-templates,
Disk layout
Containers are stored in /var/lib/lxc which is mounted on it's own partition.
root@com-prod-lxc01:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/cciss/c0d0p1 9.1G 919M 7.7G 11% / /dev/cciss/c0d0p2 7.3G 815M 6.1G 12% /var /dev/cciss/c0d0p4 113G 397M 107G 1% /var/lib/lxc root@com-prod-lxc01:~# ls -l /var/lib/lxc total 20 drwx------ 2 root root 16384 Dec 11 15:27 lost+found drwxrwx--- 3 root root 4096 Dec 15 16:26 ubuntu-test01 root@com-prod-lxc01:~# ls -l /var/lib/lxc/ubuntu-test01/ total 8 -rw-r--r-- 1 root root 574 Dec 15 16:26 config -rw-r--r-- 1 root root 0 Dec 15 16:26 fstab drwxr-xr-x 21 root root 4096 Dec 15 16:29 rootfs root@com-prod-lxc01:~#
Container management
Creating a new container
root@com-prod-lxc01:~# lxc-create -n ubuntu-test01 -t ubuntu Checking cache download in /var/cache/lxc/trusty/rootfs-amd64 ... Installing packages in template: ssh,vim,language-pack-en Downloading ubuntu trusty minimal ... I: Retrieving Release I: Retrieving Release.gpg I: Checking Release signature I: Valid Release signature (key id 790BC7277767219C42C86F933B4FE6ACC0B21F32) I: Retrieving Packages I: Validating Packages ... edited - lots of lines! ... Setting up language-pack-en (1:14.04+20150219) ... Setting up language-pack-en-base (1:14.04+20150219) ... Generating locales... en_AG.UTF-8... up-to-date Ubuntu 14.04.3 LTS ubuntu-test01 tty1 ubuntu-test01 login: ubuntu Password: Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.16.0-30-generic x86_64) ubuntu@ubuntu-test01:~$
Default login is ubuntu and ubuntu
Starting and Stopping a container
root@com-prod-lxc01:~# lxc-start -n ubuntu-test01 -d
The -d option starts the container in the background (Run the container as a daemon). Without this it will start in the foreground.
You can see the status with lxc-info:-
root@com-prod-lxc01:~# lxc-info --name ubuntu-test01 Name: ubuntu-test01 State: RUNNING PID: 9555 IP: 10.0.3.160 CPU use: 1.23 seconds BlkIO use: 148.00 KiB Memory use: 7.71 MiB KMem use: 0 bytes Link: vethG4W9L4 TX bytes: 2.22 KiB RX bytes: 2.34 KiB Total bytes: 4.56 KiB
Stopping is not very exciting:-
root@com-prod-lxc01:~# lxc-stop -n ubuntu-test01 root@com-prod-lxc01:~# root@com-prod-lxc01:~# lxc-stop -n ubuntu-test01 ubuntu-test01 is not running
The only really useful info on what happened is from the exit codes:-
0 The container was successfully stopped.
1 An error occurred while stopping the container.
2 The specified container exists but was not running.
Auto start of containers
root@com-prod-lxc01:~# lxc-ls --fancy NAME STATE IPV4 IPV6 AUTOSTART -------------------------------------------------------------------------------------- com6prod-hmaclient01 RUNNING 10.10.110.181, 10.10.61.1, 10.200.1.4 - YES com6prod-hmaclient02 RUNNING 10.10.110.182, 10.10.62.1, 10.200.1.63 - YES
This is set in the config file for each container:-
from:- /var/lib/lxc/com6prod-hmaclient01/config # Autostart lxc.start.auto = 1 lxc.start.delay = 5 lxc.start.order = 100 (increments of 100 for each instance)
Connecting to a container
To connect to the container console, use the lxc-console command, quit from the console with “Ctrl-a q”, this just disconnects from the console, if you have not logged out from the console, you will remain logged in, quitting the console does not log you out (just like vnc etc):-
root@com-prod-lxc01:~# root@com-prod-lxc01:~# lxc-console -n com6prod-hmaclient01 Connected to tty 1 Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself fmadmin@com6prod-hmaclient01:~$ cat /etc/hostname com6prod-hmaclient01 fmadmin@com6prod-hmaclient01:~$ <ctrl-a q pressed> root@com-prod-lxc01:~# root@com-prod-lxc01:~#
Info about containers
root@com-prod-lxc01:~# lxc-ls ubuntu-test01 root@com-prod-lxc01:~# lxc-ls --fancy ubuntu-test01 NAME STATE IPV4 IPV6 AUTOSTART --------------------------------------------------- ubuntu-test01 RUNNING 10.0.3.160 - NO root@com-prod-lxc01:~# lxc-info -n ubuntu-test01 Name: ubuntu-test01 State: RUNNING PID: 5266 IP: 10.0.3.160 CPU use: 1.35 seconds BlkIO use: 140.00 KiB Memory use: 7.94 MiB KMem use: 0 bytes Link: veth2O0CUI TX bytes: 3.41 KiB RX bytes: 2.54 KiB Total bytes: 5.95 KiB
Cloning a container
root@com-prod-lxc01:/var/lib/lxc# ls -l total 20 drwx------ 2 root root 16384 Dec 11 15:27 lost+found drwxrwx--- 3 root root 4096 Dec 15 16:26 ubuntu-test01 root@com-prod-lxc01:/var/lib/lxc# lxc-clone -o ubuntu-test01 -n com6-prod-hmaclient01 Created container com6-prod-hmaclient01 as copy of ubuntu-test01 root@com-prod-lxc01:/var/lib/lxc# ls -l total 24 drwxrwx--- 3 root root 4096 Jan 19 15:12 com6-prod-hmaclient01 drwxrwx--- 3 root root 4096 Dec 15 16:26 ubuntu-test01 root@com-prod-lxc01:/var/lib/lxc#
Networking
System
The server is fitted with two physical interfaces, eth0 and eth1. These are combined into a bond0 for redundancy, and should be connected to different switches (
). Over this there is a VLAN driver run which allows individual vlans to be mapped to an interface. For this to work, the uplinks to the switch are TRUNKS and should be configured as trunks on the switch not access ports.
sw# interface g 0/1 sw(config-if)# switchport mode trunk
Each vlan enabled sub interface is bridged to a brXYZ device which corresponds to the VLAN XYZ on the network. Each container is then connected appropriate interfaces.
The outbound internet is accessed through VLAN 110, so the egress from each container is over br110 which is mapped to eth0. The vpn protected interface is mapped to isolated VLANs which do not route to anything.
The iptv boxes are connected to a switch port which is attached to the appropriate vpn protected VLAN.
This diagram summarises the setup:-
LXC
/etc/init/lxc-net.conf sets the default network config. This uses a NAT to a lxcbr0 bridge device.
Specific configs are kept in /var/lib/lxc/<HOSTNAME>/config
From the client, the interfaces file looks wrong, this is because the network config is set in the lxc config file:_
lxc.network.type = veth lxc.network.flags = up lxc.network.link = br110 lxc.network.name = eth0 lxc.network.hwaddr = 00:16:3e:7d:56:93 lxc.network.ipv4.gateway = 10.10.110.1 lxc.network.ipv4 = 10.10.110.181/24 lxc.network.type = veth lxc.network.flags = up lxc.network.link = br61 lxc.network.name = eth1 lxc.network.hwaddr = 02:00:00:00:00:61 lxc.network.ipv4 = 10.10.61.1/24
DHCP
Each container runs it's own dhcp server for just that container's protected subnet. This config is kelp in /etc/dhcp/dhcpd.conf:-
#leases are stores in /var/lib/dhcp/dhcpd.leases
ddns-update-style none;
default-lease-time 600;
max-lease-time 7200;
subnet 10.10.110.0 netmask 255.255.255.0 {
}
subnet 10.10.61.0 netmask 255.255.255.0 {
range 10.10.61.11 10.10.61.100;
option domain-name-servers 8.8.8.8, 8.8.4.4;
option domain-name "vlan61.fmts.int";
option routers 10.10.61.1;
option broadcast-address 10.10.61.255;
#set lease to 3 months
default-lease-time 7776000;
max-lease-time 7776000;
}
This info can be extracted with a simple api call to the container running the VPN service:-
http://10.10.110.181/getipfrommac.php?mac=00-0e:08e1:7c7b gives:-
{"deviceip": { "exists": "true", "macaddress": "000e08e17c7b", "ipaddress": "10.10.61.11" } }
But a request for an invalid mac gives:-
http://10.10.110.181/getipfrommac.php?mac=00-0e:08e1:7c7a
{"deviceip": { "exists": "false", "macaddress": "000e08e17c7a", } }
VPN service
This is provided by HMA, https://www.hidemyass.com/.
The HMA client needs installing in each container NOT on the main server. (https://hmastuff.com/linux/hma-linux.zip)
Initially, container did not have tun device, so it needed to be created (error was ERROR: Cannot open TUN/TAP dev /dev/net/tun: No such file or directory (errno=2) )
root@ubuntu-test01:~# mkdir /dev/net root@ubuntu-test01:~# mknod /dev/net/tun c 10 200
HMA username: nodes0001
HMA password: nodes00011000sedon
Start the vpn with:-
root@ubuntu-test01:~# /root/hma-openvpn.sh -d ============================== | __ ____ ______ __ | | / / / / |/ / | / / | | / /_/ / /|_/ / /| | / / | | / __ / / / / ___ |/_/ | | /_/ /_/_/ /_/_/ |_(_) | | | | HMA! OpenVPN Script v0.23 | ============================== -> https://hidemyass.com/vpn -> https://support.hidemyass.com - Getting .ovpn template... - Current IP: 83.244.144.52 - Starting OpenVPN connection to: UK, Berkshire, Maidenhead (LOC1 S10) - 78.129.150.2 : 53 (udp) ... (Please enter your HMA! Pro VPN account username and password when asked) Enter Auth Username:nodes0001 Enter Auth Password: - Waiting for connection process to complete... - IP has changed (83.244.144.52 -> 78.129.150.43) Connection successful. Disconnect via: 'sudo killall openvpn'
Automating HMA
According to https://support.hidemyass.com/hc/en-us/articles/202721456-Recommended-Linux-CLI-OpenVPN-Client, it is possible to save username and password info so the vpn can be brought up with a single command, this uses the hma-grabber script, but there is a problem with it. Basically, wget will respect robots.txt unless it is told to ignore it and the site's robots.txt disallow downloads from
/vpn-config/TCP/ causing the script to fail.
This can be fixed by adding a -e robots=off directive to the wget command line.
Now the ovpn profiles are downloaded, they can be used.
Before and after
root@ubuntu-test01:~# curl freegeoip.net/xml/`curl curlmyip.com`
<?xml version="1.0" encoding="UTF-8"?>
<Response>
<IP>83.244.144.52</IP>
<CountryCode>GB</CountryCode>
<CountryName>United Kingdom</CountryName>
<RegionCode>ENG</RegionCode>
<RegionName>England</RegionName>
<City>Coventry</City>
<ZipCode>CV1</ZipCode>
<TimeZone>Europe/London</TimeZone>
<Latitude>52.4066</Latitude>
<Longitude>-1.5122</Longitude>
<MetroCode>0</MetroCode>
</Response>
root@ubuntu-test01:~#
root@ubuntu-test01:~# openvpn /etc/openvpn/Kenya.Nairobi.UDP.ovpn &
[1] 16792
root@ubuntu-test01:~# Fri Jan 8 14:40:06 2016 OpenVPN 2.3.2 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Dec 1 2014
Fri Jan 8 14:40:06 2016 Socket Buffers: R=[212992->131072] S=[212992->131072
....edited....
Fri Jan 8 14:40:16 2016 /sbin/ip link set dev tun0 up mtu 1500
Fri Jan 8 14:40:16 2016 /sbin/ip addr add dev tun0 10.200.7.190/22 broadcast 10.200.7.255
Fri Jan 8 14:40:16 2016 /sbin/ip route add 154.70.152.238/32 via 10.10.110.1
Fri Jan 8 14:40:16 2016 /sbin/ip route add 0.0.0.0/1 via 10.200.4.1
Fri Jan 8 14:40:16 2016 /sbin/ip route add 128.0.0.0/1 via 10.200.4.1
Fri Jan 8 14:40:16 2016 add_route_ipv6(): not adding 2000::/3, no IPv6 on if tun0
Fri Jan 8 14:40:16 2016 Initialization Sequence Completed
root@ubuntu-test01:~# ifconfig tun0
tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.200.7.190 P-t-P:10.200.7.190 Mask:255.255.252.0
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:152 (152.0 B) TX bytes:152 (152.0 B)
root@ubuntu-test01:~# curl freegeoip.net/xml/`curl curlmyip.com`
<?xml version="1.0" encoding="UTF-8"?>
<Response>
<IP>154.127.56.250</IP>
<CountryCode>KE</CountryCode>
<CountryName>Kenya</CountryName>
<RegionCode>110</RegionCode>
<RegionName>Nairobi Province</RegionName>
<City>Nairobi</City>
<ZipCode></ZipCode>
<TimeZone>Africa/Nairobi</TimeZone>
<Latitude>-1.2834</Latitude>
<Longitude>36.8167</Longitude>
<MetroCode>0</MetroCode>
</Response>
root@ubuntu-test01:~#
Troubleshooting
root@com-prod-lxc01:~# lxc-checkconfig Kernel configuration not found at /proc/config.gz; searching... Kernel configuration found at /boot/config-3.16.0-30-generic --- Namespaces --- Namespaces: enabled Utsname namespace: enabled Ipc namespace: enabled Pid namespace: enabled User namespace: enabled Network namespace: enabled Multiple /dev/pts instances: enabled --- Control groups --- Cgroup: enabled Cgroup clone_children flag: enabled Cgroup device: enabled Cgroup sched: enabled Cgroup cpu account: enabled Cgroup memory controller: enabled Cgroup cpuset: enabled --- Misc --- Veth pair device: enabled Macvlan: enabled Vlan: enabled Bridges: enabled Advanced netfilter: enabled CONFIG_NF_NAT_IPV4: enabled CONFIG_NF_NAT_IPV6: enabled CONFIG_IP_NF_TARGET_MASQUERADE: enabled CONFIG_IP6_NF_TARGET_MASQUERADE: enabled CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled --- Checkpoint/Restore --- checkpoint restore: enabled CONFIG_FHANDLE: enabled CONFIG_EVENTFD: enabled CONFIG_EPOLL: enabled CONFIG_UNIX_DIAG: enabled CONFIG_INET_DIAG: enabled CONFIG_PACKET_DIAG: enabled CONFIG_NETLINK_DIAG: enabled File capabilities: enabled