2,009 瀏覽數

CoreOS – Flannel

Container 網路要對外時需要設定 port-mapping 或是將網路設定成不同的模式,這時可以使用 flannel (SDN : Software Define Network) 來讓不同的 Host (CoreOS) 的 Container 之間透過虛擬網路來溝通.要啟動 Flannel 服務還需要 CoreOS 下這幾個服務 Container , Docker , ETCD2 , Fleet 也是啟動的.設定可以參考下面連結.

完整的 CoreOS 設定檔 http://benjr.tw/96511

#cloud-config
hostname: coreos1
ssh_authorized_keys:
  - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5wZYPD/mBs+9O9CrUxdg9kpOus24VrMuNncdt4BRc4iF5npV90HYe5j/y3IG6+2MRbAb2edyf/FUcaJHN/V+i123456yuqyAT2rv9T0eB2+wpmYCUQzqZscJP2uLK8jMhezKWS0l7X5CgJf+d17VooS6CADR9MyTbku3upKp5yEnsCfB+pBLGdrqCUTnGHPfJcLTBIvuMriz/kae0azxcderfbw7YWR8oKdWjKYKlznnBmH6VYFcgv/jSXbRbdZjKNSXIm2xIj6TIIJmo6sWhptcGohi467ODyrzCDioXD1MsYx6ImTMcY5mzL2RDePAW7CM4gWIMaIxDeL5e10SX ben@appledeAir

coreos:
  units:
    - name: etcd2.service
      command: start
    - name: fleet.service
      command: start
    - name: systemd-networkd.service
      command: stop
    - name: 00-eth0.network
      runtime: true
      content: |
        [Match]
        Name=ens32

        [Network]
        Address=172.16.15.21/24
        Gateway=172.16.15.2
        DNS=168.95.1.1
    - name: systemd-networkd.service
      command: start
    - name: flanneld.service
      drop-ins:
       - name: 50-network-config.conf
         content: |
          [Service]
          ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.1.0.0/16", "Backend": {"Type": "vxlan"}}'
      command: start
  etcd2:
    name: "node01"
    discovery: https://discovery.etcd.io/9dd875ca6dd759d67445a681adde3875
    advertise-client-urls: http://172.16.15.21:2379
    initial-advertise-peer-urls: http://172.16.15.21:2380
    listen-client-urls: http://0.0.0.0:2379
    listen-peer-urls: http://172.16.15.21:2380

裡面關於 Flannel 的設定.

coreos:
  units:
    - name: flanneld.service
      drop-ins:
       - name: 50-network-config.conf
         content: |
          [Service]
          ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.1.0.0/16", "Backend": {"Type": "vxlan"}}'
      command: start
  • Network (string): IPv4 network in CIDR format to use for the entire flannel network. This is the only mandatory key.
  • Backend (dictionary): Type of backend to use and specific configurations for that backend. The list of available backends and the keys that can be put into the this dictionary are listed below. Defaults to “udp” backend.
    • vxlan: use in-kernel VXLAN to encapsulate the packets.

其他設定請參考 https://github.com/coreos/flannel

目前我系統上有兩台 CoreOS 並組合成為一個 Cluster.

  1. CoreOS1 (Node1) , IP: 172.16.15.21
  2. CoreOS2 (Node2) , IP: 172.16.15.22

安裝完成的 CoreOS 可以先確認 etcd2 , flannel 的服務是否正常啟動.

core@coreos1 ~ $ systemctl status etcd2
● etcd2.service - etcd2
   Loaded: loaded (/usr/lib/systemd/system/etcd2.service; disabled; vendor preset: 
  Drop-In: /run/systemd/system/etcd2.service.d
           └─20-cloudinit.conf
   Active: active (running) since Tue 2017-03-28 07:37:46 UTC; 8min ago
 Main PID: 966 (etcd2)
    Tasks: 7
   Memory: 22.1M
      CPU: 3.039s
   CGroup: /system.slice/etcd2.service
           └─966 /usr/bin/etcd2
core@coreos1 ~ $ etcdctl cluster-health
member 3036b880d97395a is healthy: got healthy result from http://172.16.15.21:2379
cluster is healthy
core@coreos1 ~ $ etcdctl member list
3036b880d97395a: name=node01 peerURLs=http://172.16.15.21:2380 clientURLs=http://172.16.15.21:2379 isLeader=true
core@coreos1 ~ $ systemctl status flanneld
● flanneld.service - flannel - Network fabric for containers (System Application Co
   Loaded: loaded (/usr/lib/systemd/system/flanneld.service; disabled; vendor prese
  Drop-In: /etc/systemd/system/flanneld.service.d
           └─50-network-config.conf
   Active: active (running) since Tue 2017-03-28 07:38:59 UTC; 8min ago
     Docs: https://github.com/coreos/flannel
  Process: 1022 ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config {"Netw
  Process: 1012 ExecStartPre=/usr/bin/rkt rm --uuid-file=/var/lib/coreos/flannel-wr
  Process: 1010 ExecStartPre=/usr/bin/mkdir --parents /var/lib/coreos /run/flannel 
  Process: 1007 ExecStartPre=/sbin/modprobe ip_tables (code=exited, status=0/SUCCES
 Main PID: 1032 (flanneld)
    Tasks: 8
   Memory: 89.7M
      CPU: 2.354s
   CGroup: /system.slice/flanneld.service
           └─1032 /opt/bin/flanneld --ip-masq=true

安裝完成的 CoreOS 可以在下面幾個檔案查詢到關於 flannel 的設定

core@coreos1 ~ $ cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.1.0.0/16
FLANNEL_SUBNET=10.1.10.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
core@coreos1 ~ $ cat /run/flannel/flannel_docker_opts.env
DOCKER_OPT_BIP="--bip=10.1.10.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
  • –bip – Specify network bridge IP
  • –ip-masq – IP masquerading
  • –mtu – Set the containers network MTU

其他參數請參考 https://docs.docker.com/engine/reference/commandline/dockerd/

會看到網路裝置多了一個 flannel.1

core@coreos1 ~ $ ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.1.10.0  netmask 255.255.0.0  broadcast 0.0.0.0
        inet6 fe80::f414:caff:fe9c:9a74  prefixlen 64  scopeid 0x20<link>
        ether f6:14:ca:9c:9a:74  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0
core@coreos1 ~ $ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.16.15.2     0.0.0.0         UG    0      0        0 ens32
10.1.0.0        0.0.0.0         255.255.0.0     U     0      0        0 flannel.1
172.16.15.0     0.0.0.0         255.255.255.0   U     0      0        0 ens32

透過 Fleet 的方式設定一個 Container (busybox2),使用的是 busybox 的 Image.

core@coreos1 ~ $ vi hello.service
[Unit]
Description=MyApp
Requires=flanneld.service
After=flanneld.service
 
[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill busybox2
ExecStartPre=-/usr/bin/docker rm busybox2
ExecStartPre=/usr/bin/docker pull busybox
ExecStart=/usr/bin/docker run --name busybox2 busybox /bin/sh -c "trap 'exit 0' INT TERM; while true; do echo Hello World; sleep 1; done"
ExecStop=/usr/bin/docker stop busybox2

透過 Load 載入 container ,start 啟動 container

core@coreos1 ~ $ fleetctl load hello.service 
Unit hello.service inactive
Unit hello.service loaded on 18514a4c.../172.16.15.21
core@coreos1 ~ $ fleetctl start hello.service 
Unit hello.service launched on 18514a4c.../172.16.15.21

觀察一下 Container 的網路裝置 IP 為 10.1.10.2

core@coreos1 ~ $ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "6c59a503f4ee35ed712ff6aaa2f7f7187f0e314a3a3c18e444305fcc2e4e98b1",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.1.10.1/24",
                    "Gateway": "10.1.10.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "30c0caba7f44ce08d3e93a8876bfff5430405a26c665e4710d9d22834ff7665b": {
                "Name": "busybox2",
                "EndpointID": "0a5f65a3f5b07e212b2a9a953c6eefdb150178c4e286ac0c876ce08768020774",
                "MacAddress": "02:42:0a:01:0a:02",
                "IPv4Address": "10.1.10.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "false",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1450"
        },
        "Labels": {}
    }
]

Routing table 也不一樣了.

core@coreos1 ~ $ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.16.15.2     0.0.0.0         UG    0      0        0 ens32
10.1.0.0        0.0.0.0         255.255.0.0     U     0      0        0 flannel.1
10.1.10.0       0.0.0.0         255.255.255.0   U     0      0        0 docker0
172.16.15.0     0.0.0.0         255.255.255.0   U     0      0        0 ens32

2 個 node 的 Cluster + flannel 設定皆相同,目前 Flannel 網路狀況如下.

  1. CoreOS1 (Node1) , IP: 172.16.15.21
    FLANNEL_NETWORK=10.1.0.0/16
    FLANNEL_SUBNET=10.1.10.1/24
    Container=10.1.10.2
  2. CoreOS2 (Node2) , IP: 172.16.15.22
    FLANNEL_NETWORK=10.1.0.0/16
    FLANNEL_SUBNET=10.1.54.1/24
    Container=10.1.54.2

當 Node1 的 Container 要與 Node2 的 Container 溝通時 (Node1 與 Node2 的 ens32 是實體相連接在同一台 switch 的.)

Node1: [ Container (IP:10.1.10.2) -> Docker0 (10.1.10.0/24) -> flannel.1 (10.1.0.0/16) -> ens32 (IP:172.16.15.21) ]

Node2: [ ens32 (IP:172.16.15.22) -> flannel.1 (10.1.0.0/16) -> Docker0 (10.1.54.0/24) -> Container (IP:10.1.54.2) ]

One Reply to “CoreOS – Flannel”

  1. 自動參照通知: CoreOS 設定檔 – Benjr.tw

發表迴響