Container 網路要對外時需要設定 port-mapping 或是將網路設定成不同的模式,這時可以使用 flannel (SDN : Software Define Network) 來讓不同的 Host (CoreOS) 的 Container 之間透過虛擬網路來溝通.要啟動 Flannel 服務還需要 CoreOS 下這幾個服務 Container , Docker , ETCD2 , Fleet 也是啟動的.設定可以參考下面連結.
- Container 與 Docker 設定與使用方式 – https://benjr.tw/96566
- etcd2 設定與使用方式 – https://benjr.tw/96404
- 新增移除 etcd2 Node – https://benjr.tw/96449
- fleet 設定與使用方式 – https://benjr.tw/96502
完整的 CoreOS 設定檔 https://benjr.tw/96511
#cloud-config
hostname: coreos1
ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5wZYPD/mBs+9O9CrUxdg9kpOus24VrMuNncdt4BRc4iF5npV90HYe5j/y3IG6+2MRbAb2edyf/FUcaJHN/V+i123456yuqyAT2rv9T0eB2+wpmYCUQzqZscJP2uLK8jMhezKWS0l7X5CgJf+d17VooS6CADR9MyTbku3upKp5yEnsCfB+pBLGdrqCUTnGHPfJcLTBIvuMriz/kae0azxcderfbw7YWR8oKdWjKYKlznnBmH6VYFcgv/jSXbRbdZjKNSXIm2xIj6TIIJmo6sWhptcGohi467ODyrzCDioXD1MsYx6ImTMcY5mzL2RDePAW7CM4gWIMaIxDeL5e10SX ben@appledeAir
coreos:
units:
- name: etcd2.service
command: start
- name: fleet.service
command: start
- name: systemd-networkd.service
command: stop
- name: 00-eth0.network
runtime: true
content: |
[Match]
Name=ens32
[Network]
Address=172.16.15.21/24
Gateway=172.16.15.2
DNS=168.95.1.1
- name: systemd-networkd.service
command: start
- name: flanneld.service
drop-ins:
- name: 50-network-config.conf
content: |
[Service]
ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.1.0.0/16", "Backend": {"Type": "vxlan"}}'
command: start
etcd2:
name: "node01"
discovery: https://discovery.etcd.io/9dd875ca6dd759d67445a681adde3875
advertise-client-urls: http://172.16.15.21:2379
initial-advertise-peer-urls: http://172.16.15.21:2380
listen-client-urls: http://0.0.0.0:2379
listen-peer-urls: http://172.16.15.21:2380
裡面關於 Flannel 的設定.
coreos:
units:
- name: flanneld.service
drop-ins:
- name: 50-network-config.conf
content: |
[Service]
ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.1.0.0/16", "Backend": {"Type": "vxlan"}}'
command: start
- Network (string): IPv4 network in CIDR format to use for the entire flannel network. This is the only mandatory key.
- Backend (dictionary): Type of backend to use and specific configurations for that backend. The list of available backends and the keys that can be put into the this dictionary are listed below. Defaults to “udp” backend.
- vxlan: use in-kernel VXLAN to encapsulate the packets.
其他設定請參考 https://github.com/coreos/flannel
目前我系統上有兩台 CoreOS 並組合成為一個 Cluster.
- CoreOS1 (Node1) , IP: 172.16.15.21
- CoreOS2 (Node2) , IP: 172.16.15.22
安裝完成的 CoreOS 可以先確認 etcd2 , flannel 的服務是否正常啟動.
core@coreos1 ~ $ systemctl status etcd2
● etcd2.service - etcd2
Loaded: loaded (/usr/lib/systemd/system/etcd2.service; disabled; vendor preset:
Drop-In: /run/systemd/system/etcd2.service.d
└─20-cloudinit.conf
Active: active (running) since Tue 2017-03-28 07:37:46 UTC; 8min ago
Main PID: 966 (etcd2)
Tasks: 7
Memory: 22.1M
CPU: 3.039s
CGroup: /system.slice/etcd2.service
└─966 /usr/bin/etcd2
core@coreos1 ~ $ etcdctl cluster-health member 3036b880d97395a is healthy: got healthy result from http://172.16.15.21:2379 cluster is healthy
core@coreos1 ~ $ etcdctl member list 3036b880d97395a: name=node01 peerURLs=http://172.16.15.21:2380 clientURLs=http://172.16.15.21:2379 isLeader=true
core@coreos1 ~ $ systemctl status flanneld
● flanneld.service - flannel - Network fabric for containers (System Application Co
Loaded: loaded (/usr/lib/systemd/system/flanneld.service; disabled; vendor prese
Drop-In: /etc/systemd/system/flanneld.service.d
└─50-network-config.conf
Active: active (running) since Tue 2017-03-28 07:38:59 UTC; 8min ago
Docs: https://github.com/coreos/flannel
Process: 1022 ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config {"Netw
Process: 1012 ExecStartPre=/usr/bin/rkt rm --uuid-file=/var/lib/coreos/flannel-wr
Process: 1010 ExecStartPre=/usr/bin/mkdir --parents /var/lib/coreos /run/flannel
Process: 1007 ExecStartPre=/sbin/modprobe ip_tables (code=exited, status=0/SUCCES
Main PID: 1032 (flanneld)
Tasks: 8
Memory: 89.7M
CPU: 2.354s
CGroup: /system.slice/flanneld.service
└─1032 /opt/bin/flanneld --ip-masq=true
安裝完成的 CoreOS 可以在下面幾個檔案查詢到關於 flannel 的設定
core@coreos1 ~ $ cat /run/flannel/subnet.env FLANNEL_NETWORK=10.1.0.0/16 FLANNEL_SUBNET=10.1.10.1/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=true
core@coreos1 ~ $ cat /run/flannel/flannel_docker_opts.env DOCKER_OPT_BIP="--bip=10.1.10.1/24" DOCKER_OPT_IPMASQ="--ip-masq=false" DOCKER_OPT_MTU="--mtu=1450"
- –bip – Specify network bridge IP
- –ip-masq – IP masquerading
- –mtu – Set the containers network MTU
其他參數請參考 https://docs.docker.com/engine/reference/commandline/dockerd/
會看到網路裝置多了一個 flannel.1
core@coreos1 ~ $ ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.1.10.0 netmask 255.255.0.0 broadcast 0.0.0.0
inet6 fe80::f414:caff:fe9c:9a74 prefixlen 64 scopeid 0x20<link>
ether f6:14:ca:9c:9a:74 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 8 overruns 0 carrier 0 collisions 0
core@coreos1 ~ $ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.16.15.2 0.0.0.0 UG 0 0 0 ens32 10.1.0.0 0.0.0.0 255.255.0.0 U 0 0 0 flannel.1 172.16.15.0 0.0.0.0 255.255.255.0 U 0 0 0 ens32
透過 Fleet 的方式設定一個 Container (busybox2),使用的是 busybox 的 Image.
core@coreos1 ~ $ vi hello.service [Unit] Description=MyApp Requires=flanneld.service After=flanneld.service [Service] TimeoutStartSec=0 ExecStartPre=-/usr/bin/docker kill busybox2 ExecStartPre=-/usr/bin/docker rm busybox2 ExecStartPre=/usr/bin/docker pull busybox ExecStart=/usr/bin/docker run --name busybox2 busybox /bin/sh -c "trap 'exit 0' INT TERM; while true; do echo Hello World; sleep 1; done" ExecStop=/usr/bin/docker stop busybox2
透過 Load 載入 container ,start 啟動 container
core@coreos1 ~ $ fleetctl load hello.service Unit hello.service inactive Unit hello.service loaded on 18514a4c.../172.16.15.21
core@coreos1 ~ $ fleetctl start hello.service Unit hello.service launched on 18514a4c.../172.16.15.21
觀察一下 Container 的網路裝置 IP 為 10.1.10.2
core@coreos1 ~ $ docker network inspect bridge
[
{
"Name": "bridge",
"Id": "6c59a503f4ee35ed712ff6aaa2f7f7187f0e314a3a3c18e444305fcc2e4e98b1",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.1.10.1/24",
"Gateway": "10.1.10.1"
}
]
},
"Internal": false,
"Containers": {
"30c0caba7f44ce08d3e93a8876bfff5430405a26c665e4710d9d22834ff7665b": {
"Name": "busybox2",
"EndpointID": "0a5f65a3f5b07e212b2a9a953c6eefdb150178c4e286ac0c876ce08768020774",
"MacAddress": "02:42:0a:01:0a:02",
"IPv4Address": "10.1.10.2/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "false",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1450"
},
"Labels": {}
}
]
Routing table 也不一樣了.
core@coreos1 ~ $ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.16.15.2 0.0.0.0 UG 0 0 0 ens32 10.1.0.0 0.0.0.0 255.255.0.0 U 0 0 0 flannel.1 10.1.10.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0 172.16.15.0 0.0.0.0 255.255.255.0 U 0 0 0 ens32
2 個 node 的 Cluster + flannel 設定皆相同,目前 Flannel 網路狀況如下.
- CoreOS1 (Node1) , IP: 172.16.15.21
FLANNEL_NETWORK=10.1.0.0/16
FLANNEL_SUBNET=10.1.10.1/24
Container=10.1.10.2 - CoreOS2 (Node2) , IP: 172.16.15.22
FLANNEL_NETWORK=10.1.0.0/16
FLANNEL_SUBNET=10.1.54.1/24
Container=10.1.54.2
當 Node1 的 Container 要與 Node2 的 Container 溝通時 (Node1 與 Node2 的 ens32 是實體相連接在同一台 switch 的.)
Node1: [ Container (IP:10.1.10.2) -> Docker0 (10.1.10.0/24) -> flannel.1 (10.1.0.0/16) -> ens32 (IP:172.16.15.21) ]
Node2: [ ens32 (IP:172.16.15.22) -> flannel.1 (10.1.0.0/16) -> Docker0 (10.1.54.0/24) -> Container (IP:10.1.54.2) ]
沒有解決問題,試試搜尋本站其他內容
One thought on “CoreOS – Flannel”