一看必會系列:k8s 練習24 監控方案之:heapster 1.6.0+influxdb+grafana詳解

來源:本站原創 Kubernetes 超過238 views圍觀 0條評論

最新版本  heapster 1.6.0
https://github.com/kubernetes-retired/heapster/releases

HOST* 為master
192.168.10.73 HOST1
192.168.10.73 host1
192.168.10.74 HOST2
192.168.10.74 host2
192.168.10.72 HOST0
192.168.10.72 host0
192.168.10.69 k8s-node1

以下所有操作在 任一master上執行

下載
cd /data/soft

wget https://github.com/kubernetes-retired/heapster/archive/v1.6.0-beta.1.tar.gz

tar -zxvf v1.6.0-beta.1.tar.gz

cd /data/soft/heapster-1.6.0-beta.1/deploy/kube-config/influxdb

常規操作,列出鏡像,先下載
參考  k8s 練習6 利用阿里云下載google k8s鏡像
[[email protected] influxdb]# grep image *.yaml
grafana.yaml:        image: k8s.gcr.io/heapster-grafana-amd64:v5.0.4
heapster.yaml:        image: k8s.gcr.io/heapster-amd64:v1.5.4
heapster.yaml:        imagePullPolicy: IfNotPresent
influxdb.yaml:        image: k8s.gcr.io/heapster-influxdb-amd64:v1.5.2

以下4個之后都要 create
[[email protected] influxdb]# ll |grep -v bak
total 28
-rw-r–r–. 1 root root 2311 Apr 18 17:21 grafana.yaml
-rw-r–r–. 1 root root  381 Apr 18 18:25 heapster_user.yaml
-rw-r–r–. 1 root root 1208 Apr 18 19:20 heapster.yaml
-rw-r–r–. 1 root root 1004 Apr 18 17:17 influxdb.yaml

部分配置需要進行修改
heapster_user.yaml  這個內容在最后

heapster.yaml 將27行改成
28         – –source=kubernetes:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true

grafana.yaml  配置增加 nodePort 下面的對應行號
               
62 spec:
63   # In a production setup, we recommend accessing Grafana through an external Loadbalancer
64   # or through a public IP.
65   # type: LoadBalancer
66   # You could also use NodePort to expose the service at a randomly-generated port
67   # add
68   type: NodePort
69   ports:
70   – port: 80
71     targetPort: 3000
72     #add
73     nodePort: 30004

#按順序創建
kubectl create -f heapster_user.yaml
kubectl create -f influxdb.yaml
kubectl create -f heapster.yaml
kubectl create -f grafana.yaml

#查看狀態
[[email protected] influxdb]#  kubectl get svc -n kube-system
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
heapster               ClusterIP   10.102.35.106    <none>        80/TCP                   62m
kube-dns               ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP   29h
kubernetes-dashboard   NodePort    10.103.100.103   <none>        443:30001/TCP            26h
monitoring-grafana     NodePort    10.97.150.187    <none>        80:30004/TCP             3h1m
monitoring-influxdb    NodePort    10.100.152.72    <none>        8086:30003/TCP           3h5m

[[email protected] influxdb]# kubectl get pods –namespace=kube-system |tail
kubernetes-dashboard-5694f87d87-dht4w      1/1     Running   0          26h
monitoring-grafana-658976d65f-95swp        1/1     Running   0          3h2m
monitoring-influxdb-866db5f944-mt42h       1/1     Running   0          3h6m
[[email protected] influxdb]#

————grafana配置
下載兩個模版
https://grafana.com/dashboards/3649
https://grafana.com/dashboards/3646

訪問地址
http://192.168.10.69:30004/plugins

點home–>import dashboard–> upload剛下載的json 即可

具體看截圖

————-報錯1

heapster 看不到圖相,查日志發現

kubectl log pod.heapster-5d4bf58946-gwk5d  -n kube-system
E0418 10:06:48.946602       1 reflector.go:190] k8s.io/heapster/metrics/heapster.go:328: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list resource "pods" in API group "" at the cluster scope
E0418 10:06:48.947773       1 reflector.go:190] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list resource "namespaces" in API group "" at the cluster scope
E0418 10:06:49.945074       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list resource "nodes" in API group "" at the cluster scope
E0418 10:06:49.945835       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list resource "nodes" in API group "" at the cluster scope

分析,基本說的是沒有權限訪問,所以需要配置一個管理用戶

配置 集群管理帳戶
vim  heapster_user.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: heapster
  name: heapster
  namespace: kube-system

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: heapster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
– kind: ServiceAccount
  name: heapster
  namespace: kube-system

應用帳戶
[[email protected] influxdb]# kubectl apply -f heapster_user.yaml
Warning: kubectl apply should be used on resource created by either kubectl create –save-config or kubectl apply
serviceaccount/heapster configured
clusterrolebinding.rbac.authorization.k8s.io/heapster created

kubectl delete -f heapster.yaml

kubectl apply -f heapster.yaml

解決
[[email protected] influxdb]# kubectl get pod -n kube-system |grep heap
heapster-5d4bf58946-wblhb                  1/1     Running   0          2m32s
[[email protected] influxdb]#

查日志原故障解決

[[email protected] influxdb]# kubectl log heapster-5d4bf58946-wblhb  -n kube-system
log is DEPRECATED and will be removed in a future version. Use logs instead.
I0418 10:27:04.966389       1 heapster.go:78] /heapster –source=kubernetes:https://kubernetes.default –sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
I0418 10:27:04.966455       1 heapster.go:79] Heapster version v1.5.4
I0418 10:27:04.966673       1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0418 10:27:04.966693       1 configs.go:62] Using kubelet port 10255
I0418 10:27:04.980760       1 influxdb.go:312] created influxdb sink with options: host:monitoring-influxdb.kube-system.svc:8086 user:root db:k8s
I0418 10:27:04.980793       1 heapster.go:202] Starting with InfluxDB Sink
I0418 10:27:04.980799       1 heapster.go:202] Starting with Metric Sink
I0418 10:27:04.989898       1 heapster.go:112] Starting heapster on port 8082
E0418 10:28:05.006213       1 manager.go:101] Error in scraping containers from kubelet:192.168.10.72:10255: failed to get all container stats from Kubelet URL "http://192.168.10.72:10255/stats/container/": Post http://192.168.10.72:10255/stats/container/: dial tcp 192.168.10.72:10255: getsockopt: connection refused
E0418 10:28:05.011846       1 manager.go:101] Error in scraping containers from kubelet:192.168.10.73:10255: failed to get all container stats from Kubelet URL "http://192.168.10.73:10255/stats/container/": Post http://192.168.10.73:10255/stats/container/: dial tcp 192.168.10.73:10255: getsockopt: connection refused
E0418 10:28:05.021833       1 manager.go:101] Error in scraping containers from kubelet:192.168.10.74:10255: failed to get all container stats from Kubelet URL "http://192.168.10.74:10255/stats/container/": Post http://192.168.10.74:10255/stats/container/: dial tcp 192.168.10.74:10255: getsockopt: connection refused

—————-報錯2

[[email protected] influxdb]# kubectl log heapster-5d4bf58946-wblhb  -n kube-system
log is DEPRECATED and will be removed in a future version. Use logs instead.

E0418 10:28:05.006213       1 manager.go:101] Error in scraping containers from kubelet:192.168.10.72:10255: failed to get all container stats from Kubelet URL "http://192.168.10.72:10255/stats/container/": Post http://192.168.10.72:10255/stats/container/: dial tcp 192.168.10.72:10255: getsockopt: connection refused
E0418 10:28:05.011846       1 manager.go:101] Error in scraping containers from kubelet:192.168.10.73:10255: failed to get all container stats from Kubelet URL "http://192.168.10.73:10255/stats/container/": Post http://192.168.10.73:10255/stats/container/: dial tcp 192.168.10.73:10255: getsockopt: connection refused
E0418 10:28:05.021833       1 manager.go:101] Error in scraping containers from kubelet:192.168.10.74:10255: failed to get all container stats from Kubelet URL "http://192.168.10.74:10255/stats/container/": Post http://192.168.10.74:10255/stats/container/: dial tcp 192.168.10.74:10255: getsockopt: connection refused

解決

vim heapster.yaml
        #將這條修改為下面一條- –source=kubernetes:https://kubernetes.default  
        – –source=kubernetes:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true
       
kubectl delete -f heapster.yaml
kubectl apply -f heapster.yaml
kubectl get pod -n kube-system |grep heap
kubectl log heapster-5d9575b66b-jv9t5  -n kube-system

驗證
[[email protected] influxdb]# kubectl log heapster-5d9575b66b-jv9t5  -n kube-system
log is DEPRECATED and will be removed in a future version. Use logs instead.
I0418 11:20:25.379427       1 heapster.go:78] /heapster –source=kubernetes:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true –sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
I0418 11:20:25.379473       1 heapster.go:79] Heapster version v1.5.4
I0418 11:20:25.379710       1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0418 11:20:25.379729       1 configs.go:62] Using kubelet port 10250
I0418 11:20:25.392192       1 influxdb.go:312] created influxdb sink with options: host:monitoring-influxdb.kube-system.svc:8086 user:root db:k8s
I0418 11:20:25.392219       1 heapster.go:202] Starting with InfluxDB Sink
I0418 11:20:25.392226       1 heapster.go:202] Starting with Metric Sink
I0418 11:20:25.399904       1 heapster.go:112] Starting heapster on port 8082
I0418 11:21:05.137427       1 influxdb.go:274] Created database "k8s" on influxDB server at "monitoring-influxdb.kube-system.svc:8086"

 

參考
https://www.cnblogs.com/justmine/p/8723467.html
https://blog.csdn.net/qq_24513043/article/details/82460759

文章出自:CCIE那點事 http://www.qdxgqk.live/ 版權所有。本站文章除注明出處外,皆為作者原創文章,可自由引用,但請注明來源。 禁止全文轉載。
本文鏈接:http://www.qdxgqk.live/?p=4145轉載請注明轉自CCIE那點事
如果喜歡:點此訂閱本站
  • 相關文章
  • 為您推薦
  • 各種觀點
?
暫時還木有人評論,坐等沙發!
發表評論

您必須 [ 登錄 ] 才能發表留言!

?
?
萌宠夺宝游戏