ELK的安裝配置

來源:本站原創 Linux 超過3,392 views圍觀 0條評論

官方文檔
https://www.elastic.co/guide/en/kibana/current/rpm.html

Logstash
1、安裝jdk

Logstash的運行依賴于Java運行環境。
# yum -y install java-1.8.0
[[email protected] ~]# java -v
Unrecognized option: -v
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
[[email protected] ~]# java -version
openjdk version “1.8.0_121”
OpenJDK Runtime Environment (build 1.8.0_121-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
[[email protected] ~]#

運行報錯
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one,
then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N

步驟:

1 準備工作
2 安裝配置 Elasticsearch
3 安裝配置 Kibana
4 安裝配置 Filebeat
5 安裝配置 Logstash (可選)
6 安裝配置 Nginx (可選)
7 單獨配置一個客戶端

直接YUM.安裝
配置 ES 的 yum 源  所有都可以yum.使用這個源

導入簽名:
導入elasticsearch PGP key
rpm –import https://artifacts.elastic.co/GPG-KEY-elasticsearch
設置 yum 源
vim /etc/yum.repo.d/elk.repo
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

1 yum intall -y e/l/k/f
2 systemctl enable e/1/k/f
3 開防火墻
4 修改配置

Elasticsearch 配置文件在 /etc/elasticsearch/elasticsearch.yml ,
如果不使用 Logstash 或者 Logstash 與 Elasticsearch 不在同一服務器,
那么需要使 Elasticsearch 監聽到指定的 IP 地址和端口,例如修改 elasticsearch.yml 中的下邊兩行:
network.host: 0.0.0.0   –允許所有IP訪問
http.port: 9200   –端口
檢查安裝,執行
[[email protected] ~]# curl 192.168.142.135 9200   –elasticsearch
<script>var hashRoute = ‘/app/kibana’;
var defaultRoute = ‘/app/kibana’;

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
}</script>curl: (7) Failed to connect to 0.0.35.240: Invalid argument

編輯配置文件 /etc/kibana/kibana.yml ,修改下列兩行
server.port 5601   –這個配置有錯自己排查
server.host 0.0.0.0  –這個配置有錯自己排查

[[email protected] ~]# curl 192.168.142.135 5601 —kibana
<script>var hashRoute = ‘/app/kibana’;
var defaultRoute = ‘/app/kibana’;

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
}</script>curl: (7) Failed to connect to 0.0.21.225: Invalid argument
[[email protected] ~]#

配置 Logstash 的輸入和輸出,新建輸入輸出配置文件:
你會發現logstash 5.x里面 logstash調試命令沒有了.坑爹

find / -name logstash -type f 查一下程序放在這里
/usr/share/logstash/bin/logstash

測試一下
/usr/share/logstash/bin/logstash -e ‘input { stdin{} } output { stdout{ codec => rubydebug} }’
結果
{
    “@timestamp” => 2017-04-21T04:51:03.336Z,
      “@version” => “1”,
          “host” => “elk02”,
       “message” => “”
}

——-報錯處理——-
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using –path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
c[ERROR] 2017-09-26 19:06:06.370 [LogStash::Runner] elasticsearch – Unknown setting ‘host’ for elasticsearch
[ERROR] 2017-09-26 19:06:06.381 [LogStash::Runner] agent – Cannot create pipeline {:reason=>”Something is wrong with your configuration.”}

agent配置錯誤修改一下.本例是host字段寫錯了.

[2017-09-26T20:09:56,369][INFO ][org.logstash.beats.BeatsHandler] Exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: -12, from: /172.30.100.139:48966
[2017-09-26T20:09:56,374][INFO ][org.logstash.beats.BeatsHandler] Exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: -1, from: /172.30.100.139:48966

Invalid Frame Type—-無效類型,本例是升級filebeat后解決.

——報錯處理——-

 

我們可以使用curl命令發送請求來查看ES是否接收到了數據:
curl ‘http://localhost:9200/_search?pretty’

使用配置文件

使用-e參數 在命令行中指定配置是很常用的方式,不過如果需要配置更多設置則需要很長的內容。這種情況,
我們首先創建一個簡單的配置文件,并且指定logstash使用這個配置文件。如我們創建一個文件名是”logstash-simple.conf”
的配置文件并且保存在和Logstash相同的目錄中。內容如下:

input { stdin { } }
output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}
接下來,執行命令:
bin/logstash -f logstash-simple.conf

測試一下寫入es
[[email protected] ~]# /usr/share/logstash/bin/logstash -e ‘input { stdin{} } output { elasticsearch { hosts => [“192.168.142.137:9200”] index => “logstash-%{+YYYY.MM.dd}” } }’
在ES上可以看到
@timestamp:April 21st 2017, 13:15:59.151 @version:1 host:elk02 message: _id:AVuO8DvRQjoNvLqUPbr9 _type:logs _index:logstash-2017.04.21 _score: –

vi /etc/logstash/conf.d/first-logstash.conf
文件內容如下:
輸入:設置監聽 5044 端口,接收 beats 的輸入數據
輸出:將數據輸出到 Elasticsearch
input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => “localhost:9200”
    manage_template => false
    index => “%{[@metadata][beat]}-%{+YYYY.MM.dd}”
    document_type => “%{[@metadata][type]}”
  }
}
測試
[[email protected] ~]# curl 192.168.142.135 5044 –filebeat
<script>var hashRoute = ‘/app/kibana’;
var defaultRoute = ‘/app/kibana’;

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
}</script>curl: (7) Failed to connect to 0.0.19.180: Invalid argument
[[email protected] ~]#

修改 Filebeat 將日志發送給 Logstash

Filebeat 可以將日志輸入到 Elasticsearh,如剛才的配置。它也可以將日志輸入給 Logstash,由 Logstash 處理日志,
再將處理后的日志數據輸入到 Elasticsearch。下邊配置 Filebeat 將日志 輸入到 Logstash。

編輯 Filebeat 配置文件:

vi /etc/filebeat/filebeat.yml
注釋掉 Elasticsearch output 的相關設置:

#—————————– Logstash output ——————————–
output.logstash:
  # The Logstash hosts
  hosts: [“localhost:5044”]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: [“/etc/pki/root/ca.pem”]

  # Certificate for SSL client authentication
  #ssl.certificate: “/etc/pki/client/cert.pem”

  # Client Certificate Key
  #ssl.key: “/etc/pki/client/cert.key”

3 ysstemctl restart e/1/k/f

配置完你會發現kibana上沒辦法看到日志

index默認顯示這個因為是默認用logstash傳送日志,但配置是用filebeat傳送日志,所以這里改一下

logstash-*

——報錯信息—
Unable to fetch mapping. Do you have indices matching the pattern?
Patterns allow you to define dynamic index names using * as a wildcard. Example: logstash-*
——報錯處理——-
index 改成
filebeat-*即可
Create a new default index ‘filebeat-*’ and click on the ‘Create’ button.
——-完成——-

排錯
問題1
5601端口是起來的 但只能本地telnet 防火墻是開的

netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
 
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      11175/nginx: master
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      972/sshd           
tcp        0      0 127.0.0.1:5601          0.0.0.0:*               LISTEN      91600/node         

tcp6       0      0 :::9200                 :::*                    LISTEN      85301/java         
tcp6       0      0 :::5044                 :::*                    LISTEN      60427/java         
tcp6       0      0 :::9300                 :::*                    LISTEN      85301/java         

[[email protected] ~]# telnet 127.0.0.1 5601
Trying 127.0.0.1…
Connected to 127.0.0.1.

問題1  kibana提示
Index Patterns: Please specify a default index pattern
且不能create index

問題2
客戶端報錯
blish.write_bytes=273
2017-04-19T06:48:43-04:00 ERR Failed to publish events caused by: read tcp 192.168.142.134:33392->192.168.142.137:5044: i/o timeout
2017-04-19T06:48:43-04:00 INFO Error publishing events (retrying): read tcp 192.168.142.134:33392->192.168.142.137:5044: i/o timeout

官方文檔
http://udn.yyuap.com/doc/logstash-best-practice-cn/codec/json.html
https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-centos-7
https://www.howtoforge.com/tutorial/how-to-install-elastic-stack-on-centos-7/#step-install-and-configure-elasticsearch

文章出自:CCIE那點事 http://www.qdxgqk.live/ 版權所有。本站文章除注明出處外,皆為作者原創文章,可自由引用,但請注明來源。 禁止全文轉載。
本文標題:ELK的安裝配置
本文鏈接:http://www.qdxgqk.live/?p=3571轉載請注明轉自CCIE那點事
如果喜歡:點此訂閱本站
?
?
萌宠夺宝游戏