简介

Logstash 是开源的服务器端数据处理管道，能够同时 从多个来源采集数据、转换数据，然后将数据发送到您最喜欢的 “存储库” 中。

输入

数据往往以各种各样的形式，或分散或集中地存在于很多系统中。Logstash 支持各种输入选择 ，可以在同一时间从众多常用来源捕捉事件。能够以连续的流式传输方式，轻松地从您的日志、指标、Web 应用、数据存储以及各种 AWS 服务采集数据。

输入插件：https://www.elastic.co/guide/en/logstash/current/input-plugins.html

过滤

数据从源传输到存储库的过程中，Logstash 过滤器能够解析各个事件，识别已命名的字段以构建结构，并将它们转换成通用格式，以便更轻松、更快速地分析和实现商业价值。

Logstash 能够动态地转换和解析数据，不受格式或复杂度的影响：

• 利用 Grok 从非结构化数据中派生出结构
• 从 IP 地址破译出地理坐标
• 将 PII 数据匿名化，完全排除敏感字段
• 整体处理不受数据源、格式或架构的影响

输出

Elasticsearch 是我们的首选输出方向，能够为我们的搜索和分析带来无限可能，但它并非唯一选择。

Logstash 提供众多输出选择，您可以将数据发送到您要指定的地方，并且能够灵活地解锁众多下游用例。

输出插件：https://www.elastic.co/guide/en/logstash/current/output-plugins.html

内容来源：https://www.elastic.co/cn/products/logstash

下载安装

下载、安装logstash:

#下载，logstash5及以上版本需要jdk8wget https://download.elastic.co/logstash/logstash/logstash-2.4.1.tar.gz#解压tar -zxvf logstash-2.4.1.tar.gz#To test your Logstash installation, run the most basic Logstash pipeline#测试logstash环境，运行如下demo(input {stdin{}}:接收终端输入；output {stdout{}}：输出到终端)，出现Pipeline main started为正常cd logstash-2.4.1./bin/logstash -e 'input {stdin{}} output {stdout{}}'#-----------------------------------start-----------------------------------Settings: Default pipeline workers: 24Pipeline main started#------------------------------------end------------------------------------#The -e flag enables you to specify a configuration directly from the command line. Specifying configurations at the command line lets you quickly test configurations without having to edit a file between iterations. The pipeline in the example takes input from the standard input, stdin, and moves that input to the standard output, stdout, in a structured format.#测试，输入hello world,然后回车#出现如下信息#-----------------------------------start-----------------------------------2018-01-04T02:44:41.024Z hostname hello world#------------------------------------end------------------------------------

下载、安装filebeat:

#下载wget https://download.elastic.co/beats/filebeat/filebeat-1.3.1-x86_64.rpm#安装rpm -ivh filebeat-1.3.1-x86_64.rpm#配置logstash接收logvim /etc/filebeat/filebeat.yml#-----------------------------------start-----------------------------------filebeat:  # List of prospectors to fetch data.  prospectors:    # Each - is a prospector. Below are the prospector specific configurations    -      # Paths that should be crawled and fetched. Glob based paths.      # To fetch all ".log" files from a specific level of subdirectories      # /var/log/*/*.log can be used.      # For each file found under this path, a harvester is started.      # Make sure not file is defined twice as this can lead to unexpected behaviour.      paths:      #  - /var/log/*.log      # - c:\programdata\elasticsearch\logs\*      - /var/log/sdk-report8281/sdk-report.log      - /var/log/sdk-report8282/sdk-report.log      # Configure the file encoding for reading files with international characters      # following the W3C recommendation for HTML5 (http://www.w3.org/TR/encoding).      # Some sample encodings:      #   plain, utf-8, utf-16be-bom, utf-16be, utf-16le, big5, gb18030, gbk,      #    hz-gb-2312, euc-kr, euc-jp, iso-2022-jp, shift-jis, ...      #encoding: plain      # Type of the files. Based on this the way the file is read is decided.      # The different types cannot be mixed in one prospector      #      # Possible options are:      # * log: Reads every line of the log file (default)      # * stdin: Reads the standard in      input_type: logoutput:  logstash:    # The Logstash hosts    hosts: ["10.135.29.215:5044", "10.135.29.216:5044"]    # Number of workers per Logstash host.    #worker: 1    # Optional load balance the events between the Logstash hosts    loadbalance: true#------------------------------------end------------------------------------#For rpm and deb, you’ll find the configuration file at /etc/filebeat/filebeat.yml. Under Docker, it’s located at /usr/share/filebeat/filebeat.yml. For mac and win, look in the archive that you just extracted. There’s also a full example configuration file called filebeat.reference.yml that shows all non-deprecated options.#启动filebeatservice filebeat start

注：filebeat为logstash提供输入

使用

配置logstash读取file文件及接收filebeat上传数据

创建file_pipeline.conf文件

#vim file_pipeline.conf#-----------------------------------start-----------------------------------input{        beats {                port => "5044"        }#       file {#               path => "/var/log/*.log"#               start_position => beginning#               ignore_older => 0#       }}filter{        grok {                match => { "message" => "(?
    
     .*).[0-9]{3} %{WORD: level} %{IP: clientip} \[(?
     
      .*)\] %{JAVACLASS:classname}\|(?
      
       .*)" }        }#       geoip {#               source => "clientip"#       }}output{        stdout { codec => rubydebug }}#------------------------------------end------------------------------------#Logstash ships with about 120 patterns by default. You can find them here: https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns
      
     
    

启动logstash

#测试配置是否正确./bin/logstash -f file_pipeline.conf --configtest#--configtest测试配置文件是否正确，新版为--config.test_and_exit#新版支持--config.reload.automatic参数，修改配置无需重启#启动./bin/logstash -f file_pipeline.conf