Graylog2+Elasticsearch+Nxlog

笔者刚进入一家新公司,发现他们用的graylog,心里面一顿鄙视,这是什么工具,为什么不用ELK呢,用这么low的工具,后来发现我错了。

日志管理工具:收集,解析,可视化

  • Elasticsearch - 一个基于Lucene的文档存储,主要用于日志索引、存储和分析。

  • Logstash -管理事件和日志的工具

  • Kibana - 可视化日志和时间戳数据

  • Graylog2 -具有报警选项的可插入日志和事件分析服务器

  • Nxlog-跨平台,模块化,具备日志缓冲与流量控制、定时作业、内置配置语言的日志采集神器



Graylog与ELK方案的对比

ELK:Elasticsearch+Logstash+Kibana

Graylog:Elasticsearch+Nxlog+Graylog-server(集成graylog-web)


Graylog架构图:


                最小化架构图:

                Graylog集群架构图:



ELK架构图:




简要说两句:商业日志工具Splunk,号称日志界的Google。还有Nxlog,它们有多牛逼我就不再赘述了。而Graylog简直就是开源版的Splunk。

Graylog搜搜结果高亮,支持自定义脚本,支持邮件报警等

此次,我们进行最小化安装,集群方案的部署后面会陆陆续续更新:

        安装组件:

                Mongodb

                Elasticsearch

                Graylog-server(集成了Graylog-web)

                Graylog Collector Sidecar(旧版本是Graylog Collector,已经废弃了 )

安装环境:

        Centos7.3+Graylog2.3+elasticsearch2.4.5+Nxlog2.9+Sidecar-0.1.3

        主机IP:192.168.55.33   此次配置服务端和客户端都在一台主机上


第一部分:    Server端的部署


MongoDB:

                1:安装mongodb的yum源

                        vim  /etc/yum.repos.d/mongodb-org-3.2.repo   

                     [mongodb-org-3.2]

                        name=MongoDB Repository

                        baseurl=https://repo.mongodb.org/yum/

                        redhat/$releasever/mongodb-org/3.2/x86_64/

                        gpgcheck=1

                        enabled=1

                        gpgkey=https://www.mongodb.org/static/pgp/server-3.2.asc

    

                2:安装mongodb

                          yum install mongodb-org


                3:添加系统服务及启动                    

                         chkconfig --add mongod

                         systemctl daemon-reload

                         /sbin/chkconfig  mongod on

                         systemctl start mongod.service

                    注意:这里没有进行mongodb的相关配置包括graylog连接

                    的配置,graylog启动时会自行创建相关数据


Elasticsearch:    

                  1:首先导入Elastic GPG密钥

                        rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

    

                  2:添加yum源

                        vim  /etc/yum.repos.d/elasticsearch.repo

                      [elasticsearch-2.x]

                         name=Elasticsearch repository for 2.x packages

                         baseurl=https://packages.elastic.co/elasticsearch/2.x/centos

                         gpgcheck=1

                         gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch

                         enabled=1   

                  3:安装elasticsearch

                         yum install elasticsearch


                  4:配置elasticsearch,修改以下几个地方,elasticsearch详细配置自行谷歌

           vim  /etc/elasticsearch/elasticsearch.yml

           cluster.name: graylog2    

               elasticsearch集群名称,若有多个集群,可根据此属性区分。

           node.name: node-142      

                集群节点名称,elasticsearch启动时会自动创建,也可手动配置

           network.host: 192.168.55.33        

                设置绑定的ip地址,可以是ipv4或ipv6的,默认为0.0.0.0 

           http.port: 9200        设置对外服务的Http端口,默认是9200

           transport.tcp.port: 9300       设置节点间交互的tcp端口,默认是9300 

           discovery.zen.ping.unicast.hosts: ["192.168.55.33"]     

            设置集群中master集群初始化列表,这个数组里的机器将被自动发现加入集群,多个用逗号隔开

                     

                  5:添加至系统服务及启动                      

                         chkconfig --add elasticsearch

                         systemctl daemon-reload

                         systemctl enable elasticsearch.service

                         systemctl restart elasticsearch.service

            

Graylog:    

                    1:安装Graylog的yum源及epel源

     rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-2.3-repository_latest.rpm

     yum install epel-release

                   

                    2:安装Graylog-server及相关组件

     yum install graylog-server pwgen

                    

                    3:配置graylog2,修改以下几个地方,其他地方保持默认,也可根据实际情况进行设置,详解自行谷歌

      password_secret            WxWxFDNy36Wgl3VMQoFVyCdJl5Tp

                    对密码进行加盐处理(就是密码加盐也就是密码后面加上很长的一串字符串再进行加密),例如

      md5(md5(password)+salt)和SHA512(SHA512(password)+salt) 方式

                     这里使用pwgen生成密码:pwgen -N 1 -s 96

      root_password_sha2     72d7c50d4e1e267df628ec2ee9eabee

                     graylog-web登录的用户密码,使用   sha256sum进行加密。

                     密码加密命令:echo -n yourpassword | sha256sum

      root_timezone = +08:00         设置时区

      rest_listen_uri = http://192.16855.33:9000/api/      

                        用于接受Graylog Collector Sidecar发送的心跳信息,collectors也可以访问次uri

      web_listen_uri = http://192.168.0.200:9000/         graylog-web访问地址

      elasticsearch_cluster_name = graylog2     必须与elasticsearch设置相同

      elasticsearch_discovery_zen_ping_unicast_hosts = 192.168.55.33:9300

                         添加elasticsearch集群的主机,多个主机用逗号隔开

      mongodb_uri = mongodb://localhost/graylog       

                         MongoDB服务器身份验证,使用默认即可


                  4:添加系统服务及启动                                  

                            chkconfig --add graylog-server

                            systemctl daemon-reload

                            systemctl enable graylog-server.service

                            systemctl start graylog-server.service


         至此,Graylog-server安装完毕,可通过浏览器输入192.168.55.33:9000进行访问,登录,以上配置文件设计的

         密码加密选项必须设置,并且必须使用指定密码工具生成密码才能正常登录,否则将失败


第二部分:    Collector端的部署

           

            1:简述                   

                         Graylog Collector Sidecar是一种用于采集日志的轻量级配置管理系统,也称为后端,作为守护进程运行。

       

                这些配置通过Graylog Web界面以图形方式进行集中管理。对于特定需求,可以将原始后端配置(称为Snippets)直接存储到Graylog中。                        

                使用REST API,Sidecar守护进程将定期获取目标的所有相关配置。实际获取哪些配置取决于主机的Sidecar配置文件中定义的“标签”。例如,Web服务器主机可以包括linux和nginx标签。              

                在第一次运行或检测到配置更改时,Sidecar将生成(呈现)相关的后端配置文件。然后它将启动或重新启动那些重新配置的日志收集器。

                目前,Sidecar支持Nxlog,Filebeat和Winlogbeat。支持的功能几乎相同,可以在web界面进行切换,对于所有收集器,可以使用带有SSL加密的GELF输出。

                在服务器端,您可以与多个收集器共享输入,例如,所有Filebeat和Winlogbeat实例都可以将日志发送到单个Graylog-Beats输入。

                此次博客我们以Nxlog作为后端来进行配置。


            1:安装Nxlog和collector-sidecar

                    yum源现在没有ollector-sidecar的,去官方下载rpm包进行安装。

yum install collector-sidecar-0.1.3-1.x86_64.rpm  nxlog-ce-2.9.1716-1_rhel7.x86_64.rpm


            2:添加系统服务及用户授权                                            

             gpasswd -a nxlog root

             chown -R nxlog.nxlog /var/spool/collector-sidecar/nxlog                       

             graylog-collector-sidecar -service install

              systemctl start collector-sidecar

    
            3:配置Nxlog

                    vim /etc/nxlog.conf  

                         <Extension _gelf>

                                Module      xm_gelf                 

                            </Extension>                          

                            <Input in>

                                Moduleim_file                  

                                File"/var/log/messages"              

                            </Input>                          

                            <Output out>

                                Moduleom_udp

                                Host 192.168.55.33

                                Port  12201

                            </Output>                         

                            <Route r>

                               Pathin => out

                            </Route>

                        
              4:配置colloector

                       vim /etc/graylog/collector-sidecar/collector_sidecar.yml    

                          server_url: http://192.168.55.33:9000/api/

                            update_interval: 10

                            tls_skip_verify: false

                            send_status: true

                            list_log_files:

                            node_id: graylog-collector-sidecar

                            collector_id: file:/etc/graylog/collector-sidecar/collector-id

                            cache_path: /var/cache/graylog/collector-sidecar

                            log_path: /var/log/graylog/collector-sidecar

                            log_rotation_time: 86400

                            log_max_age: 604800

                            tags:

                                - nginx

                            backends:

                                - name: nxlog

                                  enabled: true

                                  binary_path: /usr/bin/nxlog

                                  configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf

              5:重启服务

              systemctl restart collector-sidecar           systemctl restart-nxlog                    

       至此,Collector相关的东西配置完成,接下来就是Graylog-Web端相关的配置 


Graylog-Web


            1:    浏览器输入192.168.55.33进入管理界面,配置input和output相关信息


点击collectors --> manage configurations --> creat configurations





采集器名字




tags标签名字(与采集器客户端配置文件相同)




设置output和input相关信息,与nxlog的配置文件相同




配置完成后,重启客户端的collector-sidecar:systemctl restart collector-sidecar


graylog web查看Collectors 是否运行正常:




graylog web 设置 日志接收

System -->  inputs --> 选择哪种方式读取日志 -->  设置相关属性(设置服务器端的ip地址) --> 保存




查看是否正常运行

search:可以进行正则匹配高亮显示结果




至此,全部安装完毕,使用正常,若有不明白或无法正常工作的地方,请留言或者加群讨论:656633543

该博客会持续优化更新   

网站推荐:艾影网(sonyf.com)

        

















评论