欢迎来到Graylog中文站

(维护:FansDoc.com,目前只翻译一部分,先挖个坑以后慢慢填)

对于架构的思考

在为Graylog扩充资源时,有一些经验法则:

  • Graylog节点应重点关注CPU性能。这对于运行在浏览器中用户界面也有改善。
  • Elasticsearch节点应该有尽可能多的 RAM 和你能得到的最快的磁盘。 一切都取决于I/O的速度。
  • MongoDB 存储元数据和配置信息,不需要太多资源。

还要记住,获取的消息只是 存储在Elasticsearch。如果您在Elasticsearch群集中有数据丢失 ,消息将彻底消失,除非您已创建索引的备份。

最小化配置

这是一个最小化的Graylog配置,可用于较小、非关键或测试用途的配置。没有一个组件是冗余的,而且它们配置起来也简捷。

_images/architec_small_setup.png

我们的 虚拟机应用程序 默认使用这种设计,将nginx部署为 前端代理.

生产环境下更大规模的配置

这是针对更大规模的生产环境的配置。在负责分发负载任务的负载均衡器之下,存在若干个Graylog节点。

负载均衡器通过HTTP协议,调用Graylog REST API来ping Graylog节点,以检查它们是否处于活动状态,并将失效节点移出群集。

_images/architec_bigger_setup.png

如何来规划配置这类环境请参考我们的多节点配置指南

Graylog市场的某些指南也提供了一些思路,指导您使用RabbitMQ (AMQP)Apache Kafka 来向您的环境中添加一些队列。

Graylog架构深入

如果您真的对更详细的Graylog架构感兴趣--无论您是想了解更多架构设计、性能调优,还是仅仅因为您喜欢这样的内容,我们厚脸皮的工程团队汇总了这个深层架构指南。我们倒不是心虚,但我们希望您喜欢它。

开始

本指南是为首次使用的用户设计的,旨在提供足够的关键信息,以便初始化安装和配置Graylog。每个部分都链接到与主题相关的其他细节阐述。

Graylog是一个非常灵活的解决方案。它能以许多不同的方式部署。对于那些希望对Graylog进行初步试验评估的用户,我们建议从虚拟机应用程序开始。

虚拟化应用是最快的入门方式。然而,由于虚拟化应用通常不适合在生产中使用,它应仅限于概念验证、评估或试验环境。对于生产环境的部署,用户应规划选择一项其他更灵活的安装方法。

如果您在规划和构建日志环境方面需要协助,我们提供专业的支持 以适应您的需求。

规划您的日志收集

我们知道您迫切需要Graylog安装和运行,但我们要求您花一些时间来回顾此部分并合理地规划部署。是否合理的规划,决定了这是一个能够满足各利益方需求的有用的解决方案,还是一个消耗资源、几乎没有价值的乱摊子。在设计日志管理的解决方案时,必须考虑许多因素。

策略

即使在小型组织中,现代环境也会生成大量日志数据。不久前,每天500MB被认为是一家小企业的正常日志量。如今,对于一个小环境来说,每天5GB并不鲜见。一个大的环境可以产生一千倍以上的日志量。

假设单个事件占用空间大小平均为500k,若一天增长5GB空间,这相当于每秒125个日志事件,每天约108万个事件。生成这么多信息后,您将需要一个有效管理的策略。主要有两种方法。

极简主义者

“只做必要的事”

在决定收集哪些事件时, 极简主义策略从“默认否”的立场出发。这意味着您不会收集任何日志,除非它是在已确定的商业案例中所必需的。此策略具有一些优点,它减少了收集的事件的数量,从而降低了许可和存储的成本。它还最大限度地减少了不相关事件产生的“杂音”,使分析者能够专注于具有最大价值的事件。最后,它提高了系统和查询效率,整体上提升了性能。

极多主义者

“收集这一切,让Graylog搞定它。”

极多主义策略是收集任何来源所产生的全部事件。人们的想法是,所有的日志数据都有潜在的价值,特别是对取证而言。收集这一切,并久保存,确保当需要的时候你就能取得它。然而,由于预算或其他因素制约,这一策略往往不合实际。这一策略的成本可能高得令人望而却步,因为必须将更多的技术和人力资源用于事件的收集、处理和储存。在线保有庞大的数据集也会造成性能上的损失,这一点也必须考虑在内。

使用案例

“您要如何处理事件数据?”

使用案例应在规划阶段为大多数决策提供信息。其中一些决策包括确定应从哪些源中收集事件、如何从这些源中收集、每种事件类型要存储的数量、应如何富态化事件以及数据应保留多久。

使用案例,广义的定义,是指实现技术和/或业务成果所需的技术性步骤。一个更简单的思路是,使用案例是您在收集事件日志后,要对其所执行的操作的描述。用例通常被按组分类,类似于“活动”。一些常见的用例类别是“安全”、“操作”和“DevOps”。安全用例的一个示例是监控用户对关键资源的登录。操作用例可能监控网络或硬件性能,而DevOps用例将侧重于应用层实时监控或故障排除。

事件日志源

“您想收集什么日志?”

在一个看起来万物都能生成事件日志的环境中,可能很难知道需要收集哪些内容。在大多数情况下,事件源的选择应由已确定的用例来决定。例如,如果用例是监控用户对关键资源的登录,则所选择的事件源应是那些关键资源。可能是LDAP目录服务器、本地服务器、防火墙、网络设备和关键应用程序。

按类别划分的其他一些潜在事件源。

安全

  • 防火墙
  • 终端安全(EDR,AV等)
  • 网页代理/网关
  • LDAP/活动目录
  • IDS
  • DNS
  • DHCP
  • 服务器
  • 工作站
  • NetFlow

Ops

  • 应用程序
  • 网络设备
  • 服务器
  • 报文捕获/网络录像机
  • DNS
  • DHCP
  • Email

DevOps

  • 应用日志
  • 负载均衡器日志
  • 自动化系统日志
  • 业务逻辑

收集方式

“您将如何收集它?”

确定好事件源的列表之后,下一步是对于每个源,确定它的收集方式。尽管许多硬件和软件产品支持常见的方式,如通过syslog发送日志数据,但许多产品并不支持这种。了解每个事件源适用的方法,以及可能需要的资源是至关重要的。例如,如果要从所有服务器上的本地文件读取日志,需要用到日志传送器。需要选择一种日志传送器,在部署前进行测试。在其他情况下,必须应用和集成专有API或软件工具。

在某些情况下,可能需要更改事件源本身 (安全设备、网络硬件或应用程序)。久而久之,部署维护这些收集方式通常需要额外的规划。

Graylog支持开箱即用的多种输入类型。Graylog市场中提供了更多的输入类型。在编写本文时,Graylog支持以下内容:

  • Syslog (TCP, UDP, AMQP, Kafka)
  • GELF (TCP, UDP, AMQP, Kafka, HTTP)
  • AWS (AWS Logs, FlowLogs, CloudTrail)
  • Beats/Logstash
  • CEF (TCP, UDP, AMQP, Kafka)
  • HTTP API 中的 JSON Path
  • Netflow (UDP)
  • 纯/原始 文本 (TCP, UDP, AMQP, Kafka)

Graylog市场是Graylog附加组件的中央目录。它包含插件、内容包、GELF库和更多由Graylog开发人员和社区成员构建的内容。

_images/marketplace.png

用户

“谁将使用该解决方案?”

需考虑的最重要的用户相关因素是用户数量。如果数量很大, 或者许多用户将同时查询数据, 则在架构设计时可能需要考虑到这一点。

应考虑用户的技能水平。技术水平较低的用户可能需要更多的预制的内容,如仪表盘。他们还可能需要更多的培训。

还应考虑每个用户组应访问哪些事件源。与所有访问控制一样, 应采用最小特权原则。

一些典型的用户组包括:

  • 安全分析师
  • 工程师
  • 管理
  • 服务台

保留期

“您要将数据保留多长时间?”

规划日志管理系统时的一个关键问题是日志保留。可以通过两种方式保留:在线或归档。在线数据存储在Elasticsearch中,可通过Graylog GUI进行搜索。归档的数据以压缩格式存储在Graylog服务器或网络共享文件中。它仍然是可搜索的,例如,通过GREP,但必须在Graylog中重组,以便能再次通过 GUI 进行搜索。

有些监管制度要求将事件日志保留到规定期限。在没有明确要求的情况下, 问题变成了如何来平衡储存成本与保有历史数据的效用。没有单一的答案, 因为每种情况都不一样。

大多数 Graylog 客户保留30~90天的在线事件日志(在Elasticsearch里能够搜索)和6~13月的归档事件日志。

计算存储需求

与大多数数据存储一样,Elasticsearch 在耗尽所有可用存储后会表现得糟糕。为了防止这种情况发生,必须进行适当的规划和监控。

许多变量会影响存储需求,例如留存了多少消息,分析完成之后是否保留原始消息,以及在存储之前进行了多少富态化。

规划存储的一个简单经验法则是,将您的数据摄取率乘以需在线保留数据的天数,然后将该值乘以1.3(考虑到元数据开销)。(GB/天 x 天数 x 1.3 = 存储量)

Elasticsearch 在其操作过程中广泛利用了松散的存储空间。上面计算出了数据摄取率对应的最小存储空间,强烈建议用户的实际配置超过这个值。在到达最大保留期时,Elasticsearch 存储空间不应超过总空间的75%。

下载&安装Graylog

Graylog能以许多不同的方式部署,您应该下载最适合您的东西。对于那些希望对Graylog进行初步试验评估的用户,我们建议从虚拟机应用程序开始。

虚拟化应用绝对是最快的入门方式。然而,由于虚拟化应用通常不适合在生产中使用,它应仅限于概念验证、评估或试验环境。.

虚拟化应用也完全是不安全的。它未进行强化,并默认开启所有服务。

对于生产部署,用户应选择并部署其他更灵活的安装方法之一。对于生产环境的部署,用户应规划选择一项其他更灵活的安装方法。

操作系统软件包

可以在以下操作系统上安装Graylog。

  • Ubuntu
  • Debian
  • RHEL/CentOS
  • SLES

大多数客户使用DEB或RPM等软件包工具来安装Graylog软件。详细信息在这节里,操作系统软件包.

配置管理

喜欢通过配置管理工具部署Graylog的客户可能会这样做。Graylog 当前支持Chef, Puppet, Ansible.

容器

Graylog支持Docker部署Graylog、MongoDB和Elasticsearch组件。可在Docker安装页找到安装配置说明。

虚拟化应用

可以在虚拟化应用下载页下载虚拟化应用。如果您不确定最新稳定版的版本号是什么,看看我们的发布页面.

_images/download.png

受支持的虚拟化应用

  • OVA
  • AWS-AMI

虚拟机应用程序部署指南。

Amazon Web Services部署指南。

虚拟化应用警告

虚拟化应用不适合生产环境的开箱即用部署。它们没有足够的存储空间,也不提供适应高可用需求的索引复制等功能。

虚拟化应用没有强化或以其他方式保护。您应自行承担风险,并采取所在机构要求的所有安全措施。

初始化配置

安装应用程序后,必须先配置几个项目,然后才能首次启动Graylog。Graylog的server.conf 和Elasticsearch的elasticsearch.yml文件都是包含了初始化配置所需的关键信息的配置文件。

本指南将为您提供必要的设置,使Graylog启动并运行。这些文件中还有许多其他重要设置,我们建议您在启动并运行后查看它们。欲了解更多详情,请参阅server.conf.

提示

如果您使用的是虚拟化应用,请跳过此部分,直接转到连接到Web控制台.

server.conf

server.conf文件是Graylog的配置文件。server.conf的默认路径是:/etc/graylog/server/server.conf.

提示

所有文件的默认路径列在这里.

  • 条目通常应是单行的形式,以下之一:
    • 属性名称=属性值
    • 属性名称:属性值
  • 在属性名称和属性值之间显示的空白将被忽略,因此以下是等同的:
    • name=Stephen
    • name = Stephen
  • 行首的空白也会被忽略。
  • 以注释符!!#开头的行会被忽略。空白行也会被忽略。
  • 属性值通常在行尾结束。
  • 属性值后面的空白不会被忽略,并被视为属性值的一部分。
  • 可用字符\n\r,和\t分别插入换行符、回车和制表符。
常规属性
  • is_master = true
    • 如果您正在运行多个Graylog实例,则必须(仅)指定一个graylog-server节点作为主节点。此节点将执行周期性和维护性的操作,从节点不会去执行。
  • password_secret = <secret>
    • 您必须设置一个用于加密和加盐的密钥。如果未设置此值,服务将拒绝启动。至少使用64个字符。如果您运行多个 graylog-server 节点,应确保对它们使用同样的password_secret

    提示

    例如,生成一个密钥pwgen -N 1 -s 96.

  • root_username = admin
    • 默认的根用户名为 admin.
  • root_password_sha2 = <SHA2>
    • 您初始登录密码的SHA2哈希值。用echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1来生成SHA2哈希值,插入到上面的选项中,您将能够以用户名 admin 和密码 您的密码 登录到 web 界面。

    注意

    您必须为根用户指定一个哈希密码(您只在系统初始化配置时用到它,以及在与身份验证后端失去连接的情况下)。不能使用API或通过web界面更改此密码。如果需要更改它,请在此文件中对其进行修改。

Web属性
  • http_bind_address = 127.0.0.1:9000
    • 由Graylog HTTP界面使用的网络接口。
    • http_publish_uri中默认使用此地址和端口。
  • http_publish_uri = http://$http_bind_address/

    • Web界面监听URI。
    • 此Graylog节点的HTTP URI,所有使用Graylog web接口的客户端都用到它。
Elasticsearch属性
  • elasticsearch_hosts = http://node1:9200,http://user:password@node2:19200
    • Graylog应连接到的Elasticsearch主机列表。
    • 需要为您的elasticsearch节点的http端口指定一个逗号分隔的有效URI列表。
    • 如果一个或多个elasticsearch主机需要身份验证,请在需要身份验证的各个节点URI中包含凭据。
    • 默认:http://127.0.0.1:9200 只有当Elasticsearch与Graylog服务安装在相同的主机上时,才可以保留默认设置。
MongoDB
  • mongodb_uri = mongdb://...
    • MongoDB连接字符串。在这里输入您的MongoDB连接和身份验证信息。
    • 详情请参阅https://docs.mongodb.com/manual/reference/connection-string/
    • 例子:
      • 简易版:mongodb_uri = mongodb://localhost/graylog
      • 针对MongoDB服务器进行身份验证:mongodb_uri = mongodb://grayloguser:secret@localhost:27017/graylog
      • 使用副本集而非单个主机:
        mongodb_uri = mongodb://grayloguser:secret@localhost:27017,localhost:27018,localhost:27019/graylog
传出的HTTP
  • http_proxy_uri =
    • 用于传出HTTP连接的HTTP代理
  • http_non_proxy_hosts =
    • 应绕过所配置的代理服务器,直接进行访问的主机列表。
    • 这是一个由“,”分隔的样式清单。这些样式可能以通配符“*”开头或结尾。
    • 任何匹配这些样式之一的主机都将通过直连而不是代理来访问。

elasticsearch.yml

Elasticsearch.yml是Elasticsearch的配置文件。server.conf的默认路径是:/etc/elasticsearch/elasticsearch.yml.

必须正确配置几个值,才能使elasticsearch正常工作。

  • cluster.name: graylog
    • 这个值可以设置为客户希望的任何值,不过我们建议使用”graylog“。
    • 对于集群中的每个Elasticsearch节点,此值必须相同。
  • network.host: 172.30.4.105
    • 默认情况下,Elasticsearch只绑定到环回地址(例如127.0.0.1)。这对于在服务器上运行单个开发节点来说足够了。
    • 为了与其他服务器上的节点通信并形成集群,您的节点需要绑定到一个非环回地址。
  • http.port: 9200
    • Elasticsearch将监听的端口。我们建议使用默认值。
  • discovery.zen.ping.unicast.hosts: ["es01.acme.org", "es02.acme.org"]

    • Elasticsearch使用称为“Zen 发现”的自定义发现,来实现节点间的集群和主节点选择。若要与其他服务器上的节点形成集群,必须提供集群中可能是活动节点且可联络的其他节点的种子列表。
    • 可以指定为 IP 地址或 FQDN。

连接到Web控制台

打开浏览器并导航到URL http://xxx.xxx.xxx.xxx:9000,替换为您的graylog服务器IP。您应该会看到一个类似于以下截图的Graylog登录页面。

如果使用VM应用,登录的用户名和密码都是admin。如果使用容器或操作系统版本的Graylog,则以admin身份登录,并使用在安装Graylog时,password_secret项衍生出的密码。

_images/graylog_login.png

登录后将来到“入门”界面。但是,如果您正在阅读这篇文章,那么您已经找到了“入门指南”,所以只要继续下去。

另外,随时可以将指南解除或保留以后使用。

_images/first_login.png

探索Graylog

一旦接收到消息,您可能想四处查看一下。有几个页面可用,但并不是全部页面都对所有用户可见,这取决于各自的权限。下面是对每个页面的用途和功能的简要描述。

数据流

数据流(Streams)是Graylog的一个核心特征,可以看作是对传入消息进行标记的一种方式。数据流是用于将消息实时转发到相应类别中的一种机制。流规则指明Graylog将哪个消息转发到哪个数据流。

数据流有很多用途。首先,它们用于将存储的数据路由到索引中。它们还用于控制对数据的访问,转发消息以供解析,富态化或其它修改,并确定将哪些消息归档。

数据流可以与告警(Alerts)一起使用,以便在消息满足一组条件时通知用户,或以其他方式响应。

消息可以属于一个或多个数据流。有关更多细节,请参阅数据流

搜索

Graylog搜索页(Search)是用来直接搜索日志的。Graylog使用了一种简化的语法,非常类似于Lucene。可从下拉菜单中配置相对或绝对时间范围。搜索结果可以保存或显示为仪表盘的小工具,可以从搜索界面将这些小工具直接添加到仪表盘。

用户可以配置自定义视图,并可以选择查看事件消息的摘要或完整数据。

有关更多细节,请参阅搜索

仪表盘

Graylog仪表盘(Dashboards)是日志事件所包含的信息的可视化或汇总。每个仪表盘由一个或多个小工具填充,小工具利用日志数据的字段值(如计数、平均值或总数),来可视化或汇总事件。趋势指示器、图表、图形和地图很容易创建,以便数据的可视化呈现。

仪表盘小工具和仪表盘布局是可配置的。通过Graylog基于角色的访问控制来限制对仪表盘的访问。可以通过内容包导入和导出仪表盘。

有关更多细节,请参阅仪表盘

告警

告警(Alerts)由两个相关元素组成,告警条件和告警通知。告警条件与数据流绑定,可能是基于字段的内容、字段的聚合值或消息计数阈值。告警通知在条件满足时触发,通常向分析师或其他系统发送电子邮件或HTTP回调。

额外的输出类型也可以通过插件创建。可以通过内容包导入和导出告警。

有关更多细节,请参阅告警

系统

总览

总览页(Overview)显示与Graylog实例管理相关的信息。它包括系统通知、系统作业状态、数据摄取率、Elasticsearch集群健康状况、索引器故障、时间配置和系统事件消息等方面的信息。

配置

配置页(Configuration)允许用户设置与搜索、消息处理器和插件相关的选项和变量。

节点

节点页(Nodes)包含每个Graylog节点的状态信息汇总。详细的健康信息和指标可从本页上显示的按钮获得

输入

输入(Inputs)用于告诉Graylog侦听哪个端口,或者如何取得事件日志,通常是在系统初始化配置之后的第一件事。输入页允许用户创建和配置新的输入、管理提取器、启动和停止输入、获取每个输入的指标,以及为传入消息添加静态字段。

输出

输出(Outputs)用于定义将数据转发到远端系统的方法,包括端口、协议和任何其他必要信息。在开箱即用的情况下,Graylog支持STDOUT和GELF输出,但是用户可以编写自己的输出,而且在Graylog市场还有更多。

身份验证

身份验证页(Authentication)用于配置Graylog的身份验证提供者,并管理这个Graylog集群的活动用户。Graylog支持以LDAP或活动目录进行身份验证和授权。

内容包

内容包(Content packs)加快了特定数据源的设置过程。内容包可以包含输入/提取器、数据流、仪表盘、告警和管道处理器。

在Graylog中创建的任何程序元素都可以作为内容包导出,以供其他系统使用。这些元素可以由作者保持私有,以便在内部快速部署新节点,也可以通过Graylog市场与社区共享。例如,用户可以创建自定义输入、数据流、仪表盘和告警来形成一个安全的使用案例。这些元素可以导出到内容包中,然后导入到新安装的Graylog实例中,以节省配置过程中的时间和精力。

用户可以通过Graylog市场来下载由其他用户创建和共享的内容包。用户创建的内容包不受Graylog支持,而是由其作者支持。

内容包中支持的元素列表

  • 输入
  • Grok模型
  • 输出
  • 数据流
  • 仪表盘
  • 查询表
  • 查询缓存
  • 查询数据适配器
索引

索引(Indices)是Elasticsearch中数据的基本存储单元。索引集作用是配置所存储数据的保留期、分片和复制。

值(如保留期和旋转策略)是根据每个索引设置的,因此不同的数据可能从属于不同的处理规则。

有关更多细节,请参阅索引模型.

Sidecars

Graylog创建了Sidecar代理来管理Beats或NXLog等日志传送的编队。这些日志传送程序用于从Linux和Windows服务器收集系统日志。日志传送程序从写入到本地的平面文件中读取,然后将其发送到集中日志管理解决方案,它通常是最简单的方法。

有关更多细节,请参阅Graylog Sidecar.

管道

Graylog的管道处理是一个功能强大的特性,允许用户针对特定类型的事件运行单条规则或一系列规则。管道与数据流绑定在一起,允许对流经Graylog的消息进行转发、黑名单、修改和富态化。基本上,如果您要解析、改变、转换、添加、删除某条消息的部分或全部时,管道可以派上用场。

有关更多细节,请参阅管道处理.

收集消息

一旦运行了Graylog和相关组件,下一步就是开始收集日志。

第一步是创建一个输入。输入定义了Graylog收集日志的方法。Graylog支持多种开箱即用的方法来收集日志,包括:

  • Syslog (TCP, UDP, AMQP, Kafka)
  • GELF(TCP, UDP, AMQP, Kafka, HTTP)
  • AWS - AWS Logs, FlowLogs, CloudTrail
  • Beats/Logstash
  • CEF (TCP, UDP, AMQP, Kafka)
  • HTTP API 中的 JSON Path
  • Netflow (UDP)
  • 纯/原始 文本(TCP, UDP, AMQP, Kafka)

内容包

可以通过内容包安装其他输入源。内容包是由能Graylog输入、提取器、数据流、仪表盘和输出这些配置信息的合集组成,它能够为数据源提供全面的支撑。有些内容包是默认随Graylog一起发布的,有些则可以从网站上获得。从Graylog市场下载的内容包可以使用Graylog web界面导入。

在Graylog的web界面System / Content Packs中,您可以加载甚至创建自己的内容包。

创建一个输入

打开上方菜单的System / Inputs页,来创建一个输入,单击下拉框中的箭头,选择您的输入类型并单击标为Launch new input的绿色按钮。

通常,默认设置是正确的,但是您可以随意更改。某些输入类型可能需要验证,或填入针对该输入源的其他信息。

提示

如果Graylog不是作为root用户运行的,那么您将无法使用低于1024的端口作为输入。可能需要重新配置发送设备。由于最佳实践规定应用程序不应以root用户身份运行,因此对于不能更改事件来源的客户,鼓励使用负载均衡器或其他外部方法来实现端口转换。

保存输入后,它会自动启动。

如果事件源已配置为将事件发送到所选端口,对于像Syslog或CEF这样的主动推送事件的源,您应该在几秒钟内开始接收消息。

如果您想了解更多关于Graylog摄取消息时所支持的选项,请查阅日志文件接入

验证是否正在收集消息

一旦定义了输入,就需要验证是否正从输入源接收消息。检查输入源右侧的Throughput / Metrics部分。您应该看到NetworkIO值开始攀升,显示在这个输入源上消耗的数据量。

_images/input_page.png

单击刚刚创建的输入旁边的Show received messages按钮。您现在应该看到在这个输入上接收到的消息。

_images/gs_10-messages.png

如果您单击顶部菜单中的Sources,您将看到向Graylog发送数据的所有设备、服务器或应用程序的详细概述,以及从每个源接收到的消息数量。最初,您可能不会在此页面上看到太多内容。然而,一旦您开始从更多的系统发送数据,他们的主机名或 IP 地址也将在此页上列出。

_images/sources_page.png

如果一切正常, 请跳过以下部分。

如果您没有看到消息
  1. 检查您在上述输入源配置中输入了正确的条目。
  2. 检查事件源的配置,并确保它与输入中定义的端口和其他选项相匹配。例如,如果将Syslog UDP的输入端口更改为5014,请确保发送设备也定义了相同的端口。
  3. 检查流量是否正在流向定义的端口。 您可以使用tcpdump命令来查看:

$ sudo tcpdump -i lo host 127.0.0.1 and udp port 5014

  1. 检查服务器是否正在侦听其他主机:

$ sudo netstat -peanut | grep ":5014"

如果您仍有问题,连接到我们的社区支持或通过我们提供的专业支持与我们联系。

安装Graylog

现代服务器架构和配置以许多不同的方式进行管理。有些人仍然对于每台服务器,手动将新软件安装在opt下的某个地方,而其他人已经跳上配置管理快车,可完全自动化、可重复地配置。

Graylog 可以通过许多不同的方式安装, 这样你就可以选择最适合你的方法。我们建议从虚拟机应用程序开始, 以最快的方式开始, 然后选择一个其他更灵活的安装方法, 以构建更易于扩展的配置。

本章解释许多了安装 Graylog 的方法, 旨在帮助选择适合您需求的方法。

虚拟机应用程序

预先考虑

请始终在与internet隔离的独立网络中运行此应用。还要阅读关于生产准备的说明!

下载

下载OVA镜像。如果您不确定最新稳定版的版本号是什么,看看我们的发布页面

运行镜像

您可以在VMwareVirtualbox等许多系统中运行OVA。在这个例子中,我们将指导您通过OSX上免费的Virtualbox来运行OVA。

在Virtualbox中,选择File -> Import appliance:

_images/virtualbox1.png

点击Continue并在下一页保留建议的设置。确保本地机器上有足够的RAM和CPU。你可以降低分配至虚拟机的资源,但我们建议不要降低,它保证了良好的Graylog体验。事实上,如果您以后计划扩展它并向Graylog发送更多消息,您可能要提高它。

点击Import,将OVA加载到Virtualbox中:

_images/virtualbox2.png

现在,您可以启动 VM,启动完成时应看到这样的登录shell:

_images/virtualbox3.png

提示

如果您的虚拟机没有能工作的DHCP服务器,您将收到错误消息:

“Your appliance came up without a configured IP address. Graylog is probably not running correctly!”

在这种情况下,您必须登录并编辑/etc/network/interfaces,以便设置固定的IP地址。然后创建文件/var/lib/graylog-server/firstboot并重新启动。

登录

您可以使用用户ubuntu和密码ubuntu登录到应用程序所在操作系统的shell。当然,您应该更改这些凭据。

可在您的虚拟机IP地址的80上端口访问web界面。Shell的登录提示也显示了此IP地址。(请参见上图)。

Web界面的标准用户是admin,密码显示在虚拟机第一次启动时的控制台中。

配置

如果需要进一步自定义应用,请查看 Graylog 配置文件 说明文档。

VMWare工具

如果您在VMWare主机上使用该应用,则可能需要安装虚拟机管理程序工具:

sudo apt-get install -y open-vm-tools

更新OVA到最新版本

2.x
不可能将以前的OVAs升级到Graylog 3.0.0。
3.0
从Graylog 3.0.0开始,OVAs使用操作系统包,所以您可以通过这个 升级指南来升级您的应用。

生产准备

创建Graylog应用不是为了提供一个现成的生产解决方案。它设计初衷是提供了一个快速和简单的方法来体验软件本身,而不是将浪费时间在如何去安装Graylog或它的组件到任何类型的服务器上。

操作系统软件包

在配置管理系统进入更广泛的市场和众多数据中心之前,在Linux服务器上的安装软件最常见的方式之一是使用操作系统包。Debian是DEB,红帽是RPM 许多其他发行版是基于这些或自带的包格式。软件包的在线存储库和相应的软件包管理器使得进行新软件的安装配置只需一个命令和几分钟的时间。

Graylog提供了DEBRPM官方软件包仓库。这些软件包已在下列操作系统上测试:

  • Ubuntu 12.04, 14.04, 16.04, 18.04
  • Debian 7, 8, 9
  • RHEL/CentOS 6, 7

可以通过安装单个包来设置存储库。一旦完成,就能通过apt-getyum来安装Graylog包。如果需要,还可以通过web浏览器访问https://packages.graylog2.org/来下载这些包。

先决条件

在安装和启动任何Graylog服务之前,请确保安装和配置以下软件:

  • Java (>= 8)
  • Elasticsearch (5.x 或 6.x)
  • MongoDB (>= 3.6)

注意

Graylog 3 不支持Elasticsearch 7.x!

DEB / APT

通过dpkg(1)下载和安装graylog-3.0-repository_latest.deb并且确保已安装apt-transport-https包:

$ sudo apt-get install apt-transport-https
$ wget https://packages.graylog2.org/repo/packages/graylog-3.0-repository_latest.deb
$ sudo dpkg -i graylog-3.0-repository_latest.deb
$ sudo apt-get update
$ sudo apt-get install graylog-server

安装成功后,可以使用以下命令启动Graylog。请确保为您的操作系统使用正确的命令。

操作系统 Init系统 命令
Ubuntu 14.04, 12.04 upstart sudo start graylog-server
Debian 7 SysV sudo service graylog-server start
Debian 8 & 9, Ubuntu 16.04, 18.04 systemd sudo systemctl start graylog-server

这些包被配置为在引导期间启动任何Graylog服务。在操作系统启动时,可以使用以下命令启动Graylog。

操作系统 Init系统 命令
Ubuntu 14.04, 12.04 upstart sudo rm -f /etc/init/graylog-server.override
Debian 7 SysV sudo update-rc.d graylog-server defaults 95 10
Debian 8 & 9, Ubuntu 16.06, 18.04 systemd sudo systemctl enable graylog-server
升级至最新版本

如果您以前一直使用存储库的包安装Graylog,那么首先必须更新它。新包将替换存储库URL,没有它,您只能获得以前安装的Graylog版本的bug修复版本。

更新的过程基本上像一个全新安装:

$ wget https://packages.graylog2.org/repo/packages/graylog-3.0-repository_latest.deb
$ sudo dpkg -i graylog-3.0-repository_latest.deb
$ sudo apt-get update
$ sudo apt-get install graylog-server
手动存储库安装

如果您不喜欢通过安装DEB存储库,来在系统中获取存储库配置,您可以手动安装(尽管我们不建议这样做)。

首先,添加Graylog GPG keyring,它用于将包签发到您的系统。

提示

我们假设您已将GPG密钥放入/etc/apt/trusted.gpg.d/中。

现在创建一个文件/etc/apt/sources.list.d/graylog.list,内容如下:

deb https://packages.graylog2.org/repo/debian/ stable 3.0

RPM / YUM / DNF

通过rpm(8)下载和安装graylog-3.0-repository_latest.rpm

$ sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-3.0-repository_latest.rpm
$ sudo yum install graylog-server

安装成功后,可以使用以下命令启动Graylog。请确保为您的操作系统使用正确的命令。

操作系统 Init系统 命令
CentOS 6 SysV sudo service graylog-server start
CentOS 7 systemd sudo systemctl start graylog-server

这些包被配置为在引导期间启动任何Graylog服务。在操作系统启动时,可以使用以下命令启动Graylog。

操作系统 Init系统 命令
CentOS 6 SysV sudo update-rc.d graylog-server defaults 95 10
CentOS 7 systemd sudo systemctl enable graylog-server
升级至最新版本

如果您以前一直使用存储库的包安装Graylog,那么首先必须更新它。新包将替换存储库URL,没有它,您只能获得以前安装的Graylog版本的bug修复版本。

更新的过程基本上像一个全新安装:

$ sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-3.0-repository_latest.rpm
$ sudo yum clean all
$ sudo yum install graylog-server

运行yum clean all是必需的,因为yum可能使用陈旧的缓存,从而可能无法找到最新版本的graylog-server包。

手动存储库安装

如果您不喜欢通过安装RPM存储库,来在系统中获取存储库配置,您可以手动安装(尽管我们不建议这样做)。

首先,添加Graylog GPG key,它用于将包签发到您的系统。

提示

我们假设您已将GPG密钥放入/etc/pki/rpm-gpg/RPM-GPG-KEY-graylog中。

现在创建一个文件/etc/yum.repos.d/graylog.repo,内容如下:

[graylog]
name=graylog
baseurl=https://packages.graylog2.org/repo/el/stable/3.0/$basearch/
gpgcheck=1
repo_gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-graylog

分步指南

Ubuntu安装

本指南描述了在Ubuntu 18.04 LTS上安装Graylog的最快方法。在编写本文时,所有链接和包都已存在,但以后可能会更新。

警告

此设置不应在公开暴露的服务器上执行。本指南不包括安全设置!

先决条件

以最小化安装的服务器作为基础,将需要这个额外的包:

$ sudo apt-get update && sudo apt-get upgrade
$ sudo apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen
MongoDB

官方的MongoDB存储库提供了最新的版本,是安装MongoDB的推荐方法:

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
$ echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
$ sudo apt-get update
$ sudo apt-get install -y mongodb-org

最后一步是在操作系统启动时开启MongoDB:

$ sudo systemctl daemon-reload
$ sudo systemctl enable mongod.service
$ sudo systemctl restart mongod.service
Elasticsearch

Graylog可以和Elasticsearch 6.x一起使用。请按照Elasticsearch安装指南中的安装说明进行安装:

$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ echo "deb https://artifacts.elastic.co/packages/oss-6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
$ sudo apt-get update && sudo apt-get install elasticsearch-oss

确保修改了auto_create_index:对配置文件为false: Make sure to modify the Elasticsearch配置文件 (/etc/elasticsearch/elasticsearch.yml),并将集群名称设置为graylog另外需要取消注释(除去行首的#)此行,并添加action.auto_create_index: false到配置文件:

cluster.name: graylog
action.auto_create_index: false

After you have modified the configuration, you can start Elasticsearch:

$ sudo systemctl daemon-reload
$ sudo systemctl enable elasticsearch.service
$ sudo systemctl restart elasticsearch.service
Graylog

Now install the Graylog repository configuration and Graylog itself with the following commands:

$ wget https://packages.graylog2.org/repo/packages/graylog-3.0-repository_latest.deb
$ sudo dpkg -i graylog-3.0-repository_latest.deb
$ sudo apt-get update && sudo apt-get install graylog-server

Follow the instructions in your /etc/graylog/server/server.conf and add password_secret and root_password_sha2. These settings are mandatory and without them, Graylog will not start!

You need to use the following command to create your root_password_sha2:

echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1

To be able to connect to Graylog you should set http_bind_address to the public host name or a public IP address of the machine you can connect to. More information about these settings can be found in Configuring the web interface.

Note

If you’re operating a single-node setup and would like to use HTTPS for the Graylog web interface and the Graylog REST API, it’s possible to use NGINX or Apache as a reverse proxy.

The last step is to enable Graylog during the operating system’s startup:

$ sudo systemctl daemon-reload
$ sudo systemctl enable graylog-server.service
$ sudo systemctl start graylog-server.service

The next step is to ingest messages into your Graylog and extract the messages with extractors or use the Pipelines to work with the messages.

Multiple Server Setup

If you plan to have multiple server taking care of different roles in your cluster like we have in this big production setup you need to modify only a few settings. This is covered in our Multi-node Setup guide. The default file location guide will give you the file you need to modify in your setup.

Debian installation

This guide describes the fastest way to install Graylog on Debian Linux 9 (Stretch). All links and packages are present at the time of writing but might need to be updated later on.

Warning

This setup should not be done on publicly exposed servers. This guide does not cover security settings!

Prerequisites

If you’re starting from a minimal server setup, you will need to install these additional packages:

$ sudo apt update && sudo apt upgrade
$ sudo apt install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen dirmngr
MongoDB

The official MongoDB repository provides the most up-to-date version and is the recommended way of installing MongoDB:

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
$ echo "deb http://repo.mongodb.org/apt/debian stretch/mongodb-org/4.0 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
$ sudo apt-get update
$ sudo apt-get install -y mongodb-org

The last step is to enable MongoDB during the operating system’s startup:

$ sudo systemctl daemon-reload
$ sudo systemctl enable mongod.service
$ sudo systemctl restart mongod.service
Elasticsearch

Graylog can be used with Elasticsearch 6.x, please follow the installation instructions from the Elasticsearch installation guide:

$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ echo "deb https://artifacts.elastic.co/packages/oss-6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
$ sudo apt update && sudo apt install elasticsearch-oss

Make sure to modify the Elasticsearch configuration file (/etc/elasticsearch/elasticsearch.yml) and set the cluster name to graylog additionally you need to uncomment (remove the # as first character) the line, and add action.auto_create_index: false to the configuration file:

cluster.name: graylog
action.auto_create_index: false

After you have modified the configuration, you can start Elasticsearch:

$ sudo systemctl daemon-reload
$ sudo systemctl enable elasticsearch.service
$ sudo systemctl restart elasticsearch.service
Graylog

Now install the Graylog repository configuration and Graylog itself with the following commands:

$ wget https://packages.graylog2.org/repo/packages/graylog-3.0-repository_latest.deb
$ sudo dpkg -i graylog-3.0-repository_latest.deb
$ sudo apt update && sudo apt install graylog-server

Follow the instructions in your /etc/graylog/server/server.conf and add password_secret and root_password_sha2. These settings are mandatory and without them, Graylog will not start!

You need to use the following command to create your root_password_sha2:

echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1

To be able to connect to Graylog you should set http_bind_address to the public host name or a public IP address of the machine you can connect to. More information about these settings can be found in Configuring the web interface.

Note

If you’re operating a single-node setup and would like to use HTTPS for the Graylog web interface and the Graylog REST API, it’s possible to use NGINX or Apache as a reverse proxy.

The last step is to enable Graylog during the operating system’s startup:

$ sudo systemctl daemon-reload
$ sudo systemctl enable graylog-server.service
$ sudo systemctl start graylog-server.service

The next step is to ingest messages into your Graylog and extract the messages with extractors or use the Pipelines to work with the messages.

Multiple Server Setup

If you plan to have multiple server taking care of different roles in your cluster like we have in this big production setup you need to modify only a few settings. This is covered in our Multi-node Setup guide. The default file location guide will give you the file you need to modify in your setup.

CentOS installation

This guide describes the fastest way to install Graylog on CentOS 7. All links and packages are present at the time of writing but might need to be updated later on.

Warning

This setup should not be done on publicly exposed servers. This guide does not cover security settings!

Prerequisites

Taking a minimal server setup as base will need this additional packages:

$ sudo yum install java-1.8.0-openjdk-headless.x86_64

If you want to use pwgen later on you need to Setup EPEL on your system with sudo yum install epel-release and install the package with sudo yum install pwgen.

MongoDB

Installing MongoDB on CentOS should follow the tutorial for RHEL and CentOS from the MongoDB documentation. First add the repository file /etc/yum.repos.d/mongodb-org.repo with the following contents:

[mongodb-org-4.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.0.asc

After that, install the latest release of MongoDB with sudo yum install mongodb-org.

Additionally, run these last steps to start MongoDB during the operating system’s boot and start it right away:

$ sudo chkconfig --add mongod
$ sudo systemctl daemon-reload
$ sudo systemctl enable mongod.service
$ sudo systemctl start mongod.service
Elasticsearch

Graylog can be used with Elasticsearch 6.x, please follow the installation instructions from the Elasticsearch installation guide.

First install the Elastic GPG key with rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch then add the repository file /etc/yum.repos.d/elasticsearch.repo with the following contents:

[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/oss-6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

followed by the installation of the latest release with sudo yum install elasticsearch-oss.

Make sure to modify the Elasticsearch configuration file (/etc/elasticsearch/elasticsearch.yml) and set the cluster name to graylog additionally you need to uncomment (remove the # as first character) the line, and add action.auto_create_index: false to the configuration file:

cluster.name: graylog
action.auto_create_index: false

After you have modified the configuration, you can start Elasticsearch:

$ sudo chkconfig --add elasticsearch
$ sudo systemctl daemon-reload
$ sudo systemctl enable elasticsearch.service
$ sudo systemctl restart elasticsearch.service
Graylog

Now install the Graylog repository configuration and Graylog itself with the following commands:

$ sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-3.0-repository_latest.rpm
$ sudo yum install graylog-server

Follow the instructions in your /etc/graylog/server/server.conf and add password_secret and root_password_sha2. These settings are mandatory and without them, Graylog will not start!

You need to use the following command to create your root_password_sha2:

echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1

To be able to connect to Graylog you should set http_bind_address to the public host name or a public IP address of the machine you can connect to. More information about these settings can be found in Configuring the web interface.

Note

If you’re operating a single-node setup and would like to use HTTPS for the Graylog web interface and the Graylog REST API, it’s possible to use NGINX or Apache as a reverse proxy.

The last step is to enable Graylog during the operating system’s startup:

$ sudo chkconfig --add graylog-server
$ sudo systemctl daemon-reload
$ sudo systemctl enable graylog-server.service
$ sudo systemctl start graylog-server.service

The next step is to ingest messages into your Graylog and extract the messages with extractors or use the Pipelines to work with the messages.

SELinux information

Hint

We assume that you have policycoreutils-python installed to manage SELinux.

If you’re using SELinux on your system, you need to take care of the following settings:

  • Allow the web server to access the network: sudo setsebool -P httpd_can_network_connect 1
  • If the policy above does not comply with your security policy, you can also allow access to each port individually:
    • Graylog REST API and web interface: sudo semanage port -a -t http_port_t -p tcp 9000
    • Elasticsearch (only if the HTTP API is being used): sudo semanage port -a -t http_port_t -p tcp 9200
  • Allow using MongoDB’s default port (27017/tcp): sudo semanage port -a -t mongod_port_t -p tcp 27017

If you run a single server environment with NGINX or Apache proxy, enabling the Graylog REST API is enough. All other rules are only required in a multi-node setup. Having SELinux disabled during installation and enabling it later, requires you to manually check the policies for MongoDB, Elasticsearch and Graylog.

Hint

Depending on your actual setup and configuration, you might need to add more SELinux rules to get to a running setup.

Multiple Server Setup

If you plan to have multiple server taking care of different roles in your cluster like we have in this big production setup you need to modify only a few settings. This is covered in our Multi-node Setup guide. The default file location guide will give you the file you need to modify in your setup.

SLES installation

This guide describes the fastest way to install Graylog on SLES 12 SP3. All links and packages are present at the time of writing but might need to be updated later on.

Warning

This setup should not be done on publicly exposed servers. This guide does not cover security settings!

Prerequisites

The following patterns are required for a minimal setup (see SLES 12 SP3 Deployment Guide):

- Base System
- Minimal System (Appliances)
- YaST configuration packages

Warning

This Guide assumes that the firewall is disabled and communication is possible to the outside world.

Assuming a minimal setup, you have to install the Java runtime environment:

$ sudo zypper install java-1_8_0-openjdk
MongoDB

Installing MongoDB on SLES should follow the tutorial for SLES from the MongoDB documentation. Add the GPG key and the repository before installing MongoDB:

$ sudo rpm --import https://www.mongodb.org/static/pgp/server-4.0.asc
$ sudo zypper addrepo --gpgcheck "https://repo.mongodb.org/zypper/suse/12/mongodb-org/4.0/x86_64/" mongodb
$ sudo zypper -n install mongodb-org

In order to automatically start MongoDB on system boot, you have to activate the MongoDB service by running the following commands:

$ sudo chkconfig mongod on
$ sudo systemctl daemon-reload
$ sudo systemctl restart mongod.service
Elasticsearch

Graylog can be used with Elasticsearch 6.x, please follow the installation instructions from the Elasticsearch installation guide.

First install the Elastic GPG key with rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch then add the repository file /etc/zypp/repos.d/elasticsearch.repo with the following contents:

[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/oss-6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

followed by the installation of the latest release with sudo zypper install elasticsearch-oss.

Make sure to modify the Elasticsearch configuration file (/etc/elasticsearch/elasticsearch.yml) and set the cluster name to graylog additionally you need to uncomment (remove the # as first character) the line, and add action.auto_create_index: false to the configuration file:

cluster.name: graylog
action.auto_create_index: false

In order to automatically start Elasticsearch on system boot, you have to activate the Elasticsearch service by running the following commands:

$ sudo chkconfig elasticsearch on
$ sudo systemctl daemon-reload
$ sudo systemctl restart elasticsearch.service
Graylog

First install the Graylog GPG Key with rpm --import https://packages.graylog2.org/repo/debian/keyring.gpg then add the repository file /etc/zypp/repos.d/graylog.repo with the following content:

[graylog]
name=graylog
baseurl=https://packages.graylog2.org/repo/el/stable/3.0/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-graylog

After that, install the latest release with sudo zypper install graylog-server.

Make sure to follow the instructions in your /etc/graylog/server/server.conf and add password_secret and root_password_sha2. These settings are mandatory and without them, Graylog will not start!

You can use the following command to create your password_secret:

cat /dev/urandom | base64 | cut -c1-96 | head -1

You need to use the following command to create your root_password_sha2:

echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1

To be able to connect to Graylog you should set http_bind_address to the public host name or a public IP address of the machine you can connect to. More information about these settings can be found in Configuring the web interface.

Note

If you’re operating a single-node setup and would like to use HTTPS for the Graylog web interface and the Graylog REST API, it’s possible to use NGINX or Apache as a reverse proxy.

The last step is to enable Graylog during the operating system’s startup:

$ sudo chkconfig graylog-server on
$ sudo systemctl daemon-reload
$ sudo systemctl start graylog-server.service

The next step is to ingest messages into your new Graylog Cluster and extract the messages with extractors or use the Pipelines to work with the messages.

Cluster Setup

If you plan to have multiple servers assuming different roles in your cluster like we have in this big production setup you need to modify only a few settings. This is covered in our Multi-node Setup guide. The default file location guide lists the locations of the files you need to modify.

Feedback

Please file a bug report in the GitHub repository for the operating system packages if you run into any packaging related issues.

If you found this documentation confusing or have more questions, please open an issue in the Github repository for the documentation.

Chef, Puppet, Ansible

The DevOps movement turbocharged market adoption of the newest generation of configuration management and orchestration tools like Chef, Puppet or Ansible. Graylog offers official scripts for all three of them:

Docker

Requirements

You will need a fairly recent version of Docker.

We will use the following Docker images in this chapter:

Quick start

If you want to checkout Graylog on your local desktop without any further customization, you can run the following three commands to create the necessary environment:

$ docker run --name mongo -d mongo:3
$ docker run --name elasticsearch \
    -e "http.host=0.0.0.0" \
    -e "ES_JAVA_OPTS=-Xms512m -Xmx512m" \
    -d docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.1
$ docker run --name graylog --link mongo --link elasticsearch \
    -p 9000:9000 -p 12201:12201 -p 1514:1514 \
    -e GRAYLOG_HTTP_EXTERNAL_URI="http://127.0.0.1:9000/" \
    -d graylog/graylog:3.0

Warning

All configuration examples are created to run on the local computer. Should those be used on external servers, adjust GRAYLOG_HTTP_EXTERNAL_URI and add GRAYLOG_HTTP_PUBLISH_URI and GRAYLOG_HTTP_EXTERNAL_URI according to the server.conf documentation.

How to get log data in

You can create different kinds of inputs under System / Inputs, however you can only use ports that have been properly mapped to your docker container, otherwise data will not go through.

For example, to start a Raw/Plaintext TCP input on port 5555, stop your container and recreate it, while appending -p 5555:5555 to your docker run command:

$ docker run --link mongo --link elasticsearch \
    -p 9000:9000 -p 12201:12201 -p 1514:1514 -p 5555:5555 \
    -e GRAYLOG_HTTP_EXTERNAL_URI="http://127.0.0.1:9000/" \
    -d graylog/graylog:3.0

Similarly, the same can be done for UDP by appending -p 5555:5555/udp.

After that you can send a plaintext message to the Graylog Raw/Plaintext TCP input running on port 5555 using the following command:

$ echo 'First log message' | nc localhost 5555
Settings

Graylog comes with a default configuration that works out of the box but you have to set a password for the admin user and the web interface needs to know how to connect from your browser to the Graylog REST API.

Both settings can be configured via environment variables (also see Configuration):

-e GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
-e GRAYLOG_HTTP_EXTERNAL_URI="http://127.0.0.1:9000/"

In this case you can login to Graylog with the username and password admin.

Generate your own admin password with the following command and put the SHA-256 hash into the GRAYLOG_ROOT_PASSWORD_SHA2 environment variable:

echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1

All these settings and command line parameters can be put in a docker-compose.yml file, so that they don’t have to be executed one after the other.

Example:

version: '2'
services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongodb:
    image: mongo:3
  # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.6/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.1
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:3.0
    environment:
      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      - GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
    links:
      - mongodb:mongo
      - elasticsearch
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 1514:1514
      # Syslog UDP
      - 1514:1514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp

After starting all three Docker containers by running docker-compose up, you can open the URL http://127.0.0.1:9000 in a web browser and log in with username admin and password admin (make sure to change the password later). Change GRAYLOG_HTTP_EXTERNAL_URI= to your server IP if you run Docker remotely.

Configuration

Every configuration option can be set via environment variables. Simply prefix the parameter name with GRAYLOG_ and put it all in upper case.

For example, setting up the SMTP configuration for sending Graylog alert notifications via email, the docker-compose.yml would look like this:

version: '2'
services:
  mongo:
    image: "mongo:3"
    # Other settings [...]
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.1
    # Other settings [...]
  graylog:
    image: graylog/graylog:3.0
    # Other settings [...]
    environment:
      GRAYLOG_TRANSPORT_EMAIL_ENABLED: "true"
      GRAYLOG_TRANSPORT_EMAIL_HOSTNAME: smtp
      GRAYLOG_TRANSPORT_EMAIL_PORT: 25
      GRAYLOG_TRANSPORT_EMAIL_USE_AUTH: "false"
      GRAYLOG_TRANSPORT_EMAIL_USE_TLS: "false"
      GRAYLOG_TRANSPORT_EMAIL_USE_SSL: "false"

Another option would be to store the configuration file outside of the container and edit it directly.

Custom configuration files

Instead of using a long list of environment variables to configure Graylog (see Configuration), you can also overwrite the bundled Graylog configuration files.

The bundled configuration files are stored in /usr/share/graylog/data/config/ inside the Docker container.

Create the new configuration directory next to the docker-compose.yml file and copy the default files from GitHub:

$ mkdir -p ./graylog/config
$ cd ./graylog/config
$ wget https://raw.githubusercontent.com/Graylog2/graylog-docker/3.0/config/graylog.conf
$ wget https://raw.githubusercontent.com/Graylog2/graylog-docker/3.0/config/log4j2.xml

The newly created directory ./graylog/config/ with the custom configuration files now has to be mounted into the Graylog Docker container.

This can be done by adding an entry to the volumes section of the docker-compose.yml file:

version: '2'
services:
  mongodb:
    image: mongo:3
    # Other settings [...]
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.1
    # Other settings [...]
  graylog:
    image: graylog/graylog:3.0
    # Other settings [...]
    volumes:
      # Mount local configuration directory into Docker container
      - ./graylog/config:/usr/share/graylog/data/config

Persisting data

In order to make the recorded data persistent, you can use external volumes to store all data.

In case of a container restart, this will simply re-use the existing data from the former instances.

Using Docker volumes for the data of MongoDB, Elasticsearch, and Graylog, the docker-compose.yml file looks as follows:

version: '2'
services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongodb:
    image: mongo:3
    volumes:
      - mongo_data:/data/db
  # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.1
    volumes:
      - es_data:/usr/share/elasticsearch/data
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:3.0
    volumes:
      - graylog_journal:/usr/share/graylog/data/journal
    environment:
      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      - GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
    links:
      - mongodb:mongo
      - elasticsearch
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 1514:1514
      # Syslog UDP
      - 1514:1514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp
# Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
  mongo_data:
    driver: local
  es_data:
    driver: local
  graylog_journal:
    driver: local

Start all services with exposed data directories:

$ docker-compose up

Plugins

In order to add plugins you can build a new image based on the existing graylog/graylog Docker image with the needed plugin included or you add a volume that points to the locally downloaded plugin file.

New Docker image

Simply create a new Dockerfile in an empty directory with the following contents:

FROM graylog/graylog:3.0
RUN wget -O /usr/share/graylog/plugin/graylog-plugin-auth-sso-3.0.0.jar https://github.com/Graylog2/graylog-plugin-auth-sso/releases/download/3.0.0/graylog-plugin-auth-sso-3.0.0.jar

Build a new image from the new Dockerfile (also see docker build):

$ docker build -t graylog-with-sso-plugin .

In this example, we created a new image with the SSO plugin installed. From now on reference to the newly built image instead of graylog/graylog.

The docker-compose.yml file has to reference the new Docker image:

version: '2'
services:
  mongo:
    image: "mongo:3"
    # Other settings [...]
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.1
    # Other settings [...]
  graylog:
    image: graylog-with-sso-plugin
    # Other settings [...]
Volume-mounted plugin

Instead of building a new docker image, you can also add additional plugins by mounting them directly and individually into the plugin folder of the original Docker image. This way, you don’t have to create a new docker image every time you want to add a new plugin (or remove an old one).

Simply create a plugin folder, download the plugin(s) you want to install into it and mount each file as an additional volume into the docker container:

$ mkdir -p ./graylog/plugin
$ wget -O ./graylog/plugin/graylog-plugin-auth-sso-2.3.0.jar https://github.com/Graylog2/graylog-plugin-auth-sso/releases/download/2.3.0/graylog-plugin-auth-sso-2.3.0.jar

The docker-compose.yml file has to reference the new Docker image:

version: '2'
services:
  mongo:
    image: "mongo:3"
    # Other settings [...]
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.1
    # Other settings [...]
  graylog:
    image: graylog/graylog:3.0
    # Other settings [...]
    volumes:
      # Mount local plugin file into Docker container
      - ./graylog/plugin/graylog-plugin-auth-sso-2.3.0.jar:/usr/share/graylog/plugin/graylog-plugin-auth-sso-2.3.0.jar

You can add as many of these links as you wish in your docker-compose.yml file. Simply restart the container and docker will recreate the graylog container with the new volumes included:

$ docker-compose restart

Kubernetes automatic master selection

Running Graylog in Kubernetes opens the challenge to set the is_master=true setting only for one node in the cluster. The problem can be solved by calculating the name of the pod if Graylog is running in a stafeful set with the following environment variable:

env:
- name: POD_NAME
  valueFrom:
    fieldRef:
      fieldPath: metadata.name

For a stateful set, the name of the first pod in a cluster always ends with -0. See the Documentation about stateful set . The master selection mechanism in docker-entrypoint.sh file does the following:

  • Examine if Graylog is running inside Kubernetes
  • Verify that the pod name ends in -0
  • Set is_master=true for this container

Troubleshooting

  • In case you see warnings regarding open file limit, try to set ulimit from the outside of the container:

    $ docker run --ulimit nofile=64000:64000 ...
    
  • The devicemapper storage driver can produce problems with Graylogs disk journal on some systems. In this case please pick another driver like aufs or overlay.

Testing a beta version

Caution

We only recommend running pre-release versions if you are an experienced Graylog user and know what you are doing.

You can also run a pre-release (alpha, beta, or release candidate) version of Graylog using Docker.

The pre-releases are tagged in the graylog/graylog Docker image.

Follow the documentation for the Graylog image on Docker Hub and pick an alpha/beta/rc tag like this:

$ docker run --link mongo --link elasticsearch -p 9000:9000 -p 12201:12201 -p 1514:1514 \
    -e GRAYLOG_HTTP_BIND_ADDRESS="127.0.0.1:9000" \
    -d graylog/graylog:3.0.0-beta.3-1

Amazon Web Services

AMIs

Select your AMI and AWS Region.

Usage

  • Click on Launch instance for your AWS region to start Graylog into.
  • Choose an instance type with at least 4GB memory.
  • Finish the wizard and spin up the VM.
  • Open port 80 and 22 in the applied security group to access the web interface.
  • Login to the instance via SSH as user ubuntu to see web login credentials.
  • additionally open more ports for ingesting log data, like 514 for syslog or 12201 for the GELF protocol.

Open http://<private ip> in your browser to access the Graylog web interface. Default username is admin with the password shown on the first SSH login.

Networking

Your browser needs access to port 80 for reaching the web interface. Make sure that a security group is opening that port. On the appliance a NGINX instance is used as proxy to simplify network access. Take a look in the configuration /etc/nginx/sites-available/default for further fine tuning.

HTTPS

In order to enable HTTPS for the web interface port 443 needs also be open. The configuration can be done with NGINX as well. See Using HTTPS for a full reference.

Basic configuration

The Graylog appliance is based on Ubuntu system packages so all configuration changes can be done analog to the rest of this documentation. See Configuring Graylog

Production readiness

The Graylog appliance is not created to provide a production ready solution. It is build to offer a fast and easy way to try the software itself and not wasting time to install Graylog and it components to any kind of server.

Microsoft Windows

Unfortunately there is no supported way to run Graylog on Microsoft Windows operating systems even though all parts run on the Java Virtual Machine. We recommend to run the virtual machine appliances on a Windows host. It should be technically possible to run Graylog on Windows but it is most probably not worth the time to work your way around the cliffs.

Should you require running Graylog on Windows, you need to disable the message journal in graylog-server by changing the following setting in the graylog.conf:

message_journal_enabled = false

Due to restrictions of how Windows handles file locking the journal will not work correctly.

Please note that this impacts Graylog’s ability to buffer messages, so we strongly recommend running Graylog on Linux. Consider a Linux virtual machine on a Windows host. Graylog setups on Windows are no fun and not officially supported.

Manual Setup

Graylog server on Linux

Prerequisites

Graylog depends on MongoDB and Elasticsearch to operate, please refer to the system requirements for details.

Downloading and extracting the server

Download the tar archive from the download pages and extract it on your system:

~$ tar xvfz graylog-VERSION.tgz
~$ cd graylog-VERSION
Configuration

Now copy the example configuration file:

~# cp graylog.conf.example /etc/graylog/server/server.conf

You can leave most variables as they are for a first start. All of them should be well documented.

Configure at least the following variables in /etc/graylog/server/server.conf:

  • is_master = true
    • Set only one graylog-server node as the master. This node will perform periodical and maintenance actions that slave nodes won’t. Every slave node will accept messages just as the master nodes. Nodes will fall back to slave mode if there already is a master in the cluster.
  • password_secret
    • You must set a secret that is used for password encryption and salting here. The server will refuse to start if it’s not set. Generate a secret with for example pwgen -N 1 -s 96. If you run multiple graylog-server nodes, make sure you use the same password_secret for all of them!
  • root_password_sha2
    • A SHA2 hash of a password you will use for your initial login. Set this to a SHA2 hash generated with echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1 and you will be able to log in to the web interface with username admin and password yourpassword.
  • elasticsearch_hosts
    • List of Elasticsearch hosts Graylog should connect to.
  • mongodb_uri
    • Enter your MongoDB connection and authentication information here.
Starting the server

You need to have Java installed. Running the OpenJDK is totally fine and should be available on all platforms. For example on Debian it is:

~$ apt-get install openjdk-8-jre

Start the server:

~$ cd bin/
~$ ./graylogctl start

The server will try to write a node_id to the graylog-server-node-id file. It won’t start if it can’t write there because of for example missing permissions.

See the startup parameters description below to learn more about available startup parameters. Note that you might have to be root to bind to the popular port 514 for syslog inputs.

You should see a line like this in the debug output of Graylog successfully connected to your Elasticsearch cluster:

2013-10-01 12:13:22,382 DEBUG: org.elasticsearch.transport.netty - [graylog-server] connected to node [[Unuscione, Angelo][thN_gIBkQDm2ab7k-2Zaaw][inet[/10.37.160.227:9300]]]

You can find the logs of Graylog in the directory logs/.

Important: All systems running Graylog must have synchronised system time. We strongly recommend to use NTP or similar mechanisms on all machines of your Graylog infrastructure.

Supplying external logging configuration

Graylog is using Apache Log4j 2 for its internal logging and ships with a default log configuration file which is embedded within the shipped JAR.

In case you need to modify Graylog’s logging configuration, you can supply a Java system property specifying the path to the configuration file in your start script (e. g. graylogctl).

Append this before the -jar parameter:

-Dlog4j.configurationFile=file:///path/to/log4j2.xml

Substitute the actual path to the file for the /path/to/log4j2.xml in the example.

In case you do not have a log rotation system already in place, you can also configure Graylog to rotate logs based on their size to prevent the log files to grow without bounds using the RollingFileAppender.

One such example log4j2.xml configuration is shown below:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="org.graylog2.log4j" shutdownHook="disable">
  <Appenders>
      <RollingFile name="RollingFile" fileName="/tmp/logs/graylog.log"
                   filePattern="/tmp/logs/graylog-%d{yyyy-MM-dd}.log.gz">
        <PatternLayout>
          <Pattern>%d %-5p: %c - %m%n</Pattern>
        </PatternLayout>
        <!-- Rotate logs every day or when the size exceeds 10 MB (whichever comes first) -->
        <Policies>
          <TimeBasedTriggeringPolicy modulate="true"/>
          <SizeBasedTriggeringPolicy size="10 MB"/>
        </Policies>
        <!-- Keep a maximum of 10 log files -->
        <DefaultRolloverStrategy max="10"/>
      </RollingFile>

      <Console name="STDOUT" target="SYSTEM_OUT">
          <PatternLayout pattern="%d %-5p: %c - %m%n"/>
      </Console>

      <!-- Internal Graylog log appender. Please do not disable. This makes internal log messages available via REST calls. -->
      <Memory name="graylog-internal-logs" bufferSize="500"/>
  </Appenders>
  <Loggers>
      <Logger name="org.graylog2" level="info"/>
      <Logger name="com.github.joschi.jadconfig" level="warn"/>
      <Logger name="org.apache.directory.api.ldap.model.message.BindRequestImpl" level="error"/>
      <Logger name="org.elasticsearch.script" level="warn"/>
      <Logger name="org.graylog2.periodical.VersionCheckThread" level="off"/>
      <Logger name="com.joestelmach.natty.Parser" level="warn"/>
      <Logger name="kafka.log.Log" level="warn"/>
      <Logger name="kafka.log.OffsetIndex" level="warn"/>
      <Logger name="org.apache.shiro.session.mgt.AbstractValidatingSessionManager" level="warn"/>
      <Root level="warn">
          <AppenderRef ref="STDOUT"/>
          <AppenderRef ref="RollingFile"/>
          <AppenderRef ref="graylog-internal-logs"/>
      </Root>
  </Loggers>
</Configuration>
Command line (CLI) parameters

There are a number of CLI parameters you can pass to the call in your graylogctl script:

  • -h, --help: Show help message
  • -f CONFIGFILE, --configfile CONFIGFILE: Use configuration file CONFIGFILE for Graylog; default: /etc/graylog/server/server.conf
  • -d, --debug: Run in debug mode
  • -l, --local: Run in local mode. Automatically invoked if in debug mode. Will not send system statistics, even if enabled and allowed. Only interesting for development and testing purposes.
  • -p PIDFILE, --pidfile PIDFILE: Set the file containing the PID of graylog to PIDFILE; default: /tmp/graylog.pid
  • -np, --no-pid-file: Do not write PID file (overrides -p/--pidfile)
  • --version: Show version of Graylog and exit
Problems with IPv6 vs. IPv4?

If your Graylog node refuses to listen on IPv4 addresses and always chooses for example a http_bind_address like :::9000 you can tell the JVM to prefer the IPv4 stack.

Add the java.net.preferIPv4Stack flag in your graylogctl script or from wherever you are calling the graylog.jar:

~$ sudo -u graylog java -Djava.net.preferIPv4Stack=true -jar graylog.jar
Create a message input and send a first message

Log in to the web interface on port 9000 (e.g. http://127.0.0.1:9000) and navigate to System -> Inputs.

_images/create_input.png

Launch a new Raw/Plaintext UDP input, listening on 127.0.0.1 on port 9099. There’s no need to configure anything else for now.

The list of running inputs on that node should show you your new input right away.

Let’s send a message in:

echo "Hello Graylog, let's be friends." | nc -w 1 -u 127.0.0.1 9099

This has sent a short string to the raw UDP input you just opened. Now search for friends using the search bar on the top and you should already see the message you just sent in. Click on it in the table and see it in detail:

_images/setup_1.png

You have just sent your first message to Graylog! Why not spawn a syslog input and point some of your servers to it? You could also create some user accounts for your colleagues.

System requirements

The Graylog server application has the following prerequisites:

  • Some modern Linux distribution (Debian Linux, Ubuntu Linux, or CentOS recommended)
  • Elasticsearch 5 or 6
  • MongoDB 3.6 or later (latest stable version is recommended)
  • Oracle Java SE 8 (OpenJDK 8 also works; latest stable update is recommended)

Caution

Graylog prior to 2.3 does not work with Elasticsearch 5.x!

Caution

Graylog 3 does not work with Elasticsearch 7.x!

Upgrading Graylog

When upgrading from a previous version of Graylog you follow the previous used installation method (ex. from image or package) using the new version numbers.

The following Upgrade notes should be read carefully before you start the upgrade process. Breaking changes and dependency upgrades are documented in those upgrade notes.

You should always follow minor versions when updating across multiple versions to make sure necessary migrations are run correctly. The upgrade notes are always written coming from the stable release before.

Upgrading to Graylog 2.0.x

Elasticsearch 2.x

The embedded Elasticsearch node being used by Graylog has been upgraded to Elasticsearch 2.x which includes some breaking changes. Graylog 2.x does not work with Elasticsearch 1.x anymore and cannot communicate with existing Elasticsearch 1.x clusters.

Please see Breaking changes in Elasticsearch 2.x for details.

The blog article Key points to be aware of when upgrading from Elasticsearch 1.x to 2.x also contains interesting information about the upgrade path from Elasticsearch 1.x to 2.x.

Multicast Discovery

Multicast discovery has been removed from Elasticsearch 2.x (although it is still provided as an Elasticsearch plugin for now).

To reflect this change, the elasticsearch_discovery_zen_ping_unicast_hosts now has to contain the address of at least one Elasticsearch node in the cluster which Graylog can connect to.

Default network host

The network interface which Elasticsearch binds to (elasticsearch_network_host) has been changed to localhost (i. e. 127.0.0.1 or ::1); see Network changes/Bind to localhost.

If Elasticsearch is not running on the same machine, elasticsearch_network_host must be set to a host name or an IP address which can be accessed by the other Elasticsearch nodes in the cluster.

Index range types

Note

This step needs to be performed before the update to Elasticsearch 2.x!

Some Graylog versions stored meta information about indices in elasticsearch, alongside the messages themselves. Since Elasticsearch 2.0 having multiple types with conflicting mappings is no longer possible, which means that the index_range type must be removed before upgrading to Elasticsearch 2.x.

Find out if your setup is affected by running (replace $elasticsearch with the address of one of your Elasticsearch nodes) curl -XGET $elasticsearch:9200/_all/_mapping/index_range; echo

If the output is {} you are not affected and can skip this step.

Otherwise, you need to delete the index_range type, Graylog does not use it anymore.

As Graylog sets older indices to read-only, first we need to remove the write block on those indices. Since we’ll be working with Elasticsearch’s JSON output, we recommend installing the jq utility which should be available on all popular package managers or directly at GitHub.

for i in `curl -s -XGET $elasticsearch:9200/_all/_mapping/index_range | jq -r "keys[]"`; do
    echo -n "Updating index $i: "
    echo -n "curl -XPUT $elasticsearch:9200/$i/_settings -d '{\"index.blocks.read_only\":false, \"index.blocks.write\":false}' : "
    curl -XPUT $elasticsearch:9200/$i/_settings -d '{"index.blocks.read_only":false, "index.blocks.write":false}'
    echo
done

The output for each of the curl commands should be {"acknowledged":true}. Next we have to delete the index_range mapping. We can perform this via the next command.

Note

We strongly recommend to perform this on a single index before running this bulk command. This operation can be expensive to perform if you have a lot of affected indices.

for i in `curl -s -XGET $elasticsearch:9200/_all/_mapping/index_range | jq -r "keys[]"`; do
    echo -n "Updating index $i: "
    curl -XDELETE $elasticsearch:9200/$i/index_range
    echo
done

It is not strictly necessary to set the indices back to read only, but if you prefer to do that, note the index names and commands during the first step and change the false into true.

Graylog Index Template

Graylog applies a custom index template to ensure that the indexed messages adhere to a specific schema.

Unfortunately the index template being used by Graylog 1.x is incompatible with Elasticsearch 2.x and has to be removed prior to upgrading.

In order to delete the index template the following curl command has to be issued against on of the Elasticsearch nodes:

curl -X DELETE http://localhost:9200/_template/graylog-internal

Graylog will automatically create the new index template on the next startup.

Dots in field names

One of the most important breaking changes in Elasticsearch 2.x is that field names may not contain dots anymore.

Using the Elasticsearch Migration Plugin might help to highlight some potential pitfalls if an existing Elasticsearch 1.x cluster should be upgraded to Elasticsearch 2.x.

MongoDB

Graylog 2.x requires MongoDB 2.4 or newer. We recommend using MongoDB 3.x and the WiredTiger storage engine.

When upgrading from MongoDB 2.0 or 2.2 to a supported version, make sure to read the Release Notes for the particular version.

Log4j 2 migration

Graylog switched its logging backend from Log4j 1.2 to Log4j 2.

Please refer to the Log4j Migration Guide for information on how to update your existing logging configuration.

Dead Letters feature removed

The Dead Letters feature, which stored messages that couldn’t be indexed into Elasticsearch for various reasons, has been removed.

This feature has been disabled by default. If you have enabled the feature the configuration file, please check the dead_letters_enabled collection in MongoDB and remove it afterwards.

Removed configuration settings

Index Retention and Rotation Settings

In 2.0.0 the index rotation and retention settings have been moved from the Graylog server config file to the database and are now configurable via the web interface.

The old settings from the graylog.conf or /etc/graylog/server/server.conf will be migrated to the database.

Warning

When you upgrade from a 1.x version and you modified any rotation/retention settings, please make sure you KEEP your old settings in the config file so the migration process will add your old settings to the database! Otherwise the retention process will use the default settings and might remove a lot of indices.

Overview

Some settings, which have been deprecated in previous versions, have finally been removed from the Graylog configuration file.

Removed configuration settings
Setting name Replacement
mongodb_host mongodb_uri
mongodb_port mongodb_uri
mongodb_database mongodb_uri
mongodb_useauth mongodb_uri
mongodb_user mongodb_uri
mongodb_password mongodb_uri
elasticsearch_node_name elasticsearch_node_name_prefix
collector_expiration_threshold (moved to collector plugin)
collector_inactive_threshold (moved to collector plugin)
rotation_strategy UI in web interface (System/Indices)
retention_strategy UI in web interface (System/Indices)
elasticsearch_max_docs_per_index UI in web interface (System/Indices)
elasticsearch_max_size_per_index UI in web interface (System/Indices)
elasticsearch_max_time_per_index UI in web interface (System/Indices)
elasticsearch_max_number_of_indices UI in web interface (System/Indices)
dead_letters_enabled None

Changed configuration defaults

For better consistency, the defaults of some configuration settings have been changed after the project has been renamed from Graylog2 to Graylog.

Configuration defaults
Setting name Old default New default
elasticsearch_cluster_name graylog2 graylog
elasticsearch_node_name graylog2-server graylog-server
elasticsearch_index_prefix graylog2 graylog
elasticsearch_discovery_zen_ping_unicast_hosts empty 127.0.0.1:9300
elasticsearch_discovery_zen_ping_multicast_enabled true false
mongodb_uri mongodb://127.0.0.1/graylog2 mongodb://localhost/graylog

Changed prefixes for configuration override

In the past it was possible to override configuration settings in Graylog using environment variables or Java system properties with a specific prefix.

For better consistency, these prefixes have been changed after the project has been renamed from Graylog2 to Graylog.

Configuration override prefixes
Override Old prefix New prefix Example
Environment variables GRAYLOG2_ GRAYLOG_ GRAYLOG_IS_MASTER
System properties graylog2. graylog. graylog.is_master

REST API Changes

The output ID key for the list of outputs in the /streams/* endpoints has been changed from _id to id.

 {
   "id": "564f47c41ec8fe7d920ef561",
   "creator_user_id": "admin",
   "outputs": [
     {
       "id": "56d6f2cce45e0e52d1e4b9cb", // ==> Changed from `_id` to `id`
       "title": "GELF Output",
       "type": "org.graylog2.outputs.GelfOutput",
       "creator_user_id": "admin",
       "created_at": "2016-03-02T14:03:56.686Z",
       "configuration": {
         "hostname": "127.0.0.1",
         "protocol": "TCP",
         "connect_timeout": 1000,
         "reconnect_delay": 500,
         "port": 12202,
         "tcp_no_delay": false,
         "tcp_keep_alive": false,
         "tls_trust_cert_chain": "",
         "tls_verification_enabled": false
       },
       "content_pack": null
     }
   ],
   "matching_type": "AND",
   "description": "All incoming messages",
   "created_at": "2015-11-20T16:18:12.416Z",
   "disabled": false,
   "rules": [],
   "alert_conditions": [],
   "title": "ALL",
   "content_pack": null
 }

Web Interface Config Changes

The web interface has been integrated into the Graylog server and was rewritten in React. Therefore configuring it has changed fundamentally since the last version(s). Please consult Web interface for details.

Please take note that the application.context configuration parameter present in Graylog 1.x (and earlier) is not existing anymore. The web interface can currently only be served without a path prefix.

Upgrading to Graylog 2.1.x

HTTPS Setup

Previous versions of Graylog were automatically generating a private key/certificate pair for HTTPS if either the private key or the certificate (or both) for rest_tls_key_file, rest_tls_cert_file, web_tls_key_file, or web_tls_cert_file couldn’t be read. While this feature is very comfortable for inexperienced users, it has lots of serious drawbacks like very weak key sizes (only 1024 bits), being untrusted by all TLS libraries used by web browsers and other client software (because they are self-signed and not included in the system’s CA/trust store), and problems with inter-node communications with other Graylog nodes.

Due to those shortcomings, the feature has been removed completely. Users need to use proper certificates or generate their own self-signed certificates and configure them with the appropriate settings, see Using HTTPS for reference.

Web Interface Listener

Graylog 2.0.x has been using separate listeners for the REST API and the web interface by default. The Graylog REST API on http://127.0.0.1:12900, the Graylog web interface on http://127.0.0.1:9000. Beginning with Graylog 2.1.0 it is possible to run both the REST API and the web interface on the same host/port-combination and this is now the default. This means that the REST API is now running on http://127.0.0.1:9000/api/ by default and the web interface is now running on http://127.0.0.1:9000/. Furthermore, all requests going to http://127.0.0.1:9000/api/ requesting a content-type of text/html or application/xhtml+xml are redirected to the web interface, therefore making it even easier to set up Graylog and use it behind proxies, expose it externally etc.

Please take note that you can still run the REST API and the web interface on two separate listeners. If you are running a Graylog 2.0.x configuration specifying web_listen_uri explicitly and you want to keep that, you do not have to change anything.

Please also take note, that when you have configured rest_listen_uri and web_listen_uri to run on the same host/port-combination, the following configuration directives will have no effect:

  • web_enable_tls, web_tls_cert_file, web_tls_key_file, web_tls_key_password (These will depend on the TLS configuration of the REST listener).
  • web_enable_cors, web_enable_gzip, web_thread_pool_size, web_max_initial_line_length, web_max_header_size (Those will depend on the corresponding settings of the REST listener).

Internal Metrics to MongoDB

Previous versions of Graylog included a (long deprecated) metrics reporter for writing internal metrics into MongoDB in a fixed interval of 1 second.

This feature has been removed completely and can be optionally pulled in by using the Graylog Metrics Reporter Plugins.

Configuration file changes

Network settings

The network settings in the Graylog configuration file (rest_listen_uri, rest_transport_uri, and web_listen_uri) are now using the default ports for the HTTP (80) and HTTPS (443) if no custom port was given. Previously those settings were using the custom ports 12900 (Graylog REST API) and 9000 (Graylog web interface) if no explicit port was given.

Examples:

Configuration setting Old effective URI New effective URI
rest_listen_uri = http://127.0.0.1:12900/ http://127.0.0.1:12900/ http://127.0.0.1:12900/
rest_listen_uri = http://127.0.0.1/ http://127.0.0.1:12900/ http://127.0.0.1:80/
rest_listen_uri = https://127.0.0.1/ https://127.0.0.1:12900/ https://127.0.0.1:443/
Collector Sidecar

The network changes are reflected in the Sidecar configuration as well and should be adopted. However it’s still possible to use the old API port by setting it explicitly. In case a mass deployment is too hard to change, just run the following to switch back to the old REST API port (OVA based installation):

sudo graylog-ctl set-listen-address --service rest --address http://0.0.0.0:12900
sudo graylog-ctl reconfigure

Graylog REST API

Removed resources
Original resource Replacement
/system/buffers /system/metrics/org.graylog2.buffers.input.size
/system/metrics/org.graylog2.buffers.input.usage
/system/metrics/org.graylog2.buffers.process.size
/system/metrics/org.graylog2.buffers.process.usage
/system/metrics/org.graylog2.buffers.output.size
/system/metrics/org.graylog2.buffers.output.usage
/system/buffers/classes None
Removed index rotation/retention settings from “/system/configuration”

The index rotation and retention settings have been moved to MongoDB in Graylog 2.0.0 but the representation of the old configuration options was still present in the /system/configuration resource.

In order to stay in sync with the actual configuration file, the following values have been removed:

  • rotation_strategy
  • retention_strategy
  • elasticsearch_max_docs_per_index
  • elasticsearch_max_size_per_index
  • elasticsearch_max_time_per_index
  • elasticsearch_max_number_of_indices

The retention and rotation configuration settings can be retrieved using the following resources:

  • /system/indices/rotation/config
  • /system/indices/retention/config

For Plugin Authors

Between Graylog 2.0.x and 2.1.0 we also made changes to the Plugin API. These include:

  • Removing org.graylog2.plugin.streams.Stream#getAlertCondition, as it was faulty and not easily replaceable with a working version without breaking our separation of models and persistence services.

If you are maintaining a plugin that was originally written for Graylog 1.x or 2.0.x, you need to make sure that your plugin is still compiling and working under Graylog 2.1.x or adapt it if necessary.

UI Plugins

The new app prefix feature requires some changes in UI plugins to make them work with that.

  • import webpackEntry from 'webpack-entry'; needs to be added at the very top of the src/web/index.jsx file
  • The Routes.pluginRoute() function needs to be used instead of a literal string to build URLs for links and buttons

Please check the updated plugins documentation for details.

Changed Elasticsearch Cluster Status Behavior

In previous versions Graylog stopped indexing into the current write index if the Elasticsearch cluster status turned RED. Since Graylog 2.1.0 only checks the status of the current write index when it tries to index messages.

If the current write index is GREEN or YELLOW, Graylog will continue to index messages even though the overall cluster status is RED. This avoids Graylog downtimes when doing Elasticsearch maintenance or when older indices have problems.

Changes in message field values trimming

Previous versions of Graylog were trimming message field values inconsistently, depending on the codec used. We have changed that behaviour in Graylog 2.1.0, so all message field values are trimmed by default. This means that leading or trailing whitespace of every field is removed during ingestion.

Important: This change will break your existing stream rules, extractors, and Drool rules if you are expecting leading or trailing white spaces in them. Please adapt them so they do not require those white spaces.

Upgrading to Graylog 2.2.x

Email Alarm Callback

Previous versions of Graylog created an implicit email alarm callback if no explicit callback existed for a stream.

Due to the extensive rework done in alerting, this behavior has been modified to be explicit, and more consistent with other entities within Graylog: from now on there will not be a default alarm callback.

To simplify the transition for people relying on this behavior, we have added a migration step that will create an email alarm callback for each stream that has alert conditions, has alert receivers, but has no associated alarm callbacks.

With to the introduction of email templates in 0.21, the transport_email_subject_prefix config setting became unused. It is now being removed completely. In early versions it was used to add a prefix to the generated subject of alerting emails. Since 0.21 it is possible to define a complete template used for the generation of alert email subjects.

Alert Notifications (previously known as Alarm Callbacks)

Graylog 2.2.0 introduces some changes in alerting. Alerts have now states, helping you to know in an easier way if something requires your attention.

These changes also affect the way we send notifications: Starting in Graylog 2.2.0, alert notifications are only executed once, just when a new alert is triggered. As long as the alert is unresolved or in grace period, Graylog will not send further notifications. This will help you reducing the noise and annoyance of getting notified way too often when a problem persists for a while.

If you are using Graylog for alerting, please take a moment to ensure this change will not break any of your processes when an alert occurs.

Default stream/Index Sets

With the introduction of index sets, and the ability to change a stream’s write target, the default stream needs additional information, which is calculated when starting a new Graylog 2.2 master node.

It requires recalculation of the index ranges of the default stream’s index set, which when updating from pre-2.2 versions is stored in the graylog_ index. This is potentially expensive, because it has to calculate three aggregations across every open index to detect which streams are stored in which index.

Please be advised that this necessary migration can put additional load on your cluster.

Warning

Make sure that all rotation and retention strategy plugins you had installed in 2.1 are updated to a version that is compatible with 2.2 before you start the Graylog 2.2 version for the first time. (e.g. Graylog Enterprise) This is needed so the required data migrations will run without problems.

Warning

The option to remove a message from the default stream is currently not available when using the pipeline function route_to_stream. This will be fixed in a subsequent bug fix release. Please see the corresponding Github issue.

RotationStrategy & RetentionStrategy Interfaces

The Java interfaces for RetentionStrategy and RotationStrategy changed in 2.2. The #rotate() and #retain() methods are now getting an IndexSet as first parameter.

This only affects you if you are using custom rotation or retention strategies.

Changes in Exposed Configuration

The exposed configuration settings on the /system/configuration resource of the Graylog REST API doesn’t contain the following (deprecated) Elasticsearch-related settings anymore:

  • elasticsearch_shards
  • elasticsearch_replicas
  • index_optimization_max_num_segments
  • disable_index_optimization

Changes in Split & Count Converter

The behavior of the split & count converter has been changed to that it resembles typical split() functions.

Previously, the split & count converter returned 0, if the split pattern didn’t occur in the string. Now it will return 1.

Examples:

String Split Pattern Old Result | New Result
<empty> - 0 0
foo - 0 1
foo-bar - 2 2

Graylog REST API

Streams API

Due to the introduction of index sets, the payload for creating, updating and cloning of streams now requires the index_set_id field. The value for this needs to be the ID of an existing index set.

Affected endpoints:

  • POST /streams
  • PUT  /streams/{streamId}
  • POST /streams/{streamId}/clone

Upgrading to Graylog 2.3.x

Graylog switches to Elasticsearch HTTP client

In all prior versions, Graylog used the Elasticsearch node client to connect to an Elasticsearch cluster, which was acting as a client-only Elasticsearch node. For compatibility reasons of the used binary transfer protocol, the range of Elasticsearch versions Graylog could connect to was limited. For more information and differences between the different ways to connect to Elasticsearch, you can check the Elasticsearch documentation.

Starting with version 2.3.0, we are switching over to using a lightweight HTTP client, which is almost version-agnostic. The biggest change is that it does not connect to the Elasticsearch native protocol port (defaulting to 9300/tcp), but the Elasticsearch HTTP port (defaulting to 9200/tcp).

Due to the differences in connecting to the Elasticsearch cluster, configuring Graylog has changed. These configuration settings have been removed:

elasticsearch_cluster_discovery_timeout
elasticsearch_cluster_name
elasticsearch_config_file
elasticsearch_discovery_initial_state_timeout
elasticsearch_discovery_zen_ping_unicast_hosts
elasticsearch_http_enabled
elasticsearch_network_bind_host
elasticsearch_network_host
elasticsearch_network_publish_host
elasticsearch_node_data
elasticsearch_node_master
elasticsearch_node_name_prefix
elasticsearch_path_data
elasticsearch_path_home
elasticsearch_transport_tcp_port

The following configuration options are now being used to configure connectivity to Elasticsearch:

Config Setting Type Comments Default
elasticsearch_connect_timeout Duration Timeout when connection to individual Elasticsearch hosts 10s (10 Seconds)
elasticsearch_hosts List<URI> Comma-separated list of URIs of Elasticsearch hosts http://127.0.0.1:9200
elasticsearch_idle_timeout Duration Timeout after which idle connections are terminated -1s (Never)
elasticsearch_max_total_connections int Maximum number of total Elasticsearch connections 20
elasticsearch_max_total_connections_per_route int Maximum number of Elasticsearch connections per route/host 2
elasticsearch_max_retries int Maximum number of retries for requests to Elasticsearch 2
elasticsearch_socket_timeout Duration Timeout when sending/receiving from Elasticsearch connection 60s (60 Seconds)
elasticsearch_discovery_enabled boolean Enable automatic Elasticsearch node discovery false
elasticsearch_discovery_filter String Filter by node attributes for the discovered nodes empty (use all nodes)
elasticsearch_discovery_frequency Duration Frequency of the Elasticsearch node discovery 30s (30 Seconds)

In most cases, the only configuration setting that needs to be set explicitly is elasticsearch_hosts. All other configuration settings should be tweaked only in case of errors.

Warning

The automatic node discovery does not work if Elasticsearch requires authentication, e. g. when using Shield (X-Pack).

Caution

Graylog does not react to externally triggered index changes (creating/closing/reopening/deleting an index) anymore. All of these actions need to be performed through the Graylog REST API in order to retain index consistency.

Special note for upgrading from an existing Graylog setup with a new Elasticsearch cluster

If you are upgrading the Elasticsearch cluster of an existing Graylog setup without migrating the indices, your Graylog setup contains stale index ranges causing nonexisting index errors upon search/alerting. To remediate this, you need to manually trigger an index range recalculation for all index sets once. This is possible using the web interface using the System->Indices functionality or by using the REST API using the /system/indices/ranges/<index set id>/rebuild endpoint.

Graylog REST API

Rotation and Retention strategies

The deprecated HTTP resources at /system/indices/rotation/config and /system/indices/retention/config, which didn’t work since Graylog 2.2.0, have been removed.

These settings are part of the index set configuration and can be configured under /system/indices/index_sets.

Stream List Response structure does not include in_grace field anymore

The response to GET /streams, GET /streams/<id> & PUT /streams/<id> does not contain the in_grace field for configured alert conditions anymore.

The value of this flag can be retrieved using the GET /alerts/conditions endpoint, or per stream using the GET /streams/<streamId>/alerts/conditions endpoint.

Upgrading to Graylog 2.4.x

You can upgrade from Graylog 2.3.x to Graylog 2.4.x without the need to change the configuration of your Graylog server.

More plugins shipped by default

The following Graylog plugins are now shipped as part of the Graylog server release.

Warning

Make sure you remove all previous versions of these plugins from your plugin/ folder before starting the new Graylog version!

Upgrading to Graylog 2.5.x

Protecting against CSRF, HTTP header required

Using the Graylog server API requires all clients sending non-GET requests to include a custom HTTP header (X-Requested-By). The value of the header is not important, but it’s presence is, as all requests without it will be ignored and will return a 400 error.

This is important for people using scripts that modify Graylog in any way through the REST API. We already adapted Graylog web interface and our plugins, so if you don’t use any scripts or 3rd party products to access Graylog, you don’t have to do anything else.

If you are using the Graylog Sidecar, you either have to use Graylog version 2.5.1 or update the Sidecar to version 0.1.7. That version is using the correct CSRF headers for HTTP requests against the Graylog server API.

Elasticsearch 6 changes

There is a breaking change in Elasticsearch 6 that may affect some queries on your searches and dashboards:

Before Elasticsearch 6, queries for keyword fields were split by whitespaces and combined with OR operators resulting, for example, in type:(ssh login) and type:(ssh OR login) being equivalent. This is no longer the case in Elasticsearch 6 and now those queries are different: the former looking for the ssh login value, the second for either ssh or login values.

Please ensure to look for those queries in your Graylog setup before upgrading to Elasticsearch 6 and add the OR operators where needed.

Upgrading to Graylog 3.0.x

Elasticsearch Version Requirements

Graylog 3.0 drops support for Elasticsearch versions before 5.6.x. That means you have to upgrade Elasticsearch to at least version 5.6.13 before upgrading Graylog to version 3.0. Make sure to read the Elasticsearch upgrade guides before doing that.

Simplified HTTP interface configuration

Graylog used to have a lot of different settings regarding the various HTTP interfaces it provides, namely the Graylog REST API and the Graylog web interface.

This mostly originates from the fact that Graylog used to consist of two components before Graylog 2.0.0, a server component and a separate web interface.

The changes in this release finally merge the HTTP listeners for the Graylog REST API and web interface into a single HTTP listener, which should make the initial configuration of Graylog simpler and reduce errors caused by conflicting settings.

The path of the Graylog REST API is now hard-coded to /api, so if you’re still using the legacy URI on port 12900/tcp or have been using a custom path (via the rest_listen_uri or rest_transport_uri settings), you’ll have to update the URI used to access the Graylog REST API.

This might also affect your Graylog Collector Sidecars. Make sure to check each collector_sidecar.yml and update the server_url accordingly.

If you are using a reverse proxy in front of Graylog (like nginx) and configured it to set the X-Graylog-Server-URL HTTP header, you have to remove the api/ suffix because that is now the default. (as mentioned above)

Example:

# This nginx setting in Graylog <3.0 ...
header_upstream X-Graylog-Server-URL http://{host}/api

# ... needs to be changed to the following with Graylog 3.0
header_upstream X-Graylog-Server-URL http://{host}/

For a more detailed description of the new HTTP settings, please consult the annotated Graylog configuration file.

Overview of removed Graylog REST API settings:

Removed Setting New Setting Default
rest_listen_uri http_bind_address 127.0.0.1:9000
rest_transport_uri http_publish_uri http://$http_bind_address/
web_endpoint_uri http_external_uri $http_publish_uri
rest_enable_cors http_enable_cors true
rest_enable_gzip http_enable_gzip true
rest_max_header_size http_max_header_size 8192
rest_max_initial_line_length http_max_initial_line_length 4096
rest_thread_pool_size http_thread_pool_size 16
rest_enable_tls http_enable_tls false
rest_tls_cert_file http_tls_cert_file Empty
rest_tls_key_file http_tls_key_file Empty
rest_tls_key_password http_tls_key_password Empty

Overview of removed Graylog web interface settings:

Removed Setting New Setting Default
web_enable None  
web_listen_uri http_bind_address 127.0.0.1:9000
web_enable_cors http_enable_cors true
web_enable_gzip http_enable_gzip true
web_max_header_size http_max_header_size 8192
web_max_initial_line_length http_max_initial_line_length 4096
web_thread_pool_size http_thread_pool_size 16
web_enable_tls http_enable_tls false
web_tls_cert_file http_tls_cert_file Empty
web_tls_key_file http_tls_key_file Empty
web_tls_key_password http_tls_key_password Empty

Plugins merged into the Graylog server

Starting with Graylog 3.0.0, the following official plugins were merged into the Graylog server:

That means these plugins are not available as separate plugins anymore. If you manually update your Graylog installation (without using operating system packages), make sure to remove all old plugin files from the plugin_dir folder.

The old issues in these repositories are still available for reference but new issues should only be created in the Graylog server issue tracker.

The following HTTP API paths changed due to the plugin merge:

Old Path New Path
/plugins/org.graylog.plugins.map/mapdata /search/mapdata
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/pipeline /system/pipelines/pipeline
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/pipeline/parse /system/pipelines/pipeline/parse
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/rule /system/pipelines/rule
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/rule/functions /system/pipelines/rule/functions
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/rule/multiple /system/pipelines/rule/multiple
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/rule/parse /system/pipelines/rule/parse
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/connections /system/pipelines/connections
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/connections/to_stream /system/pipelines/connections/to_stream
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/connections/to_pipeline /system/pipelines/connections/to_pipeline
/plugins/org.graylog.plugins.pipelineprocessor/system/pipelines/simulate /system/pipelines/simulate

New “bin_dir” and “data_dir” configuration parameters

We introduced two new configuration parameters related to file system paths.

  • bin_dir config option points to the directory that contains scripts like graylogctl.
  • data_dir option configures the base directory for Graylog server state.

Please check the updated default graylog.conf configuration file for required changes to your existing file.

Removed support for Drools-based filters

For a long time, Graylog allowed to use Drools to filter messages. Unfortunately, using Drools to perform complex filter logic came with a performance penalty and wasn’t as flexible as we would have liked it to be.

Starting with Graylog 3.0.0, the support for Drools-based message filters has been removed from Graylog. The rules_file configuration setting has been removed accordingly.

We recommend migrating the Drools-based logic to Processing Pipelines.

Drools-based blacklist

Graylog provided undocumented blacklist-functionality based on Drools. This blacklist could only be modified via the Graylog REST API on the /filters/blacklist resource.

If you’ve been using this functionality, you’ll have to migrate these blacklist rules to the Processing Pipelines.

To check if you’re using the Drools-based blacklist in Graylog prior to version 3.0.0, you can run the following command:

# curl -u admin:password -H 'Accept: application/json' 'http://graylog.example.com/api/filters/blacklist?pretty=true'
String-based blacklist rule

Old blacklist rule:

{
   "id" : "54e300001234123412340001",
   "type" : "string",
   "name" : "String Blacklist",
   "description" : "Drop messages based on case-insensitive string comparison",
   "fieldName" : "custom_field",
   "pattern" : "EXAMPLE pattern",
   "creator_user_id" : "admin",
   "created_at" : "2018-04-04T12:00:00.000Z"
}

New pipeline rule:

rule "string-blacklist"
when
  has_field("custom_field") &&
  lowercase(to_string($message.custom_field)) == "example pattern"
then
  drop_message();
end

See also:

Regex-based blacklist rule

Old blacklist rule:

{
   "id" : "54e300001234123412340002",
   "type" : "regex",
   "name" : "Regex Blacklist",
   "description" : "Drop messages based on regular expression",
   "fieldName" : "custom_field",
   "pattern" : "^EXAMPLE.*",
   "creator_user_id" : "admin",
   "created_at" : "2018-04-04T12:00:00.000Z"
}

New pipeline rule:

rule "regex-blacklist"
when
  has_field("custom_field") &&
  regex("^EXAMPLE.*", to_string($message.custom_field)).matches == true
then
  drop_message();
end

See also:

IP Range-based blacklist rule

Old blacklist rule:

{
   "id" : "54e300001234123412340003",
   "type" : "iprange",
   "name" : "IP Blacklist",
   "description" : "Drop messages based on IP address",
   "fieldName" : "custom_field",
   "pattern" : "192.168.0.0/16",
   "creator_user_id" : "admin",
   "created_at" : "2018-04-04T12:00:00.000Z"
}

New pipeline rule:

rule "ip-blacklist"
when
  has_field("custom_field") &&
  cidr_match("192.168.0.0/16", to_ip($message.custom_field))
then
  drop_message();
end

See also:

Changed metrics name for stream rules

The name of the metrics for stream rules have been changed to include the stream ID which helps identifying the actual stream they are related to.

Old metric name:

org.graylog2.plugin.streams.StreamRule.${stream-rule-id}.executionTime

New metric name:

org.graylog2.plugin.streams.Stream.${stream-id}.StreamRule.${stream-rule-id}.executionTime

Email alarm callback default settings

The defaults of the configuration settings for the email alarm callback with regard to encrypted connections have been changed.

Setting Old default New default
transport_email_use_tls false true
transport_email_use_ssl true false

Furthermore, it’s not possible anymore to enable both settings (SMTP with STARTTLS and SMTP over SSL) at the same time because this led to errors at runtime when Graylog tried to upgrade the connection to TLS with STARTTLS in an already existing SMTPS connection.

Most SMTP services prefer SMTP with STARTTLS to provide an encrypted connection.

Collector Sidecar is deprecated

Graylog 3.0 comes with a new Sidecar implementation. We still support the old Collector Sidecars, which can be found in the System / Collectors (legacy) menu entry. For more information check the Sidecar documentation and the Upgrade guide.

Legacy Content Packs

The implementation of content packs where fundamentally reworked. Parameters were added and checks implemented to give the user better usability. This rework did come with the cost that old content packs might not work any longer and stop the new content packs from loading. If the content packs page does not finish loading we recommend to remove the old content packs from your MongoDB. For that, please connect to your MongoDB shell and remove all content packs with the following command:

> db.content_packs.deleteMany({})

This command will only remove the content packs, it will not remove the installed configurations.

Elasticsearch 6 changes

There is a breaking change in Elasticsearch 6 that may affect some queries on your searches and dashboards:

Before Elasticsearch 6, queries for keyword fields were split by whitespaces and combined with OR operators resulting, for example, in type:(ssh login) and type:(ssh OR login) being equivalent. This is no longer the case in Elasticsearch 6 and now those queries are different: the former looking for the ssh login value, the second for either ssh or login values.

Please ensure to look for those queries in your Graylog setup before upgrading to Elasticsearch 6 and add the OR operators where needed.

Upgrading Graylog Originally Installed from Image

2.x
It is not possible to upgrade previous OVAs to Graylog 3.0.0.
3.0
Starting with Graylog 3.0.0, OVAs use the Operating System packages, so you can upgrade your appliance by following this update guide.

Upgrading Graylog Originally Installed from Package

If the current installation was installed using a package manager (ex. yum, apt), update the repository package to the target version, and use the system tools to upgrade the package. For .rpm based systems this update guide and for .deb based systems this update guide should help.

Upgrading Elasticsearch

Since Graylog 2.3 Elasticsearch 5.x is supported. This Graylog version supports Elasticsearch 2.x and 5.x. It is recommended to update Elasticsearch 2.x to the latest stable 5.x version, after you have Graylog 2.3 or later running. This Elasticsearch upgrade does not need to be made during the Graylog update.

When upgrading from Elasticsearch 2.x to Elasticsearch 5.x, make sure to read the upgrade guide provided by Elastic. The Graylog Elasticsearch configuration documentation contains information about the compatible Elasticsearch version. After the upgrade you must rotate the indices once manually.

Graylog 2.5 is the first Graylog version that supports Elasticsearch 6, the upgrade is recommended as soon as possible but might need more attention and include the need to reindex your data. Make sure to check our Elasticsearch 6 upgrade notes for this and other requirements.

When upgrading from Elasticsearch 5.x to Elasticsearch 6.x, make sure to read the upgrade guide provided by Elastic. The Graylog Elasticsearch configuration documentation contains information about the compatible Elasticsearch version. After the upgrade you must rotate the indices once manually.

Elasticsearch 6 notes

Upgrades from Elasticsearch 2.x direct to the latest Elasticsearch 6.x are not supported. Only the upgrade from Elasticsearch 5.x is supported and covered by this document.

At first check if the data in Elasticsearch need to be re-indexed:

$ http :9200/_settings | \
 jq '[ path(.[] |select(.settings.index.version.created < "5000000"))[] ]'

The above example use the tools httpie and jq to query the Elasticsearch API and check if any indices are created with Elasticsearch prior to Version 5. If that returns any index names, you need to re-index your data to make them work with Elasticseach 6.

Upgrading to Elasticsearch 6 is always a full-cluster-restart and all breaking changes need to checked carefully. Once started their is no going back or downgrade possible.

The Elasticsearch breaking changes notes contain a complete list of changes in Elasticsearch that should be checked against your configuration. The most notable is the cluster name is no longer allowed in path.data (see breaking changes in Elasticsearch 6) and the release of Elasticseach OSS Packages.

Upgrade without re-index

When no re-index is needed the easiest way is to follow the elastic upgrade guide for Elasticsearch this gives all needed commands.

Upgrade with re-index

First a brief overview what steps need to be performed followed by the list of commands. Once you started the process of reindex your data you need to finish all steps to get a working Graylog and Elasticsearch cluster again.

  1. Upgrade to Graylog latest 2.4 release (2.4.6 at time of writing)
  2. Upgrade ES to the latest 5.6.x on all cluster nodes. (5.6.14 as of this writing)
  3. Wait until all shards are initialized after updating ES
    • If the active write index is still a 2.x ES index, a manual index rotation needs to be triggered
  4. Upgrade to Graylog 2.5 (2.5.1 at time of writing)
  5. Update the index template for every index set to the latest ES 6 one by using the Graylog HTTP API. (otherwise a user has to rotate the active write index just to install the latest index template)
  6. Check which indices have been created with ES 2.x and need re-indexing
    • For each index to re-index do the following
      • Check that index is not the active write index
      • Create a re-index target index: <index-name>_reindex (e.g. graylog_0_reindex) (with correct settings for shards and replicas)
      • Check that mapping and settings of the new index are correct
      • Start re-index task in ES (using requests_per_second URL param and size param in the payload to avoid overloading the ES cluster)
      • Check progress of re-index task and wait until it is done
      • Check that the document counts in the old and the new index match
      • Delete old index
      • Recreate the old index: <index-name> (e.g. graylog_0) (with correct settings for shards and replicas)
      • Check that mapping and settings of the new index are correct
      • Start re-index task in ES to re-index documents back into the old index (using requests_per_second URL param and size param in the payload to avoid overloading the ES cluster)
      • Check that the document counts in the old and the new index match
      • Recreate Graylog index ranges for the old index
      • Delete temporary re-index target index (e.g. graylog_0_reindex)
  7. Delete old re-index tasks from ES
  8. Upgrade to the latest ES 6.x version. (6.5.1 as of this writing)
Detailed list of commands

Note

This is not a copy&paste tutorial and you need to read and adjust the commands to your local needs. We use the tools httpie and jq in the following commands.

Prerequisites
Check ES versions of all nodes

The ES version needs to be the same on all ES nodes in the cluster before we can start the re-index process!:

http ":9200/_cat/nodes?v&h=name,ip,version"
Check that all shards are initialized (“green”)

All shards need to be initialized before we can start the re-index process.:

http ":9200/_cat/indices?h=health,status,index" | grep -v '^green'
Update Graylog index templates in Elasticsearch

The index templates that Graylog writes to Elasticsearch need to be updated before we can start the re-index process.:

http post :9000/api/system/indexer/indices/templates/update x-requested-by:httpie
Collect indices that need a re-index to work with ES 6

All indices which have not been created with ES 5 need to be re-index to work with ES 6. (or deleted if they are not needed anymore…):

http :9200/_settings | jq '[ path(.[] | select(.settings.index.version.created < "5000000"))[] ]'
Re-Index commands for every index

The following commands need to be executed for every index that needs to be re-indexed. Replace the graylog_0 index name in the examples below with the index name you are currently working on.

Check if index is an active write index

We should never re-index the active write target because that index is actively written to. If the active write index is still a 2.x ES index, a manual index rotation needs to be triggered.:

http :9200/*_deflector/_alias | jq 'keys'
Create new index

The new index needs to be created before it can be used as a re-index target. The request needs to include the correct settings for the number of shards and replicas. These settings can be different for each index set! (actual settings can be found in the Graylog “System / Indices” page for each index set):

http put :9200/graylog_0_reindex settings:='{"number_of_shards":4,"number_of_replicas":0}'
Check mapping and index settings

Use these commands to check if the settings and index mapping for the new index are correct.:

http :9200/graylog_0_reindex/_mapping
http :9200/graylog_0_reindex/_settings
Start re-index process

This command starts the actual re-index process. It will return a task ID that can be used to check the progress of the re-index task in Elasticsearch.

The size value in the payload is the batch size that will be used for the re-index process. It defaults to 1000 and can be adjusted to tune the re-indexing process.:

http post :9200/_reindex wait_for_completion==false source:='{"index":"graylog_0","size": 1000}' dest:='{"index":"graylog_0_reindex"}'

The re-index API supports the requests_per_second URL parameter to throttle the re-index process. This can be useful to make sure that the re-index process doesn’t take too much resources. See this document for an explanation on how the parameter works: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/docs-reindex.html#_url_parameters_3:

http post :9200/_reindex wait_for_completion==false requests_per_second==500 source:='{"index":"graylog_0","size": 1000}' dest:='{"index":"graylog_0_reindex"}'
Wait for the re-index to complete and check re-index progress

The re-index progress can be checked with the following command using the task ID that has been returned by the re-index request.:

http :9200/_tasks/<task-id>
Compare documents in the old and new index

Before we continue, we should check that all documents have been re-indexed into the new index by comparing the document counts.:

http :9200/graylog_0/_count
http :9200/graylog_0_reindex/_count
Delete old index

Now delete the old index so we can recreate it for re-indexing.:

http delete :9200/graylog_0
Recreate old index

Recreate the old index again so we can use it as a re-index target. The request needs to include the correct settings for the number of shards and replicas. These settings can be different for each index set! (actual settings can be found in the Graylog “System / Indices” page for each index set):

http put :9200/graylog_0 settings:='{"number_of_shards":4,"number_of_replicas":0}'
Check mapping and index settings

Use these commands to check if the settings and index mapping for the recreated index are correct.:

http :9200/graylog_0/_mapping
http :9200/graylog_0/_settings
Start re-index process for old index

This command starts the re-index process to move back the documents into the old index. It will return a task ID that can be used to check the progress of the re-index task in Elasticsearch.

The size value in the payload is the batch size that will be used for the re-index process. It defaults to 1000 and can be adjusted to tune the re-indexing process.:

http post :9200/_reindex wait_for_completion==false source:='{"index":"graylog_0_reindex","size": 1000}' dest:='{"index":"graylog_0"}'

The re-index API supports the requests_per_second URL parameter to throttle the re-index process. This can be useful to make sure that the re-index process doesn’t take too much resources. See this document for an explanation on how the parameter works: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/docs-reindex.html#_url_parameters_3:

http post :9200/_reindex wait_for_completion==false requests_per_second==500 source:='{"index":"graylog_0_reindex","size": 1000}' dest:='{"index":"graylog_0"}'
Compare documents in the old and new index

Before we continue, we should check that all documents have been re-indexed into the re-created old index by comparing the document counts with the temporary index.:

http :9200/graylog_0/_count
http :9200/graylog_0_reindex/_count
Create index range for the recreated index

Graylog needs to know about the recreated index by creating an index range for it.:

http post :9000/api/system/indices/ranges/graylog_0/rebuild x-requested-by:httpie
Delete temporary re-index target index

The temporary re-index target index can now be deleted because we don’t use it anymore.:

http delete :9200/graylog_0_reindex
Cleanup

The re-index process leaves some tasks in Elasticsearch that need to be cleaned up automatically.

Find completed re-index tasks for deletion

Execute the following command to get all the tasks we should remove.:

http :9200/.tasks/_search | jq '[.hits.hits[] | select(._source.task.action == "indices:data/write/reindex" and ._source.completed == true) | {"task_id": ._id, "description": ._source.task.description}]'
Remove completed re-index tasks

Execute the following command for every completed task ID. Re-Index Commands:

http delete :9200/.tasks/task/<task-id>

Configuring Graylog

server.conf

The file server.conf is the Graylog configuration file.

Note

Check Default file locations to locate it in your installation.

It has to use ISO 8859-1/Latin-1 character encoding. Characters that cannot be directly represented in this encoding can be written using Unicode escapes as defined in Java SE Specifications, using the u prefix. For example, u002c.

  • Entries are generally expected to be a single line of the form, one of the following:
    • propertyName=propertyValue
    • propertyName:propertyValue
  • White space that appears between the property name and property value is ignored, so the following are equivalent:
    • name=Stephen
    • name = Stephen
  • White space at the beginning of the line is also ignored.

  • Lines that start with the comment characters ! or # are ignored. Blank lines are also ignored.

  • The property value is generally terminated by the end of the line. White space following the property value is not ignored, and is treated as part of the property value.

  • A property value can span several lines if each line is terminated by a backslash (\) character. For example:

    targetCities=\
           Detroit,\
           Chicago,\
           Los Angeles
    

    This is equivalent to targetCities=Detroit,Chicago,Los Angeles (white space at the beginning of lines is ignored).

  • The characters newline, carriage return, and tab can be inserted with characters \n, \r, and \t, respectively.

  • The backslash character must be escaped as a double backslash. For example:

    path=c:\\docs\\doc1
    

Properties

General
  • is_master = true
    • If you are running more than one instances of Graylog server you have to select only one graylog-server node as the master. This node will perform periodical and maintenance actions that slave nodes won’t.
    • Every slave node will accept messages just as the master nodes. Nodes will fall back to slave mode if there already is a master in the cluster.
  • node_id_file = /etc/graylog/server/<node-id>
    • The auto-generated node ID will be stored in this file and read after restarts. It is a good idea to use an absolute file path here if you are starting Graylog server from init scripts or similar.
  • password_secret = <secret>
    • You MUST set a secret that is used for password encryption and salting. The server will refuse to start if it’s not set. Use at least 64 characters. If you run multiple graylog-server nodes, make sure you use the same password_secret for all of them!

    Note

    Generate a secret with for example pwgen -N 1 -s 96

  • root_username = admin
    • The default root user is named admin.
  • root_password_sha2 = <SHA2>
    • A SHA2 hash of a password you will use for your initial login. Set this to a SHA2 hash generated with echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1 and you will be able to log in to the web interface with username admin and password yourpassword.

    Caution

    You MUST specify a hash password for the root user (which you only need to initially set up the system and in case you lose connectivity to your authentication backend). This password cannot be changed using the API or via the web interface. If you need to change it, modify it in this file.

  • root_email = ""
    • The email address of the root user. Default is empty.
  • root_timezone = UTC
  • bin_dir = bin
    • This directory contains binaries that are used by the Graylog server. (relative or absolute)
  • data_dir = data
    • This directory is used to store Graylog server state. (relative or absolute)
  • plugin_dir = plugin
    • Set plugin directory here (relative or absolute)
Web & REST API
  • http_bind_address = 127.0.0.1:9000
    • The network interface used by the Graylog HTTP interface.
    • This network interface must be accessible by all Graylog nodes in the cluster and by all clients using the Graylog web interface.
    • If the port is omitted, Graylog will use port 9000 by default.
  • http_publish_uri = http://$http_bind_address/
    • The HTTP URI of this Graylog node which is used to communicate with the other Graylog nodes in the cluster and by all clients using the Graylog web interface.
    • The URI will be published in the cluster discovery APIs, so that other Graylog nodes will be able to find and connect to this Graylog node.
    • This configuration setting has to be used if this Graylog node is available on another network interface than $http_bind_address, for example if the machine has multiple network interfaces or is behind a NAT gateway.
    • This configuration setting must not be configured to a wildcard address!
    • If http_bind_address contains a wildcard IPv4 address (0.0.0.0), http_publish_uri will be filled with the first non-loopback IPv4 address of this machine instead.
  • http_external_uri = $http_publish_uri
    • The public URI of Graylog which will be used by the Graylog web interface to communicate with the Graylog REST API.
    • The external Graylog URI usually has to be specified, if Graylog is running behind a reverse proxy or load-balancer and it will be used to generate URLs addressing entities in the Graylog REST API (see $http_bind_address).
    • When using Graylog Collector, this URI will be used to receive heartbeat messages and must be accessible for all collectors.
    • This setting can be overriden on a per-request basis with the “X-Graylog-Server-URL” HTTP request header.
  • http_enable_cors = true
    • Enable CORS headers for HTTP interface.
    • This is necessary for JS-clients accessing the server directly.
    • If these are disabled, modern browsers will not be able to retrieve resources from the server.
  • http_enable_gzip = true
    • This compresses API responses and therefore helps to reduce overall round trip times.
  • http_max_header_size = 8192
    • The maximum size of the HTTP request headers in bytes.
  • http_thread_pool_size = 16
    • The size of the thread pool used exclusively for serving the HTTP interface.
  • http_enable_tls = false
    • This secures the communication with the HTTP interface with TLS to prevent request forgery and eavesdropping.
  • http_tls_cert_file = /path/to/graylog.crt
    • The X.509 certificate chain file in PEM format to use for securing the HTTP interface.
  • http_tls_key_file = /path/to/graylog.key
    • The PKCS#8 private key file in PEM format to use for securing the HTTP interface.
  • http_tls_key_password = secret
    • The password to unlock the private key used for securing the HTTP interface. (if key is encrypted)
  • trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128
    • Comma separated list of trusted proxies that are allowed to set the client address with X-Forwarded-For header. May be subnets, or hosts.
Elasticsearch
  • elasticsearch_hosts = http://node1:9200,http://user:password@node2:19200
    • List of Elasticsearch hosts Graylog should connect to.
    • Need to be specified as a comma-separated list of valid URIs for the http ports of your elasticsearch nodes.
    • If one or more of your elasticsearch hosts require authentication, include the credentials in each node URI that requires authentication.
    • Default: http://127.0.0.1:9200
  • elasticsearch_connect_timeout = 10s
    • Maximum amount of time to wait for successfull connection to Elasticsearch HTTP port.
    • Default: 10 seconds
  • elasticsearch_socket_timeout = 60s
    • Maximum amount of time to wait for reading back a response from an Elasticsearch server.
    • Default: 60 seconds
  • elasticsearch_idle_timeout = -1s
    • Maximum idle time for an Elasticsearch connection. If this is exceeded, this connection will be tore down.
    • Default: infinity
  • elasticsearch_max_total_connections = 20
    • Maximum number of total connections to Elasticsearch.
    • Default: 20
  • elasticsearch_max_total_connections_per_route = 2
    • Maximum number of total connections per Elasticsearch route (normally this means per elasticsearch server).
    • Default: 2
  • elasticsearch_max_retries = 2
    • Maximum number of times Graylog will retry failed requests to Elasticsearch.
    • Default: 2
  • elasticsearch_discovery_enabled = false

    Warning

    Automatic node discovery does not work if Elasticsearch requires authentication, e. g. with Shield.

    Warning

    This setting must be false on AWS Elasticsearch Clusters (the hosted ones) and should be used carefully. In case of trouble with connections to ES this should be the first option to be disabled. See Automatic node discovery for more details.

  • elasticsearch_discovery_filter = rack:42
  • elasticsearch_discovery_frequency = 30s
    • Frequency of the Elasticsearch node discovery.
    • Default: 30 seconds
  • elasticsearch_compression_enabled = false
    • Enable payload compression for Elasticsearch requests.
    • Default: false
Rotation

Attention

The following settings identified with ! in this section have been moved to the database in Graylog 2.0. When you upgrade, make sure to set these to your previous 1.x settings so they will be migrated to the database!

  • rotation_strategy = count !
    • Graylog will use multiple indices to store documents in. You can configured the strategy it uses to determine when to rotate the currently active write index.
    • It supports multiple rotation strategies: - count of messages per index, use elasticsearch_max_docs_per_index - size per index, use elasticsearch_max_size_per_index
    • valid values are count, size and time, default is count.
  • elasticsearch_max_docs_per_index = 20000000 !
    • (Approximate) maximum number of documents in an Elasticsearch index before a new index is being created, also see no_retention and elasticsearch_max_number_of_indices.
    • Configure this if you used rotation_strategy = count above.
  • elasticsearch_max_size_per_index = 1073741824 !
    • (Approximate) maximum size in bytes per Elasticsearch index on disk before a new index is being created, also see no_retention and `elasticsearch_max_number_of_indices`. Default is 1GB.
    • Configure this if you used rotation_strategy = size above.
  • elasticsearch_max_time_per_index = 1d !
    • (Approximate) maximum time before a new Elasticsearch index is being created, also see no_retention and elasticsearch_max_number_of_indices. Default is 1 day.
    • Configure this if you used rotation_strategy = time above.
    • Please note that this rotation period does not look at the time specified in the received messages, but is using the real clock value to decide when to rotate the index!
    • Specify the time using a duration and a suffix indicating which unit you want:
      • 1w = 1 week
      • 1d = 1 day
      • 12h = 12 hours
    • Permitted suffixes are: d for day, h for hour, m for minute, s for second.
  • elasticsearch_max_number_of_indices = 20 !
    • How many indices do you want to keep?
  • retention_strategy = delete !
    • Decide what happens with the oldest indices when the maximum number of indices is reached.
    • The following strategies are availble:
      • delete - Deletes the index completely (Default)
      • close - Closes the index and hides it from the system. Can be re-opened later.

  • elasticsearch_disable_version_check = true
    • Disable checking the version of Elasticsearch for being compatible with this Graylog release.

    Warning

    Using Graylog with unsupported and untested versions of Elasticsearch may lead to data loss!

  • no_retention = false
    • Disable message retention on this node, i. e. disable Elasticsearch index rotation.

Attention

The following settings identified with !! have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these to your previous settings so they will be migrated to the database. This settings are read once at the very first startup to be the initial settings in the database.

  • elasticsearch_shards = 4 !!
    • The number of shards for your indices. A good setting here highly depends on the number of nodes in your Elasticsearch cluster. If you have one node, set it to 1.
  • elasticsearch_replicas = 0 !!
    • The number of replicas for your indices. A good setting here highly depends on the number of nodes in your Elasticsearch cluster. If you have one node, set it to 0.

    Note

    elasticsearch_shards and elasticsearch_replicas only applies to newly created indices.

  • elasticsearch_index_prefix = graylog !!
    • Prefix for all Elasticsearch indices and index aliases managed by Graylog.
  • elasticsearch_template_name = graylog-internal !!
    • Name of the Elasticsearch index template used by Graylog to apply the mandatory index mapping.
    • Default: graylog-internal
  • elasticsearch_analyzer = standard !!
    • Analyzer (tokenizer) to use for message and full_message field. The “standard” filter usually is a good idea.
    • All supported analyzers are: standard, simple, whitespace, stop, keyword, pattern, language, snowball, custom
    • Elasticsearch documentation: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/analysis.html
    • Note that this setting only takes effect on newly created indices.
  • disable_index_optimization = false !!
    • Disable the optimization of Elasticsearch indices after index cycling. This may take some load from Elasticsearch on heavily used systems with large indices, but it will decrease search performance. The default is to optimize cycled indices.
  • index_optimization_max_num_segments = 1 !!
    • Optimize the index down to <= index_optimization_max_num_segments. A higher number may take some load from Elasticsearch on heavily used systems with large indices, but it will decrease search performance. The default is 1.

  • allow_leading_wildcard_searches = false
    • Do you want to allow searches with leading wildcards? This can be extremely resource hungry and should only be enabled with care.
    • See also: Searching
  • allow_highlighting = false
    • Do you want to allow searches to be highlighted? Depending on the size of your messages this can be memory hungry and should only be enabled after making sure your Elasticsearch cluster has enough memory.
  • elasticsearch_request_timeout = 1m
    • Global request timeout for Elasticsearch requests (e. g. during search, index creation, or index time-range calculations) based on a best-effort to restrict the runtime of Elasticsearch operations.
    • Default: 1m
  • elasticsearch_index_optimization_timeout = 1h
    • Global timeout for index optimization (force merge) requests.
    • Default: 1h
  • elasticsearch_index_optimization_jobs = 20
    • Maximum number of concurrently running index optimization (force merge) jobs.
    • If you are using lots of different index sets, you might want to increase that number.
    • Default: 20
  • index_ranges_cleanup_interval = 1h
    • Time interval for index range information cleanups. This setting defines how often stale index range information is being purged from the database.
    • Default: 1h
  • output_batch_size = 500
    • Batch size for the Elasticsearch output. This is the maximum (!) number of messages the Elasticsearch output module will get at once and write to Elasticsearch in a batch call. If the configured batch size has not been reached within output_flush_interval seconds, everything that is available will be flushed at once. Remember that every output buffer processor manages its own batch and performs its own batch write calls. (outputbuffer_processors variable)
  • output_flush_interval = 1
    • Flush interval (in seconds) for the Elasticsearch output. This is the maximum amount of time between two batches of messages written to Elasticsearch. It is only effective at all if your minimum number of messages for this time period is less than output_batch_size * outputbuffer_processors.
  • output_fault_count_threshold = 5

  • output_fault_penalty_seconds = 30
    • As stream outputs are loaded only on demand, an output which is failing to initialize will be tried over and over again. To prevent this, the following configuration options define after how many faults an output will not be tried again for an also configurable amount of seconds.
  • processbuffer_processors = 5

  • outputbuffer_processors = 3
    • The number of parallel running processors.
    • Raise this number if your buffers are filling up.
  • outputbuffer_processor_keep_alive_time = 5000

  • outputbuffer_processor_threads_core_pool_size = 3

  • outputbuffer_processor_threads_max_pool_size = 30

  • udp_recvbuffer_sizes = 1048576
    • UDP receive buffer size for all message inputs (e. g. SyslogUDPInput).
  • processor_wait_strategy = blocking
    • Wait strategy describing how buffer processors wait on a cursor sequence. (default: sleeping)
    • Possible types:
      • yielding - Compromise between performance and CPU usage.
      • sleeping - Compromise between performance and CPU usage. Latency spikes can occur after quiet periods.
      • blocking - High throughput, low latency, higher CPU usage.
      • busy_spinning - Avoids syscalls which could introduce latency jitter. Best when threads can be bound to specific CPU cores.
  • ring_size = 65536
    • Size of internal ring buffers. Raise this if raising outputbuffer_processors does not help anymore.
    • For optimum performance your LogMessage objects in the ring buffer should fit in your CPU L3 cache.
    • Must be a power of 2. (512, 1024, 2048, …)
  • inputbuffer_ring_size = 65536

  • inputbuffer_processors = 2

  • inputbuffer_wait_strategy = blocking

  • message_journal_enabled = true
    • Enable the disk based message journal.
  • message_journal_dir = data/journal
    • The directory which will be used to store the message journal. The directory must me exclusively used by Graylog and must not contain any other files than the ones created by Graylog itself.

    Attention

    If you create a seperate partition for the journal files and use a file system creating directories like ‘lost+found’ in the root directory, you need to create a sub directory for your journal. Otherwise Graylog will log an error message that the journal is corrupt and Graylog will not start.

  • message_journal_max_age = 12h

  • message_journal_max_size = 5gb
    • Journal hold messages before they could be written to Elasticsearch.
    • For a maximum of 12 hours or 5 GB whichever happens first.
    • During normal operation the journal will be smaller.
  • message_journal_flush_age = 1m
    • This setting allows specifying a time interval at which we will force an fsync of data written to the log. For example if this was set to 1000 we would fsync after 1000 ms had passed.
  • message_journal_flush_interval = 1000000
    • This setting allows specifying an interval at which we will force an fsync of data written to the log. For example if this was set to 1 we would fsync after every message; if it were 5 we would fsync after every five messages.
  • message_journal_segment_age = 1h
    • This configuration controls the period of time after which Graylog will force the log to roll even if the segment file isn’t full to ensure that retention can delete or compact old data.
  • message_journal_segment_size = 100mb

Attention

When the journal is full and it keeps receiving messages, it will start dropping messages as a FIFO queue: The first dropped message will be the first inserted and so on (and not some random).

  • async_eventbus_processors = 2
    • Number of threads used exclusively for dispatching internal events. Default is 2.
  • lb_recognition_period_seconds = 3
    • How many seconds to wait between marking node as DEAD for possible load balancers and starting the actual shutdown process. Set to 0 if you have no status checking load balancers in front.
  • lb_throttle_threshold_percentage = 95
    • Journal usage percentage that triggers requesting throttling for this server node from load balancers. The feature is disabled if not set.
  • stream_processing_timeout = 2000
  • stream_processing_max_faults = 3
    • Every message is matched against the configured streams and it can happen that a stream contains rules which take an unusual amount of time to run, for example if its using regular expressions that perform excessive backtracking.
    • This will impact the processing of the entire server. To keep such misbehaving stream rules from impacting other streams, Graylog limits the execution time for each stream.
    • The default values are noted below, the timeout is in milliseconds.
    • If the stream matching for one stream took longer than the timeout value, and this happened more than “max_faults” times that stream is disabled and a notification is shown in the web interface.
  • alert_check_interval = 60
    • Length of the interval in seconds in which the alert conditions for all streams should be checked and alarms are being sent.

Note

Since 0.21 the Graylog server supports pluggable output modules. This means a single message can be written to multiple outputs. The next setting defines the timeout for a single output module, including the default output module where all messages end up.

  • output_module_timeout = 10000
    • Time in milliseconds to wait for all message outputs to finish writing a single message.
  • stale_master_timeout = 2000
    • Time in milliseconds after which a detected stale master node is being rechecked on startup.
  • shutdown_timeout = 30000
    • Time in milliseconds which Graylog is waiting for all threads to stop on shutdown.
MongoDB
  • mongodb_uri = mongodb://...
    • MongoDB connection string. Enter your MongoDB connection and authentication information here.
    • See https://docs.mongodb.com/manual/reference/connection-string/ for details.
    • Take notice that +-signs in the username or password need to be replaced by %2B.
    • Examples:
      • Simple: mongodb://localhost/graylog
      • Authenticate against the MongoDB server: mongodb_uri = mongodb://grayloguser:secret@localhost:27017/graylog
      • Use a replica set instead of a single host: mongodb://grayloguser:secret@localhost:27017,localhost:27018,localhost:27019/graylog
      • DNS Seedlist is set as mongodb+srv://server.example.org/graylog.
  • mongodb_max_connections = 1000
    • Increase this value according to the maximum connections your MongoDB server can handle from a single client if you encounter MongoDB connection problems.
  • mongodb_threads_allowed_to_block_multiplier = 5
Email
  • transport_email_enabled = false
  • transport_email_hostname = mail.example.com
  • transport_email_port = 587
  • transport_email_use_auth = true
  • transport_email_use_tls = true
    • Enable SMTP with STARTTLS for encrypted connections.
  • transport_email_use_ssl = false
    • Enable SMTP over SSL (SMTPS) for encrypted connections.

Attention

Make sure to enable only one of these two settings because most (or all) SMTP services only support one of the encryption mechanisms on the same port. Most SMTP services support SMTP with STARTTLS while SMTPS is deprecated on most SMTP services. Setting both to false is needed when you want to sent via unencrypted connection.

  • transport_email_auth_username = you@example.com
  • transport_email_auth_password = secret
  • transport_email_subject_prefix = [graylog]
  • transport_email_from_email = graylog@example.com
  • transport_email_web_interface_url = https://graylog.example.com
    • Specify this to include links to the stream in your stream alert mails.
    • This should define the fully qualified base url to your web interface exactly the same way as it is accessed by your users.
HTTP
  • http_connect_timeout = 5s
    • The default connect timeout for outgoing HTTP connections.
    • Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
    • Default: 5s
  • http_read_timeout = 10s
    • The default read timeout for outgoing HTTP connections.
    • Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
    • Default: 10s
  • http_write_timeout = 10s
    • The default write timeout for outgoing HTTP connections.
    • Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
    • Default: 10s
  • http_proxy_uri =
    • HTTP proxy for outgoing HTTP connections

Attention

If you configure a proxy, make sure to also configure the “http_non_proxy_hosts” option so internal HTTP connections with other nodes does not go through the proxy.

  • http_non_proxy_hosts =
    • A list of hosts that should be reached directly, bypassing the configured proxy server.
    • This is a list of patterns separated by “,”. The patterns may start or end with a “*” for wildcards.
    • Any host matching one of these patterns will be reached through a direct connection instead of through a proxy.
Script alert notification
  • integrations_web_interface_uri = https://graylog.example.com
    • Specify this to include a search page link (that displays relevant alert messages) in the script arguments or standard in JSON.
    • This should define the fully qualified base url to your web interface exactly the same way as it is accessed by your users.
    • Default: none
  • integrations_scripts_dir = /usr/share/graylog-server/scripts
    • An absolute or relative path where scripts are permitted to be executed from.
    • If specified, this overrides the default location (see the File Locations document.
Others
  • gc_warning_threshold = 1s
    • The threshold of the garbage collection runs. If GC runs take longer than this threshold, a system notification will be generated to warn the administrator about possible problems with the system. Default is 1 second.
  • ldap_connection_timeout = 2000
    • Connection timeout for a configured LDAP server (e. g. ActiveDirectory) in milliseconds.
  • disable_sigar = false
    • Disable the use of SIGAR for collecting system stats.
  • dashboard_widget_default_cache_time = 10s
    • The default cache time for dashboard widgets. (Default: 10 seconds, minimum: 1 second)
  • proxied_requests_thread_pool_size = 32
    • For some cluster-related REST requests, the node must query all other nodes in the cluster. This is the maximum number of threads available for this. Increase it, if /cluster/* requests take long to complete.
    • Should be http_thread_pool_size * average_cluster_size if you have a high number of concurrent users.

Web interface

When your Graylog instance/cluster is up and running, the next thing you usually want to do is check out our web interface, which offers you great capabilities for searching and analyzing your indexed data and configuring your Graylog environment. Per default you can access it using your browser on http://<graylog-server>:9000/.

Overview

The Graylog web interface was rewritten in JavaScript for 2.0 to be a client-side single-page browser application. This means its code is running solely in your browser, fetching all data via HTTP(S) from the REST API of your Graylog server.

Note

The HTTP address must be accessible by everyone using the web interface. This means that Graylog must listen on a public network interface or be exposed to one using a proxy, NAT or a load balancer!

Configuration Options

If our default settings do not work for you, there is a number of options in the Graylog server configuration file which you can change to influence its behavior:

Setting Default Explanation
http_bind_address 127.0.0.1:9000 The network interface used by the Graylog HTTP interface.
http_publish_uri If not set, http://$http_bind_address will be used. The HTTP URI of this Graylog node which is used to communicate with the other Graylog nodes in the cluster and by all clients using the Graylog web interface.
http_external_uri If not set, $http_publish_uri will be used. The public URI of Graylog which will be used by the Graylog web interface to communicate with the Graylog REST API. Graylog web interface.
http_enable_cors true This is necessary for JS-clients accessing the server directly. If disabled, modern browsers will not be able to retrieve resources from the server.
http_enable_gzip true Serve web interface assets using compression to reduce overall roundtrip times.
http_max_header_size 8192 The maximum size of the HTTP request headers in bytes.
http_thread_pool_size 16 The size of the thread pool used exclusively for serving the HTTP interface.
http_enable_tls false This secures the communication with the HTTP interface with TLS to prevent request forgery and eavesdropping.
http_tls_cert_file (no default) The X.509 certificate chain file in PEM format to use for securing the HTTP interface.
http_tls_key_file (no default) The PKCS#8 private key file in PEM format to use for securing the HTTP interface.
http_tls_key_password (no default) The password to unlock the private key used for securing the HTTP interface. (only needed if the key is encryped)

How does the web interface connect to the Graylog server?

The web interface is fetching all information it is showing from the REST API of the Graylog server. Therefore it needs to connect to it using HTTP(S). There are several ways how you can define which way the web interface connects to the Graylog server. The URI used by the web interface is determined in this exact order:

  • If the HTTP(S) client going to the web interface port sends a X-Graylog-Server-URL header, which contains a valid URL, then this is overriding everything else.
  • If http_external_uri is defined in the Graylog configuration file, this is used if the aforementioned header is not set.
  • If http_publish_uri is defined in the Graylog configuration file, this is used if the aforementioned http_external_uri is not set.
  • If none of the above are defined, http://$http_bind_address is used.

The web interface assets (e.g. the index.html, CSS and JavaScript files) are accessible at the URI root (/ by default) and the REST API endpoints are accessible at the /api path.

Example:

Setting http_bind_address to 10.0.0.1:9000 configures the Graylog server with the following URLs.

  • Web interface: http://10.0.0.1:9000/
  • REST API: http://10.0.0.1:9000/api/

Browser Compatibility

Writing the web interface as a single-page application is a challenging task. We want to provide the best possible experience to everyone, which often means using modern web technology only available in recent browsers, while keeping a reasonable compatibility with old and less-capable browsers. These browsers are officially supported in Graylog 3.0:

Browser OS Minimum Version
Chrome Windows, OS X, Linux 50
Firefox Windows, OS X, Linux 45 / 38 ESR
Internet Explorer Windows 11
Microsoft Edge Windows 25
Safari OS X 9

Please take into account that you need to enable JavaScript in order to use Graylog web interface.

Making the web interface work with load balancers/proxies

If you want to run a load balancer/reverse proxy in front of Graylog, you need to make sure that:

  • The HTTP port of the load balancer/reverse proxy is accessible for clients
  • The HTTP address for the Graylog server is properly set (as explained in How does the web interface connect to the Graylog server?), so it is resolvable and accessible for the load balancer/reverse proxy.
  • If you use SSL, your certificates must be valid and trusted by your clients.

Note

To help you with your specific environment, we show some example configuration use cases.

For the configuration use cases below we assume the following:

  • Your Graylog server configuration contains http_bind_address = 127.0.0.1:9000
  • The hostname for the setup is graylog.example.org
  • The IP address for that hostname is 192.168.0.10
Using a Layer 3 load balancer (forwarding TCP Ports)
  1. Configure your load balancer to forward connections going to 192.168.0.10:80 to 127.0.0.1:9000.
  2. Start the Graylog server as usual.
  3. Access the web interface on http://graylog.example.org.
  4. Read up on Using HTTPS.
NGINX

Proxy web interface and API traffic using HTTP:

server
{
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;
    server_name graylog.example.org;

    location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Server $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Graylog-Server-URL http://$server_name/;
      proxy_pass       http://127.0.0.1:9000;
    }
}

NGINX can be used for SSL Termination, you would only need to modify the server listen directive and add all Information about your certificate.

If you are running multiple Graylog Server you might want to use HTTPS/SSL to connect to the Graylog Servers (on how to Setup read Using HTTPS) and use HTTPS/SSL on NGINX. The configuration for TLS certificates, keys and ciphers is omitted from the sample config for brevity’s sake.

Proxy web interface and API traffic using HTTPS (TLS):

server
{
    listen      443 ssl spdy;
    server_name graylog.example.org;
    # <- your SSL Settings here!

    location /
    {
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Server $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Graylog-Server-URL https://$server_name/;
      proxy_pass       http://127.0.0.1:9000;
    }
}

If you want to serve serveral different applications under one domain name, you can also serve the Graylog web interface using a path prefix.

Proxy web interface and API traffic under a path prefix using HTTP:

server
{
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;
    server_name applications.example.org;

    location /graylog/
    {
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Server $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Graylog-Server-URL http://$server_name/graylog/;
      rewrite          ^/graylog/(.*)$  /$1  break;
      proxy_pass       http://127.0.0.1:9000;
    }
}

This makes your Graylog setup available under the following URLs:

  • Web interface: http://applications.example.org/graylog/
  • REST API: http://applications.example.org/graylog/api/
Apache httpd 2.x

Proxy web interface and API traffic using HTTP:

<VirtualHost *:80>
    ServerName graylog.example.org
    ProxyRequests Off
    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>

    <Location />
        RequestHeader set X-Graylog-Server-URL "http://graylog.example.org/"
        ProxyPass http://127.0.0.1:9000/
        ProxyPassReverse http://127.0.0.1:9000/
    </Location>

</VirtualHost>

Proxy web interface and API traffic using HTTPS (TLS):

<VirtualHost *:443>
    ServerName graylog.example.org
    ProxyRequests Off
    SSLEngine on
    # <- your SSL Settings here!

    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>

    <Location />
        RequestHeader set X-Graylog-Server-URL "https://graylog.example.org/"
        ProxyPass http://127.0.0.1:9000/
        ProxyPassReverse http://127.0.0.1:9000/
    </Location>

</VirtualHost>
HAProxy 1.6

Proxy web interface and API traffic using HTTP:

frontend http
    bind 0.0.0.0:80

    option forwardfor
    http-request add-header X-Forwarded-Host %[req.hdr(host)]
    http-request add-header X-Forwarded-Server %[req.hdr(host)]
    http-request add-header X-Forwarded-Port %[dst_port]
    acl is_graylog hdr_dom(host) -i -m str graylog.example.org
    use_backend     graylog if is_graylog

backend graylog
    description     The Graylog Web backend.
    http-request set-header X-Graylog-Server-URL http://graylog.example.org/
    use-server graylog_1
    server graylog_1 127.0.0.1:9000 maxconn 20 check

Multiple Backends (roundrobin) with Health-Check (using HTTP):

frontend graylog_http
    bind *:80
    option forwardfor
    http-request add-header X-Forwarded-Host %[req.hdr(host)]
    http-request add-header X-Forwarded-Server %[req.hdr(host)]
    http-request add-header X-Forwarded-Port %[dst_port]
    acl is_graylog hdr_dom(host) -i -m str graylog.example.org
    use_backend     graylog

backend graylog
    description     The Graylog Web backend.
    balance roundrobin
    option httpchk HEAD /api/system/lbstatus
    http-request set-header X-Graylog-Server-URL http://graylog.example.org/
    server graylog1 192.168.0.10:9000 maxconn 20 check
    server graylog2 192.168.0.11:9000 maxconn 20 check
    server graylog3 192.168.0.12:9000 maxconn 20 check

Load balancer integration

When running multiple Graylog servers a common deployment scenario is to route the message traffic through an IP load balancer. By doing this we can achieve both a highly available setup, as well as increasing message processing throughput, by simply adding more servers that operate in parallel.

Load balancer state

However, load balancers usually need some way of determining whether a backend service is reachable and healthy or not. For this purpose Graylog exposes a load balancer state that is reachable via its REST API.

There are two ways the load balancer state can change:

  • due to a lifecycle change (e.g. the server is starting to accept messages, or shutting down)
  • due to manual intervention via the REST API

Note

In the following examples we assume that the Graylog REST API is available on the URI path /api/ (e. g. http://graylog.example.com/api/).

To query the current load balancer status of a Graylog instance, all you need to do is to issue a HTTP call to its REST API:

GET /api/system/lbstatus

The status knows two different states, ALIVE and DEAD, which is also the text/plain response of the resource. Additionally, the same information is reflected in the HTTP status codes: If the state is ALIVE the return code will be 200 OK, for DEAD it will be 503 Service unavailable. This is done to make it easier to configure a wide range of load balancer types and vendors to be able to react to the status.

The resource is accessible without authentication to make it easier for load balancers to access it.

To programmatically change the load balancer status, an additional endpoint is exposed:

PUT /api/system/lbstatus/override/alive
PUT /api/system/lbstatus/override/dead

Only authenticated and authorized users are able to change the status, in the currently released Graylog version this means only admin users can change it.

Graceful shutdown

Often, when running a service behind a load balancer, the goal is to be able to perform zero-downtime upgrades, by taking one of the servers offline, upgrading it, and then bringing it back online. During that time the remaining servers can take the load seamlessly.

By using the load balancer status API described above one can already perform such a task. However, it would still be guesswork when the Graylog server is done processing all the messages it already accepted.

For this purpose Graylog supports a graceful shutdown command, also accessible via the web interface and API. It will set the load balancer status to DEAD, stop all inputs, turn on messages processing (should it have been disabled manually previously), and flush all messages in memory to Elasticsearch. After all buffers and caches are processed, it will shut itself down safely.

_images/nodes_more_action.png

Web Interface

It is possible to use the Graylog web interface behind a load balancer for high availability purposes.

Note

Take care of the configuration you need with a proxy setup, as it will not work out of the box.

Using HTTPS

We highly recommend securing your Graylog installation using SSL/TLS to make sure that no sensitive data is sent over the wire in plain text. To make this work, you need to enable the http_enable_tls setting in your Graylog server configuration.

You also need to make sure that you have proper certificates in place, which are valid and trusted by the clients.

Note

If you’re operating a single-node setup and would like to use HTTPS for the Graylog web interface and the Graylog REST API, it’s possible to use NGINX or Apache as a reverse proxy.

Things to consider

You have multiple options to ensure that your connection is secure and safe. The first would be to create a self-signed certificate, add that to the previously copied java keystore and use this keystore with your Graylog java options. Since you will need to do this for every certificate and every trust store, this quickly becomes unmanageable in a clustered architecture. Each node needs to trust all certificates from all other nodes.

The second option would be to create your own certificate authority. You only add the certificate authority once to the key store and all certificates that are created with this authority will be trusted.

The same can be done if you have already your own certificate authority, you only need the certificates and keys in the format that can be used with Graylog. Add the certificate authority key to the keystore and all certificates that are signed by this certificate authority will be trusted. Same when you pay for certificates or use a free Certificate authority like let’s encrypt to get the server certificates.

Just add the certificate authority to the keystore and all certificates are trusted.

Certificate/Key file format

When you are configuring TLS, you need to make sure that your certificate/key files are in the right format, which is X.509 for certificates and PKCS#8 for the private keys. Both must to be stored in PEM format.

Creating a self-signed private key/certificate

Create a file named openssl-graylog.cnf with the following content (customized to your needs):

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

# Details about the issuer of the certificate
[req_distinguished_name]
C = US
ST = Some-State
L = Some-City
O = My Company
OU = My Division
CN = graylog.example.com

[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

# IP addresses and DNS names the certificate should include
# Use IP.### for IP addresses and DNS.### for DNS names,
# with "###" being a consecutive number.
[alt_names]
IP.1 = 203.0.113.42
DNS.1 = graylog.example.com

Create PKCS#5 private key and X.509 certificate:

$ openssl version
OpenSSL 0.9.8zh 14 Jan 2016
$ openssl req -x509 -days 365 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem
Generating a 2048 bit RSA private key
............................+++
.+++
writing new private key to 'pkcs5-plain.pem'
-----

Convert PKCS#5 private key into a unencrypted PKCS#8 private key:

$ openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem

Convert PKCS#5 private key into an encrypted PKCS#8 private key (using the passphrase secret):

$ openssl pkcs8 -in pkcs5-plain.pem -topk8 -out pkcs8-encrypted.pem -passout pass:secret

Converting a PKCS #12 (PFX) file to private key and certificate pair

PKCS #12 key stores (PFX files) are commonly used on Microsoft Windows. This needs to be done only if you have to convert PKCS #12 Keys to be used with Graylog.

In this example, the PKCS #12 (PFX) file is named keystore.pfx:

$ openssl pkcs12 -in keystore.pfx -nokeys -out graylog-certificate.pem
$ openssl pkcs12 -in keystore.pfx -nocerts -out graylog-pkcs5.pem
$ openssl pkcs8 -in graylog-pkcs5.pem -topk8 -out graylog-key.pem

The resulting graylog-certificate.pem and graylog-key.pem can be used in the Graylog configuration file.

Converting an existing Java Keystore to private key/certificate pair

This section describes how to export a private key and certificate from an existing Java KeyStore in JKS format. This is needed if you want to export the certificates from the Java KeyStore.

The starting point is an existing Java KeyStore in JKS format which contains a private key and certificate which should be used in Graylog:

$ keytool -list -v -keystore keystore.jks -alias graylog.example.com
Enter keystore password:
Alias name: graylog.example.com
Creation date: May 10, 2016
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=graylog.example.com, OU=Unknown, O="Graylog, Inc.", L=Hamburg, ST=Hamburg, C=DE
Issuer: CN=graylog.example.com, OU=Unknown, O="Graylog, Inc.", L=Hamburg, ST=Hamburg, C=DE
Serial number: 2b33832d
Valid from: Tue May 10 10:02:34 CEST 2016 until: Mon Aug 08 10:02:34 CEST 2016
Certificate fingerprints:
       MD5:  8A:3D:9F:ED:69:93:1B:6C:E3:29:66:EA:82:8D:42:BE
       SHA1: 5B:27:92:25:46:36:BC:F0:82:8F:9A:30:D8:50:D0:ED:32:4D:C6:A0
       SHA256: 11:11:77:F5:F6:6A:20:A8:E6:4A:5D:B5:20:21:4E:B8:FE:B6:38:1D:45:6B:ED:D0:7B:CE:B8:C8:BC:DD:B4:FB
       Signature algorithm name: SHA256withRSA
       Version: 3

Extensions:

#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: AC 79 64 9F A1 60 14 B9   51 F4 F5 0B B3 B5 02 A5  .yd..`..Q.......
0010: B8 07 DC 7B                                        ....
]
]

The Java KeyStore in JKS format has to be converted to a PKCS#12 keystore, so that OpenSSL can work with it:

$ keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.p12 -deststoretype PKCS12
Enter destination keystore password:
Re-enter new password:
Enter source keystore password:
Entry for alias graylog.example.com successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

After the keystore has been successfully converted into PKCS#12 format, OpenSSL can export the X.509 certificate with PEM encoding:

$ openssl pkcs12 -in keystore.p12 -nokeys -out graylog-certificate.pem
Enter Import Password:
MAC verified OK

The private key can only be exported in PKCS#5 format with PEM encoding:

$ openssl pkcs12 -in keystore.p12 -nocerts -out graylog-pkcs5.pem
Enter Import Password:
MAC verified OK
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:

Graylog currently only supports PKCS#8 private keys with PEM encoding, so OpenSSL has to convert it into the correct format:

$ openssl pkcs8 -in graylog-pkcs5.pem -topk8 -out graylog-key.pem
Enter pass phrase for graylog-pkcs5.pem:
Enter Encryption Password:
Verifying - Enter Encryption Password:

The working directory should now contain the PKCS#8 private key (graylog-key.pem) and the X.509 certificate (graylog-certificate.pem) to be used with Graylog:

$ head graylog-key.pem graylog-certificate.pem
==> graylog-key.pem <==
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIE6TAbBgkqhkiG9w0BBQMwDgQIwMhLa5bw9vgCAggABIIEyN42AeYJJNBEiqhI
mWqJDot4Jokw2vB4abcIJ5Do4+7tjtMrecVRCDSvBZzjkXjnbumBHEoxexe5f0/z
wgq6f/UDyTM3uKYQTG91fcqTyMDUlo3Wc8OqSqsNehOAQzA7hMCehqgNJHO0Zfny
EFvrXHurJWi4eA9vLRup86dbm4Wp3o8pmjOLduXieHfcgVtm5jfd7XfL5cRFS8kS
bSFH4v8xDxLNaJmKkKl9gPCACMRbO9nGk/Z9q9N8zkj+xG9lxlNRMX51SRzg20E0
nyyKTb39tJF35zjroB2HfiFWyrPQ1uF6yGoroGvu0L3eWosjBLjdRs0eBgjJCm5P
ic9zSVqMH6/4CPKJqvB97vP4QhpYcr9jlYJsbn6Zg4OIELpM00VLvp0yU9tqTuRR
TDPYAlNMLZ2RrV52CEsh3zO21WHM7r187x4WHgprDFnjkXf02DrFhgCsGwkEQnb3
vj86q13RHhqoXT4W0zugvcv2/NBLMv0HNQBAfEK3X1YBmtQpEJhwSxeszA1i7CpU

==> graylog-certificate.pem <==
Bag Attributes
    friendlyName: graylog.example.com
    localKeyID: 54 69 6D 65 20 31 34 36 32 38 36 37 38 32 33 30 39 32
subject=/C=DE/ST=Hamburg/L=Hamburg/O=Graylog, Inc./OU=Unknown/CN=graylog.example.com
issuer=/C=DE/ST=Hamburg/L=Hamburg/O=Graylog, Inc./OU=Unknown/CN=graylog.example.com
-----BEGIN CERTIFICATE-----
MIIDkTCCAnmgAwIBAgIEKzODLTANBgkqhkiG9w0BAQsFADB5MQswCQYDVQQGEwJE
RTEQMA4GA1UECBMHSGFtYnVyZzEQMA4GA1UEBxMHSGFtYnVyZzEWMBQGA1UEChMN
R3JheWxvZywgSW5jLjEQMA4GA1UECxMHVW5rbm93bjEcMBoGA1UEAxMTZ3JheWxv
Zy5leGFtcGxlLmNvbTAeFw0xNjA1MTAwODAyMzRaFw0xNjA4MDgwODAyMzRaMHkx

The resulting PKCS#8 private key (graylog-key.pem) and the X.509 certificate (graylog-certificate.pem) can now be used to enable encrypted connections with Graylog by enabling TLS for the Graylog REST API and the web interface in the Graylog configuration file:

# Enable HTTPS support for the HTTP interface.
# This secures the communication with the HTTP interface with TLS to prevent request forgery and eavesdropping.
http_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the HTTP interface.
http_tls_cert_file = /path/to/graylog-certificate.pem

# The PKCS#8 private key file in PEM format to use for securing the HTTP interface.
http_tls_key_file = /path/to/graylog-key.pem

# The password to unlock the private key used for securing the HTTP interface. (if key is encrypted)
http_tls_key_password = secret

Sample files

This section shows the difference between following private key formats with samples. It will help you to identify between the following private key formats and provides samples.

PKCS#5 plain private key:

-----BEGIN RSA PRIVATE KEY-----
MIIBOwIBAAJBANxtmQ1Kccdp7HBNt8zgTai48Vv617bj4SnhkcMN99sCQ2Naj/sp
[...]
NiCYNLiCawBbpZnYw/ztPVACK4EwOpUy+u19cMB0JA==
-----END RSA PRIVATE KEY-----

PKCS#8 plain private key:

-----BEGIN PRIVATE KEY-----
MIIBVAIBADANBgkqhkiG9w0BAQEFAASCAT4wggE6AgEAAkEA6GZN0rQFKRIVaPOz
[...]
LaLGdd9G63kLg85eldSy55uIAXsvqQIgfSYaliVtSbAgyx1Yfs3hJ+CTpNKzTNv/
Fx80EltYV6k=
-----END PRIVATE KEY-----

PKCS#5 encrypted private key:

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,E83B4019057F55E9

iIPs59nQn4RSd7ppch9/vNE7PfRSHLoQFmaAjaF0DxjV9oucznUjJq2gphAB2E2H
[...]
y5IT1MZPgN3LNkVSsLPWKo08uFZQdfu0JTKcn7NPyRc=
-----END RSA PRIVATE KEY-----

PKCS#8 encrypted private key:

-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIBpjBABgkqhkiG9w0BBQ0wMzAbBgkqhkiG9w0BBQwwDgQIU9Y9p2EfWucCAggA
[...]
IjsZNp6zmlqf/RXnETsJjGd0TXRWaEdu+XOOyVyPskX2177X9DUJoD31
-----END ENCRYPTED PRIVATE KEY-----

Adding a self-signed certificate to the JVM trust store

Graylog nodes inside a cluster need to communicate with each other using the Graylog REST API. When using HTTPS for the Graylog REST API, the X.509 certificate must be trusted by the JVM trust store (similar to the trusted CA bundle in an operating system), otherwise communication will fail.

Important

If you are using different X.509 certificates for each Graylog node, you have to add all of them into the JVM trust store of each Graylog node.

The default trust store of an installed Java runtime environment can be found at $JAVA_HOME/jre/lib/security/cacerts. In order not to “pollute” the official trust store, we make a copy of it which we will use with Graylog instead:

$ cp -a "${JAVA_HOME}/jre/lib/security/cacerts" /path/to/cacerts.jks

After the original key store file has been copied, we can add the self-signed certificate (cert.pem, see Creating a self-signed private key/certificate) to the key store (the default password is changeit):

$ keytool -importcert -keystore /path/to/cacerts.jks -storepass changeit -alias graylog-self-signed -file cert.pem
Owner: CN=graylog.example.com, O="Graylog, Inc.", L=Hamburg, ST=Hamburg, C=DE
Issuer: CN=graylog.example.com, O="Graylog, Inc.", L=Hamburg, ST=Hamburg, C=DE
Serial number: 8c80134cee556734
Valid from: Tue Jun 14 16:38:17 CEST 2016 until: Wed Jun 14 16:38:17 CEST 2017
Certificate fingerprints:
       MD5:  69:D1:B3:01:46:0D:E9:45:FB:C6:6C:69:EA:38:ED:3E
       SHA1: F0:64:D0:1B:3B:6B:C8:01:D5:4D:33:36:87:F0:FB:10:E1:36:21:9E
       SHA256: F7:F2:73:3D:86:DC:10:22:1D:14:B8:5D:66:B4:EB:48:FD:3D:74:89:EC:C4:DF:D0:D2:EC:F8:5D:78:49:E7:2F
       Signature algorithm name: SHA1withRSA
       Version: 3

Extensions:

[Other details about the certificate...]

Trust this certificate? [no]:  yes
Certificate was added to keystore

To verify that the self-signed certificate has indeed been added, it can be listed with the following command:

$ keytool -keystore /path/to/cacerts.jks -storepass changeit -list | grep graylog-self-signed -A1
graylog-self-signed, Jun 14, 2016, trustedCertEntry,
Certificate fingerprint (SHA1): F0:64:D0:1B:3B:6B:C8:01:D5:4D:33:36:87:F0:FB:10:E1:36:21:9E

The printed certificate fingerprint (SHA1) should match the one printed when importing the self-signed certificate.

In order for the JVM to pick up the new trust store, it has to be started with the JVM parameter -Djavax.net.ssl.trustStore=/path/to/cacerts.jks. If you’ve been using another password to encrypt the JVM trust store than the default changeit, you additionally have to set the JVM parameter -Djavax.net.ssl.trustStorePassword=secret.

Most start and init scripts for Graylog provide a JAVA_OPTS variable which can be used to pass the javax.net.ssl.trustStore and (optionally) javax.net.ssl.trustStorePassword system properties.

Note

The default location to change the JVM parameter depends on your installation type and is documented with all other default locations.

Warning

Without adding the previously created Java keystore to the JVM parameters, Graylog won’t be able to verify any self-signed certificates or custom CA certificates.

Disabling specific TLS ciphers and algorithms

Since Java 7u76 it is possible to disable specific TLS algorithms and ciphers for secure connections.

In order to disable specific TLS algorithms and ciphers, you need to provide a properties file with a list of disabled algorithms and ciphers. Take a look at the example security.properties in the Graylog source repository.

For example, if you want to disable all algorithms except for TLS 1.2, the properties file has to contain the following line:

jdk.tls.disabledAlgorithms=SSLv2Hello, SSLv3, TLSv1, TLSv1.1

If additionally you want to disable DSA/RSA key sizes lower than 2048 bits and EC key sizes lower than 160 bits, the properties file has to contain the following line:

jdk.tls.disabledAlgorithms=SSLv2Hello, SSLv3, TLSv1, TLSv1.1, EC keySize < 160, RSA keySize < 2048, DSA keySize < 2048

To load the properties file into a JVM, you have to pass it to Java using the java.security.properties system property:

java -Djava.security.properties=/path/to/security.properties -jar /path/to/graylog.jar server

Most start and init scripts for Graylog provide a JAVA_OPTS variable which can be used to pass the java.security.properties system property.

Multi-node Setup

This guide doesn’t provide a step-by-step tutorial for building a multi-node Graylog cluster but does simply give some advice for questions that might arise during the setup.

It’s important for such a project that you understand each step in the setup process and do some planning upfront. Without a proper roadmap of all the things you want to achieve with a Graylog cluster, you will be lost on the way.

Graylog should be the last component you install in this setup. Its dependencies, namely MongoDB and Elasticsearch, have to be up and running first.

Important

This guide doesn’t include instructions for running a multi-node Graylog cluster in an untrusted network. We assume that the connection between the hosts is trusted and doesn’t have to be secured individually.

Prerequisites

Every server which is part of this setup should have the software requirements installed to run the targeted software. All software requirements can be found in the installation manual.

We highly recommend that the system time on all systems is kept in sync via NTP or a similar mechanism. Needless to say that DNS resolution must be working, too. Because everything is a freaking DNS problem.

In order to simplify the installation process, the servers should have a working Internet connection.

MongoDB replica set

We recommend to deploy a MongoDB replica set.

MongoDB doesn’t have to run on dedicated servers for the workload generated by Graylog, but you should follow the recommendations given in the MongoDB documentation about architecture. Most important is that you have an odd number of MongoDB servers in the replica set.

In most setups, each Graylog server will also host an instance of MongoDB which is part of the same replica set and shares the data with all other nodes in the cluster.

Note

To avoid unauthorized access to your MongoDB database, the MongoDB replica set should be setup with authentication.

The correct order of working steps should be as follows:

  1. Create the replica set (rs01)
  2. Create the database (graylog)
  3. Create a user account for accessing the database, which has the roles readWrite and dbAdmin.

If your MongoDB needs to be reachable over network you should set the IP with bind_ip in the configuration.

Elasticsearch cluster

The Elasticsearch setup documentation should help you to install Elasticsearch with a robust base configuration.

It is important to name the Elasticsearch cluster not simply named elasticsearch to avoid accidental conflicts with Elasticsearch nodes using the default configuration. Just choose anything else (we recommend graylog), because this is the default name and any Elasticsearch instance that is started in the same network will try to connect to this cluster.

The Elasticsearch servers need one IP that can be reached over network set in network.host and some participants of the cluster in discovery.zen.ping.unicast.hosts. That is enough to have a minimal cluster setup.

When you secure your Elasticsearch with User Authentification you need to add credentials to the Graylog configuration to be able to use the secured Elasticsearch cluster with Graylog.

Graylog Multi-node

After the installation of Graylog, you should take care that only one Graylog node is configured to be master with the configuration setting is_master = true.

The HTTP settings configured in http_bind_address (or http_publish_uri) must be accessable for all Graylog nodes of the cluster.

Graylog to MongoDB connection

The mongodb_uri configuration setting must include all MongoDB nodes forming the replica set, the name of the replica set, as well as the previously configured user account with access to the replica set. The configuration setting is a normal MongoDB connection string.

Finally, the MongoDB connection string in the Graylog configuration file should look like this:

mongodb_uri = mongodb://USERNAME:PASSWORD@mongodb-node01:27017,mongodb-node02:27017,mongodb-node03:27017/graylog?replicaSet=rs01
Graylog to Elasticsearch connection

Graylog will connect to the Elasticsearch REST API.

To avoid issues with the connection to the Elasticsearch cluster you should add some of the network addresses of the Elasticsearch nodes to elasticsearch_hosts.

Graylog web interface

It’s possible to use a loadbalancer in front of all Graylog servers, please refer to Making the web interface work with load balancers/proxies for more details.

Depending on your setup, it’s possible to either use a hardware loadbalancer for TLS/HTTPS termination, a reverse proxy, or to simply enable it in the Graylog node.

Scaling

Each component in this multi-node setup can be scaled on the individual needs.

Depending on the amount of messages ingested and how long messages should be available for direct search, the Elasticsearch cluster will need most of the resources on your setup.

Keep an eye on the Metrics of each part of the cluster. One option is to use telegraf to fetch importand metrics and store them in your favorite metric system (e. g. Graphite, Prometheus or Influx).

Elasticseach Metrics and some administration can be done with Elastic HQ or Cerebro. Those will help you to understand the Elasticsearch cluster health and behavior.

Graylog Metrics can be monitored with the Graylog Metrics Reporter plugins which are able to send the internal Graylog metrics to your favorite metrics collector (e. g. Graphite or Prometheus).

Up until today, we have almost never faced the issue that the MongoDB replica set needed special attention. But of course you should still monitor it and store its metrics - just to be sure.

Troubleshooting

  • After every configuration change or service restart, watch the logfile of the applications you have worked on. Sometimes other log files can also give you hints about what went wrong. For example if you’re configuring Graylog and try to find out why the connection to the MongoDB isn’t working, the MongoDB logs can help to identify the problem.
  • If HTTPS has been enabled for the Graylog REST API, it need to be setup for the Graylog web interface, too.

Elasticsearch

We strongly recommend to use a dedicated Elasticsearch cluster for your Graylog setup.

If you are using a shared Elasticsearch setup, a problem with indices unrelated to Graylog might turn the cluster status to YELLOW or RED and impact the availability and performance of your Graylog setup.

Elasticsearch versions

Starting with version 2.3, Graylog uses the HTTP protocol to connect to your Elasticsearch cluster, so it does not have a hard requirement for the Elasticsearch version anymore. We can safely assume that any version starting from 2.x is working.

Caution

Graylog 2.4 does not work with Elasticsearch 6.x!

Note

Graylog works fine with the Amazon Elasticsearch Service using Elasticsearch 5.3.x or later.

Configuration

Caution

As Graylog has switched from an embedded Elasticsearch node client to a lightweight HTTP client in version 2.3, please check the upgrade notes how to migrate your configuration if you are switching from an earlier version.

Graylog

The most important setting to make a successful connection is a list of comma-separated URIs to one or more Elasticsearch nodes. Graylog needs to know the address of at least one other Elasticsearch node given in the elasticsearch_hosts setting. The specified value should at least contain the scheme (http:// for unencrypted, https:// for encrypted connections), the hostname or IP and the port of the HTTP listener (which is 9200 unless otherwise configured) of this node. Optionally, you can also specify an authentication section containing a user name and a password, if either your Elasticsearch node uses Shield/X-Pack or Search Guard, or you have an intermediate HTTP proxy requiring authentication in between the Graylog server and the Elasticsearch node. Additionally you can specify an optional path prefix at the end of the URI.

A sample specification of elasticsearch_hosts could look like this:

elasticsearch_hosts = http://es-node-1.example.org:9200/foo,https://someuser:somepassword@es-node-2.example.org:19200

Caution

Graylog assumes that all nodes in the cluster are running the same versions of Elasticsearch. While it might work when patch-levels differ, we highly encourage to keep versions consistent.

Warning

Graylog does not react to externally triggered index changes (creating/closing/reopening/deleting an index) anymore. All of these actions need to be performed through the Graylog REST API in order to retain index consistency.

Available Elasticsearch configuration tunables

The following configuration options are now being used to configure connectivity to Elasticsearch:

Config Setting Type Comments Default
elasticsearch_connect_timeout Duration Timeout when connection to individual Elasticsearch hosts 10s (10 Seconds)
elasticsearch_hosts List<URI> Comma-separated list of URIs of Elasticsearch hosts http://127.0.0.1:9200
elasticsearch_idle_timeout Duration Timeout after which idle connections are terminated -1s (Never)
elasticsearch_max_total_connections int Maximum number of total Elasticsearch connections 20
elasticsearch_max_total_connections_per_route int Maximum number of Elasticsearch connections per route/host 2
elasticsearch_socket_timeout Duration Timeout when sending/receiving from Elasticsearch connection 60s (60 Seconds)
elasticsearch_discovery_enabled boolean Enable automatic Elasticsearch node discovery false
elasticsearch_discovery_filter String Filter by node attributes for the discovered nodes empty (use all nodes)
elasticsearch_discovery_frequency Duration Frequency of the Elasticsearch node discovery 30s (30 Seconds)
elasticsearch_compression_enabled boolean Enable GZIP compression of Elasticseach request payloads false
Automatic node discovery

Caution

Authentication with the Elasticsearch cluster will not work if the automatic node discovery is being used.

Caution

Automatic node discovery does not work when using the Amazon Elasticsearch Service because Amazon blocks certain Elasticsearch API endpoints.

Graylog uses automatic node discovery to gather a list of all available Elasticsearch nodes in the cluster at runtime and distribute requests among them to potentially increase performance and availability. To enable this feature, you need to set the elasticsearch_discovery_enabled to true. Optionally, you can define the a filter allowing to selectively include/exclude discovered nodes (details how to specify node filters are found in the Elasticsearch cluster documentation) using the elasticsearch_discovery_filter setting, or tuning the frequency of the node discovery using the elasticsearch_discovery_frequency configuration option.

Configuration of Elasticsearch nodes
Control access to Elasticsearch ports

If you are not using Shield/X-Pack or Search Guard to authenticate access to your Elasticsearch nodes, make sure to restrict access to the Elasticsearch ports (default: 9200/tcp and 9300/tcp). Otherwise the data is readable by anyone who has access to the machine over network.

Open file limits

Because Elasticsearch has to keep a lot of files open simultaneously it requires a higher open file limit that the usual operating system defaults allow. Set it to at least 64000 open file descriptors.

Graylog will show a notification in the web interface when there is a node in the Elasticsearch cluster which has a too low open file limit.

Read about how to raise the open file limit in the corresponding 5.x / 6.x documentation pages.

Heap size

It is strongly recommended to raise the standard size of heap memory allocated to Elasticsearch. Just set the ES_HEAP_SIZE environment variable to for example 24g to allocate 24GB. We recommend to use around 50% of the available system memory for Elasticsearch (when running on a dedicated host) to leave enough space for the system caches that Elasticsearch uses a lot. But please take care that you don’t cross 32 GB!

Merge throttling

Elasticsearch is throttling the merging of Lucene segments to allow extremely fast searches. This throttling however has default values that are very conservative and can lead to slow ingestion rates when used with Graylog. You would see the message journal growing without a real indication of CPU or memory stress on the Elasticsearch nodes. It usually goes along with Elasticsearch INFO log messages like this:

now throttling indexing

When running on fast IO like SSDs or a SAN we recommend to increase the value of the indices.store.throttle.max_bytes_per_sec in your elasticsearch.yml to 150MB:

indices.store.throttle.max_bytes_per_sec: 150mb

Play around with this setting until you reach the best performance.

Tuning Elasticsearch

Graylog is already setting specific configuration for every index it is managing. This is enough tuning for a lot of use cases and setups.

More detailed information about the configuration of Elasticsearch can be found in the official documentation.

Avoiding split-brain and shard shuffling

Split-brain events

Elasticsearch sacrifices consistency in order to ensure availability, and partition tolerance. The reasoning behind that is that short periods of misbehaviour are less problematic than short periods of unavailability. In other words, when Elasticsearch nodes in a cluster are unable to replicate changes to data, they will keep serving applications such as Graylog. When the nodes are able to replicate their data, they will attempt to converge the replicas and to achieve eventual consistency.

Elasticsearch tackles the previous by electing master nodes, which are in charge of database operations such as creating new indices, moving shards around the cluster nodes, and so forth. Master nodes coordinate their actions actively with others, ensuring that the data can be converged by non-masters. The cluster nodes that are not master nodes are not allowed to make changes that would break the cluster.

The previous mechanism can in some circumstances fail, causing a split-brain event. When an Elasticsearch cluster is split into two sides, both thinking they are the master, data consistency is lost as the masters work independently on the data. As a result the nodes will respond differently to same queries. This is considered a catastrophic event, because the data from two masters can not be rejoined automatically, and it takes quite a bit of manual work to remedy the situation.

Avoiding split-brain events

Elasticsearch nodes take a simple majority vote over who is master. If the majority agrees that they are the master, then most likely the disconnected minority has also come to conclusion that they can not be the master, and everything is just fine. This mechanism requires at least 3 nodes to work reliably however, because one or two nodes can not form a majority.

The minimum amount of master nodes required to elect a master must be configured manually in elasticsearch.yml:

# At least NODES/2+1 on clusters with NODES > 2, where NODES is the number of master nodes in the cluster
discovery.zen.minimum_master_nodes: 2

The configuration values should typically for example:

Master nodes minimum_master_nodes Comments
1 1  
2 1 With 2 the other node going down would stop the cluster from working!
3 2  
4 3  
5 3  
6 4  

Some of the master nodes may be dedicated master nodes, meaning they are configured just to handle lightweight operational (cluster management) responsibilities. They will not handle or store any of the cluster’s data. The function of such nodes is similar to so called witness servers on other database products, and setting them up on dedicated witness sites will greatly reduce the chance of Elasticsearch cluster instability.

A dedicated master node has the following configuration in elasticsearch.yml:

node.data: false
node.master: true
Shard shuffling

When cluster status changes, for example because of node restarts or availability issues, Elasticsearch will start automatically rebalancing the data in the cluster. The cluster works on making sure that the amount of shards and replicas will conform to the cluster configuration. This is a problem if the status changes are just temporary. Moving shards and replicas around in the cluster takes considerable amount of resources, and should be done only when necessary.

Avoiding unnecessary shuffling

Elasticsearch has couple configuration options, which are designed to allow short times of unavailability before starting the recovery process with shard shuffling. There are 3 settings that may be configured in elasticsearch.yml:

# Recover only after the given number of nodes have joined the cluster. Can be seen as "minimum number of nodes to attempt recovery at all".
gateway.recover_after_nodes: 8
# Time to wait for additional nodes after recover_after_nodes is met.
gateway.recover_after_time: 5m
# Inform ElasticSearch how many nodes form a full cluster. If this number is met, start up immediately.
gateway.expected_nodes: 10

The configuration options should be set up so that only minimal node unavailability is tolerated. For example server restarts are common, and should be done in managed manner. The logic is that if you lose large part of your cluster, you probably should start re-shuffling the shards and replicas without tolerating the situation.

Custom index mappings

Sometimes it’s useful to not rely on Elasticsearch’s dynamic mapping but to define a stricter schema for messages.

Note

If the index mapping is conflicting with the actual message to be sent to Elasticsearch, indexing that message will fail.

Graylog itself is using a default mapping which includes settings for the timestamp, message, full_message, and source fields of indexed messages:

$ curl -X GET 'http://localhost:9200/_template/graylog-internal?pretty'
{
  "graylog-internal" : {
    "order" : -2147483648,
    "template" : "graylog_*",
    "settings" : { },
    "mappings" : {
      "message" : {
        "_ttl" : {
          "enabled" : true
        },
        "_source" : {
          "enabled" : true
        },
        "dynamic_templates" : [ {
          "internal_fields" : {
            "mapping" : {
              "index" : "not_analyzed",
              "type" : "string"
            },
            "match" : "gl2_*"
          }
        }, {
          "store_generic" : {
            "mapping" : {
              "index" : "not_analyzed"
            },
            "match" : "*"
          }
        } ],
        "properties" : {
          "full_message" : {
            "analyzer" : "standard",
            "index" : "analyzed",
            "type" : "string"
          },
          "streams" : {
            "index" : "not_analyzed",
            "type" : "string"
          },
          "source" : {
            "analyzer" : "analyzer_keyword",
            "index" : "analyzed",
            "type" : "string"
          },
          "message" : {
            "analyzer" : "standard",
            "index" : "analyzed",
            "type" : "string"
          },
          "timestamp" : {
            "format" : "yyyy-MM-dd HH:mm:ss.SSS",
            "type" : "date"
          }
        }
      }
    },
    "aliases" : { }
  }
}

In order to extend the default mapping of Elasticsearch and Graylog, you can create one or more custom index mappings and add them as index templates to Elasticsearch.

Let’s say we have a schema for our data like the following:

Field Name Field Type Example
http_method string GET
http_response_code long 200
ingest_time date 2016-06-13T15:00:51.927Z
took_ms long 56

This would translate to the following additional index mapping in Elasticsearch:

"mappings" : {
  "message" : {
    "properties" : {
      "http_method" : {
        "type" : "string",
        "index" : "not_analyzed"
      },
      "http_response_code" : {
        "type" : "long"
      },
      "ingest_time" : {
        "type" : "date",
        "format": "strict_date_time"
      },
      "took_ms" : {
        "type" : "long"
      }
    }
  }
}

The format of the ingest_time field is described in the Elasticsearch documentation about the format mapping parameter. Also make sure to check the Elasticsearch documentation about Field datatypes.

In order to apply the additional index mapping when Graylog creates a new index in Elasticsearch, it has to be added to an index template. The Graylog default template (graylog-internal) has the lowest priority and will be merged with the custom index template by Elasticsearch.

Warning

If the default index mapping and the custom index mapping cannot be merged (e. g. because of conflicting field datatypes), Elasticsearch will throw an exception and won’t create the index. So be extremely cautious and conservative about the custom index mappings!

Creating a new index template

Save the following index template for the custom index mapping into a file named graylog-custom-mapping.json:

{
  "template": "graylog_*",
  "mappings" : {
    "message" : {
      "properties" : {
        "http_method" : {
          "type" : "string",
          "index" : "not_analyzed"
        },
        "http_response_code" : {
          "type" : "long"
        },
        "ingest_time" : {
          "type" : "date",
          "format": "strict_date_time"
        },
        "took_ms" : {
          "type" : "long"
        }
      }
    }
  }
}

Finally, load the index mapping into Elasticsearch with the following command:

$ curl -X PUT -d @'graylog-custom-mapping.json' -H 'Content-Type: application/json' 'http://localhost:9200/_template/graylog-custom-mapping?pretty'
{
  "acknowledged" : true
}

Every Elasticsearch index created from that time on, will have an index mapping consisting of the original graylog-internal index template and the new graylog-custom-mapping template:

$ curl -X GET 'http://localhost:9200/graylog_deflector/_mapping?pretty'
{
  "graylog_2" : {
    "mappings" : {
      "message" : {
        "_ttl" : {
          "enabled" : true
        },
        "dynamic_templates" : [ {
          "internal_fields" : {
            "mapping" : {
              "index" : "not_analyzed",
              "type" : "string"
            },
            "match" : "gl2_*"
          }
        }, {
          "store_generic" : {
            "mapping" : {
              "index" : "not_analyzed"
            },
            "match" : "*"
          }
        } ],
        "properties" : {
          "full_message" : {
            "type" : "string",
            "analyzer" : "standard"
          },
          "http_method" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "http_response_code" : {
            "type" : "long"
          },
          "ingest_time" : {
            "type" : "date",
            "format" : "strict_date_time"
          },
          "message" : {
            "type" : "string",
            "analyzer" : "standard"
          },
          "source" : {
            "type" : "string",
            "analyzer" : "analyzer_keyword"
          },
          "streams" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "timestamp" : {
            "type" : "date",
            "format" : "yyyy-MM-dd HH:mm:ss.SSS"
          },
          "took_ms" : {
            "type" : "long"
          }
        }
      }
    }
  }
}

Note

When using different index sets every index set can have its own mapping.

Deleting custom index templates

If you want to remove an existing index template from Elasticsearch, simply issue a DELETE request to Elasticsearch:

$ curl -X DELETE 'http://localhost:9200/_template/graylog-custom-mapping?pretty'
{
  "acknowledged" : true
}

After you’ve removed the index template, new indices will only have the original index mapping:

$ curl -X GET 'http://localhost:9200/graylog_deflector/_mapping?pretty'
{
  "graylog_3" : {
    "mappings" : {
      "message" : {
        "_ttl" : {
          "enabled" : true
        },
        "dynamic_templates" : [ {
          "internal_fields" : {
            "mapping" : {
              "index" : "not_analyzed",
              "type" : "string"
            },
            "match" : "gl2_*"
          }
        }, {
          "store_generic" : {
            "mapping" : {
              "index" : "not_analyzed"
            },
            "match" : "*"
          }
        } ],
        "properties" : {
          "full_message" : {
            "type" : "string",
            "analyzer" : "standard"
          },
          "message" : {
            "type" : "string",
            "analyzer" : "standard"
          },
          "source" : {
            "type" : "string",
            "analyzer" : "analyzer_keyword"
          },
          "streams" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "timestamp" : {
            "type" : "date",
            "format" : "yyyy-MM-dd HH:mm:ss.SSS"
          }
        }
      }
    }
  }
}

Additional information on Elasticsearch Index Templates can be found in the official Elasticsearch Template Documentation

Note

Settings and index mappings in templates are only applied to new indices. After adding, modifying, or deleting an index template, you have to manually rotate the write-active indices of your index sets for the changes to take effect.

Rotate indices manually

Select the desired index set on the System / Indices page in the Graylog web interface by clicking on the name of the index set, then select “Rotate active write index” from the “Maintenance” dropdown menu.

_images/rotate_index_1.png _images/rotate_index_2.png

Cluster Status explained

Elasticsearch provides a classification for the cluster health.

The cluster status applies to different levels:

  • Shard level - see status descriptions below
  • Index level - inherits the status of the worst shard status
  • Cluster level - inherits the status of the worst index status

That means that the Elasticsearch cluster status can turn red if a single index or shard has problems even though the rest of the indices/shards are okay.

Note

Graylog checks the status of the current write index while indexing messages. If that one is GREEN or YELLOW, Graylog will continue to write messages into Elasticsearch regardless of the overall cluster status.

Explanation of the different status levels:

RED

The RED status indicates that some or all of the primary shards are not available.

In this state, no searches can be performed until all primary shards have been restored.

YELLOW

The YELLOW status means that all of the primary shards are available but some or all shard replicas are not.

When the index configuration include replication with a count that is equal or higher than the number of nodes, your cluster cannot become green. In most cases, this can be solved by adding another Elasticsearch node to the cluster or by reducing the replication factor of the indices.

GREEN

The cluster is fully operational. All primary and replica shards are available.

Index model

Overview

Graylog is transparently managing one or more sets of Elasticsearch indices to optimize search and analysis operations for speed and low resource consumption.

To enable managing indices with different mappings, analyzers, and replication settings Graylog is using so-called index sets which are an abstraction of all these settings.

_images/index_set_overview.png

Each index set contains the necessary settings for Graylog to create, manage, and fill Elasticsearch indices and handle index rotation and data retention for specific requirements.

_images/index_set_details.png

Graylog is maintaining an index alias per index set which is always pointing to the current write-active index from that index set. There is always exactly one index to which new messages are written until the configured rotation criterion (number of documents, index size, or index age) has been met.

A background task continuously checks if the rotation criterion of an index set has been met and a new index is created and prepared when that happens. Once the index is ready, the index alias is atomically switched to it. That means that all Graylog nodes can write messages to the into alias without even knowing what the currently write-active index of the index set is.

_images/index_model_write.png

Almost every read operation is performed with a given time range. Because Graylog is writing messages sequentially into Elasticsearch it can keep information about the time range each index covers. It selects a lists of indices to query when having a time range provided. If no time range was provided, it will search in all indices it knows.

_images/index_model_read.png
Eviction of indices and messages

There are configuration settings for the maximum number of indices Graylog is managing in a given index set.

Depending on the configured retention strategy, the oldest indices of an index set will automatically be closed, deleted, or exported when the configured maximum number of indices has been reached.

The deletion is performed by the Graylog master node in a background thread which is continuously comparing the number of indices with the configured maximum:

INFO : org.graylog2.indexer.rotation.strategies.AbstractRotationStrategy - Deflector index <graylog_95> should be rotated, Pointing deflector to new index now!
INFO : org.graylog2.indexer.MongoIndexSet - Cycling from <graylog_95> to <graylog_96>.
INFO : org.graylog2.indexer.MongoIndexSet - Creating target index <graylog_96>.
INFO : org.graylog2.indexer.indices.Indices - Created Graylog index template "graylog-internal" in Elasticsearch.
INFO : org.graylog2.indexer.MongoIndexSet - Waiting for allocation of index <graylog_96>.
INFO : org.graylog2.indexer.MongoIndexSet - Index <graylog_96> has been successfully allocated.
INFO : org.graylog2.indexer.MongoIndexSet - Pointing index alias <graylog_deflector> to new index <graylog_96>.
INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <f1018ae0-dcaa-11e6-97c3-6c4008b8fc28> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
INFO : org.graylog2.indexer.MongoIndexSet - Successfully pointed index alias <graylog_deflector> to index <graylog_96>.

Index Set Configuration

Index sets have a variety of different settings related to how Graylog will store messages into the Elasticsearch cluster.

_images/index_set_create.png
  • Title: A descriptive name of the index set.
  • Description: A description of the index set for human consumption.
  • Index prefix: A unique prefix used for Elasticsearch indices managed by the index set. The prefix must start with a letter or number, and can only contain letters, numbers, _, - and +. The index alias will be named accordingly, e. g. graylog_deflector if the index prefix was graylog.
  • Analyzer: (default: standard) The Elasticsearch analyzer for the index set.
  • Index shards: (default: 4) The number of Elasticsearch shards used per index.
  • Index replicas: (default: 0) The number of Elasticsearch replicas used per index.
  • Max. number of segments: (default: 1) The maximum number of segments per Elasticsearch index after index optimization (force merge), see Segment Merging for details.
  • Disable index optimization after rotation: Disable Elasticsearch index optimization (force merge) after index rotation. Only activate this if you have serious problems with the performance of your Elasticsearch cluster during the optimization process.
Index rotation
  • Message count: Rotates the index after a specific number of messages have been written.
  • Index size: Rotates the index after an approximate size on disk (before optimization) has been reached.
  • Index time: Rotates the index after a specific time (e. g. 1 hour or 1 week).
_images/index_set_create_rotation.png
Index retention
  • Delete: Delete indices in Elasticsearch to minimize resource consumption.
  • Close: Close indices in Elasticsearch to reduce resource consumption.
  • Do nothing
  • Archive: Commercial feature, see Archiving.
_images/index_set_create_retention.png

Maintenance

Keeping the index ranges in sync

Graylog will take care of calculating index ranges automatically as soon as a new index has been created.

In case the stored metadata about index time ranges has run out of sync, Graylog will notify you in the web interface. This can happen if an index was deleted manually or messages from already “closed” indices were removed.

The system will offer you to just re-generate all time range information. This may take a few seconds but is an easy task for Graylog.

You can easily re-build the information yourself after manually deleting indices or doing other changes that might cause synchronization problems:

$ curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild

This will trigger a system job:

INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Recalculating index ranges.
INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <9b64a9d0-dcac-11e6-97c3-6c4008b8fc28> [org.graylog2.indexer.ranges.RebuildIndexRangesJob]
INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Recalculating index ranges for index set Default index set (graylog2_*): 5 indices affected.
INFO : org.graylog2.indexer.ranges.MongoIndexRangeService - Calculated range of [graylog_96] in [7ms].
INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Created ranges for index graylog_96: MongoIndexRange{id=null, indexName=graylog_96, begin=2017-01-17T11:49:02.529Z, end=2017-01-17T12:00:01.492Z, calculatedAt=2017-01-17T12:00:58.097Z, calculationDuration=7, streamIds=[000000000000000000000001]}
[...]
INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Done calculating index ranges for 5 indices. Took 44ms.
INFO : org.graylog2.system.jobs.SystemJobManager - SystemJob <9b64a9d0-dcac-11e6-97c3-6c4008b8fc28> [org.graylog2.indexer.ranges.RebuildIndexRangesJob] finished in 46ms.
Manually rotating the active write index

Sometimes you might want to rotate the active write index manually and not wait until the configured rotation criterion for in the latest index has been met, for example if you’ve changed the index mapping or the number of shards per index.

You can do this either via an HTTP request against the REST API of the Graylog master node or via the web interface:

$ curl -XPOST http://127.0.0.1:9000/api/system/deflector/cycle
_images/index_set_maintenance.png

Triggering this job produces log output similar to the following lines:

INFO : org.graylog2.rest.resources.system.DeflectorResource - Cycling deflector for index set <58501f0b4a133077ecd134d9>. Reason: REST request.
INFO : org.graylog2.indexer.MongoIndexSet - Cycling from <graylog_97> to <graylog_98>.
INFO : org.graylog2.indexer.MongoIndexSet - Creating target index <graylog_98>.
INFO : org.graylog2.indexer.indices.Indices - Created Graylog index template "graylog-internal" in Elasticsearch.
INFO : org.graylog2.indexer.MongoIndexSet - Waiting for allocation of index <graylog_98>.
INFO : org.graylog2.indexer.MongoIndexSet - Index <graylog_98> has been successfully allocated.
INFO : org.graylog2.indexer.MongoIndexSet - Pointing index alias <graylog_deflector> to new index <graylog_98>.
INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <024aac80-dcad-11e6-97c3-6c4008b8fc28> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
INFO : org.graylog2.indexer.MongoIndexSet - Successfully pointed index alias <graylog_deflector> to index <graylog_98>.
INFO : org.graylog2.indexer.retention.strategies.AbstractIndexCountBasedRetentionStrategy - Number of indices (5) higher than limit (4). Running retention for 1 index.
INFO : org.graylog2.indexer.retention.strategies.AbstractIndexCountBasedRetentionStrategy - Running retention strategy [org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy] for index <graylog_94>
INFO : org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy - Finished index retention strategy [delete] for index <graylog_94> in 23ms.

Backup

When it comes to backup in a Graylog setup it is not easy to answer. You need to consider what type of backup will suit your needs.

Your Graylog Server setup and settings are easy to backup with a MongoDB dump and a filesystem backup of all configuration files.

The data within your Elasticsearch Cluster can take the advantage of the Snapshot and Restore function that are offered by Elasticsearch.

Disaster recovery

To be able to restore Graylog after a total System crash you need the Graylog server.conf``file - to be exact you need the key you used for ``password_secret in the configuration. The second important part is the MongoDB. This database contains all configuration. Possible options how-to backup MongoDB can be found at the MongoDB documentation.

If you need to restore log data, you can do this using the archiving feature of Graylog enterprise or any other elasticsearch backup and restore option. It is not enough to copy the data directories of your Elasticsearch nodes, you might not be able to restore from that.

Elasticsearch and MongoDB are databases, for both you should implement the ability to make a data dump and restore that - if you need want to be able to restore the current state.

Default file locations

Each installation flavor of Graylog will place the configuration files into a specific location on the local files system. The goal of this section is to provide a short overview about the most common and most important default file locations.

DEB package

This paragraph covers Graylog installations on Ubuntu Linux, Debian Linux, and Debian derivates installed with the DEB package.

Graylog
  File system path
Configuration /etc/graylog/server/server.conf
Logging configuration /etc/graylog/server/log4j2.xml
Plugins /usr/share/graylog-server/plugin
Binaries /usr/share/graylog-server/bin
Scripts /usr/share/graylog-server/scripts
JVM settings /etc/default/graylog-server
Message journal files /var/lib/graylog-server/journal
Log Files /var/log/graylog-server/
Elasticsearch

Note

These are only the most common file locations. Please refer to the Elasticsearch documentation for a comprehensive list of default file locations.

  File system path
Configuration /etc/elasticsearch
JVM settings /etc/default/elasticsearch
Data files /var/lib/elasticsearch/data
Log files /var/log/elasticsearch/
MongoDB
  File system path
Configuration /etc/mongod.conf
Data files /var/lib/mongodb/
Log files /var/log/mongodb/

RPM package

This paragraph covers Graylog installations on Fedora Linux, Red Hat Enterprise Linux, CentOS Linux, and other Red Hat Linux derivates installed with the RPM package.

Graylog
  File system path
Configuration /etc/graylog/server/server.conf
Logging configuration /etc/graylog/server/log4j2.xml
Plugins /usr/share/graylog-server/plugin
Binaries /usr/share/graylog-server/bin
Scripts /usr/share/graylog-server/scripts
JVM settings /etc/sysconfig/graylog-server
Message journal files /var/lib/graylog-server/journal
Log Files /var/log/graylog-server/
Elasticsearch

Note

These are only the most common file locations. Please refer to the Elasticsearch documentation for a comprehensive list of default file locations.

  File system path
Configuration /etc/elasticsearch
JVM settings /etc/sysconfig/elasticsearch
Data files /var/lib/elasticsearch/
Log files /var/log/elasticsearch/
MongoDB
  File system path
Configuration /etc/mongod.conf
Data files /var/lib/mongodb/
Log files /var/log/mongodb/

Graylog REST API

The functionality Graylog REST API is very comprehensive; even the Graylog web interface is exclusively using Graylog REST API to interact with the Graylog cluster.

To connect to the Graylog REST API with a web browser, just add api/api-browser to your current http_publish_uri setting or use the API browser button on the nodes overview page (System / Nodes in the web interface).

For example if your Graylog REST API is listening on http://192.168.178.26:9000/api/, the API browser will be available at http://192.168.178.26:9000/api/api-browser/.

_images/system_nodes_overview.png

Note

The customized version of Swagger UI used by Graylog does currently only work in Google Chrome and Firefox.

Using the API browser

After providing the credentials (username and password), you can browse all available HTTP resources of the Graylog REST API.

_images/use_api_browser.png

Interacting with the Graylog REST API

While having a graphical UI for the Graylog REST API is perfect for interactive usage and exploratory learning, the real power unfolds when using the Graylog REST API for automation or integrating Graylog into another system, such as monitoring or ticket systems.

Naturally, the same operations the API browser offers can be used on the command line or in scripts. A very common HTTP client being used for this kind of interaction is curl.

Note

In the following examples, the username GM and password superpower will be used to demonstrate how to work with the Graylog REST API running at http://192.168.178.26:9000/api.

Warning

Since Graylog 2.5.0, all non-GET API requests must include and set a value for the X-Requested-By HTTP header. This is needed to prevent CSRF attacks.

The following command displays Graylog cluster information as JSON, exactly the same information the web interface is displaying on the System / Nodes page:

curl -u GM:superpower -H 'Accept: application/json' -X GET 'http://192.168.178.26:9000/api/cluster?pretty=true'

The Graylog REST API will respond with the following information:

{
  "71ab6aaa-cb39-46be-9dac-4ba99fed3d66" : {
    "facility" : "graylog-server",
    "codename" : "Smuttynose",
    "node_id" : "71ab6aaa-cb39-46be-9dac-4ba99fed3d66",
    "cluster_id" : "3adaf799-1551-4239-84e5-6ed939b56f62",
    "version" : "2.1.1+01d50e5",
    "started_at" : "2016-09-23T10:39:00.179Z",
    "hostname" : "gm-01-c.fritz.box",
    "lifecycle" : "running",
    "lb_status" : "alive",
    "timezone" : "Europe/Berlin",
    "operating_system" : "Linux 3.10.0-327.28.3.el7.x86_64",
    "is_processing" : true
  },
  "ed0ad32d-8776-4d25-be2f-a8956ecebdcf" : {
    "facility" : "graylog-server",
    "codename" : "Smuttynose",
    "node_id" : "ed0ad32d-8776-4d25-be2f-a8956ecebdcf",
    "cluster_id" : "3adaf799-1551-4239-84e5-6ed939b56f62",
    "version" : "2.1.1+01d50e5",
    "started_at" : "2016-09-23T10:40:07.325Z",
    "hostname" : "gm-01-d.fritz.box",
    "lifecycle" : "running",
    "lb_status" : "alive",
    "timezone" : "Europe/Berlin",
    "operating_system" : "Linux 3.16.0-4-amd64",
    "is_processing" : true
  },
  "58c57924-808a-4fa7-be09-63ca551628cd" : {
    "facility" : "graylog-server",
    "codename" : "Smuttynose",
    "node_id" : "58c57924-808a-4fa7-be09-63ca551628cd",
    "cluster_id" : "3adaf799-1551-4239-84e5-6ed939b56f62",
    "version" : "2.1.1+01d50e5",
    "started_at" : "2016-09-30T13:31:39.051Z",
    "hostname" : "gm-01-u.fritz.box",
    "lifecycle" : "running",
    "lb_status" : "alive",
    "timezone" : "Europe/Berlin",
    "operating_system" : "Linux 4.4.0-36-generic",
    "is_processing" : true
  }
Creating and using Access Token

For security reasons, using the username and password directly on the command line or in some third party application is undesirable.

To prevent having to use the clear text credentials, Graylog allows to create access tokens which can be used for authentication instead.

Note

Users require the permissions users:tokenlist, users:tokencreate, and users:tokenremove to manage their access tokens. Please check the documentation on Permission system for more information. Also note that users, even administrators, may only manage their own tokens.

The following example will create an access token named agents for the user graylog-sidecar:

  • Navigate to the users configuration menu System /  Authentication.
_images/api_tokens_1-fs8.png
  • Select the user you want to create a token for and click on Edit tokens.
_images/api_tokens_2-fs8.png
  • Give the token a name and create it.
_images/api_tokens_3-fs8.png
  • You should see now the token in the list.
_images/api_tokens_4-fs8.png

Either by unchecking the hide option or by copying the token to the clipboard you can access the token. The received access token can now be used as username in a request to the Graylog REST API using Basic Auth together with the literal password token.

When an access token is no longer needed, it can be delete on the Graylog UI via the Delete button.

Creating and using Session Token

While access tokens can be used for permanent access, session tokens will expire after a certain time. The expiration time can be adjusted in the user’s profile.

Getting a new session token can be obtained via POST request to the Graylog REST API. Username and password are required to get a valid session ID. The following example will create an session token for the user GM:

curl -i -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'X-Requested-By: cli' 'http://192.168.178.26:9000/api/system/sessions' -d '{"username":"GM", "password":"superpower", "host":""}'

The response will include the session token in the field session_id and the time of expiration:

{
    "valid_until" : "2016-10-24T16:08:57.854+0000",
    "session_id" : "cf1df45c-53ea-446c-8ed7-e1df64861de7"
}

The received token can now be used as username in a request to the Graylog REST API using Basic Auth together with the literal password session.

Now a curl command to get a list of access tokens would look as follows:

curl -u cf1df45c-53ea-446c-8ed7-e1df64861de7:session -H 'Accept: application/json' -X GET 'http://192.168.178.26:9000/api/cluster?pretty=true'

Securing Graylog

To secure your Graylog setup, you should not use one of our pre-configured images, create your own unique installation where you understand each component and secure the environment by design. Expose only the services that are needed and secure them whenever possible with TLS/SSL and some kind of authentication. Do not use the pre-created appliances for critical production environments.

On the Graylog appliances MongoDB and Elasticsearch is listening on the external interface. This makes the creation of a cluster easier and demonstrates the way Graylog works. Never run this in an insecure network.

When using Amazon Web Services and our pre-configured AMI, never open all ports in the security group. Do not expose the server to the internet. Access Graylog only from within your VPC. Enable encryption for the communication.

Default ports

All parts of one Graylog installation will communicate over network sockets. Depending on your setup and number of nodes this might be exposed or can be bound to localhost. The SELinux configuration is already covered in our step-by-step guide for CentOS Linux.

Default network communication ports
Component Port
Graylog (web interface / API) 9000 (tcp)
Graylog to Elasticsearch 9200 (tcp)
Elasticsearch node communication 9300 (tcp)
MongoDB 27017 (tcp)

Each setup is unique in the requirements and ports might be changed by configuration, but you should limit who is able to connect to which service. In the architecture description you can see what components need to be exposed and communicate with each other.

Configuring TLS ciphers

When running Graylog in untrusted environments such as the Internet, we strongly recommend to use SSL/TLS for all connections.

It’s possible to disable unsafe or deprecated TLS ciphers in Graylog. When using nginx or Apache httpd for SSL termination the Mozilla SSL Configuration Generator will help to create a reasonably secure configuration for them.

Sending in log data

A Graylog setup is pretty worthless without any data in it. This page explains the basic principles of getting your data into the system and also explains common fallacies.

What are Graylog message inputs?

Message inputs are the Graylog parts responsible for accepting log messages. They are launched from the web interface (or the REST API) in the System -> Inputs section and are launched and configured without the need to restart any part of the system.

Content packs

Content packs are bundles of Graylog input, extractor, stream, dashboard, and output configurations that can provide full support for a data source. Some content packs are shipped with Graylog by default and some are available from the website. Content packs that were downloaded from the Graylog Marketplace can be imported using the Graylog web interface.

You can load and even create own content packs from the System -> Content Packs section of your Graylog web interface.

Syslog

Graylog is able to accept and parse RFC 5424 and RFC 3164 compliant syslog messages and supports TCP transport with both the octet counting or termination character methods. UDP is also supported and the recommended way to send log messages in most architectures.

Many devices, especially routers and firewalls, do not send RFC compliant syslog messages. This might result in wrong or completely failing parsing. In that case you might have to go with a combination of raw/plaintext message inputs that do not attempt to do any parsing and Extractors.

Rule of thumb is that messages forwarded by rsyslog or syslog-ng are usually parsed flawlessly.

Sending syslog from Linux hosts

Sending syslog data from Linux hosts is described on the Graylog Marketplace.

Sending syslog from MacOS X hosts

Sending log messages from MacOS X syslog daemons is easy. Just define a graylog-server instance as UDP log target by adding this line in your /etc/syslog.conf:

*.* @graylog.example.org:514

Now restart syslogd:

$ sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist
$ sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist

Important: If syslogd was running as another user you might end up with multiple syslogd instances and strange behavior of the whole system. Please check that only one syslogd process is running:

$ ps aux | grep syslog
lennart         58775   0.0  0.0  2432768    592 s004  S+    6:10PM   0:00.00 grep syslog
root            58759   0.0  0.0  2478772   1020   ??  Ss    6:09PM   0:00.01 /usr/sbin/syslogd

That’s it! Your MacOS X syslog messages should now appear in your Graylog system.

GELF / Sending from applications

The Graylog Extended Log Format (GELF) is a log format that avoids the shortcomings of classic plain syslog and is perfect to logging from your application layer. It comes with optional compression, chunking and most importantly a clearly defined structure. There are dozens of GELF libraries for many frameworks and programming languages to get you started.

Read more about GELF in the specification.

GELF via HTTP

You can send in all GELF types via HTTP, including uncompressed GELF that is just a plain JSON string.

After launching a GELF HTTP input you can use the following endpoints to send messages:

http://graylog.example.org:[port]/gelf (POST)

Try sending an example message using curl:

curl -XPOST http://graylog.example.org:12202/gelf -p0 -d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}'

Both keep-alive and compression are supported via the common HTTP headers. The server will return a 202 Accepted when the message was accepted for processing.

Using Apache Kafka as transport queue

Graylog supports Apache Kafka as a transport for various inputs such as GELF, syslog, and Raw/Plaintext inputs. The Kafka topic can be filtered by a regular expression and depending on the input, various additional settings can be configured.

Learn how to use rsyslog and Apache Kafka in the Sending syslog via Kafka into Graylog guide.

Using RabbitMQ (AMQP) as transport queue

Graylog supports AMQP as a transport for various inputs such as GELF, syslog, and Raw/Plaintext inputs. It can connect to any AMQP broker supporting AMQP 0-9-1 such as RabbitMQ.

Learn how to use rsyslog and RabbitMQ in the Sending syslog via AMQP into Graylog guide.

Microsoft Windows

Sending syslog data from Windows is described on the Graylog Marketplace.

Ruby on Rails

This is easy: You just need to combine a few components.

Log all requests and logger calls into Graylog

The recommended way to send structured information (i.e. HTTP return code, action, controller, … in additional fields) about every request and explicit Rails.logger calls is easily accomplished using the GELF gem and lograge. Lograge builds one combined log entry for every request (instead of several lines like the standard Rails logger) and has a Graylog output since version 0.2.0.

Start by adding Lograge and the GELF gem to your Gemfile:

gem "gelf"
gem "lograge"

Now configure both in your Rails application. Usually config/environments/production.rb is a good place for that:

config.lograge.enabled = true
config.lograge.formatter = Lograge::Formatters::Graylog2.new
config.logger = GELF::Logger.new("graylog.example.org", 12201, "WAN", { :host => "hostname-of-this-app", :facility => "heroku" })

This configuration will also send all explicit Rails.logger calls (e.g. Rails.logger.error "Something went wrong") to Graylog.

Log only explicit logger calls into Graylog

If you don’t want to log information about every request, but only explicit Rails.logger calls, it is enough to only configure the Rails logger.

Add the GELF gem to your Gemfile:

gem "gelf"

…and configure it in your Rails application. Usually config/environments/production.rb is a good place for that:

config.logger = GELF::Logger.new("graylog.example.org", 12201, "WAN", { :host => "hostname-of-this-app", :facility => "heroku" })

Heroku

You need to apply a workaround if you want custom logging on Heroku. The reason for this is that Heroku injects an own logger (rails_log_stdout), that overwrites your custom one. The workaround is to add a file that makes Heroku think that the logger is already in your application:

$ touch vendor/plugins/rails_log_stdout/heroku_fix

Raw/Plaintext inputs

The built-in raw/plaintext inputs allow you to parse any text that you can send via TCP or UDP. No parsing is applied at all by default until you build your own parser using custom Extractors. This is a good way to support any text-based logging format.

You can also write Plugins if you need extreme flexibility.

JSON path from HTTP API input

The JSON path from HTTP API input is reading any JSON response of a REST resource and stores a field value of it as a Graylog message.

Example

Let’s try to read the download count of a release package stored on GitHub for analysis in Graylog. The call looks like this:

$ curl -XGET https://api.github.com/repos/YourAccount/YourRepo/releases/assets/12345
{
  "url": "https://api.github.com/repos/YourAccount/YourRepo/releases/assets/12345",
  "id": 12345,
  "name": "somerelease.tgz",
  "label": "somerelease.tgz",
  "content_type": "application/octet-stream",
  "state": "uploaded",
  "size": 38179285,
  "download_count": 9937,
  "created_at": "2013-09-30T20:05:01Z",
  "updated_at": "2013-09-30T20:05:46Z"
}

The attribute we want to extract is download_count so we set the JSON path to $.download_count.

This will result in a message in Graylog looking like this:

_images/jsonpath_1.png

You can use Graylog to analyze your download counts now.

JSONPath

JSONPath can do much more than just selecting a simple known field value. You can for example do this to select the first download_count from a list of releases where the field state has the value uploaded:

$.releases[?(@.state == 'uploaded')][0].download_count

…or only the first download count at all:

$.releases[0].download_count

You can learn more about JSONPath here.

Reading from files

Log files come in a lot of different flavors and formats, much more than any single program could handle.

To support this use case, we provide the Sidecar which acts as a supervisor process for other programs, such as nxlog and Filebeats, which have specifically been built to collect log messages from local files and ship them to remote systems like Graylog.

Of course you can still use any program supporting the GELF or syslog protocol (among others) to send your logs to Graylog.

Input Throttling

Throttling allows certain Graylog Inputs to slow their message intake rates (by temporarily pausing intake processing) if contention occurs in the Graylog Journal.

Graylog Inputs that support throttling

  • AWS Flow Logs
  • AWS Logs
  • CEF AMQP Input
  • CEF Kafka Input
  • GELF AMQP
  • GELF Kafka
  • JSON path from HTTP API
  • Raw/Plaintext AMQP
  • Raw/Plaintext Kafka
  • Syslog AMQP
  • Syslog Kafka

Enabling throttling

To enable throttling for one of these inputs, edit it in System > Inputs and check the Allow throttling this input checkbox.

Throttling criteria

When enabled, the following criteria will be used to determine if throttling will occur:

  1. If there are zero uncommitted entries in the Graylog Journal, throttling will not occur. No further checks will be performed.
  2. Throttling will occur if the Journal has more than 100k uncommitted entries.
  3. Throttling will occur if the Journal is growing in size rapidly (approximately 20k entries per second or greater).
  4. Throttling will occur if the process ring buffer is full.
  5. Nothing is currently being written to the Journal, throttling will not occur. No further checks will be performed.
  6. Throttling will occur if the Journal is more than 90% full.
  7. Throttling will occur if the Journal write rate is more than twice as high as the read rate.

Graylog Sidecar

Note

Graylog 3.0 comes with a new Sidecar implementation. We still support the old Collector Sidecars, which can be found in the System / Collectors (legacy) menu entry. In case you need to configure legacy Collector Sidecar please refer to the Graylog Collector Sidecar documentation. We encourage users to migrate to the new Sidecar, which is covered by this document.

Graylog Sidecar is a lightweight configuration management system for different log collectors, also called Backends. The Graylog node(s) act as a centralized hub containing the configurations of log collectors. On supported message-producing devices/hosts, Sidecar can run as a service (Windows host) or daemon (Linux host).

_images/sidecar_overview.png

The log collector configurations are centrally managed through the Graylog web interface. Periodically, the Sidecar daemon will fetch all relevant configurations for the target, using the REST API. On its first run, or when a configuration change has been detected, Sidecar will generate (render) relevant backend configuration files. Then it will start, or restart, those reconfigured log collectors.

Installation

Currently we provide pre-compiled packages on the Github releases page of the project. Once the Sidecar project is settled and matured we will add the packages to the DEB and YUM online repositories. To get the Sidecar working Download a package and install it on the target system.

Please follow the version matrix to pick the right package:

Sidecar version Graylog server version
1.0.x 3.0.x
0.1.x 2.2.x,2.3.x,2.4.x,2.5.x,3.0.x
0.0.9 2.1.x

All following commands should be executed on the remote machine where you want to collect log data from.

Install the Sidecar

Ubuntu
$ sudo dpkg -i graylog-sidecar_1.0.0-1_amd64.deb

Edit the configuration (see Configuration) and activate the Sidecar as a system service:

$ vi /etc/graylog/sidecar/sidecar.yml

$ sudo graylog-sidecar -service install

[Ubuntu 14.04 with Upstart]
$ sudo start graylog-sidecar

[Ubuntu 16.04 and later with Systemd]
$ sudo systemctl start graylog-sidecar
CentOS

Install the RPM package on RedHat based systems

$ sudo rpm -i graylog-sidecar-1.0.0-1.x86_64.rpm

Edit the configuration (see Configuration) and activate the Sidecar as a system service:

$ vi /etc/graylog/sidecar/sidecar.yml

$ sudo graylog-sidecar -service install
$ sudo systemctl start graylog-sidecar
Windows

Use the Windows installer, it can be run interactively:

$ graylog_sidecar_installer_1.0.0-1.exe

Or in silent mode with:

$ graylog_sidecar_installer_1.0.0-1.exe /S -SERVERURL=http://10.0.2.2:9000/api -APITOKEN=yourapitoken

Optionally edit the configuration (see Configuration) and register the system service:

notepad.exe C:\Program Files\Graylog\sidecar\sidecar.yml

& "C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service install
& "C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service start

Install collectors

Next up, you can decide which collectors you want to use with your Sidecar and install them as well. We only cover the installation of the most common ones here, but you are free to use other collectors as well. Graylog contains default collector configurations for Filebeat, Winlogbeat and NXLog. But since you’re able to define your own collector backends, there is nothing stopping you from running e.g. sysmon, auditd or packetbeat.

Beats on Linux

Install Filebeat or another Beats package by following the instructions on the official Filebeat download page.

Beats on Windows

The Windows Sidecar package already includes Filebeat and Winlogbeat. For other Beats packages follow the instructions on the official Beats download page.

NXLog on Ubuntu

Install the NXLog package from the official NXLog download page. Because the Sidecar takes control of stopping and starting NXlog it is necessary to stop all running instances of NXlog and unconfigure the default system service:

$ sudo /etc/init.d/nxlog stop
$ sudo update-rc.d -f nxlog remove
$ sudo gpasswd -a nxlog adm
$ sudo chown -R nxlog.nxlog /var/spool/collector-sidecar/nxlog
NXLog on CentOS

The same on a RedHat based system:

$ sudo service nxlog stop
$ sudo chkconfig --del nxlog
$ sudo gpasswd -a nxlog root
$ sudo chown -R nxlog.nxlog /var/spool/collector-sidecar/nxlog
NXlog on Windows

Install the NXLog package from the official download page and deactivate the system service. We just need the binaries installed on the system:

$ C:\Program Files (x86)\nxlog\nxlog -u

Sidecar Configuration

On the command line you can provide a path to the configuration file with the -c switch. The default configuration path on Linux systems is /etc/graylog/sidecar/sidecar.yml on Linux and C:\Program Files\Graylog\sidecar\sidecar.yml on Windows.

Most configuration parameters come with built-in defaults. The only parameters that need adjustment are server_url and server_api_token. You can get your API token by following the link on the Sidecars Overview page.

sidecar.yml Reference

Parameter Description
server_url URL to the Graylog API, e.g. http://192.168.1.1:9000/api/
server_api_token The API token to use to authenticate against the Graylog server API.
e.g 1jq26cssvc6rj4qac4bt9oeeh0p4vt5u5kal9jocl1g9mdi4og3n
The token is mandatory and needs to be configured.
node_id The node ID of the sidecar. This can be a path to a file or an ID string.
Example file path: file:/etc/graylog/sidecar/node-id
Example ID string: 6033137e-d56b-47fc-9762-cd699c11a5a9
ATTENTION: Every sidecar instance needs a unique ID!
Default: file:/etc/graylog/sidecar/node-id
node_name Name of the Sidecar instance, will also show up in the web interface.
The hostname will be used if not set.
update_interval The interval in seconds the sidecar will fetch new configurations from the Graylog server
Default: 10
tls_skip_verify This configures if the sidecar should skip the verification of TLS connections. Default: false
send_status This controls the transmission of detailed sidecar information like collector status,
metrics and log file lists. It can be disabled to reduce load on the Graylog server if needed.
Default: true
list_log_files Send a directory listing to Graylog and display it on the host status page,
e.g. /var/log. This can also be a list of directories. Default: []
cache_path The directory where the sidecar stores internal data. Default: /var/cache/graylog-sidecar
collector_configuration_directory The directory where the sidecar generates configurations for collectors.
Default: /var/lib/graylog-sidecar/generated
log_path The directory where the sidecar stores its logs. Default: /var/log/graylog-sidecar
log_rotate_max_file_size The maximum size of the log file before it gets rotated. Default: 10MiB
log_rotate_keep_files The maximum number of old log files to retain.
collector_binaries_whitelist A list of binaries which are allowed to be executed by the Sidecar.
An empty list disables the white list feature.
Default: /usr/bin/filebeat, /usr/bin/packetbeat, /usr/bin/metricbeat, /usr/bin/heartbeat,
/usr/bin/auditbeat, /usr/bin/journalbeat, /usr/share/filebeat/bin/filebeat,
/usr/share/packetbeat/bin/packetbeat, /usr/share/metricbeat/bin/metricbeat,
/usr/share/heartbeat/bin/heartbeat, /usr/share/auditbeat/bin/auditbeat,
/usr/share/journalbeat/bin/journalbeat, /usr/bin/nxlog, /opt/nxlog/bin/nxlog

First start

Once you installed the Sidecar package and started the service for the first time, you can verify that it shows up in the Sidecars Overview page. A new sidecar instance will not have any configurations assigned yet. Take the Step-by-step guide to create your first configuration.

Mode of Operation

When the Sidecar is assigned a configuration via the Graylog web interface, it will write a configuration file into the collector_configuration_directory directory for each collector backend. E.g. if you assigned a Filebeat collector you will find a filebeat.yml file in that directory. All changes have to be made in the Graylog web interface. Every time the Sidecar detects an update to its configuration it will rewrite the corresponding collector configuration file. Manually editing these files is not recommended.

Every time a collector configuration file is changed the collector process is restarted. The Sidecar takes care of the collector processes and reports the status back to the web interface

Sidecar Status

Each Sidecar instance is able to send status information back to Graylog. By enabling the option send_status metrics like load or the IP address of the host Sidecar is running on are sent. Also metrics that are relevant for a stable operation e.g. disk volumes over 75% utilization are included. Additionally with the list_log_files option a directory listing is displayed in the Graylog web interface. In that way an administrator can see which files are available for collecting. The list is periodically updated and files with write access are highlighted for easy identification. After enabling send_status or send_status + list_log_files go to the collector overview and click on one of them, a status page with the configured information will be displayed.

Step-by-step guide

We have prepared an example on how to configure the Sidecar using the Graylog web interface. The assumption is that we want to collect Apache logfiles and ship them with a Filebeat collector to a Beats input that is listening on Port 5044 on your Graylog Server.

  • The first step is to create a Beats input where collectors can send data to. Click on System / Inputs and start a global Beats input on the listening address 0.0.0.0 and port 5044.
_images/sidecar_sbs0.png
  • Navigate to the Sidecars overview. In your Graylog web interface click on System / Sidecars.
_images/sidecars_overview.png
  • Navigate to the Sidecar Configuration page.
_images/sidecar_sbs1.png
  • Next we create a new configuration: We give the configuration a name and select filebeat on Linux as collector. (This collector definition is shipped with Graylog, and comes with a default configuration template). Most of the configuration defaults should work for you. However you need to change the hosts: setting and point it to your Beats input. You also might want to change the paths: to the location of your Apache logs. When done click Create to save your configuration.
_images/sidecar_sbs2.png
  • Next we need to assign our newly created configuration (and therefore the Filebeat collector) to our sidecar. Go to the Collector Administration page.
_images/sidecar_sbs3.png
  • You will see a list of sidecars and underneath them a list of collectors that could be assigned to them. Please note that collectors are assigned to sidecars by means of applying a collector configuration to the sidecar. Therefore, we first select the filebeat collector and then click on the Configure menu, where we can select the filebeat-conf configuration we created earlier.
_images/sidecar_sbs4.png
  • Confirming the assignment, will directly push this configuration to your sidecar which will go and start the Filebeat collector with this configuration.
_images/sidecar_sbs5.png
  • If everything went fine, you should see the status running on the administration page.
_images/sidecar_sbs6.png
  • Congratulations your collector setup is working now! You can go back to the Sidecars overview and click on the Show messages button to search for logs that have been collected via your sidecar.
_images/sidecar_sbs7.png

Creating a new Log Collector

Graylog comes with a few predefined log collectors which can be easily extended and changed to your needs. Let’s assume you want your sidecar to run rsyslogd(8) for you.

  • Navigate to the Sidecars overview. In your Graylog web interface click on System / Sidecars.
_images/sidecars_overview.png
  • Navigate to the Sidecar Configuration page.
_images/sidecar_sbs1.png
  • After we click on Create Log Collector, we are presented with the following page, where we have to fill out some fields for our new collector. We give the collector a unique name and select Linux and Foreground Execution. Given that you installed rsyslogd(8) under /usr/sbin/rsyslogd we configure the executable path accordingly. If you are using Foreground Execution make sure that the collector you are running does not daemonize itself. Otherwise the sidecar has no way of controlling the collector once it has forked off into the background. For rsyslogd we therefore provide -n as Execute Parameter. If your collector supports configuration validation, it is advised to use it. This acts as a pre-check, so that sidecar won’t restart a collector with a broken configuration. For rsyslogd the option to do a configuration check is -N 1. Optionally you can provide a Default Template which will be proposed once you create a configuration for this collector.
_images/sidecar_new_collector.png
  • Next up you can use your newly created collector by creating a configuration for it and assign it to a Sidecar. Please follow the Step-by-step guide accordingly.
  • Note: Your Sidecar might refuse to start your collector, because it needs to be added to the collector_binaries_whitelist first. Please edit your Configuration and restart your Sidecar.

Using Configuration Variables

Configuration variables can contain arbitrary strings like the IP address of your Graylog server or the port of an input. The variables can then be used in multiple collector configurations, which avoids duplication and simplifies management.

To create a configuration variable go any Collector Configuration page:

_images/sidecar_sbs2.png

On the right you’ll find a box Collector Configuration Reference which contains Runtime Variables and Variables. Click on Variables and then Create Variable to receive the following modal:

_images/sidecar_conf_variable.png

In this example we replace the hard coded IP and Port from our Beats input with a new variable named ${user.BeatsInput}:

_images/sidecar_conf_variable2.png

We can now use this variable in all our configurations. If we ever need to change the IP/port of our input, we just change the variable.

Runtime Variables

Runtime variables contain runtime informations from each Sidecar that is requesting this configuration. An important example is the ${sidecar.nodeId} variable. The collector configuration should contain an instruction to fill that variable in an extra field gl2_source_collector. This allows Graylog to relate messages to the Sidecar that produced them. (This is what makes the Show messages button on the Sidecars overview page work)

Secure Sidecar Communication

The Communication between Sidecar and Graylog will be secured if your API uses SSL.

To secure the communication between the Collector and Graylog you just need to mark Enable TLS in your Beats Input. Without giving additional Information, Graylog will now create a self-signed certificate for this Input. Now in the Sidecar Beats Output Configuration you just mark Enable TLS Support and Insecure TLS connection. After this is saved, the communication between Beats and Graylog will use TLS.

Certificate based client authentication

If you want Graylog to only accept data from authenticated Collectors please follow the steps at Secured Graylog and Beats input

Run Sidecar as non-root user

The default is that the Sidecar is started with the root user to allow access to all log files. But this is not mandatory. If you like to start it with a daemon user, proceed like the following:

  • Create a daemon user e.g. sidecar

The Sidecar itself is accessing the following files and directories:

  • sidecar.yml - /etc/graylog/sidecar/sidecar.yml
  • collector_configuration_directory - /var/lib/graylog-sidecar/generated/
  • node_id - /etc/graylog/sidecar/node-id
  • cache_path - /var/cache/graylog-sidecar/
  • log_path - /var/log/graylog-sidecar/

So to make these directories readable for the sidecar user, use:

  • chown -R sidecar /etc/graylog/sidecar
  • chown -R sidecar /var/cache/graylog-sidecar
  • chown -R sidecar /var/lib/graylog-sidecar
  • chown -R sidecar /var/log/graylog-sidecar

You can change all paths to different places in the file system. If you prefer to store all Sidecar data in the home directory of the sidecar user, just change the paths accordingly.

Now systemd needs to know that the Sidecar should be started with a non-root user. Open /etc/systemd/system/collector-sidecar.service with an editor and navigate to the [Service] section, add:

User=sidecar
Group=sidecar

To make use of these settings reload systemd:

$ sudo systemctl daemon-reload
$ sudo systemctl restart graylog-sidecar

Check the log files in /var/log/graylog-sidecar for any errors. Understand that not only the Sidecar but also all backends, like filebeat, will be started as sidecar user after these changes. So all log files that the backend should observe also need to be readable by the sidecar user. Depending on the Linux distribution there is usually an administrator group which has access to most log files. By adding the sidecar user to that group you can grant access fairly easy. For example on Debian/Ubuntu systems this group is called adm (see System Groups in Debian Wiki or Security/Privileges - Monitor system logs in Ubuntu wiki).

Upgrading from the Collector Sidecar

This guide describes how you can perform an upgrade from the deprecated Collector Sidecars (0.1.x) to the new Sidecars (1.x).

One major difference between the old and the new Sidecars, is that we replaced the UI based collector configuration approach with one where you can manage the plain text configuration of the collectors directly. This might seem like an inconvenience at first, but gives you the flexibility to configure any collector backend you want.

Additionally, the new Sidecars don’t assign configurations based on tags anymore. Instead you have to assign configurations explicitly (see Step-by-Step guide).

1. Install New Sidecar

The new Sidecar has different paths and executable names, so it can coexist with the old one. Install the new Sidecar by following the Installation instructions and have your Sidecar running as described in First Start.

Note: In case you were using filebeat on Linux, please make sure to also install the official collector package, since the filebeat binary is not part of the Sidecar package anymore.

2. Migrate configuration

Next, we need to migrate the configuration that was previously rendered on each host by the Collector Sidecar, to a new Collector Configuration.

We recommend to use the Sidecar Configuration Migrator. However, retrieving the old configuration can also be done manually by fetching it from your host at the /etc/graylog/collector-sidecar/generated/ directory.

3. Adopt configuration to Graylog 3.0

There are a few things that might need attention after an upgrade:

  • Use Runtime variables for static fields

    The imported configuration contains instructions that add static fields which allows Graylog to relate messages to a Sidecar. You should replace the hardcoded values of gl2_source_collector and collector_node_id with runtime variables.

    In case of a Beats collector this would be:

    fields.gl2_source_collector: ${sidecar.nodeId}
    fields.collector_node_id: ${sidecar.nodeName}
    
  • Migrate to the new Beats input

    Graylog 3.0 comes with a new Beats input. The former one was renamed to Beats (deprecated). The new input handles fields a little different. Therefore you should define fields_under_root: true for the new input to get the Graylog fields work.

4. Switch over to the new Sidecar

Once you’re done creating a new configuration, you can assign it to your Sidecar (see Step-by-Step guide). If everything works as expected, make sure to uninstall the old Collector Sidecar to avoid collecting your logs twice.

Sidecar Configuration Migrator

The task of the Sidecar configuration migrator is to extract the configuration from existing Collector Sidecars and convert it into new Sidecar configurations.

This feature needs a Collector Sidecar with version 0.1.8 or greater. Please upgrade the instance you want to import configurations from, if necessary.

  • Navigate to the Collectors (legacy) overview. In your Graylog web interface click on System / Collectors (legacy).
_images/sidecar_mig_1.png
  • Click on the name of the Collector you want to import configurations from
_images/sidecar_mig_2.png
  • Click the Import Configuration button on a backend to import a configuration. If the import was successful, follow the link to create a new Sidecar configuration:
_images/sidecar_mig_3.png
  • After clicking on Create Configuration use the Migrate button underneath the configuration editor:
_images/sidecar_mig_4.png
  • A window opens up and lets you pick already imported configurations. Clicking Apply will paste the configuration into the editor. Afterwards you can edit and save the configuration as usual.
_images/sidecar_mig_5.png

Sidecar Glossary

To understand the different parts of the Graylog Sidecar they are explained in the following section.

Configuration

A configuration is the representation of a log collector configuration file in the Graylog web interface. A configuration can be assigned to Sidecars, which also assigns the corresponding collector. You can have multiple configurations for a single log collector. However, you can not assign the same collector twice to a Sidecar.

Inputs

Inputs are the way how collectors ingest data. An input can be a log file that the collector should continuously read or a connection to the Windows event system that emits log events. An input is connected to an output, otherwise there would be no way of sending the data to the next hop. So first create an output and then associate one or many inputs with it.

Debug

The Sidecar is writing log files to the directory configured in log_path. One file for each backend, there you can check for general issues like file permissions or log transmission problems. The Sidecar itself is writing to sidecar.log. Problems like failed connection to the Graylog API can be found there.

You can also start the Sidecar in foreground and monitor the output of the process:

$ graylog-sidecar -debug

Uninstall

On Linux just uninstall the package, to perform an uninstall on Windows run:

& "C:\Program Files\Graylog\graylog-sidecar.exe" -service stop
& "C:\Program Files\Graylog\graylog-sidecar.exe" -service uninstall

Known Problems

Currently we know of two problems with NXLog:

  • Since version 2.9.17 timestamps are transmitted without millisecond precision
  • On Windows machines NXlog is not able to store its collector state so features like file tailing don’t work correctly in combination with Sidecar. Use Sidecar version 0.1.0-alpha.1 or newer.

Known issue if you use a loadbalancer or firewall in front of Graylog’s API:

  • The Sidecar is using a persistent connection for API requests. Therefore it logs 408 Request Time-out if the loadbalancer session or http timeout is lower than the configured update_interval.

Searching

Search query language

Syntax

The search syntax is very close to the Lucene syntax. By default all message fields are included in the search if you don’t specify a message field to search in.

Messages that include the term ssh:

ssh

Messages that include the term ssh or login:

ssh login

Messages that include the exact phrase ssh login:

"ssh login"

Messages where the field type includes ssh:

type:ssh

Messages where the field type includes ssh or login:

type:(ssh OR login)

Note

Elasticsearch 2.x and 5.x split queries on whitespace, so the query type:(ssh login) was equivalent to type:(ssh OR login). This is no longer the case in Elasticsearch 6.0 and you must now include an OR operator between each term.

Messages where the field type includes the exact phrase ssh login:

type:"ssh login"

Messages that have the field type:

_exists_:type

Messages that do not have the field type:

NOT _exists_:type

Note

Elasticsearch 2.x allows to use _missing_:type instead of NOT _exists_:type. This query syntax has been removed in Elasticsearch 5.0.

Messages that match regular expression ethernet[0-9]+:

/ethernet[0-9]+/

Note

Please refer to the Elasticsearch documentation about the Regular expression syntax for details about the supported regular expression dialect.

By default all terms or phrases are OR connected so all messages that have at least one hit are returned. You can use Boolean operators and groups for control over this:

"ssh login" AND source:example.org
("ssh login" AND (source:example.org OR source:another.example.org)) OR _exists_:always_find_me

You can also use the NOT operator:

"ssh login" AND NOT source:example.org
NOT example.org

Note that AND, OR, and NOT are case sensitive and must be typed in all upper-case.

Wildcards: Use ? to replace a single character or * to replace zero or more characters:

source:*.org
source:exam?le.org
source:exam?le.*

Note that leading wildcards are disabled to avoid excessive memory consumption! You can enable them in your Graylog configuration file:

allow_leading_wildcard_searches = true

Also note that message, full_message, and source are the only fields that are being analyzed by default. While wildcard searches (using * and ?) work on all indexed fields, analyzed fields will behave a little bit different. See wildcard and regexp queries for details.

Fuzziness: You can search for similar terms:

ssh logni~
source:exmaple.org~

This example is using the Damerau–Levenshtein distance with a default distance of 2 and will match “ssh login” and “example.org” (intentionally misspelled in the query).

You can change the distance like this:

source:exmaple.org~1

You can also use the fuzzyness operator to do a proximity search where the terms in a phrase can have different/fuzzy distances from each other and don’t have to be in the defined order:

"foo bar"~5

Numeric fields support range queries. Ranges in square brackets are inclusive, curly brackets are exclusive and can even be combined:

http_response_code:[500 TO 504]
http_response_code:{400 TO 404}
bytes:{0 TO 64]
http_response_code:[0 TO 64}

You can also do searches with one side unbounded:

http_response_code:>400
http_response_code:<400
http_response_code:>=400
http_response_code:<=400

It is also possible to combine unbounded range operators:

http_response_code:(>=400 AND <500)

Escaping

The following characters must be escaped with a backslash:

&& || : \ / + - ! ( ) { } [ ] ^ " ~ * ?

Example:

resource:\/posts\/45326

Time frame selector

The time frame selector defines in what time range to search in. It offers three different ways of selecting a time range and is vital for search speed: If you know you are only interested in messages of the last hour, only search in that time frame. This will make Graylog search in relevant indices only and greatly reduce system load and required resources.

_images/queries_time_range_selector.png

Relative time frame selector

The relative time frame selector lets you look for messages from the selected option to the time you hit the search button. The selector offers a wide set of relative time frames that fit most of your search needs.

Absolute time frame selector

When you know exactly the boundaries of your search, you want to use the absolute time frame selector. Simply introduce the dates and times for the search manually or click in the input field to open up a calendar where you can choose the day with your mouse.

Keyword time frame selector

Graylog offers a keyword time frame selector that allows you to specify the time frame for the search in natural language like last hour or last 90 days. The web interface shows a preview of the two actual timestamps that will be used for the search.

_images/queries_keyword_time_selector.png

Here are a few examples for possible values.

  • “last month” searches between one month ago and now
  • “4 hours ago” searches between four hours ago and now
  • “1st of april to 2 days ago” searches between 1st of April and 2 days ago
  • “yesterday midnight +0200 to today midnight +0200” searches between yesterday midnight and today midnight in timezone +0200 - will be 22:00 in UTC

The time frame is parsed using the natty natural language parser. Please consult its documentation for details.

Saved searches

Sometimes you may want to search a specific search configuration to be used later. Graylog provides a saved search functionality to accomplish exactly that.

Once you submitted your search, selected the fields you want to show from the search sidebar, and chosen a resolution for the histogram, click on the Save search criteria button on the sidebar.

_images/saved_search_create.png

Give a name to the current search and click on save. When you want to use the saved search later on, you only need to select it from the saved search selector.

_images/saved_search_selector.png

Of course, you can always update the selected fields or name of your saved search. To do so, select the saved search from the saved search selector, update the field selection or histogram resolution, and click on Saved search -> Update search criteria. It is also possible to delete the saved search by selecting Saved search -> Delete saved search.

_images/saved_search_update.png

Histogram

The search page includes a search result histogram, where you can view in a concise way the number of messages received grouped by a certain time period that Graylog will adjust for you.

The histogram also allows you to further narrow down the cause for an issue:

  • Delimit the search time range by brushing over the histogram. Just click and drag with your mouse over the chart to select the time range you want to use, and click on the search button to perform that search
  • See the time where alerts are triggered in the graph annotations. If you are searching in a stream, you will only see alerts related to that stream
_images/search_histogram.png

Analysis

Graylog provides several tools to analyze your search results. It is possible to save these analysis into dashboards, so you can check them over time in a more convenient way. To analyze a field from your search results, expand the field in the search sidebar and click on the button of the analysis you want to perform.

_images/search_analysis.png

Field statistics

Compute different statistics on your fields, to help you better summarize and understand the data in them.

The statistical information consist of: total, mean, minimum, maximum, standard deviation, variance, sum, and cardinality. On non-numeric fields, you can only see the total amount of messages containing that field, and the cardinality of the field, i.e. the number of unique values it has.

_images/field_statistics.png

Quick values

Quick values helps you to find out the distribution of values for a field. Alongside a graphic representation of the common values contained in a field, Graylog will display a table with all different values, allowing you to see the number of times they appear. You can include any value in your search query by clicking on the magnifying glass icon located in the value row.

_images/quick_values.png

Field graphs

You can create field graphs for any numeric field, by clicking on the Generate chart button in the search sidebar. Using the options in the Customize menu on top of the field graph, you can change the statistical function used in the graph, the kind of graph to use to represent the values, the graph interpolation, as well as the time resolution.

_images/field_graph.png

Once you have customized some field graphs, you can also combine them by dragging them from the hamburger icon on the top corner of the graph, and dropping them into another field graph. You can see the location of the hamburger icon and the end result in the following screenshots:

_images/stacked_graph_1.png _images/stacked_graph_2.png

Field graphs appear every time you perform a search, allowing you to compare data, or combine graphs coming from different streams.

Decorators

Decorators allow you to alter message fields during search time automatically, while preserving the unmodified message on disk. Decorators are specially useful to make some data in your fields more readable, combine data in some field, or add new fields with more information about the message. As decorators are configured per stream (including the default stream), you are also able to present a single message in different streams differently.

As changes made by decorators are not persisted, you cannot search for decorated values or use field analyzers on them. You can still use those features in the original non-decorated fields.

Decorators are applied on a stream-level, and are shared among all users capable of accessing a stream, so all users can share the same results and benefit from the advantages decorators add.

Graylog includes some message decorators out of the box, but you can add new ones from pipelines or by writing your own as plugins.

In order to apply decorators to your search results, click on the Decorators tab in your search sidebar, select the decorator you want to apply from the dropdown, and click on Apply. Once you save your changes, the search results will already contain the decorated values.

_images/create_decorator.png

When you apply multiple decorators to the same search results, you can change the order in which they are applied at any time by using drag and drop in the decorator list.

Syslog severity mapper

The syslog severity mapper decorator lets you convert the numeric syslog level of syslog messages, to a human readable string. For example, applying the decorator to the level field in your logs would convert the syslog level 4 to Warning (4).

To apply a syslog severity mapper decorator, you need to provide the following data:

  • Source field: Field containing the numeric syslog level
  • Target field: Field to store the human readable string. It can be the same one as the source field, if you wish to replace the numeric value on your search results

Format string

The format string decorator provides a simple way of combining several fields into one. It can also be used to modify the content of a field in, without altering the stored result in Elasticsearch.

To apply a format string decorator you need to provide the following data:

  • Format string: Pattern used to format the resulting string. You can provide fields in the message by enclosing them in ${}. E.g. ${source} will add the contents of the source message field into the resulting string
  • Target field: Field to store the resulting value
  • Require all fields (optional): Check this box to only format the string when all other fields are present

For example, using the format string Request to ${controller}#${action} finished in ${took_ms}ms with code ${http_response_code}, could produce the text Request to PostsController#show finished in 57ms with code 200, and make it visible in one of the message fields in your search results.

Pipeline Decorator

The pipeline decorator provides a way to decorate messages by processing them with an existing processing pipeline. In contrast to using a processing pipeline, changes done to the message by the pipeline are not persisted. Instead, the pipeline is used at search time to modify the presentation of the message.

The prerequisite of using the pipeline decorator is that an existing pipeline is required.

Note

Please take note, that the pipeline you use for decoration should not be connected to a stream. This would mean that it is run twice (during indexing and search time) for each message, effectively rendering the second run useless.

When you are done creating a pipeline, you can now add a decorator using it on any number of streams. In order to create one, you proceed just like for any other decorator type, by clicking on the Decorator sidebar, selecting the type (“Pipeline Processor Decorator” in this case) and clicking the Apply button next to one.

_images/pipeline_decorator_select_type.png

Upon clicking Apply, the pipeline to be used for decorating can be selected.

_images/pipeline_decorator_select_pipeline.png

After selecting a pipeline and clicking Save, you are already set creating a new pipeline decorator.

Debugging decorators

When a message is not decorated as expected, or you need to know how it looked like originally, you can see all changes that were done during decoration by clicking “Show changes” in the message details.

_images/pipeline_decorator_show_changes.png

In this view, deleted content is shown in red, while added content is shown in green. This means that added fields will have a single green entry, removed fields a single red entry and modified fields will have two entries, a red and a green one.

Further functionality

If the existing decorators are not sufficient for your needs, you can either search the Graylog marketplace, or write your own decorator.

Export results as CSV

It is also possible to export the results of your search as a CSV document. To do so, select all fields you want to export in the search sidebar, click on the More actions button, and select Export as CSV.

_images/export_as_csv.png

Hint: Some Graylog inputs keep the original message in the full_message field. If you need to export the original message, you can do so by clicking on the List all fields link at the bottom of the sidebar, and then selecting the full_message field.

Warning

Exporting results to a CSV will not preserve sorting because Graylog is using the virtual _doc field to “sort” documents for performance reasons. If you need to have the exported data ordered you will need to either make a scroll query to ElasticSearch and process it after, or to download the file and post process it via other means.

Search result highlighting

Graylog supports search result highlighting since v0.20.2:

_images/search_result_highlighting.png

Enabling/Disabling search result highlighting

Using search result highlighting will result in slightly higher resource consumption of searches. You can enable and disable it using a configuration parameter in the graylog.conf of your Graylog nodes:

allow_highlighting = true

Search configuration

Graylog allows customizing the options allowed to search queries, like limiting the time range users can select or configuring the list of displayed relative time ranges.

_images/queries_search_configuration.png

All search configuration settings can be customized using the web interface on the System -> Configurations page in the Search configuration section.

Query time range limit

Sometimes the amount of data stored in Graylog is quite big and spans a wide time range (e. g. multiple years). In order to prevent normal users from accidentally running search queries which could use up lots of resources, it is possible to limit the time range that users are allowed to search in.

Using this feature, the time range of a search query exceeding the configured query time range limit will automatically be adapted to the given limit.

_images/queries_query_time_range_limit.png

The query time range limit is a duration formatted according to ISO 8601 following the basic format P<date>T<time> with the following rules:

Designator Description
P Duration designator (for period) placed at the start of the duration representation
Y Year designator that follows the value for the number of years
M Month designator that follows the value for the number of months
W Week designator that follows the value for the number of weeks
D Day designator that follows the value for the number of days
T Time designator that precedes the time components of the representation
H Hour designator that follows the value for the number of hours
M Minute designator that follows the value for the number of minutes
S Second designator that follows the value for the number of seconds

Examples:

ISO 8601 duration Description
P30D 30 days
PT1H 1 hour
P1DT12H 1 day and 12 hours

More details about the format of ISO 8601 durations can be found on Wikipedia.

Relative time ranges

The list of time ranges displayed in the Relative time frame selector can be configured, too. It consists of a list of ISO 8601 durations which the users can select on the search page.

_images/queries_relative_timerange_options.png

Streams

What are streams?

The Graylog streams are a mechanism to route messages into categories in realtime while they are processed. You define rules that instruct Graylog which message to route into which streams. Imagine sending these three messages to Graylog:

message: INSERT failed (out of disk space)
level: 3 (error)
source: database-host-1

message: Added user 'foo'.
level: 6 (informational)
source: database-host-2

message: smtp ERR: remote closed the connection
level: 3 (error)
source: application-x

One of the many things that you could do with streams is creating a stream called Database errors that is catching every error message from one of your database hosts.

Create a new stream with these rules, selecting the option to match all rules:

  • Field level must be greater than 4
  • Field source must match regular expression ^database-host-\d+

This will route every new message with a level higher than WARN and a source that matches the database host regular expression into the stream.

A message will be routed into every stream that has all (or any) of its rules matching. This means that a message can be part of many streams and not just one.

The stream is now appearing in the streams list and a click on its title will show you all database errors.

Streams can be used to be alerted in case certain condition happens. We cover more topics related to alerts in Alerts.

What’s the difference to saved searches?

The biggest difference is that streams are processed in realtime. This allows realtime alerting and forwarding to other systems. Imagine forwarding your database errors to another system or writing them to a file by regularly reading them from the message storage. Realtime streams do this much better.

Another difference is that searches for complex stream rule sets are always comparably cheap to perform because a message is tagged with stream IDs when processed. A search for Graylog internally always looks like this, no matter how many stream rules you have configured:

streams:[STREAM_ID]

Building a query with all rules would cause significantly higher load on the message storage.

How do I create a stream?

  1. Navigate to the streams section from the top navigation bar.
  2. Click “Create stream”.
  3. Save the stream after entering a name and a description. For example All error messages and Catching all error messages from all sources. The stream is now saved but not yet activated.
  4. Click on “Edit rules” for the stream you just created. That will open a page where you can manage and test stream rules.
  5. Choose how you want to evaluate the stream rules to decide which messages go into the stream:
    • A message must match all of the following rules (logical AND): Messages will only be routed into the stream if all rules in the stream are fulfilled. This is the default behavior
    • A message must match at least one of the following rules (logical OR): Messages will be routed into the stream if one or more rules in the stream are fulfilled
  6. Add stream rules, by indicating the field that you want to check, and the condition that should satisfy. Try the rules against some messages by loading them from an input or manually giving a message ID. Once you are satisfied with the results, click on “I’m done”.
  7. The stream is still paused, click on the “Start stream” button to activate the stream.

Index Sets

For starters, you should read Index model for a comprehensive description of the index set functionality in Graylog.

Every stream is assigned to an index set which controls how messages routed into that stream are being stored into Elasticsearch. The stream overview in the web interface shows the assigned index set for each stream.

_images/stream_overview.png

Index sets can be assigned to a stream when creating the stream and changed later when editing the stream settings.

Important

Graylog will not automatically copy messages into new Elasticsearch indices if another index set is being assigned to a stream.

_images/stream_create.png

Graylog routes every message into the All messages stream by default, unless the message is removed from this stream with a pipeline rule (see Processing Pipelines) or it’s routed into a stream marked with Remove matches from ‘All messages’ stream.

The latter is useful if messages should be stored with different settings than the ones in the Default index set, for example web server access logs should only be stored for 4 weeks while all other messages should be stored for 1 year.

Storage requirements

Graylog writes messages once for each index set into Elasticsearch. This means that if all streams are using the Default index set, each message will be written exactly once into Elasticsearch, no matter into how many streams the message has been sent. This can be thought of a kind of de-duplication.

If some streams use other index sets and the Remove matches from ‘All messages’ stream setting is not enabled, messages will be written into Elasticsearch at least twice, once for the Default index set and once for the assigned index set. This means that the same message will be stored in two or more indices in Elasticsearch with different index settings.

Unless you explicitly want to store messages multiple times in different Elasticsearch indices, either assign the Default index set to the respective streams or enable the Remove matches from ‘All messages’ stream setting for the respective streams.

Outputs

The stream output system allows you to forward every message that is routed into a stream to other destinations.

Outputs are managed globally (like message inputs) and not for single streams. You can create new outputs and activate them for as many streams as you like. This way you can configure a forwarding destination once and select multiple streams to use it.

Graylog ships with default outputs and can be extended with Plugins.

Use cases

These are a few example use cases for streams:

  • Forward a subset of messages to other data analysis or BI systems to reduce their license costs.
  • Monitor exception or error rates in your whole environment and broken down per subsystem.
  • Get a list of all failed SSH logins and use quick values to analyze which user names where affected.
  • Catch all HTTP POST requests to /login that were answered with a HTTP 302 and route them into a stream called Successful user logins. Now get a chart of when users logged in and use quick values to get a list of users that performed the most logins in the search time frame.

How are streams processed internally?

Every message that comes in is matched against the rules of a stream. For messages satisfying all or at least one of the stream rules (as configured in the stream), the internal ID of that stream is stored in the streams array of the processed message.

All analysis methods and searches that are bound to streams can now easily narrow their operation by searching with a streams:[STREAM_ID] limit. This is done automatically by Graylog and does not have to be provided by the user.

_images/internal_stream_processing.png

Stream Processing Runtime Limits

An important step during the processing of a message is the stream classification. Every message is matched against the user-configured stream rules. The message is added to the stream if all or any rules of a stream matches, depending on what the user chose. Applying stream rules is done during the indexing of a message only, so the amount of time spent for the classification of a message is crucial for the overall performance and message throughput the system can handle.

There are certain scenarios when a stream rule takes very long to match. When this happens for a number of messages, message processing can stall, messages waiting for processing accumulate in memory and the whole system could become non-responsive. Messages are lost and manual intervention would be necessary. This is the worst case scenario.

To prevent this, the runtime of stream rule matching is limited. When it is taking longer than the configured runtime limit, the process of matching this exact message against the rules of this specific stream is aborted. Message processing in general and for this specific message continues though. As the runtime limit needs to be configured pretty high (usually a magnitude higher as a regular stream rule match takes), any excess of it is considered a fault and is recorded for this stream. If the number of recorded faults for a single stream is higher than a configured threshold, the stream rule set of this stream is considered faulty and the stream is disabled. This is done to protect the overall stability and performance of message processing. Obviously, this is a tradeoff and based on the assumption, that the total loss of one or more messages is worse than a loss of stream classification for these.

There are scenarios where this might not be applicable or even detrimental. If there is a high fluctuation of the message load including situations where the message load is much higher than the system can handle, overall stream matching can take longer than the configured timeout. If this happens repeatedly, all streams get disabled. This is a clear indicator that your system is overutilized and not able to handle the peak message load.

How to configure the timeout values if the defaults do not match

There are two configuration variables in the configuration file of the server, which influence the behavior of this functionality.

  • stream_processing_timeout defines the maximum amount of time the rules of a stream are able to spend. When this is exceeded, stream rule matching for this stream is aborted and a fault is recorded. This setting is defined in milliseconds, the default is 2000 (2 seconds).
  • stream_processing_max_faults is the maximum number of times a single stream can exceed this runtime limit. When it happens more often, the stream is disabled until it is manually reenabled. The default for this setting is 3.

What could cause it?

If a single stream has been disabled and all others are doing well, the chances are high that one or more stream rules are performing bad under certain circumstances. In most cases, this is related to stream rules which are utilizing regular expressions. For most other stream rules types the general runtime is constant, while it varies very much for regular expressions, influenced by the regular expression itself and the input matched against it. In some special cases, the difference between a match and a non-match of a regular expression can be in the order of 100 or even 1000. This is caused by a phenomenon called catastrophic backtracking. There are good write-ups about it on the web which will help you understanding it.

Summary: How do I solve it?

  1. Check the rules of the stream that is disabled for rules that could take very long (especially regular expressions).
  2. Modify or delete those stream rules.
  3. Re-enable the stream.

Programmatic access via the REST API

Many organisations already run monitoring infrastructure that are able to alert operations staff when incidents are detected. These systems are often capable of either polling for information on a regular schedule or being pushed new alerts - this article describes how to use the Graylog Stream Alert API to poll for currently active alerts in order to further process them in third party products.

Checking for currently active alert/triggered conditions

Graylog stream alerts can currently be configured to send emails when one or more of the associated alert conditions evaluate to true. While sending email solves many immediate problems when it comes to alerting, it can be helpful to gain programmatic access to the currently active alerts.

Each stream which has alerts configured also has a list of active alerts, which can potentially be empty if there were no alerts so far. Using the stream’s ID, one can check the current state of the alert conditions associated with the stream using the authenticated API call:

GET /streams/<streamid>/alerts/check

It returns a description of the configured conditions as well as a count of how many triggered the alert. This data can be used to for example send SNMP traps in other parts of the monitoring system.

Sample JSON return value:

{
  "total_triggered": 0,
  "results": [
    {
      "condition": {
        "id": "984d04d5-1791-4500-a17e-cd9621cc2ea7",
        "in_grace": false,
        "created_at": "2014-06-11T12:42:50.312Z",
        "parameters": {
          "field": "one_minute_rate",
          "grace": 1,
          "time": 1,
          "backlog": 0,
          "threshold_type": "lower",
          "type": "mean",
          "threshold": 1
        },
        "creator_user_id": "admin",
        "type": "field_value"
      },
      "triggered": false
    }
  ],
  "calculated_at": "2014-06-12T13:44:20.704Z"
}

Note that the result is cached for 30 seconds.

List of already triggered stream alerts

Checking the current state of a stream’s alerts can be useful to trigger alarms in other monitoring systems, but if one wants to send more detailed messages to operations, it can be very helpful to get more information about the current state of the stream, for example the list of all triggered alerts since a certain timestamp.

This information is available per stream using the call:

GET /streams/<streamid>/alerts?since=1402460923

The since parameter is a unix timestamp value. Its return value could be:

{
  "total": 1,
  "alerts": [
    {
      "id": "539878473004e72240a5c829",
      "condition_id": "984d04d5-1791-4500-a17e-cd9621cc2ea7",
      "condition_parameters": {
        "field": "one_minute_rate",
        "grace": 1,
        "time": 1,
        "backlog": 0,
        "threshold_type": "lower",
        "type": "mean",
        "threshold": 1
      },
      "description": "Field one_minute_rate had a mean of 0.0 in the last 1 minutes with trigger condition lower than 1.0. (Current grace time: 1 minutes)",
      "triggered_at": "2014-06-11T15:39:51.780Z",
      "stream_id": "53984d8630042acb39c79f84"
    }
  ]
}

Using this information more detailed messages can be produced, since the response contains more detailed information about the nature of the alert, as well as the number of alerts triggered since the timestamp provided.

Note that currently a maximum of 300 alerts will be returned.

FAQs

Using regular expressions for stream matching

Stream rules support matching field values using regular expressions. Graylog uses the Java Pattern class to execute regular expressions.

For the individual elements of regular expression syntax, please refer to Oracle’s documentation, however the syntax largely follows the familiar regular expression languages in widespread use today and will be familiar to most.

However, one key question that is often raised is matching a string in case insensitive manner. Java regular expressions are case sensitive by default. Certain flags, such as the one to ignore case sensitivity can either be set in the code, or as an inline flag in the regular expression.

To for example route every message that matches the browser name in the following user agent string:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36

the regular expression .*applewebkit.* will not match because it is case sensitive. In order to match the expression using any combination of upper- and lowercase characters use the (?i) flag as such:

(?i).*applewebkit.*

Most of the other flags supported by Java are rarely used in the context of matching stream rules or extractors, but if you need them their use is documented on the same Javadoc page by Oracle.

Can I add messages to a stream after they were processed and stored?

No. Currently there is no way to re-process or re-match messages into streams.

Only new messages are routed into the current set of streams.

Can I write own outputs, alert conditions or notifications?

Yes. Please refer to the Plugins documentation page.

Alerts

Alerts are always based on streams. You can define conditions that trigger alerts. For example whenever the stream All production exceptions has more than 50 messages per minute or when the field milliseconds had a too high standard deviation in the last five minutes.

Navigate to the alerts section from the top navigation bar to see already configured alerts, alerts that were fired in the past or to configure new alert conditions and notifications.

Graylog ships with default alert conditions and alert notifications, and both can be extended with Plugins.

Alert states

Graylog alerts are periodical searches that can trigger some notifications when a defined condition is satisfied. Since Graylog 2.2.0, alerts can have two states:

Unresolved
Alerts have an unresolved state while the defined condition is satisfied. New alerts are triggered in this state, and they also execute the notifications attached to the stream. These alerts usually require an action on your side.
Resolved
Graylog automatically resolves alerts once their alert condition is no longer satisfied. This is the final state of an alert, as Graylog will create a new alert if the alert condition is satisfied again in the future. After an alert is resolved, Graylog will apply the grace period you defined in the alert condition, waiting a certain time before creating a new alert for this alert condition.

Alerts overview

The alerts overview page lets you find out which alerts currently require your attention in an easy way, while also allowing you to check alerts that were triggered in the past and are now resolved.

_images/alerts_alerts_overview.png

You can click on an alert name at any time to see more details about it.

Alert details

From the alert details page you can quickly check the reason why an alert was triggered, the status and configuration of notifications sent by Graylog, and the search results in the time frame when the alert was unresolved.

_images/alerts_alert_details.png
Alert timeline

From within the alert details page, you can see a timeline of what occurred since Graylog detected an alert condition was satisfied. This includes the time when Graylog evaluated the condition that triggered the alert, the time when notifications were executed and the results of executing them, and the time when the alert was resolved (if that is the case).

Triggered notifications

Sometimes sending alert notifications may fail for some reason. Graylog includes details of the configured notifications at the time an alert was triggered and the result of executing those notifications, helping you to debug and fix any problems that may arise.

Search results

You can quickly look at messages received while the alert was unresolved from within the alert details page. It is also possible to open that search in the search page, allowing you to further analyse the problem at hand.

Conditions

The first step of managing alerts with Graylog is defining alert conditions.

Alert conditions specify searches that Graylog will execute periodically, and also indicate under which circumstances Graylog should consider those search results as exceptional, triggering an alert in that case.

Click on Manage conditions in the Alerts section to see your current conditions details, modify them, or add new ones. Clicking on an alert condition’s title will open a detail page where you can also see the notifications that will be executed when the condition is satisfied.

_images/alerts_alert_condition.png

Alert condition types explained

In this section we explain what the default alert conditions included in Graylog do, and how to configure them. Since Graylog 2.2.0, alert conditions can be extended via Plugins, you can find more types in the Graylog Marketplace or even create your own.

Message count condition

This condition triggers whenever the stream received more than X messages in the last Y minutes. Perfect for example to be alerted when there are many exceptions on your platform. Create a stream that catches every error message and be alerted when that stream exceeds normal throughput levels.

Field aggregation condition

This condition triggers whenever the result of a statistical computation of a numerical message field in the stream is higher or lower than a given threshold. Perfect to monitor performance problems: Be alerted whenever the standard deviation of the response time of your application was higher than X in the last Y minutes.

Field content condition

This condition triggers whenever the stream received at least one message since the last alert run that has a field set to a given value. Get an alert when a message with the field `type` set to `security` arrives in the stream.

Important

We do not recommend to run this on analyzed fields like message or full_message because it is broken down to terms and you might get unexpected alerts. For example a check for security would also alert if a message with the field set to no security is received because it was broken down to no and security. This only happens on the analyzed message and full_message in Graylog.

Please also take note that only a single alert is raised for this condition during the alerting interval, although multiple messages containing the given value may have been received since the last alert. The alerting interval is the time that is configured as alert_check_interval in the Graylog server.conf.

Notifications

Warning

Starting in Graylog 2.2.0, alert notifications are only triggered once, just when a new alert is created. As long as the alert is unresolved or in grace period, Graylog will not send further notifications. This will help you reducing the noise and annoyance of getting notified way too often when a problem persists for a while. Should your setup require repeated notifications you can enable this during the creation of the alert condition since Graylog 2.2.2.

Notifications (previously known as Alarm Callbacks) enable you to take actions on external systems when an alert is triggered. In this way, you can rely on Graylog to know when something is not right in your logs.

Click on Manage notifications in the Alerts section to see your current notification details, modify them, test them, or add new ones. Remember that notifications are associated to streams, so all conditions evaluated in a stream will share the same notifications.

_images/alerts_alert_notification.png

Alert notifications types explained

In this section we explain what the default alert notifications included in Graylog do, and how to configure them. Alert notifications are meant to be extensible through Plugins, you can find more types in the Graylog Marketplace or even create your own.

Important

In previous versions of Graylog (before 2.2.0), the email alarm notification was used, when alert conditions existed for a stream, but no alarm notification had been created before. This has been changed, so that if there is no alarm notification existing for a stream, alerts will be shown in the interface but no other action is performed. To help users coming from earlier version, there is a migration job which is being run once, creating the email alarm notification explicitly for qualifying streams, so the old behavior is preserved.

Email alert notification

The email alert notification can be used to send an email to the configured alert receivers when the conditions are triggered.

Make sure to check the email-related configuration settings in the Graylog configuration file.

Three configuration options are available for the alert notification to customize the email that will be sent. The email body and email subject are JMTE templates. JMTE is a minimal template engine that supports variables, loops and conditions. See the JMTE documentation for a language reference.

We expose the following objects to the templates.

stream

The stream this alert belongs to.

  • stream.id ID of the stream
  • stream.title title of the stream
  • stream.description stream description
stream_url
A string that contains the HTTP URL to the stream.
check_result

The check result object for this stream.

  • check_result.triggeredCondition string representation of the triggered alert condition
  • check_result.triggeredAt date when this condition was triggered
  • check_result.resultDescription text that describes the check result
backlog
A list of message objects. Can be used to iterate over the messages via foreach.
message (only available via iteration over the backlog object)

The message object has several fields with details about the message. When using the message object without accessing any fields, the toString() method of the underlying Java object is used to display it.

  • message.id autogenerated message id
  • message.message the actual message text
  • message.source the source of the message
  • message.timestamp the message timestamp
  • message.fields map of key value pairs for all the fields defined in the message

The message.fields fields can be useful to get access to arbitrary fields that are defined in the message. For example message.fields.full_message would return the full_message of a GELF message.

_images/alerts_email_notification.png
HTTP alert notification

The HTTP alert notification lets you configure an endpoint that will be called when the alert is triggered.

Graylog will send a POST request to the notification URL including information about the alert. Here is an example of the payload included in a notification:

{
    "check_result": {
        "result_description": "Stream had 2 messages in the last 1 minutes with trigger condition more than 1 messages. (Current grace time: 1 minutes)",
        "triggered_condition": {
            "id": "5e7a9c8d-9bb1-47b6-b8db-4a3a83a25e0c",
            "type": "MESSAGE_COUNT",
            "created_at": "2015-09-10T09:44:10.552Z",
            "creator_user_id": "admin",
            "grace": 1,
            "parameters": {
                "grace": 1,
                "threshold": 1,
                "threshold_type": "more",
                "backlog": 5,
                "time": 1
            },
            "description": "time: 1, threshold_type: more, threshold: 1, grace: 1",
            "type_string": "MESSAGE_COUNT",
            "backlog": 5
        },
        "triggered_at": "2015-09-10T09:45:54.749Z",
        "triggered": true,
        "matching_messages": [
            {
                "index": "graylog2_7",
                "message": "WARN: System is failing",
                "fields": {
                    "gl2_remote_ip": "127.0.0.1",
                    "gl2_remote_port": 56498,
                    "gl2_source_node": "41283fec-36b4-4352-a859-7b3d79846b3c",
                    "gl2_source_input": "55f15092bee8e2841898eb53"
                },
                "id": "b7b08150-57a0-11e5-b2a2-d6b4cd83d1d5",
                "stream_ids": [
                    "55f1509dbee8e2841898eb64"
                ],
                "source": "127.0.0.1",
                "timestamp": "2015-09-10T09:45:49.284Z"
            },
            {
                "index": "graylog2_7",
                "message": "ERROR: This is an example error message",
                "fields": {
                    "gl2_remote_ip": "127.0.0.1",
                    "gl2_remote_port": 56481,
                    "gl2_source_node": "41283fec-36b4-4352-a859-7b3d79846b3c",
                    "gl2_source_input": "55f15092bee8e2841898eb53"
                },
                "id": "afd71342-57a0-11e5-b2a2-d6b4cd83d1d5",
                "stream_ids": [
                    "55f1509dbee8e2841898eb64"
                ],
                "source": "127.0.0.1",
                "timestamp": "2015-09-10T09:45:36.116Z"
            }
        ]
    },
    "stream": {
        "creator_user_id": "admin",
        "outputs": [],
        "matching_type": "AND",
        "description": "test stream",
        "created_at": "2015-09-10T09:42:53.833Z",
        "disabled": false,
        "rules": [
            {
                "field": "gl2_source_input",
                "stream_id": "55f1509dbee8e2841898eb64",
                "id": "55f150b5bee8e2841898eb7f",
                "type": 1,
                "inverted": false,
                "value": "55f15092bee8e2841898eb53"
            }
        ],
        "alert_conditions": [
            {
                "creator_user_id": "admin",
                "created_at": "2015-09-10T09:44:10.552Z",
                "id": "5e7a9c8d-9bb1-47b6-b8db-4a3a83a25e0c",
                "type": "message_count",
                "parameters": {
                    "grace": 1,
                    "threshold": 1,
                    "threshold_type": "more",
                    "backlog": 5,
                    "time": 1
                }
            }
        ],
        "id": "55f1509dbee8e2841898eb64",
        "title": "test",
        "content_pack": null
    }
}
Script alert notification

The Script Alert Notification lets you configure a script that will be executed when the alert is triggered.

Important

Script Alert Notification is an Enterprise Integrations plugin feature and thus requires an Enterprise license.

_images/alerts_script_notification.png

These are the supported configuration options.

Script Path
The path to where the script is located. Must me within the permitted script path (which is customizable).
Script Timeout
The maximum time (in milliseconds) the script will be allowed to execute before being forcefully terminated.
Script Arguments

String of parameters in which the delimiters are either a space-delimited or a new-line. The following argument variables may be used:

Stream

The stream this alert belongs to.

  • stream_id ID of the stream
  • stream_name title of the stream
  • stream_description stream description
  • stream_url a string that contains the URL to the view the relevant messages for the alert. Make sure to set the HTTP URL configuration parameter, as there is no default.
Alert

The check result object for this stream.

  • alert_description text that describes the check result
  • alert_triggered_at date when this condition was triggered
Condition

The available conditions to request are

  • condition_id ID of the condition
  • condition_description description of the condition
  • condition_title title of the condition
  • condition_type type of condition
  • condition_grace grace period for the condition
  • condition_repeat_notification repeat notification of the script
Send Alert Data Through STDIN

Sends JSON alert data through standard in. You can use a JSON parser in your script.

{
  "stream_id": "000000000000000000000001",
  "stream_name": "All messages",
  "stream_description": "Stream containing all messages",
  "stream_url": "http://localhost:8080///streams/000000000000000000000001/messages?rangetype=absolute&from=2019-01-25T20:57:50.793Z&to=2019-01-25T21:02:50.793Z&q=*",
  "alert_description": "Stream received messages matching <has_field:\"true\"> (Current grace time: 0 minutes)",
  "alert_triggered_at": "2019-01-25T21:02:50.793Z",
  "condition_id": "ea9fcdff-2037-44f9-801e-099bf4bb3dbd",
  "condition_description": "field: has_field, value: true, grace: 0, repeat notifications: false",
  "condition_title": "has_field",
  "condition_type": "field_content_value",
  "condition_grace": 0,
  "condition_parameters": {
    "backlog": 10,
    "repeat_notifications": false,
    "field": "has_field",
    "query": "*",
    "grace": 0,
    "value": "true"
  },
  "condition_repeat_notifications": false,
  "message_backlog": [
    {
      "has_field": "true",
      "gl2_remote_ip": "127.0.0.1",
      "gl2_remote_port": 56246,
      "streams": [
        "000000000000000000000001"
      ],
      "gl2_source_node": "e065896b-8a9a-4f45-83f2-e740525ed035",
      "_id": "92839500-20e4-11e9-8175-0637e3f7ecfc",
      "source": "example.org",
      "message": "Hello there",
      "gl2_source_input": "5c2e99687a90e30a3512f766",
      "facility": "test",
      "timestamp": "2019-01-25T21:02:49.423Z"
    },
    {
      "has_field": "true",
      "gl2_remote_ip": "127.0.0.1",
      "gl2_remote_port": 56245,
      "streams": [
        "000000000000000000000001"
      ],
      "gl2_source_node": "e065896b-8a9a-4f45-83f2-e740525ed035",
      "_id": "928087c0-20e4-11e9-8175-0637e3f7ecfc",
      "source": "example.org",
      "message": "Hello there",
      "gl2_source_input": "5c2e99687a90e30a3512f766",
      "facility": "test",
      "timestamp": "2019-01-25T21:02:49.403Z"
    }
  ],
  "message_backlog_size": 5
}

Script Alert Notification success is determined by its exit value; success equals zero. Any non-zero exit value will cause it to fail. Returning any error text through STDERR will also cause the alarm callback to fail.

Here is a sample Python script that shows all of the supported Script Alert Notification functionality (argument parsing, STDIN JSON parsing, STDOUT, exit values, and returning an exit value).:

#!/usr/bin/env python3
import json
import sys
import time


# Function that prints text to standard error
def print_stderr(*args, **kwargs):
    print(*args, file=sys.stderr, **kwargs)

# Main function
if __name__ == "__main__":

    # Print out all input arguments.
    sys.stdout.write("All Arguments Passed In: " + ' '.join(sys.argv[1:]) + "\n")
    sys.stdout.write("Stream Name: " + sys.argv[2] + "\n")
    sys.stdout.write("Stream Description: " + sys.argv[3] + "\n")
    sys.stdout.write("Alert Triggered At: " + sys.argv[6] + "\n")

    # Turn stdin.readlines() array into a string
    std_in_string = ''.join(sys.stdin.readlines())

    # Load JSON
    alert_object = json.loads(std_in_string)

    # Extract some values from the JSON.
    sys.stdout.write("Values from JSON: \n")
    sys.stdout.write("Stream ID: " + alert_object["stream_id"] + "\n")
    sys.stdout.write("Stream Name: " + alert_object["stream_name"] + "\n")
    sys.stdout.write("Alert Triggered At: " + alert_object["alert_triggered_at"] + "\n")

    # Extract Message Backlog field from JSON.
    sys.stdout.write("\n\nFields:\n")
    for message in alert_object["message_backlog"]:
        for field in message.keys():
            print("Field: " + field)
            print("Value: " + str(message[field]))

    # Write to stderr if desired
    # print_stderr("Test return through standard error")

    # Return an exit value. Zero is success, non-zero indicates failure.
    exit(0)

Dashboards

Why dashboards matter

Using dashboards allows you to build pre-defined views on your data to always have everything important just one click away.

Sometimes it takes domain knowledge to be able to figure out the search queries to get the correct results for your specific applications. People with the required domain knowledge can define the search query once and then display the results on a dashboard to share them with co-workers, managers, or even sales and marketing departments.

This guide will take you through the process of creating dashboards and storing information on them. At the end you will have a dashboard with automatically updating information that you can share with anybody or just a subset of people based on permissions.

_images/dashboards_1.png

How to use dashboards

Creating an empty dashboard

Navigate to the Dashboards section using the link in the top menu bar of your Graylog web interface. The page is listing all dashboards that you are allowed to view. (More on permissions later.) Hit the Create dashboard button to create a new empty dashboard.

The only required information is a title and a description of the new dashboard. Use a specific but not too long title so people can easily see what to expect on the dashboard. The description can be a bit longer and could contain more detailed information about the displayed data or how it is collected.

Hit the Create button to create the dashboard. You should now see your new dashboard on the dashboards overview page. Click on the title of your new dashboard to see it. Next, we will be adding widgets to the dashboard we have just created.

_images/dashboards_2.png

Adding widgets

You should have your empty dashboard in front of you. Let’s add some widgets! You can add search result information to dashboards with a couple of clicks. The following search result types can be added to dashboards:

  • Search result counts
  • Search result histogram charts
  • Statistical values
  • Field value charts
  • Stacked charts
  • Quick values results

You can learn more about the different widget types in Widget types explained.

Once you can see the results of your search, you will see buttons with the “Add to dashboard” text, that will allow you to select the dashboard where the widget will be displayed, and configure the widget.

_images/dashboards_3.png _images/dashboards_4.png

Examples

It is strongly recommended to read the getting started guide on basic searches and analysis first. This will make the following examples more obvious for you.

  • Top log sources today
    • Example search: *, timeframe: Last 24 hours
    • Expand the source field in the sidebar and hit Quick values
    • Add quick values to dashboard
  • Number of exceptions in a given app today
    • Example search: source:myapp AND Exception, timeframe: Last 24 hours
    • Add search result count to dashboard
  • Response time chart of a given app
    • Example search: source:myapp2, any timeframe you want
    • Expand a field representing the response time of requests in the sidebar and hit Generate chart
    • Add chart to dashboard

Widgets from streams

You can of course also add widgets from stream search results. Every widget added this way will always be bound to streams. If you have a stream that contains every SSH login you can just search for everything (*) in that stream and store the result count as SSH logins on a dashboard.

Result

You should now see widgets on your dashboard. You will learn how to modify the dashboard, and edit widgets in the next chapter.

_images/dashboards_1.png

Widget types explained

Graylog supports a wide variety of widgets that allow you to quickly visualize data from your logs. This section intends to give you some information to better understand each widget type, and how they can help you to see relevant details from the many logs you receive.

Search result counts

This kind of widget includes a count of the number of search results for a given search. It can help you to quickly visualize things like the number of exceptions an application logs, or the number of requests your site receives.

All search result counts created with a relative time frame can additionally display trend information. The trend is calculated by comparing the count for the given time frame, with the one resulting from going further back the same amount of time. For example, to calculate the trend in a search result count with a relative search of 5 minutes ago, Graylog will count the messages in the last 5 minutes, and compare that with the count of the previous 5 minutes.

Search result histogram charts

The search result histogram displays a chart using the time frame of your search, graphing the number of search result counts over time. It may help you to visualize how the number of request to your site change over time, or to see how many downloads a file has over time.

Changing the graph resolution, you can decide how much time each bar of the graph represents.

Statistical values

You can add to your dashboard any statistical value calculated for a field. This may help you to see the mean time response for your application, or how many unique servers are handling requests to your application, by using the cardinality value of that field. Please refer to Field statistics for more information on the available statistical functions and how to display them in your searches.

As with search result counts, you can also add trend information to statistical value widgets created with a relative time frame.

Field value charts

To draw an statistical value over time, you can use a field value chart. They could help you to see the evolution of the number of unique users visiting your site in the last week. In the Field graphs section we explain how to create these charts and ways you can customize them.

Stacked charts

Stacked charts group several field value charts under the same axes. They let you compare different values in a compact way, like the number of visits to two different websites. As explained in Field graphs, stacked charts are basically field value charts represented in the same axes.

Quick values results

In order to show a list of values a certain field contains and their distribution, you can use a quick value widget. This may help you to see the percentage of failed requests in your application, or which parts of your application experience more problems. Please refer to Quick values to see how to request this information in your search result page.

The quick values information can be represented as a pie chart and/or as a table, so you can choose what is the best fit for your needs.

Modifying dashboards

You need to unlock dashboards to make any changes to them. Hit the “Unlock/Edit” button in the top right corner of a dashboard to unlock it. You should now see different icons at the bottom of each widget, that allow you to perform more actions.

Unlocked dashboard widgets explained

Unlocked dashboard widgets have two buttons that should be pretty self-explanatory.

  • Delete widget
  • Edit widget configuration
  • Change widget size (when you hover over the widget)
_images/dashboards_5.png

Widget cache times

Widget values are cached in graylog-server by default. This means that the cost of value computation does not grow with every new device or even browser tab displaying a dashboard. Some widgets might need to show real-time information (set cache time to 1 second) and some widgets might be updated way less often (like Top SSH users this month, cache time 10 minutes) to save expensive computation resources.

Repositioning widgets

Just grab a widget with your mouse in unlocked dashboard mode and move it around. Other widgets should adopt and re-position intelligently to make place for the widget you are moving. The positions are automatically saved when dropping a widget.

Resizing widgets

When hovering over a widget, you will see that a gray arrow appears in its bottom-right corner. You can use that icon to resize widgets. Their contents will adapt to the new size automatically!

_images/dashboards_7.png

Dashboard permissions

Graylog users in the Admin role are always allowed to view and edit all dashboards. Users in the Reader role are by default not allowed to view or edit any dashboard.

_images/dashboards_6.png

Navigate to System -> Roles and create a new role that grant the permissions you wish. You can then assign that new role to any users you wish to give dashboard permissions in the System -> Users page.

You can read more about user permissions and roles.

That’s it!

Congratulations, you have just gone through the basic principles of Graylog dashboards. Now think about which dashboards to create. We suggest:

  • Create dashboards for yourself and your team members
  • Create dashboards to share with your manager
  • Create dashboards to share with the CIO of your company

Think about which information you need access to frequently. What information could your manager or CIO be interested in? Maybe they want to see how the number of exceptions went down or how your team utilized existing hardware better. The sales team could be interested to see signup rates in realtime and the marketing team will love you for providing insights into low level KPIs that is just a click away.

Extractors

The problem explained

Syslog (RFC3164, RFC5424) is the de facto standard logging protocol since the 1980s and was originally developed as part of the sendmail project. It comes with some annoying shortcomings that we tried to improve in GELF for application logging.

Because syslog has a clear specification in its RFCs it should be possible to parse it relatively easy. Unfortunately there are a lot of devices (especially routers and firewalls) out there that send logs looking like syslog but actually breaking several rules stated in the RFCs. We tried to write a parser that reads all of them as good as possible and failed. Such a loosely defined text message usually breaks the compatibility in the first date field already. Some devices leave out hostnames completely, some use localized time zone names (e. g. “MESZ” instead of “CEST”), and some just omit the current year in the timestamp field.

Then there are devices out there that at least do not claim to send syslog when they don’t but have another completely separate log format that needs to be parsed specifically.

We decided not to write custom message inputs and parsers for all those thousands of devices, formats, firmwares and configuration parameters out there but came up with the concept of Extractors introduced the v0.20.0 series of Graylog.

Graylog extractors explained

The extractors allow you to instruct Graylog nodes about how to extract data from any text in the received message (no matter from which format or if an already extracted field) to message fields. You may already know why structuring data into fields is important if you are using Graylog: There are a lot of analysis possibilities with full text searches but the real power of log analytics unveils when you can run queries like http_response_code:>=500 AND user_id:9001 to get all internal server errors that were triggered by a specific user.

Wouldn’t it be nice to be able to search for all blocked packages of a given source IP or to get a quickterms analysis of recently failed SSH login usernames? Hard to do when all you have is just a single long text message.

Attention

Graylog extractors only work on text fields but won’t be executed for numeric fields or anything other than a string.

Creating extractors is possible via either Graylog REST API calls or from the web interface using a wizard. Select a message input on the System -> Inputs page and hit Manage extractors in the actions menu. The wizard allows you to load a message to test your extractor configuration against. You can extract data using for example regular expressions, Grok patterns, substrings, or even by splitting the message into tokens by separator characters. The wizard looks like this and should be pretty intuitive:

_images/extractors_1.png

You can also choose to apply so called converters on the extracted value to for example convert a string consisting of numbers to an integer or double value (important for range searches later), anonymize IP addresses, lower-/uppercase a string, build a hash value, and much more.

Import extractors

The recommended way of importing extractors in Graylog is using Content packs. The Graylog Marketplace provides access to many content packs that you can easily download and import into your Graylog setup.

You can still import extractors from JSON if you want to. Just copy the JSON extractor export into the import dialog of a message input of the fitting type (every extractor set entry in the directory tells you what type of input to spawn, e. g. syslog, GELF, or Raw/plaintext) and you are good to go. The next messages coming in should already include the extracted fields with possibly converted values.

A message sent by Heroku and received by Graylog with the imported Heroku extractor set on a plaintext TCP input looks like this: (look at the extracted fields in the message detail view)

_images/extractors_2.png

Using regular expressions to extract data

Extractors support matching field values using regular expressions. Graylog uses the Java Pattern class to evaluate regular expressions.

For the individual elements of regular expression syntax, please refer to Oracle’s documentation, however the syntax largely follows the familiar regular expression languages in widespread use today and will be familiar to most.

However, one key question that is often raised is matching a string in case insensitive manner. Java regular expressions are case sensitive by default. Certain flags, such as the one to ignore case sensitivity can either be set in the code, or as an inline flag in the regular expression.

For example, to create an extractor that matches the browser name in the following user agent string:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36

the regular expression (applewebkit) will not match because it is case sensitive. In order to match the expression using any combination of upper- and lowercase characters use the (?i) flag as such:

(?i)(applewebkit)

Most of the other flags supported by Java are rarely used in the context of matching stream rules or extractors, but if you need them their use is documented on the same Javadoc page by Oracle. One common reason to use regular expression flags in your regular expression is to make use of what is called non-capturing groups. Those are parentheses which only group alternatives, but do not make Graylog extract the data they match and are indicated by (?:).

Using Grok patterns to extract data

Graylog also supports the extracting data using the popular Grok language to allow you to make use of your existing patterns.

Grok is a set of regular expressions that can be combined to more complex patterns, allowing to name different parts of the matched groups.

By using Grok patterns, you can extract multiple fields from a message field in a single extractor, which often simplifies specifying extractors.

Simple regular expressions are often sufficient to extract a single word or number from a log line, but if you know the entire structure of a line beforehand, for example for an access log, or the format of a firewall log, using Grok is advantageous.

For example a firewall log line could contain:

len=50824 src=172.17.22.108 sport=829 dst=192.168.70.66 dport=513

We can now create the following patterns on the System/Grok Patterns page in the web interface:

BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM})
IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?
IPV4 (?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])
IP (?:%{IPV6}|%{IPV4})
DATA .*?

Then, in the extractor configuration, we can use these patterns to extract the relevant fields from the line:

len=%{NUMBER:length} src=%{IP:srcip} sport=%{NUMBER:srcport} dst=%{IP:dstip} dport=%{NUMBER:dstport}

This will add the relevant extracted fields to our log message, allowing Graylog to search on those individual fields, which can lead to more effective search queries by allowing to specifically look for packets that came from a specific source IP instead of also matching destination IPs if one would only search for the IP across all fields.

If the Grok pattern creates many fields, which can happen if you make use of heavily nested patterns, you can tell Graylog to skip certain fields (and the output of their subpatterns) by naming a field with the special keyword UNWANTED.

Let’s say you want to parse a line like:

type:44 bytes:34 errors:122

but you are only interested in the second number bytes. You could use a pattern like:

type:%{BASE10NUM:type} bytes:%{BASE10NUM:bytes} errors:%{BASE10NUM:errors}

However, this would create three fields named type, bytes, and errors. Even not naming the first and last patterns would still create a field names BASE10NUM. In order to ignore fields, but still require matching them use UNWANTED:

type:%{BASE10NUM:UNWANTED} bytes:%{BASE10NUM:bytes} errors:%{BASE10NUM:UNWANTED}

This now creates only a single field called bytes while making sure the entire pattern must match.

If you already know the data type of the extracted fields, you can make use of the type conversion feature built into the Graylog Grok library. Going back to the earlier example:

len=50824 src=172.17.22.108 sport=829 dst=192.168.70.66 dport=513

We know that the content of the field len is an integer and would like to make sure it is stored with that data type, so we can later create field graphs with it or access the field’s statistical values, like average etc.

Grok directly supports converting field values by adding ;datatype at the end of the pattern, like:

len=%{NUMBER:length;int} src=%{IP:srcip} sport=%{NUMBER:srcport} dst=%{IP:dstip} dport=%{NUMBER:dstport}

The currently supported data types, and their corresponding ranges and values, are:

Type Range Example
byte -128 … 127 %{NUMBER:fieldname;byte}
short -32768 … 32767 %{NUMBER:fieldname;short}
int -2^31 … 2^31 -1 %{NUMBER:fieldname;int}
long -2^63 … 2^63 -1 %{NUMBER:fieldname;long}
float 32-bit IEEE 754 %{NUMBER:fieldname;float}
double 64-bit IEEE 754 %{NUMBER:fieldname;double}
boolean true, false %{DATA:fieldname;boolean}
string Any UTF-8 string %{DATA:fieldname;string}
date See SimpleDateFormat %{DATA:timestamp;date;dd/MMM/yyyy:HH:mm:ss Z}
datetime Alias for date  

There are many resources are the web with useful patterns, and one very helpful tool is the Grok Debugger, which allows you to test your patterns while you develop them.

Graylog uses Java Grok to parse and run Grok patterns.

Using the JSON extractor

Since version 1.2, Graylog also supports extracting data from messages sent in JSON format.

Using the JSON extractor is easy: once a Graylog input receives messages in JSON format, you can create an extractor by going to System -> Inputs and clicking on the Manage extractors button for that input. Next, you need to load a message to extract data from, and select the field containing the JSON document. The following page let you add some extra information to tell Graylog how it should extract the information. Let’s illustrate how a message would be extracted with an example message:

{"level": "ERROR", "details": {"message": "This is an example error message", "controller": "IndexController", "tags": ["one", "two", "three"]}}

Using the default settings, that message would be extracted into these fields:

details_tags
one, two, three
level
ERROR
details_controller
IndexController
details_message
This is an example error message

In the create extractor page, you can also customize how to separate list of elements, keys, and key/values. It is also possible to flatten JSON structures or expand them into multiple fields, as shown in the example above.

Automatically extract all key=value pairs

Sometimes you will receive messages like this:

This is a test message with some key/value pairs. key1=value1 some_other_key=foo

You might want to extract all key=value pairs into Graylog message fields without having to specify all possible key names or even their order. This is how you can easily do this:

Create a new extractor of type “Copy Input” and select to read from the field message. (Or any other string field that contains key=value pairs.) Configure the extractor to store the (copied) field value to the same field. In this case message. The trick is to add the “Key=Value pairs to fields” converter as last step. Because we use the “Copy Input” extractor, the converter will run over the complete field you selected and convert all key=value pairs it can find.

This is a screenshot of the complete extractor configuration:

_images/keyvalue_converter_1.png

… and this is the resulting message:

_images/keyvalue_converter_2.png

Normalization

Many log formats are similar to each other, but not quite the same. In particular they often only differ in the names attached to pieces of information.

For example, consider different hardware firewall vendors, whose models log the destination IP in different fields of the message, some use dstip, some dst and yet others use destination-address:

2004-10-13 10:37:17 PDT Packet Length=50824, Source address=172.17.22.108, Source port=829, Destination address=192.168.70.66, Destination port=513
2004-10-13 10:37:17 PDT len=50824 src=172.17.22.108 sport=829 dst=192.168.70.66 dport=513
2004-10-13 10:37:17 PDT length="50824" srcip="172.17.22.108" srcport="829" dstip="192.168.70.66" dstport="513"

You can use one or more non-capturing groups to specify the alternatives of the field names, but still be able to extract the a parentheses group in the regular expression. Remember that Graylog will extract data from the first matched group of the regular expression. An example of a regular expression matching the destination IP field of all those log messages from above is:

(?:dst|dstip|[dD]estination\saddress)="?(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"?

This will only extract the IP address without caring about which of the three naming schemes was used in the original log message. This way you don’t have to set up three different extractors.

The standard date converter

Date parser converters for extractors allow you to convert extracted data into timestamps - Usually used to set the timestamp of a message based on some date it contains. Let’s assume we have this message from a network device:

<131>: foo-bar-dc3-org-de01: Mar 12 00:45:38: %LINK-3-UPDOWN: Interface GigabitEthernet0/31, changed state to down

Extracting most of the data is not a problem and can be done easily. Using the date in the message (Mar 12 00:45:38) as Graylog message timestamp however needs to be done with a date parser converter.

Use a copy input extractor rule to select the timestamp and apply the Date converter with a format string:

MMM dd HH:mm:ss

(format string table at the end of this page)

_images/dateparser_1.png _images/dateparser_2.png
Standard date converter format string table
Symbol Meaning Presentation Examples
G era text AD
C century of era (>=0) number 20
Y year of era (>=0) year 1996
x weekyear year 1996
w week of weekyear number 27
e day of week number 2
E day of week text Tuesday; Tue
y year year 1996
D day of year number 189
M month of year month July; Jul; 07
d day of month number 10
a halfday of day text PM
K hour of halfday (0~11) number 0
h clockhour of halfday (1~12) number 12
H hour of day (0~23) number 0
k clockhour of day (1~24) number 24
m minute of hour number 30
s second of minute number 55
S fraction of second millis 978
z time zone text Pacific Standard Time; PST
Z time zone offset/id zone -0800; -08:00; America/Los_Angeles
escape for text delimiter  
‘’ single quote literal

The flexible date converter

Now imagine you had one of those devices that send messages that are not so easy to parse because they do not follow a strict timestamp format. Some network devices for example like to send days of the month without adding a padding 0 for the first 9 days. You’ll have dates like Mar 9 and Mar 10 and end up having problems defining a parser string for that. Or maybe you have something else that is really exotic like just last wednesday as timestamp. The flexible date converter is accepting any text data and tries to build a date from that as good as it can.

Examples:

  • Mar 12, converted at 12:27:00 UTC in the year 2014: 2014-03-12T12:27:00.000
  • 2014-3-12 12:27: 2014-03-12T12:27:00.000
  • Mar 12 2pm: 2014-03-12T14:00:00.000

Note that the flexible date converter is using UTC as time zone by default unless you have time zone information in the parsed text or have configured another time zone when adding the flexible date converter to an extractor (see this comprehensive list of time zones available for the flexible date converter).

Processing Pipelines

Graylog’s new processing pipelines plugin allows greater flexibility in routing, blacklisting, modifying, and enriching messages as they flow through Graylog.

Pipelines and rules are not configuration for pre-built code, as extractors and stream rules are, but are instead represented as code, much like Drools rules. This gives them great flexibility and extensibility, and enables live changes to Graylog’s message processing behavior.

The language used for pipeline rules is very simple and can be extended by functions, which are fully pluggable.

The following pages introduce the concepts of pipelines, rules, stream connections, and the built-in functions.

Pipelines

Overview

Pipelines are the central concept tying together the processing steps applied to your messages.

Pipelines contain rules and can be connected to one or more streams, enabling fine-grained control of the processing applied to messages.

Processing rules are simply conditions followed by a list of actions, and do not have control flow by themselves. Therefore, pipelines have one additional concept: stages.

Think of stages as groups of conditions and actions which need to run in order. All stages with the same priority run at the same time across all connected pipelines. Stages provide the necessary control flow to decide whether or not to run the remaining stages in a pipeline.

Pipeline structure

Internally pipelines are represented as code. Let’s have a look at a simple example and understand what each part does:

pipeline "My new pipeline"
stage 1 match all
  rule "has firewall fields";
  rule "from firewall subnet";
stage 2 match either
  rule "geocode IPs";
  rule "anonymize source IPs";
end

This code snippet declares a new pipeline named My new pipeline, which has two stages.

Stages are run in the order of their given priority, and aren’t otherwise named. Stage priorities can be any integer, positive or negative, you prefer. In our example the first stage has a priority of 1 and the second stage a priority of 2, however -99 and 42 could be used instead. Ordering based upon stage priority gives you the ability to run certain rules before or after others, which might exist in other connected pipelines, without modifying those other connected pipelines. This is particularly handy when dealing with changing data formats.

For example, if there was a second pipeline declared with a stage assigned priority 0, that stage’s rules would run before either of the ones from the example (priorities 1 and 2, respectively). Note that the order in which stages are declared is irrelevant, since they are sorted according to their priority.

Stages then list the rule references they want to be executed, as well as whether any or all of the rules’ conditions need to be satisfied to continue running the pipeline.

In our example, imagine rule “has firewall fields” checks for the presence of message fields src_ip and dst_ip, but does not have any actions to run. For a message without both fields the rule’s condition would evaluate to false and the pipeline would abort after stage 1, as the stage requires all rules be satisfied (match all). With the pipeline aborted, stage 2 would not run.

match either acts as an OR operator, only requiring a single rule’s condition evaluate to true in order to continue pipeline processing. Note that actions are still ran for all matching rules in the stage, even if it is the final stage in the pipeline.

Rules are referenced by their names, and can therefore be shared among many different pipelines. The intention is to enable creation of reusable building blocks, making it easier to process the data specific to your organization or use case.

Read more about Rules in the next section.

Rules

Overview

Rules are the cornerstone of processing pipelines. They contain the logic about how to change, enrich, route, and drop messages.

To avoid the complexities of a complete programming language, Graylog supports a small rule language to express processing logic. The rule language is intentionally limited to allow for easier understanding, faster learning, and better runtime optimization.

The real work of rules is done in functions, which are completely pluggable. Graylog already ships with a great number of built-in functions, providing data conversion, string manipulation, data retrieval using lookup tables, JSON parsing, and much more.

We expect that special purpose functions will be written and shared by the community, enabling faster innovation and problem solving than previously possible.

Rule Structure

Building upon the previous example in the Pipelines section, let’s look at examples of some of the rules we’ve referenced:

rule "has firewall fields"
when
    has_field("src_ip") && has_field("dst_ip")
then
end
rule "from firewall subnet"
when
    cidr_match("10.10.10.0/24", to_ip($message.gl2_remote_ip))
then
end

Firstly, apart from naming the rule structure follows a simple when, then pattern. In the when clause we specify a boolean expression which is evaluated in the context of the current message in the pipeline. These are the conditions used by the pipeline processor to determine whether to run a rule, and collectively (when evaluating the containing stage’s match all or match any requirement) whether to continue in a pipeline.

Note that the has firewall fields rule uses the built-in function has_field to check whether the message has the src_ip and dst_ip fields, as we want to use them in a later stage of the pipeline. This rule has no actions to run in its then clause, since we only want to use it to determine whether subsequent stages should run.

The second rule, from firewall subnet, uses the built-in function cidr_match, which takes a CIDR pattern and an IP address. In this case we reference a field from the currently-processed message using the message reference syntax $message.

Graylog always sets the gl2_remote_ip field on messages, so we don’t need to check whether that field exists. If we wanted to use a field that might not exist on all messages we’d first use the has_field function to ensure its presence.

Note the call to to_ip around the gl2_remote_ip field reference. This is necessary since the field is stored as a string internally, and cidr_match requires an IP address object for its ip parameter.

Requiring an explicit conversion to an IP address object demonstrates an important feature of Graylog’s rule language: enforcement of type safety to ensure that you end up with the data in the correct format. All too often everything is treated as a string, which wastes enormous amounts of cycles on data conversion and prevents proper analysis of the data.

We again have no actions to run, since we’re just using the rule to manage the pipeline’s flow, so the then block is empty.

You might be wondering why we didn’t just combine the has firewall fields and from firewall subnet rules, since they seem to be serving the same purpose. While we could absolutely do so, recall that rules are intended to be reusable building blocks. Imagine you have a another pipeline for a different firewall subnet. Rather than duplicating the logic to check for src_ip and dst_ip, and updating each rule if anything ever changes (e.g. additional fields), you can simply add the has firewall fields rule to your new stage. With this approach you only need to update a single rule, with the change immediatedly taking effect for all pipelines referencing it. Nice!

Data Types

As we have seen in the previous section, we need to make sure to use the proper data types when calling functions.

Graylog’s rule language parser rejects invalid use of types, making it safe to write rules.

The six built-in types in Graylog are string (a UTF-8 string), double (corresponds to Java’s Double), long (Java’s Long), boolean (Boolean), void (indicating a function has no return value to prevent it being used in a condition), and ip (a subset of InetAddress), but plugins are free to add additional types as they see fit. The rule processor takes care of ensuring that values and functions agree on the types being used.

By convention, functions that convert types start with the prefix to_. Please refer to the Functions index for a list.

Conditions

In Graylog’s rules the when clause is a boolean expression, which is evaluated against the processed message.

Expressions support the common boolean operators AND (or &&), OR (||), NOT (!), and comparison operators (<, <=, >, >=, ==, !=).

Any function that returns a value can be called in the when clause, but it must eventually evaluate to a boolean. For example: we were able to use to_ip in the from firewall subnet since it was being passed to cidr_match, which returns a boolean, but could not use route_to_stream since it doesn’t return a value.

The condition must not be empty, but can simply consist of the boolean literal true. This is useful when you always want to execute a rule’s actions.

If a condition calls a function which is not present (perhaps due to a typo or missing plugin) the call evaluates to false.

Note

Comparing two fields can be done when you use the same data type, e.g. to_string($message.src_ip) == to_string($message.dst_ip) will compare the two strings and will become true on match. Comparing different data types evaluates to false.

Actions

A rule’s then clause contains a list of actions which are evaluated in the order they appear.

There are two different types of actions:

  • Function calls
  • Variable assignments

Function calls look exactly like they do in conditions. All functions, including those which do not return a value, may be used in the then clause.

Variable assignments have the following form:

let name = value;

Variables are useful to avoid recomputing expensive parsing of data, holding on to temporary values, or making rules more readable.

Variables need to be defined before they can be used. Their fields (if any) can be accessed using the name.field notation in any place where a value of the field’s type is required.

The list of actions can be empty, in which case the rule is essentially a pluggable condition to help manage a pipeline’s processing flow.

Stream connections

Overview

Pipelines by themselves do not process any messages. For a pipeline to actually do any work it must first be connected to one or more streams, which enables fine-grained control of the messages processed by that pipeline.

Note that the built-in function route_to_stream causes a message to be routed to a particular stream. After the routing occurs, the pipeline engine will look up and start evaluating any pipelines connected to that stream.

Although pipelines can trigger other pipelines via message routing, incoming messages must be processed by an initial set of pipelines connected to one or more streams.

The All messages stream

All messages received by Graylog are initially routed into the All messages stream. You can use this stream as the entry point to pipeline processing, allowing incoming messages to be routed to more streams and being processed subsequently.

However, if you prefer to use the original stream matching functionality (i.e. stream rules), you can configure the Pipeline Processor to run after the Message Filter Chain (in the Message Processors Configuration section of the System -> Configurations page) and connect pipelines to existing streams. This gives you fine-grained control over the extraction, conversion, and enrichment process.

The importance of message processor ordering

It’s important to note that the order of message processors may have a significant impact on how your messages get processed.

For example: Message Filter Chain is responsible for setting static fields and running extractors defined on inputs, as well as evaluation of stream rules. If you create a pipeline that expects the presence of a static field, but the Pipeline Processor runs before Message Filter Chain, that field will not be available for use in your pipeline.

When designing your streams and pipelines be aware of the message processor order, especially if you have dependencies on earlier message processing.

Functions

Overview

Functions are the means of interacting with the messages Graylog processes.

Functions are written in Java and are pluggable, allowing Graylog’s pipeline processing capabilities to be easily extended.

Conceptually a function receives parameters, the current message context, and (potentially) returns a value. The data types of its return value and parameters determine where it can be used in a rule. Graylog ensures the rules are sound from a data type perspective.

A function’s parameters can either be passed as named pairs or position, as long as optional parameters are declared as coming last. The functions’ documentation below indicates which parameters are optional by wrapping them in square brackets.

Let’s look at a small example to illustrate these properties:

rule "function howto"
when
    has_field("transaction_date")
then
    // the following date format assumes there's no time zone in the string
    let new_date = parse_date(to_string($message.transaction_date), "yyyy-MM-dd HH:mm:ss");
    set_field("transaction_year", new_date.year);
end

In this example, we check if the current message contains the field transaction_date and then, after converting it to a string, try to parse it according to the format string yyyy-MM-dd HH:mm:ss, so for example the string 2016-03-05 14:45:02 would match. The parse_date function returns a DateTime object from the Java Joda-Time library, allowing easier access to the date’s components.

We then add the transaction’s year as a new field, transaction_year to the message.

You’ll note that we didn’t specify a time zone for our date, but Graylog still had to pick one. Graylog never relies on the local time of your server, as that makes it nearly impossible to figure out why date handling came up with its result.

The reason Graylog knows which timezone to use is because parse_date actually takes four parameters, rather than the two we’ve given it in this example. The other two parameters are a String called timezone (default value: "UTC") and a String called locale (default value: the default locale of the system running Graylog) which both are optional.

Let’s assume we have another message field called transaction_timezone, which is sent by the application and contains the time zone ID the transaction was done in (hopefully no application in the world sends its data like this, though):

rule "function howto"
when
    has_field("transaction_date") && has_field("transaction_timezone")
then
    // the following date format assumes there's no time zone in the string
    let new_date = parse_date(
                        to_string($message.transaction_date),
                        "yyyy-MM-dd HH:mm:ss",
                        to_string($message.transaction_timezone)
                );
    set_field("transaction_year", new_date.year);
end

Now we’re passing the parse_date function its timezone parameter the string value of the message’s transaction_timezone field.

In this case we only have a single optional parameter, which makes it easy to simply omit it from the end of the function call. However, if there are multiple optional parameters, or if there are so many parameters that it gets difficult to keep track of which positions correspond to which parameters, you can also use the named parameter variant of function calls. In this mode the order of the parameters does not matter, but all required ones still need to be there.

In our case the alternative version of calling parse_date would look like this:

rule "function howto"
when
    has_field("transaction_date") && has_field("transaction_timezone")
then
    // the following date format assumes there's no time zone in the string
    let new_date = parse_date(
                        value: to_string($message.transaction_date),
                        pattern: "yyyy-MM-dd HH:mm:ss",
                        timezone: to_string($message.transaction_timezone)
                );
    set_field("transaction_year", new_date.year);
end

All parameters in Graylog’s processing functions, listed below, are named.

Function Index

The following list describes the built-in functions that ship with Graylog. Additional third party functions are available via plugins in the marketplace.

Built-in Functions
Name Description
debug Print the passed value as string in the Graylog log.
to_bool Converts the single parameter to a boolean value using its string value.
to_double Converts the first parameter to a double floating point value.
to_long Converts the first parameter to a long integer value.
to_string Converts the first parameter to its string representation.
to_url Converts a value to a valid URL using its string representation.
to_map Converts a value to a map.
is_null Checks whether a value is null.
is_not_null Checks whether a value is not null.
is_boolean Checks whether a value is a boolean value (true or false).
is_number Checks whether a value is a numeric value (of type long or double).
is_double Checks whether a value is a floating point value (of type double).
is_long Checks whether a value is an integer value (of type long).
is_string Checks whether a value is a string.
is_collection Checks whether a value is an iterable collection.
is_list Checks whether a value is an iterable list.
is_map Checks whether a value is a map.
is_date Checks whether a value is a date (of type DateTime).
is_period Checks whether a value is a time period (of type Period).
is_ip Checks whether a value is an IP address (IPv4 or IPv6).
is_json Checks whether a value is a parsed JSON tree.
is_url Checks whether a value is a parsed URL.
abbreviate Abbreviates a String using ellipses.
capitalize Capitalizes a String changing the first letter to title case.
uncapitalize Uncapitalizes a String changing the first letter to lower case.
uppercase Converts a String to upper case.
lowercase Converts a String to lower case.
swapcase Swaps the case of a String.
contains Checks if a string contains another string.
starts_with Checks if a string starts with a given prefix.
ends_with Checks if a string ends with a given suffix.
substring Returns a substring of value with the given start and end offsets.
concat Concatenates two strings.
split Split a string around matches of this pattern (Java syntax).
regex Match a regular expression against a string, with matcher groups.
grok Applies a Grok pattern to a string.
key_value Extracts key/value pairs from a string.
crc32 Returns the hex encoded CRC32 digest of the given string.
crc32c Returns the hex encoded CRC32C (RFC 3720, Section 12.1) digest of the given string.
md5 Returns the hex encoded MD5 digest of the given string.
murmur3_32 Returns the hex encoded MurmurHash3 (32-bit) digest of the given string.
murmur3_128 Returns the hex encoded MurmurHash3 (128-bit) digest of the given string.
sha1 Returns the hex encoded SHA1 digest of the given string.
sha256 Returns the hex encoded SHA256 digest of the given string.
sha512 Returns the hex encoded SHA512 digest of the given string.
parse_json Parse a string into a JSON tree.
select_jsonpath Selects one or more named JSON Path expressions from a JSON tree.
to_ip Converts the given string to an IP object.
cidr_match Checks whether the given IP matches a CIDR pattern.
from_input Checks whether the current message was received by the given input.
route_to_stream Assigns the current message to the specified stream.
remove_from_stream Removes the current message from the specified stream.
create_message Currently incomplete Creates a new message which will be evaluated by the entire processing pipeline.
clone_message Clones a message.
drop_message This currently processed message will be removed from the processing pipeline after the rule finishes.
has_field Checks whether the currently processed message contains the named field.
remove_field Removes the named field from the currently processed message.
set_field Sets the name field to the given value in the currently processed message.
set_fields Sets multiple fields to the given values in the currently processed message.
rename_field Rename a message field.
syslog_facility Converts a syslog facility number to its string representation.
syslog_level Converts a syslog level number to its string representation.
expand_syslog_priority Converts a syslog priority number to its level and facility.
expand_syslog_priority_as_string Converts a syslog priority number to its level and facility string representations.
now Returns the current date and time.
parse_date Parses a date and time from the given string, according to a strict pattern.
flex_parse_date Attempts to parse a date and time using the Natty date parser.
parse_unix_milliseconds Attempts to parse a UNIX millisecond timestamp (milliseconds since 1970-01-01T00:00:00.000Z).
format_date Formats a date and time according to a given formatter pattern.
to_date Converts a type to a date.
years Create a period with a specified number of years.
months Create a period with a specified number of months.
weeks Create a period with a specified number of weeks.
days Create a period with a specified number of days.
hours Create a period with a specified number of hours.
minutes Create a period with a specified number of minutes.
seconds Create a period with a specified number of seconds.
millis Create a period with a specified number of millis.
period Parses an ISO 8601 period from the specified string.
lookup Looks up a multi value in the named lookup table.
lookup_value Looks up a single value in the named lookup table.
debug

debug(value: any)

Print any passed value as string in the Graylog log.

Note

The debug message will only appear in the log of the Graylog node that was processing the message you are trying to debug.

Example:

// Print: "INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Dropped message from <source>"
let debug_message = concat("Dropped message from ", to_string($message.source));
debug(debug_message);
to_bool

to_bool(value: any)

Converts the single parameter to a boolean value using its string value.

to_double

to_double(value: any, [default: double])

Converts the first parameter to a double floating point value.

to_long

to_long(value: any, [default: long])

Converts the first parameter to a long integer value.

to_string

to_string(value: any, [default: string])

Converts the first parameter to its string representation.

to_url

to_url(url: any, [default: string])

Converts the given url to a valid URL.

to_map

to_map(value: any)

Converts the given map-like value to a valid map.

The to_map() function currently only supports converting a parsed JSON tree into a map so that it can be used together with set_fields.

Example:

let json = parse_json(to_string($message.json_payload));
let map = to_map(json);
set_fields(map);

See also:

is_null

is_null(value: any)

Checks if the given value is null.

Example:

// Check if the `src_addr` field is null (empty).
// If null, boolean true is returned. If not null, boolean false is returned.
is_null(src_addr)
is_not_null

is_not_null(value: any)

Checks if the given value is not null.

Example:

// Check if the `src_addr` field is not null.
// If not null, boolean true is returned. If null, boolean false is returned.
is_not_null(src_addr)
is_boolean

is_boolean(value: any)

Checks whether the given value is a boolean value (true or false).

is_number

is_number(value: any)

Checks whether the given value is a numeric value (of type long or double).

See also:

is_double

is_double(value: any)

Checks whether the given value is a floating point value (of type double).

See also:

is_long

is_long(value: any)

Checks whether the given value is an integer value (of type long).

See also:

is_string

is_string(value: any)

Checks whether the given value is a string.

See also:

is_collection

is_collection(value: any)

Checks whether the given value is an iterable collection.

is_list

is_list(value: any)

Checks whether the given value is an iterable list.

is_map

is_map(value: any)

Checks whether the given value is a map.

See also:

is_date

is_date(value: any)

Checks whether the given value is a date (of type DateTime).

See also:

is_period

is_period(value: any)

Checks whether the given value is a time period (of type Period).

See also:

is_ip

is_ip(value: any)

Checks whether the given value is an IP address (IPv4 or IPv6).

See also:

is_json

is_json(value: any)

Checks whether the given value is a parsed JSON tree.

See also:

is_url

is_url(value: any)

Checks whether the given value is a parsed URL.

See also:

abbreviate

abbreviate(value: string, width: long)

Abbreviates a String using ellipses, the width defines the maximum length of the resulting string.

capitalize

capitalize(value: string)

Capitalizes a String changing the first letter to title case.

uncapitalize

uncapitalize(value: string)

Uncapitalizes a String changing the first letter to lower case.

uppercase

uppercase(value: string, [locale: string])

Converts a String to upper case. The locale (IETF BCP 47 language tag) defaults to “en”.

lowercase

lowercase(value: string, [locale: string])

Converts a String to lower case. The locale (IETF BCP 47 language tag) defaults to “en”.

swapcase

swapcase(value: string)

Swaps the case of a String changing upper and title case to lower case, and lower case to upper case.

contains

contains(value: string, search: string, [ignore_case: boolean])

Checks if value contains search, optionally ignoring the case of the search pattern.

Example:

// Check if the `example.org` is in the `hostname` field. Ignore case.
contains(to_string($message.hostname), "example.org", true)
starts_with

starts_with(value: string, prefix: string, [ignore_case: boolean])

Checks if value starts with prefix, optionally ignoring the case of the string.

Example:

// Returns true
starts_with("Foobar Baz Quux", "foo", true);
// Returns false
starts_with("Foobar Baz Quux", "Quux");
ends_with

ends_with(value: string, suffix: string, [ignore_case: boolean])

Checks if value ends with suffix, optionally ignoring the case of the string.

Example:

// Returns true
starts_with("Foobar Baz Quux", "quux", true);
// Returns false
starts_with("Foobar Baz Quux", "Baz");
substring

substring(value: string, start: long, [end: long])

Returns a substring of value starting at the start offset (zero based indices), optionally ending at the end offset. Both offsets can be negative, indicating positions relative to the end of value.

concat

concat(first: string, second: string)

Returns a new string combining the text of first and second.

Note

The concat() function only concatenates two strings. If you want to build a string from more than two sub-strings, you’ll have to use concat() multiple times, see the example below.

Example:

// Build a message like:
// 'TCP connect from 88.99.35.172 to 192.168.1.10 Port 443'
let build_message_0 = concat(to_string($message.protocol), " connect from ");
let build_message_1 = concat(build_message_0, to_string($message.src_ip));
let build_message_2 = concat(build_message_1, " to ");
let build_message_3 = concat(build_message_2, to_string($message.dst_ip));
let build_message_4 = concat(build_message_3, " Port ");
let build_message_5 = concat(build_message_4, to_string($message.dst_port));
set_field("message", build_message_5);
split

split(pattern: string, value: string, [limit: int])

Split a value around matches of pattern. Use limit to indicate the number of times the pattern should be applied.

Note

Patterns have to be valid Java String literals, please ensure you escape any backslashes in your regular expressions!

regex

regex(pattern: string, value: string, [group_names: array[string])

Match the regular expression in pattern against value. Returns a match object, with the boolean property matches to indicate whether the regular expression matched and, if requested, the matching groups as groups. The groups can optionally be named using the group_names array. If not named, the groups names are strings starting with "0".

Note

Patterns have to be valid Java String literals, please ensure you escape any backslashes in your regular expressions!

grok

grok(pattern: string, value: string, [only_named_captures: boolean])

Applies the grok pattern grok to value. Returns a match object, containing a Map of field names and values. You can set only_named_captures to true to only return matches using named captures.

Tip

The result of executing the grok function can be passed as argument for set_fields to set the extracted fields into a message.

See also:

key_value
key_value(
  value: string,
  [delimiters: string],
  [kv_delimiters: string],
  [ignore_empty_values: boolean],
  [allow_dup_keys: boolean],
  [handle_dup_keys: string],
  [trim_key_chars: string],
  [trim_value_chars: string]
)

Extracts key-value pairs from the given value and returns them as a Map of field names and values. You can optionally specify:

delimiters
Characters used to separate pairs. We will use each character in the string, so you do not need to separate them. Default value: <whitespace>.
kv_delimiters
Characters used to separate keys from values. Again, there is no need to separate each character. Default value: =.
ignore_empty_values
Ignores keys containing empty values. Default value: true.
allow_dup_keys
Indicates if duplicated keys are allowed. Default value: true.
handle_dup_keys
How to handle duplicated keys (if allow_dup_keys is set). It can take the values take_first, which will only use the first value for the key; or take_last, which will only use the last value for the key. Setting this option to any other value will change the handling to concatenate, which will combine all values given to the key, separating them with the value set in this option. For example, setting handle_dup_keys: ",", would combine all values given to a key a, separating them with a comma, such as 1,2,foo. Default value: take_first.
trim_key_chars
Characters to trim (remove from the beginning and end) from keys. Default value: no trim.
trim_value_chars
Characters to trim (remove from the beginning and end) from values. Default value: no trim.

Tip

The result of executing the key_value function can be passed as argument for set_fields to set the extracted fields into a message.

See also:

crc32

crc32(value: string)

Creates the hex encoded CRC32 digest of the value.

crc32c

crc32c(value: string)

Creates the hex encoded CRC32C (RFC 3720, Section 12.1) digest of the value.

md5

md5(value: string)

Creates the hex encoded MD5 digest of the value.

murmur3_32

murmur3_32(value: string)

Creates the hex encoded MurmurHash3 (32-bit) digest of the value.

murmur3_128

murmur3_128(value: string)

Creates the hex encoded MurmurHash3 (128-bit) digest of the value.

sha1

sha1(value: string)

Creates the hex encoded SHA1 digest of the value.

sha256

sha256(value: string)

Creates the hex encoded SHA256 digest of the value.

sha512

sha512(value: string)

Creates the hex encoded SHA512 digest of the value.

parse_json

parse_json(value: string)

Parses the value string as JSON, returning the resulting JSON tree.

See also:

select_jsonpath

select_jsonpath(json: JsonNode, paths: Map<string, string>)

Evaluates the given paths against the json tree and returns the map of the resulting values.

See also:

to_ip

to_ip(ip: string)

Converts the given ip string to an IpAddress object.

See also:

cidr_match

cidr_match(cidr: string, ip: IpAddress)

Checks whether the given ip address object matches the cidr pattern.

See also:

from_input

from_input(id: string | name: string)

Checks whether the currently processed message was received on the given input. The input can be looked up by either specifying its name (the comparison ignores the case) or the id.

route_to_stream

route_to_stream(id: string | name: string, [message: Message], [remove_from_default: boolean])

Routes the message to the given stream. The stream can be looked up by either specifying its name or the id.

If message is omitted, this function uses the currently processed message.

This causes the message to be evaluated on the pipelines connected to that stream, unless the stream has already been processed for this message.

If remove_from_default is true, the message is also removed from the default stream “All messages”.

Example:

// Route the current processed message to a stream with ID `512bad1a535b43bd6f3f5e86` (preferred method)
route_to_stream("512bad1a535b43bd6f3f5e86");

// Route the current processed message to a stream named `Custom Stream`
route_to_stream("Custom Stream");
remove_from_stream

remove_from_stream(id: string | name: string, [message: Message])

Removes the message from the given stream. The stream can be looked up by either specifying its name or the id.

If message is omitted, this function uses the currently processed message.

If the message ends up being on no stream anymore, it is implicitly routed back to the default stream “All messages”. This ensures that you the message is not accidentally lost due to complex stream routing rules. If you want to discard the message entirely, use the drop_message function.

create_message

create_message([message: string], [source: string], [timestamp: DateTime])

Creates a new message with from the given parameters. If any of them is omitted, its value is taken from the corresponding fields of the currently processed message. If timestamp is omitted, the timestamp of the created message will be the timestamp at that moment.

clone_message

clone_message([message: Message])

Clones a message. If message is omitted, this function uses the currently processed message.

drop_message

drop_message(message: Message)

The processing pipeline will remove the given message after the rule is finished executing.

If message is omitted, this function uses the currently processed message.

This can be used to implement flexible blacklisting based on various conditions.

Example:

rule "drop messages over 16383 characters"
when
    has_field("message") AND
    regex(pattern: "^.{16383,}$", value: to_string($message.message)).matches == true
then
    drop_message();
    // added debug message to be notified about the dropped message
    debug( concat("dropped oversized message from ", to_string($message.source)));
end
has_field

has_field(field: string, [message: Message])

Checks whether the given message contains a field with the name field.

If message is omitted, this function uses the currently processed message.

remove_field

remove_field(field: string, [message: Message])

Removes the given field with the name field from the given message, unless the field is reserved.

If message is omitted, this function uses the currently processed message.

set_field

set_field(field: string, value: any, [prefix: string], [suffix: string], [message: Message])

Sets the given field named field to the new value. The field name must be valid, and specifically cannot include a . character. It is trimmed of leading and trailing whitespace. String values are trimmed of whitespace as well.

The optional prefix and suffix parameters specify which prefix or suffix should be added to the inserted field name.

If message is omitted, this function uses the currently processed message.

See also:

set_fields

set_fields(fields: Map<string, any>, [prefix: string], [suffix: string], [message: Message])

Sets all of the given name-value pairs in field in the given message. This is a convenience function acting like set_field. It can be helpful for using the result of a function like select_jsonpath or regex in the currently processed message especially when the key names are the result of a regular expression.

The optional prefix and suffix parameters specify which prefix or suffix should be added to the inserted field names.

If message is omitted, this function uses the currently processed message.

See also:

rename_field

rename_field(old_field: string, new_field: string, [message: Message])

Modifies the field name old_field to new_field in the given message, keeping the field value unchanged.

syslog_facility

syslog_facility(value: any)

Converts the syslog facility number in value to its string representation.

syslog_level

syslog_level(value: any)

Converts the syslog severity number in value to its string representation.

expand_syslog_priority

expand_syslog_priority(value: any)

Converts the syslog priority number in value to its numeric severity and facility values.

expand_syslog_priority_as_string

expand_syslog_priority_as_string(value: any)

Converts the syslog priority number in value to its severity and facility string representations.

now

now([timezone: string])

Returns the current date and time. Uses the default time zone UTC.

See also:

parse_date

parse_date(value: string, pattern: string, [locale: string], [timezone: string])

Parses the value into a date and time object, using the pattern. If no timezone is detected in the pattern, the optional timezone parameter is used as the assumed timezone. If omitted the timezone defaults to UTC.

The format used for the pattern parameter is identical to the pattern of the Joda-Time DateTimeFormat.

Symbol Meaning Presentation Examples
G era text AD
C century of era (>=0) number 20
Y year of era (>=0) year 1996
x weekyear year 1996
w week of weekyear number 27
e day of week number 2
E day of week text Tuesday; Tue
y year year 1996
D day of year number 189
M month of year month July; Jul; 07
d day of month number 10
a halfday of day text PM
K hour of halfday (0~11) number 0
h clockhour of halfday (1~12) number 12
H hour of day (0~23) number 0
k clockhour of day (1~24) number 24
m minute of hour number 30
s second of minute number 55
S fraction of second millis 978
z time zone text Pacific Standard Time; PST
Z time zone offset/id zone -0800; -08:00; America/Los_Angeles
' escape for text delimiter  
'' single quote literal

The format used for the locale parameter is a valid language tag according to IETF BCP 47 which can be parsed by the Locale#forLanguageTag(String) method.

Also see IANA Language Subtag Registry.

If no locale was specified, the locale of the system running Graylog (the default locale) is being used.

Examples:

Language Tag Description
en English
en-US English as used in the United States
de-CH German for Switzerland

See also:

flex_parse_date

flex_parse_date(value: string, [default: DateTime], [timezone: string])

Uses the Natty date parser to parse a date and time value. If no timezone is detected in the pattern, the optional timezone parameter is used as the assumed timezone. If omitted the timezone defaults to UTC.

In case the parser fails to detect a valid date and time the default date and time is being returned, otherwise the expression fails to evaluate and will be aborted.

See also:

parse_unix_milliseconds

parse_unix_milliseconds(value: long)

Attempts to parse a UNIX millisecond timestamp (milliseconds since 1970-01-01T00:00:00.000Z) into a proper DateTime object.

Example:

// 1519902000000 == 2018-03-01T12:00:00.000Z
let timestamp = parse_unix_milliseconds(1519902000000);
set_field("timestamp", timestamp);

See also:

format_date

format_date(value: DateTime, format: string, [timezone: string])

Returns the given date and time value formatted according to the format string. If no timezone is given, it defaults to UTC.

to_date

to_date(value: any, [timezone: string])

Converts value to a date. If no timezone is given, it defaults to UTC.

See also:

years

years(value: long)

Create a time period with value number of years.

See also:

months

months(value: long)

Create a time period with value number of months.

See also:

weeks

weeks(value: long)

Create a time period with value number of weeks.

See also:

days

days(value: long)

Create a time period with value number of days.

See also:

hours

hours(value: long)

Create a time period with value number of hours.

See also:

minutes

minutes(value: long)

Create a time period with value number of minutes.

See also:

seconds

seconds(value: long)

Create a time period with value number of seconds.

See also:

millis

millis(value: long)

Create a time period with value number of milliseconds.

See also:

period

period(value: string)

Parses an ISO 8601 time period from value.

See also:

lookup

lookup(lookup_table: string, key: any, [default: any])

Looks up a multi value in the named lookup table.

Example:

rule "dst_ip geoip lookup"
when
  has_field("dst_ip")
then
  let geo = lookup("geoip-lookup", to_string($message.dst_ip));
  set_field("dst_ip_geolocation", geo["coordinates"]);
  set_field("dst_ip_geo_country_code", geo["country"].iso_code);
  set_field("dst_ip_geo_country_name", geo["country"].names.en);
  set_field("dst_ip_geo_city_name", geo["city"].names.en);
end
lookup_value

lookup_value(lookup_table: string, key: any, [default: any])

Looks up a single value in the named lookup table.

Example:

// Lookup a value in lookup table "ip_lookup" where the key is the string representation of the src_addr field.
lookup_value("ip_lookup", to_string($message.src_addr));

Usage

Overview

Once you understand the concepts explained in Pipelines, Rules, and Stream connections, you’re ready to start creating your own processing pipelines. This page gives you the information you need to get started with the user interface.

Configuration

Configure the message processor

Before start using the processing pipelines you need to ensure the Pipeline Processor message processor is enabled and correctly configured. You can do so by going to the System -> Configurations page, and checking the configuration in the Message Processors Configuration section.

_images/pipelines_message_processor.png

On the Configurations page, you need to enable the Pipeline Processor message processor and, if you want your pipelines to have access to static fields set on inputs and/or fields set by extractors, set the Pipeline Processor after the Message Filter Chain.

Manage rules

You can create, edit, and delete your pipeline rules in the Manage rules page, under System -> Pipelines.

_images/pipelines_manage_rules.png

Clicking on Create Rule or Edit in one of the rules will open a page where you can write your own rule. The page lists available functions and their details to make the task a bit more manageable.

_images/pipelines_edit_rule.png

Managing pipelines

Once there are some rules in Graylog, you can create pipelines that use them to modify and enrich your messages.

To manage your pipelines, access Manage pipelines page under System -> Pipelines. This page is where you can create, edit, and delete pipelines.

_images/pipelines_manage_pipelines.png

In order to create or edit pipelines, and as explained in Pipelines, you need to add your rules to a stage, which has a certain priority. The Web interface will let you add rules to the default stage (priority 0), and to create new stages with potentially different priorities.

_images/pipelines_show_pipeline.png

A pipeline can have more than one stage, and when you create or edit a stage you need to select how to proceed to the next stage in the pipeline:

All rules on this stage match the message
This option will only consider further stages in the pipeline when all conditions in rules evaluated in this stage are true. This is equivalent to match all in the Pipelines section.
At least one of the rules on this stage matches the message
Selecting this option will continue to further stages in the pipeline when one or more of the conditions in rules evaluated in this stage are true. This is equivalent to match either in the Pipelines section.

Connect pipelines to streams

You can decide which streams are connected to a pipeline from the pipeline details page. Under System -> Pipelines, click on the title of the pipeline you want to connect to a stream, and then click on the Edit connections button.

_images/pipelines_manage_connections.png

You can assign many pipelines to the same stream, in which case all connected pipelines will process messages routed into that stream based upon the overall order of stage priorities.

_images/pipelines_edit_connections.png

Remember, as mentioned in the Stream connections documentation, the All messages stream is where all messages are initially routed, and is therefore a good place to apply pipelines applicable to all of your messages. Such pipelines might be responsible for stream routing, blacklisting, field manipulation, etc.

Simulate your changes

After performing some changes in a processing pipeline, you most likely want to see how they are applied to incoming messages. This is what the pipeline simulator is for.

Click the Simulate processing button under System -> Pipelines or in the pipeline details page to access the pipeline simulator.

_images/pipelines_simulation_1.png

In order to test the message processing you need to provide a raw message that will be routed into the stream you want to simulate. The raw message should use the same format Graylog will receive. For example: you can type a GELF message, in the same format your GELF library would send, in the Raw message field. Don’t forget to select the correct codec for the message you provide.

After specifying the message and codec, click Load message to start the simulation and display the results.

_images/pipelines_simulation_2.png

The simulation provides the following results:

Changes summary
Provides a summary of modified fields in the original message, as well as a list of added and dropped messages.
Results preview
Shows all fields in the processed message.
Simulation trace
Displays a trace of the processing, indicating which rules were evaluated and which were executed. It also includes a timeline, in microseconds, to allow you to see which rules and pipelines are taking up the most time during message processing.

Lookup Tables

Graylog 2.3 introduced the lookup tables feature. It allows you to lookup/map/translate message field values into new values and write them into new message fields or overwrite existing fields. A simple example is to use a static CSV file to map IP addresses to host names.

Components

The lookup table systems consists of four components.

  • Data adapters
  • Caches
  • Lookup tables
  • Lookup results

Data Adapters

Data adapters are used to do the actual lookup for a value. They might read from a CSV file, connect to a database or execute HTTP requests to receive the lookup result.

Data adapter implementations are pluggable and new ones can be added through plugins.

Caches

The caches are responsible for caching the lookup results to improve the lookup performance and/or to avoid overloading databases and APIs. They are separate entities to make it possible to reuse a cache implementation for different data adapters. That way, the data adapters do not have to care about caching and do not have to implement it on their own.

Cache implementations are pluggable and new ones can be added through plugins.

Important

The CSV file adapter reads the entire contents of the file into HEAP memory. Ensure that you size the HEAP accordingly.

Note

The CSV file adapter refreshes its contents within each check interval if the file was changed. If the cache was purged but the check interval has not elapsed, lookups might return expired values.

Lookup Tables

The lookup table component ties together a data adapter instance and a cache instance. It is needed to actually enable the usage of the lookup table in extractors, converters, pipeline functions and decorators.

Lookup Results

The lookup result is returned by a lookup table through the data adapter and can contain two types of data. A single value and a multi value.

The single value can be a string, number or boolean and will be used in extractors, converters, decorators and pipeline rules. In our CSV example to lookup host names for IP addresses, this would be the host name string.

A multi value is a map/dictionary-like data structure and can contain several different values. This is useful if the data adapter can provide multiple values for a key. A good example for this would be the geo-ip data adapter which does not only provide the latitude and longitude for an IP address, but also information about the city and country of the location. Currently, the multi value can only be used in a pipeline rule when using the lookup() pipeline function.

Example 1: Output for a CSV data adapter including a single value and a multi value.

_images/example-single-value.png

Example 2: Output for the geo-ip data adapter including a single value and a multi value.

_images/example-multi-value.png

Setup

The lookup tables can be configured on the “System/Lookup Tables” page.

You need to create at least one data adapter and one cache before you can create your first lookup table. The following example setup creates a lookup table with a CSV file data adapter and an in-memory cache.

Create Data Adapter

Navigate to “System/Lookup Tables” and click the “Data Adapters” button in the top right corner. Then you first have to select a data adapter type.

Every data adapter form includes data adapter specific documentation that helps you to configure it correctly.

_images/setup-data-adapter.png

Create Cache

Navigate to “System/Lookup Tables” and click the “Caches” button in the top right corner. Then you first have to select a cache type.

Every cache form includes cache specific documentation that helps you to configure it correctly.

_images/setup-cache.png

Create Lookup Table

Now you can create a lookup table with the newly created data adapter and cache by navigating to “System/Lookup Tables” and clicking “Create lookup table”.

Make sure to select the data adapter and cache instances in the creation form.

_images/setup-table.png
Default Values

Every lookup table can optionally be configured with default values which will be used if a lookup operation does not return any result.

_images/setup-table-defaults.png

Usage

Lookup tables can be used with the following Graylog components.

  • Extractors
  • Converters
  • Decorators
  • Pipeline rules

Extractors

A lookup table extractor can be used to lookup the value of a message field in a lookup table and write the result into a new field or overwrite an existing field.

_images/usage-extractor.png

Converters

When you use an extractor to get values out of a text message, you can use a lookup table converter to do a lookup on the extracted value.

_images/usage-converter.png

Decorators

A lookup table decorator can be used to enrich messages by looking up values at search time.

_images/usage-decorator.png

Pipeline Rules

There are two lookup functions that can be used in a pipeline rule, lookup() and lookup_value(). The first returns the multi value data of the lookup result, the second returns the single value.

_images/usage-pipeline-rule.png

Built-in Data Adapters

The following Data Adapters are shipped with Graylog by default. Detailed on-screen documentation for each is available on the Add/Edit Data Adapter page in Graylog.

CSV File Adapter

Performs key/value lookups from a CSV file.

DNS Lookup Adapter

Provides the ability to perform the following types of DNS resolutions:

  • Resolve hostname to IPv4 address (A records)
  • Resolve hostname to IPv6 address (AAAA records)
  • Resolve hostname to IPv4 and IPv6 address (A and AAAA records)
  • Reverse lookup (PTR record)
  • Text lookup (TXT records)

DSV File from HTTP Adapter

Performs key/value from a DSV file. This adapter supports more advanced customization than the CSV File adapter (such a custom delimiter and customizable key/value columns).

HTTP JSONPath Adapter

Executes HTTP GET requests to lookup a key and parses the result based on configured JSONPath expressions.

Geo IP - MaxMind Databases

Provides the ability to extract geolocation information of IP addresses from MaxMind Country and City databases.

Geolocation

Graylog lets you extract and visualize geolocation information from IP addresses in your logs. Here we will explain how to configure the geolocation resolution, and how to create a map with the extracted geo-information.

Setup

Graylog ships with geolocation capabilities by default but some configuration is still required on your side. This section explains how to configure the functionality in detail.

On Graylog 3.0, the preferred way of configuring geolocation is by using Lookup Tables, as it provides more flexibility and is compatible with more database types. If you would rather use the old Message Processor, please check the 2.5 documentation.

Note

Before you get started, we recommend taking a look at some Lookup Table concepts in the documentation.

Download the database

In the first place, you need to download a geolocation database. The Lookup Table Geo IP Data Adapter supports both MaxMind City and Country databases in the MaxMind DB format, as the GeoIP2 Databases or GeoLite2 Databases that MaxMind provides.

The next step is to store the geolocation database on all servers running Graylog. Make sure you grant the right permissions to the file so the user running Graylog can read the database.

Configure Lookup Table

The next step is to configure a Graylog Lookup Table that is able to use the geolocation database. Follow the Lookup Tables setup documentation to see what you need to do. In most common cases you need to:

  1. Create a Geo IP Data Adapter and point it to the location where you store the database. You can additionally test the Data Adapter to ensure it all works as expected.
  2. Create a Cache (if needed) to make your lookups faster.
  3. Create a Lookup Table that uses the Data Adapter and Cache you created in previous steps.

Use the Lookup Table

Now you are almost ready to extract geolocation information from IP addresses. All you need to do is to use the Lookup Table you created in the previous step in a Extractor, Converter, Decorator or Pipeline Rule. Take a look at the Lookup Tables usage documentation for more information.

Note

Make sure to read The importance of message processor ordering, specially if you will use the Lookup Table with a Pipeline, in order to better understand how Graylog will process messages.

Visualize geolocations in a map

Graylog can display maps from geolocation stored in any field, as long as the geo-points are using the latitude,longitude format. The default return value of the Geo IP Data Adapter returns the coordinates in the right format, so you most likely don’t need to do anything special if you are using a Lookup Table for extracting geolocation information.

Display a map in the search results page

On any search result page, you can expand the field you want to use to draw a map in the search sidebar, and click on the World Map link. That will show a map with all different points stored in that field.

_images/geolocation_7.png

Add map to a dashboard

You can add the map visualization into any dashboards as you do with other widgets. Once you displayed a map in the search result page, click on Add to dashboard, and select the dashboard where you want to add the map.

_images/geolocation_8.png _images/geolocation_9.png

FAQs

Will Graylog extract IPs from all fields?

No, you can configure which fields you want to extract data from in the Pipeline Rule or Extractor using the Lookup Table configured in the setup section.

What geo-information is extracted from IPs?

Depending on the database you use, the extracted information will be different. By using a Pipeline Rule alongside a Lookup Table, you can extract any information returned by the MaxMind Database for the IP in your logs.

Where is the extracted geo-information stored?

Extracted geo-information is stored in message fields, which you can name as you wish.

Which geo-points format does Graylog use to store coordinates?

Graylog returns the geolocation information in the latitude,longitude format. The Map visualization also requires that format to be able to draw the coordinates on a map.

I have a field in my messages with coordinates information already, can I use it in Graylog?

Yes, you can display a map for coordinates as long as they are in the latitude,longitude format.

Not all fields containing IP addresses are resolved. Why does this happen?

Most likely it is a misconfiguration issue. It is easier to extract information if IP addresses are in their own field. You should also make sure your Message Processors are in the right order in the Message Processors Configuration, as explained in The importance of message processor ordering.

Indexer failures

Every Graylog node is constantly keeping track about every indexing operation it performs. This is important for making sure that you are not silently losing any messages. The web interface can show you a number of write operations that failed and also a list of failed operations. Like any other information in the web interface this is also available via the REST APIs so you can hook it into your own monitoring systems.

_images/indexerfailures_1.png

Information about the indexing failure is stored in a capped MongoDB collection that is limited in size. A lot (many tens of thousands) of failure messages should fit in there but it should not be considered a complete collection of all errors ever thrown.

Common indexer failure reasons

There are some common failures that can occur under certain circumstances. Those are explained here:

MapperParsingException

An error message would look like this:

MapperParsingException[failed to parse [failure]]; nested: NumberFormatException[For input string: "some string value"];

You tried to write a string into a numeric field of the index. The indexer tried to convert it to a number, but failed because the string did contain characters that could not be converted.

This can be triggered by for example sending GELF messages with different field types or extractors trying to write strings without converting them to numeric values first. The recommended solution is to actively decide on field types. If you sent in a field like http_response_code with a numeric value then you should never change that type in the future.

The same can happen with all other field types like for example booleans.

Note that index cycling is something to keep in mind here. The first type written to a field per index wins. If the Graylog index cycles then the field types are starting from scratch for that index. If the first message written to that index has the http_response_code set as string then it will be a string until the index cycles the next time. Take a look at Index model for more information.

Users and Roles

Graylog has a granular permission system which secures the access to its features. Each interaction which can look at data or change configuration in Graylog must be performed as an authenticated user.

Each user can have varying levels of access to Graylog’s features, which can be controlled with assigning roles to users.

The following sections describe the capabilities of users and roles and also how to use LDAP for authentication.

Users

It is recommended to create an account for each individual user accessing Graylog.

User accounts have the usual properties such as a login name, email address, full name, password etc. In addition to these fields, you can also configure the session timeout, roles and timezone.

_images/create_user.png

Sessions

Each login for a user creates a session, which is bound to the browser the user is currently using. Whenever the user interacts with Graylog this session is extended.

For security reasons you will want to have Graylog expire sessions after a certain period of inactivity. Once the interval specified by timeout expires the user will be logged out of the system. Requests like displaying throughput statistics do not extend the session, which means that if the user keeps Graylog open in a browser tab, but does not interact with it, their session will expire as if the browser was closed.

Logging out explicitly terminates the session.

Timezone

Since Graylog internally processes and stores messages in the UTC timezone, it is important to set the correct timezone for each user.

Even though the system defaults are often enough to display correct times, in case your team is spread across different timezones, each user can be assigned and change their respective timezone setting. You can find the current timezone settings for the various components on the System / Overview page of your Graylog web interface.

Initial Roles

Each user needs to be assigned at least one role, which governs the basic set of permissions this user has in Graylog.

Normal users, which do not need to create inputs, outputs or perform administrative tasks like managing access control etc, should be assigned the built in Reader role in addition to the custom roles which grant access to streams and dashboards.

Roles

In Graylog, roles are named collections of individual permissions which can be assigned to users. Previous Graylog versions could only assign individual permissions to each user in the system, making updating stream or dashboard permissions for a large group of users difficult to deal with.

Starting with Graylog 1.2 you can create roles which bundle permissions to streams and dashboards. These roles can then be assigned to any number of users and later be updated to include new streams and dashboards.

_images/roles.png

The two roles Admin and Reader are built in and cannot be changed. The Admin role grants all permissions and should only be assigned to users operating Graylog. The Reader role grants the basic permissions every user needs to be able to use Graylog. The interface will ensure that every user at least has the Reader role in addition to more business specific roles you create.

Roles cannot be deleted as long as users are still assigned to them to prevent accidentally locking users out.