Self Host 使用 Exceptionless 实时监控程序运行日志服务
2017-10-14

Exceptionless 是一个可以对 ASP.NET Core, ASP.NET MVC,WebAPI, WebForms, WPF, Console 应用提供系统的日志,错误监控、报表等服务实时日志收集框架 。具体介绍:免费开源分布式系统日志收集框架 Exceptionless

image

日志分析与程序异常在系统开发与运行中占非常重要的地位,在技术选型的过程中需要考虑一些问题

  • 数据分散在多台服务器中,难以查找
  • 数据量大,查询速度慢
  • 涉及多个系统调用,难以快速定位数据问题

开源日志项目
https://github.com/Graylog2/graylog2-server
https://github.com/exceptionless/Exceptionless
https://github.com/elastic
https://github.com/getsentry/sentry
https://github.com/dianping/cat

配置 Elasticsearch 集群

最终我们还是选用个了 Exceptionless ,于是就自己搭建了一套本地环境(Elasticsearch 配置官方推荐:three node  with two master nodes on Linux)。

image

#配置JAVA运行环境
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install software-properties-common htop
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
java -version

# centos
yum install java(openjdk)
tar -zxvf jdk-8u131-linux-x64.tar.gz
#配置环境变量

vi /etc/profile
JAVA_HOME=/opt/jdk1.8.0_131
CLASSPATH=.:$JAVA_HOME/lib
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME CLASSPATH PATH

 #生效环境变量

source /etc/profile

#安装elasticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.tar.gz
tar -xzf elasticsearch-5.5.0.tar.gz

#安装插件
sudo bin/elasticsearch-plugin install mapper-size
#配置elasticsearch-head
Running with docker
https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/
for Elasticsearch 5.x open http://localhost:9100/: 
docker run -p 9100:9100 mobz/elasticsearch-head:5

#Enable CORS in elasticsearch
http.cors.enabled: true
http.cors.allow-origin: "*"

#创建elastic用户
useradd elastic (userdel elastic)
#修改所有者
chown -R elastic /opt/elasticsearch-5.5.0

#切换到elastic用户启动 
su elastic
#以后台方式启动
./bin/elasticsearch –d

Elasticsearch 配置


# ======================== Elasticsearch Configuration =========================#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html#
# ---------------------------------- Cluster -----------------------------------#
# Use a descriptive name for your cluster:

# multi node configuration
cluster.name: exceptionless
action.destructive_requires_name: trueaction.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*#bootstrap.memory_lock: true#
# ------------------------------------ Node ------------------------------------#
# Use a descriptive name for the node:
#
node.name: p2_es_node_03

#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0#
# Set a custom port for HTTP:
#
http.port: 9200#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true# 启用跨域
#Enable CORS in elasticsearch
http.cors.enabled: truehttp.cors.allow-origin: "*"# 集群配置
# 这个配置限制了单机上可以开启的ES存储实例的个数,当我们需要单机多实例,则需要把这个配置赋值2,或者更高。
node.max_local_storage_nodes: 1#设置这个集群中节点的数量,默认为2,一旦这N个节点启动,就会立即进行数据恢复。
gateway.expected_nodes: 3#设置这个参数来保证集群中的节点可以知道其它N个有master资格的节点。默认为1,对于大的集群来说,可以设置大一点的值(2-4)
discovery.zen.minimum_master_nodes: 2#设置集群中master节点的初始列表,可以通过这些节点来自动发现新加入集群的节点。
discovery.zen.ping.unicast.hosts: ["10.255.131.162","10.255.131.163","10.255.131.164"]
#network.bind_host: [ '_site_', '_local_' ]
#network.publish_host: '_site_'

JVM 配置


## JVM configuration

################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html## for more information
##
################################################################

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space-Xms4g-Xmx4g

################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless## you understand what you are doing
##
################################################################

## GC configuration-XX:+UseConcMarkSweepGC-XX:CMSInitiatingOccupancyFraction=75-XX:+UseCMSInitiatingOccupancyOnly

## optimizations

# disable calls to System#gc-XX:+DisableExplicitGC

# pre-touch memory pages used by the JVM during initialization-XX:+AlwaysPreTouch

## basic

# force the server VM (remove on 32-bit client JVMs)-server

# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)-Xss1m

# set to headless, just in case-Djava.awt.headless=true# ensure UTF-8 encoding by default (e.g. filenames)-Dfile.encoding=UTF-8# use our provided JNA always versus the system one-Djna.nosys=true# use old-style file permissions on JDK9-Djdk.io.permissionsUseCanonicalPath=true# flags to configure Netty-Dio.netty.noUnsafe=true-Dio.netty.noKeySetOptimization=true-Dio.netty.recycler.maxCapacityPerThread=0# log4j 2-Dlog4j.shutdownHookEnabled=false-Dlog4j2.disable.jmx=true-Dlog4j.skipJansi=true## heap dumps

# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps
# ensure the directory exists and has sufficient space
#-XX:HeapDumpPath=${heap.dump.path}

## GC logging

#-XX:+PrintGCDetails
#-XX:+PrintGCTimeStamps
#-XX:+PrintGCDateStamps
#-XX:+PrintClassHistogram
#-XX:+PrintTenuringDistribution
#-XX:+PrintGCApplicationStoppedTime

# log GC status to a file with time stamps
# ensure the directory exists
#-Xloggc:${loggc}

# By default, the GC log file will not rotate.
# By uncommenting the lines below, the GC log file# will be rotated every 128MB at most 32 times.
#-XX:+UseGCLogFileRotation
#-XX:NumberOfGCLogFiles=32#-XX:GCLogFileSize=128M

# Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON.
# If documents were already indexed with unquoted fields in a previous version
# of Elasticsearch, some operations may throw errors.
#
# WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided
# only for migration purposes.
#-Delasticsearch.json.allow_unquoted_field_names=true

运行日志


LOG:
elastic@iZuf60cj5pna5im3va46nlZ:/opt/elasticsearch-5.5.0$ lsbin  config  lib  LICENSE.txt  modules  NOTICE.txt  plugins  README.textile
elastic@iZuf60cj5pna5im3va46nlZ:/opt/elasticsearch-5.5.0$ ./bin/elasticsearch
[2017-07-12T14:57:36,651][INFO ][o.e.n.Node               ] [] initializing ...
[2017-07-12T14:57:36,721][INFO ][o.e.e.NodeEnvironment    ] [9rZOxk3] using [1] data paths, mounts [[/ (/dev/vda1)]], net usable_space [230.6gb], net total_space [245.9gb], spins? [possibly], types [ext4]
[2017-07-12T14:57:36,722][INFO ][o.e.e.NodeEnvironment    ] [9rZOxk3] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-07-12T14:57:36,723][INFO ][o.e.n.Node               ] node name [9rZOxk3] derived from node ID [9rZOxk3hS0Wmj6SPvU0ZSg]; set [node.name] to override
[2017-07-12T14:57:36,723][INFO ][o.e.n.Node               ] version[5.5.0], pid[27006], build[260387d/2017-06-30T23:16:05.735Z], OS[Linux/4.4.0-63-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11]
[2017-07-12T14:57:36,723][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/elasticsearch-5.5.0]
[2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService     ] [9rZOxk3] loaded module [aggs-matrix-stats]
[2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService     ] [9rZOxk3] loaded module [ingest-common]
[2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService     ] [9rZOxk3] loaded module [lang-expression]
[2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService     ] [9rZOxk3] loaded module [lang-groovy]
[2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService     ] [9rZOxk3] loaded module [lang-mustache]
[2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService     ] [9rZOxk3] loaded module [lang-painless]
[2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService     ] [9rZOxk3] loaded module [parent-join]
[2017-07-12T14:57:37,514][INFO ][o.e.p.PluginsService     ] [9rZOxk3] loaded module [percolator]
[2017-07-12T14:57:37,514][INFO ][o.e.p.PluginsService     ] [9rZOxk3] loaded module [reindex]
[2017-07-12T14:57:37,514][INFO ][o.e.p.PluginsService     ] [9rZOxk3] loaded module [transport-netty3]
[2017-07-12T14:57:37,514][INFO ][o.e.p.PluginsService     ] [9rZOxk3] loaded module [transport-netty4]
[2017-07-12T14:57:37,514][INFO ][o.e.p.PluginsService     ] [9rZOxk3] loaded plugin [mapper-size]
[2017-07-12T14:57:38,768][INFO ][o.e.d.DiscoveryModule    ] [9rZOxk3] using discovery type [zen]
[2017-07-12T14:57:39,323][INFO ][o.e.n.Node               ] initialized
[2017-07-12T14:57:39,324][INFO ][o.e.n.Node               ] [9rZOxk3] starting ...
[2017-07-12T14:57:39,463][INFO ][o.e.t.TransportService   ] [9rZOxk3] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2017-07-12T14:57:39,475][WARN ][o.e.b.BootstrapChecks    ] [9rZOxk3] max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]
[2017-07-12T14:57:39,476][WARN ][o.e.b.BootstrapChecks    ] [9rZOxk3] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2017-07-12T14:57:42,521][INFO ][o.e.c.s.ClusterService   ] [9rZOxk3] new_master {9rZOxk3}{9rZOxk3hS0Wmj6SPvU0ZSg}{Khs76coWR6qWwC9DjuhyeQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-07-12T14:57:42,543][INFO ][o.e.h.n.Netty4HttpServerTransport] [9rZOxk3] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2017-07-12T14:57:42,543][INFO ][o.e.n.Node               ] [9rZOxk3] started
[2017-07-12T14:57:42,555][INFO ][o.e.g.GatewayService     ] [9rZOxk3] recovered [0] indices into cluster_state

 

问题一:警告提示

[2016-11-06T16:27:21,712][WARN ][o.e.b.JNANatives ] unable to install syscall filter:

Java.lang.UnsupportedOperationException: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in
at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:349) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:630) ~[elasticsearch-5.0.0.jar:5.0.0]

报了一大串错误,其实只是一个警告。

解决:使用新的linux版本,就不会出现此类问题了。

问题二:ERROR: bootstrap checks failed

max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]
max number of threads [1024] for user [lishang] likely too low, increase to at least [2048]

vi /etc/security/limits.conf
添加如下内容:

* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096

问题三:max number of threads [1024] for user [lish] likely too low, increase to at least [2048]

vi /etc/security/limits.d/90-nproc.conf
#修改为
* soft nproc 2048

问题四:max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

解决:切换到root用户修改配置sysctl.conf

vi /etc/sysctl.conf

添加下面配置:
vm.max_map_count=655360
并执行命令:
sysctl -p
然后,重新启动elasticsearch,即可启动成功。

#安装kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.4.0-linux-x86_64.tar.gz
tar -xzf kibana-5.4.0-linux-x86_64.tar.gz
cd kibana/

#查看运行进程
ps -ef |grep java
#kill 进程
kill -9 pid

配置 Exceptionless

  • 配置项目文件权限为users
  • 修改 app.config.*.js 的 BASE_URL 为本机访问IP
  • 修改 Web.config 配置

<configuration>
  <connectionStrings>
    <add name="RedisConnectionString" connectionString="" />
    <add name="ElasticSearchConnectionString" connectionString="http://10.255.131.162:9200" />
    <add name="LdapConnectionString" connectionString="" />
  </connectionStrings>
  <appSettings>
    <!-- Base url for the ui used to build links in emails and other places. -->
    <add key="BaseURL" value="http://10.255.130.67/#" />
    <!-- Controls whether SSL is required. Only enable this if you have SSL configured. -->
    <add key="EnableSSL" value="false" />
    <!--
    Dev: Use this mode when debugging. (Outbound emails will not be sent)
    QA: Use this mode when deployed to staging. (Outbound emails restricted)
    Production: Use this mode when deployed to production.    -->
    <add key="WebsiteMode" value="Production" />
    <!-- Controls whether users can signup. -->
    <add key="EnableAccountCreation" value="false" />
    <!-- Controls whether daily summary emails are sent -->
    <add key="EnableDailySummary" value="true" />
    <!--
      Email Client Settings (Uncomment the section below to change the default email settings)
      There are three valid SmtpEncryption settings: None, SSL and StartTLS    -->
    <!-- Email -->
    <add key="SmtpHost" value="smtp.exmail.qq.com" />
    <add key="SmtpPort" value="465" />
    <add key="SmtpEncryption" value="SSL" />
    <add key="SmtpFrom" value="test@test.com" />
    <add key="SmtpUser" value="test@test.com" />
    <add key="SmtpPassword" value="!test." />

    <!-- Folder used to store event post data -->
    <add key="StorageFolder" value="|DataDirectory|\storage" />
    <!-- Runs the jobs in the current website process -->
    <add key="RunJobsInProcess" value="true" />
    <add key="aspnet:UseTaskFriendlySynchronizationContext" value="true" />
    <!--保留日志天数 -->
    <add key="MaximumRetentionDays" value="90" />
    <!--分片数量-->
    <add key="ElasticSearchNumberOfShards" value="3" /> 
    <!--复制分片数量-->
    <add key="ElasticSearchNumberOfReplicas" value="1" />
  </appSettings>

 首次访问注册用户后创建项目即可(EnableAccountCreation 需要配置成 true)

检查服务状态
http://ip/api/v2/status 
开放API
http://api.exceptionless.com/docs/index
客户端调用示列
https://github.com/exceptionless/Exceptionless.Net/tree/master/samples

QA:
本地部署使用哪个版本?
使用最新的版本即可
Elasticsearch(v5.5.0)
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.tar.gz
Exceptionless(v4.0.2)
https://github.com/exceptionless/Exceptionless/releases

大量记录日志会影响性能吗?
支持 python 或者其他语言收集日志吗?
默认可以使用相关的扩展来实现写入到ES 
https://github.com/ReactiveMarkets/NLog.Targets.ElasticSearch 
https://github.com/jptoto/log4net.ElasticSearch
也可以使用 Exceptionless 的扩展
https://github.com/serilog/serilog-sinks-exceptionless
https://github.com/exceptionless/Exceptionless.Net/tree/master/src/Platforms
如NLLog:


<nlog>

  <extensions>

    <add assembly="Exceptionless.NLog"/>

  </extensions>  

  <targets async="true">

    <target name="exceptionless" apiKey="API_KEY_HERE" xsi:type="Exceptionless">

      <field name="host" layout="${machinename}" />

      <field name="identity" layout="${identity}" />

      <field name="windows-identity" layout="${windows-identity:userName=True:domain=False}" />

      <field name="process" layout="${processname}" />

    </target>

  </targets>

  <rules>

    <logger name="*" minlevel="Trace" writeTo="exceptionless" />

  </rules>

</nlog>


Elasticsearch 主分片与副本数量设定?

修改分片数
curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
  "index.number_of_replicas" : "1",
  "index.number_of_shards" : "5"
}'
Exceptionless 配置 QQ 邮箱发送不了邮件?

配置 <add key="SmtpFrom" value="test@test.com" /> 节点 
怎么配置保留日志的天数?
配置 MaximumRetentionDays 属性即可(https://github.com/exceptionless/Exceptionless/issues/139)
每秒产生大量的日志请求 ,会有性能问题吗?
可以使用使用内存存储(UseInMemoryStorage),相对会稳定很多; Exceptionless 前端做负载(需要配置Redis)
ExceptionlessClient.Default.Configuration.UseInMemoryStorage();
怎么标识是哪个用户操作发生的事件?
https://github.com/exceptionless/Exceptionless/wiki/User-Sessions
用户标识访问者归属地的 DB 下载失败?
怎么样快速查询日志?
怎么样监控日志服务是否正常?
Redis服务是否是必须配置的?
开放了哪些 API 接口 ?
http://api.exceptionless.com/docs/index
怎么分配 Elasticsearch 中的 JVM 的内存大小?
Exceptionless 日志没有实时显示?
检查 IIS 是否配置了 websocket 协议 ; 检查 SignalR 是否连接成功
ws://xx.xx.xxx.xxx/api/v2/push/connect?transport=webSockets&clientProtocol=1.5&access_token=hkm9YhSalvakZWkJmI3ek2M7rB8WTeuANmy0kp54&connectionToken=YQdrdiaOnprcRq%2BsaMBKfdpRwLnUtc5NPZ0ZExJVfb3h4j5SrHkUDqQOOq8NAgzGx7YADZ3azpJ3JIFFt217sSlE7pJf%2B7848sxFTfzXg%2FmAHvvcHjipHirL0Xl3FvfqaQaqo50Tyo5m77GvBM1YrQ%3D%3D&tid=8

REFER:
https://github.com/medcl/elasticsearch-analysis-ik

原文地址 : http://www.cnblogs.com/Irving/p/6912684.html