基于kafka的flink1.13.2的log4j日志系统（免费版）

伐树熊的数仓建设大数据实战 2023-05-09 12:59:19

日志管理规划设计

编辑flink的log4j.properties文件

status = DEBUG
property.CONSOLE_LOG_LEVEL=INFO

property.OUTPUT_LOG_LEVEL=INFO
property.EVERY_FILE_SIZE=500M
property.FILE_MAX=10

appender.console.type = Console
appender.console.name = STDOUT
appender.console.filter.threshold.level = DEBUG
appender.console.filter.threshold.type = ThresholdFilter
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %5p %t %c{1}:%L - %m%n

kafka appender config
appender.kafka.type=Kafka
appender.kafka.name=Kafka
appender.kafka.syncSend=false
appender.kafka.ignoreExceptions=false
appender.kafka.topic=elk_log_info
appender.kafka.property.type=Property
appender.kafka.property.name=bootstrap.servers
appender.kafka.property.value=host1:9092,host2:9092,host3:9092
appender.kafka.layout.type=JSONLayout
apender.kafka.layout.value=net.logstash.log4j.JSONEventLayoutV1
appender.kafka.layout.compact=true
appender.kafka.layout.complete=false
appender.kafka.layout.additionalField1.type=KeyValuePair
appender.kafka.layout.additionalField1.key=logdir
appender.kafka.layout.additionalField1.value=${sys:log.file}
appender.kafka.layout.additionalField2.type=KeyValuePair
appender.kafka.layout.additionalField2.key=flink_job_name
appender.kafka.layout.additionalField2.value=${sys:flink_job_name}
appender.kafka.layout.additionalField3.type=KeyValuePair
appender.kafka.layout.additionalField3.key=yarnContainerId
appender.kafka.layout.additionalField3.value=${sys:yarnContainerId}

appender.kafka.layout.type=PatternLayout
appender.kafka.layout.pattern={"log_level":"%p","log_timestamp":"%d{ISO8601}","log_thread":"%t","log_file":"%F", "log_line":"%L","log_message":"'%m'","log_path":"%X{log_path}","job_name":"${sys:flink_job_name}"}%n
############################################################################################
# Allows this configuration to be modified at runtime. The file will be checked every 30 seconds.
monitorInterval=30
# Uncomment this if you want to _only_ change Flink's logging
logger.flink.name = org.apache.flink
logger.flink.level = INFO

logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO
logger.shaded_zookeeper.name = org.apache.flink.shaded.zookeeper3
logger.shaded_zookeeper.level = INFO
# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF
############################################################################################
rootLogger.level = INFO
rootLogger.appenderRef.stdout.ref = STDOUT
rootLogger.appenderRef.kafka.ref = Kafka

创建kafka日志主题（自行创建）

主题：elk_log_info

搭建ELK环境（自行搭建）

编辑logstash.conf文件

input {
  kafka {
    bootstrap_servers => ["host1:9092,host2:9092,host3:9092"]
    group_id => "elk-consumer"
    auto_offset_reset => "latest"
    consumer_threads => 3
    decorate_events => true
    topics => ["elk_log_info"]
    type => "zc"
    codec => "json"
    max_partition_fetch_bytes => "5242940"
  }
}
output {
  elasticsearch {
    hosts => "es:9200"
    index => "zc-logstash-%{+YYYY.MM.dd}"
  }
}

编辑flink-config.yaml文件

追加：

env.java.opts:  -Dlog4j2.formatMsgNoLookups=true
env.java.opts.taskmanager: -DyarnContainerId=$CONTAINER_ID
env.java.opts.jobmanager: -DyarnContainerId=$CONTAINER_ID

任务启动追加job名称

追加参数：env.java.opts=“-Dflink_job_name=DemoTask”
示例：

flink run -m yarn-cluster -d  -yjm 1024 -ytm 1048 -c cn.gwm.flink.streaming.task.DemoTask -yD env.java.opts="-Dflink_job_name=DemoTask"  -ynm  demoTask  /vmdata/flink/demoTask-lxy.jar

...全文