How Do I Configure Azure Databricks with Log4J2
Log4J: Warn Failed to Set Property [Rollingpolicy] to Value "Org. Apache. Log4J. Rolling. Timebasedrollingpolicy". Log4J: Warn Please Set a Rolling Policy for...
log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.TimeBasedRollingPolicy". log4j:WARN Please set a rolling policy for the DatabricksRollingFileAppender named 'publicFile' log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.TimeBasedRollingPolicy". log4j:WARN Please set a rolling policy for the DatabricksRollingFileAppender named 'privateFile' log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.TimeBasedRollingPolicy". log4j:WARN Please set a rolling policy for the DatabricksRollingFileAppender named 'product' log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.TimeBasedRollingPolicy". log4j:WARN Please set a rolling policy for the DatabricksRollingFileAppender named 'metrics' log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.TimeBasedRollingPolicy". log4j:WARN Please set a rolling policy for the DatabricksRollingFileAppender named 'usage
2 Answers
As of Datatbricks runtime 11.0 with Spark 3.3.0, log4j has been upgraded to log4j2. I haven't seen official documentation for custom log support but this is what's been working for me. Similar with log4j 1 support, you need a custom init script to run on the cluster and create a log4j2.properties file. This script might look something like:
#! /bin/bash
set -euxo pipefail
echo "Running on the driver? ${DB_IS_DRIVER}"
echo "Driver ip: ${DB_DRIVER_IP}"
cat >>/databricks/spark/dbconf/log4j/driver/log4j2.properties <<EOL
appender.customFile.type = RollingFile
appender.customFile.name = customFile
appender.customFile.layout.type = PatternLayout
appender.customFile.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
appender.customFile.filePattern = logs/log4j.custom.%d{yyyy-MM-dd-HH}.log.gz
appender.customFile.policies.type = Policies
appender.customFile.policies.time.type = TimeBasedTriggeringPolicy
appender.customFile.policies.time.interval = 1
appender.customFile.fileName = logs/stdout.custom-active.log
logger.custom=DEBUG, customFile
logger.custom.name = com.custom
logger.custom.additivity = true
EOL
Chloe Bennett explores the intersection of pop culture, streaming entertainment, digital trends, and contemporary lifestyle. Her weekly commentary reaches thousands of culture enthusiasts.
