What Is a Flume Job

Apache Flume is a tool/service/data ingestion mechanism for collecting aggregating and transporting large amounts of streaming data such as log files, events (etc…) from various sources to a centralized data store. … It is principally designed to copy streaming data (log data) from various web servers to HDFS.

What is flume used for *?

Flume. Apache Flume. Apache Flume is an open-source, powerful, reliable and flexible system used to collect, aggregate and move large amounts of unstructured data from multiple data sources into HDFS/Hbase (for example) in a distributed fashion via it’s strong coupling with the Hadoop cluster.

How do I start flume agent?

1. To start Flume directly, run the following command on the Flume host: /usr/hdp/current/flume-server/bin/flume-ng agent -c /etc/flume/conf -f /etc/flume/conf/ flume.conf -n agent.
2. To start Flume as a service, run the following command on the Flume host: service flume-agent start.