This module can be found under the Log processors module after selecting modules in the shortcut bar in the new unified interface.
Apache NiFi is an open-source data ingestion platform that was built to automate the flow of data between systems (For example transfer a JSON document and add that in a database, transfer all the FTP files directly to Hadoop, transfer data from Apache Kafka to Elasticsearch). Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
It was developed by the National Security Agency (NSA) and is now being maintained and further development is supported by Apache foundation. It is based on Java and runs in Jetty server. NiFi Supports any device which runs Java and you can easily install NiFi on AWS. NiFi is used in varied industries such as healthcare, insurance, telecom, manufacturing, finance, oil and gas among others. As a best practice organize your projects into three parts, ingestion, test and monitoring.
Apache NiFi is now used in many top organizations that want to harness the power of their fast data by sourcing and transferring information from and to their database and big data lakes. It is a key tool to learn for the analyst and data scientists alike.
Apache NiFi has an easy to use drag and drop user interface, and it focuses on the configuration of the processors. It guarantees that you do not lose your data through its guaranteed delivery feature.
Some of the high-level capabilities and objectives of Apache NiFi include:
Web-based user interface
Seamless experience between design, control, feedback, and monitoring
Loss tolerant vs guaranteed delivery
Low latency vs high throughput
Flow can be modified at runtime
Track dataflow from beginning to end
Designed for extension
Build your own processors and more
Enables rapid development and effective testing
SSL, SSH, HTTPS, encrypted content, etc...
Multi-tenant authorization and internal authorization/policy management
Apache NiFi is good at:
Reliable and secure transfer of data between systems
Delivery of data from sources to analytic platforms
Enrichment and preparation of data:
Conversation between formats
What Apache NiFi should not be used for:
Complex event processing
Joins, rolling windows, aggregate operations