Event Processor [Deprecated]

The following section is deprecated and only applicable to SIEMonster versions below 5.0.

This module can be found under the Log processors module after selecting modules in the shortcut bar in the new unified interface.

Apache NiFi is an open-source data ingestion platform that was built to automate the flow of data between systems (For example transfer a JSON document and add that in a database, transfer all the FTP files directly to Hadoop, transfer data from Apache Kafka to Elasticsearch). Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic.

It was developed by the National Security Agency (NSA) and is now being maintained and further development is supported by Apache foundation. It is based on Java and runs in Jetty server. NiFi Supports any device which runs Java and you can easily install NiFi on AWS. NiFi is used in varied industries such as healthcare, insurance, telecom, manufacturing, finance, oil and gas among others. As a best practice organize your projects into three parts, ingestion, test and monitoring.

Apache NiFi is now used in many top organizations that want to harness the power of their fast data by sourcing and transferring information from and to their database and big data lakes. It is a key tool to learn for the analyst and data scientists alike.

Apache NiFi has an easy to use drag and drop user interface, and it focuses on the configuration of the processors. It guarantees that you do not lose your data through its guaranteed delivery feature.

Some of the high-level capabilities and objectives of Apache NiFi include:

• Web-based user interface

  • Seamless experience between design, control, feedback, and monitoring

• Highly configurable

  • Loss tolerant vs guaranteed delivery

  • Low latency vs high throughput

  • Dynamic prioritization

  • Flow can be modified at runtime

  • Back pressure

• Data Provenance

  • Track dataflow from beginning to end

• Designed for extension

  • Build your own processors and more

  • Enables rapid development and effective testing

• Secure

  • SSL, SSH, HTTPS, encrypted content, etc...

  • Multi-tenant authorization and internal authorization/policy management

Apache NiFi is good at:

• Reliable and secure transfer of data between systems

• Delivery of data from sources to analytic platforms

• Enrichment and preparation of data:

  • Conversation between formats

  • Extraction/parsing

  • Routing decisions

What Apache NiFi should not be used for:

• Distributed computation

• Complex event processing

• Joins, rolling windows, aggregate operations