Configure ELK,Kibana,Filebeat to collect and analyse netflow data on ubuntu 20.04

[muruganandan]

Hi Team,

I have configured ELK,Kibana and filebeat to collect netflow data from the router. Somehow I managed to collect the data successfully and Kibana started to show netflow data. But after then I have added to few lines in the filebeat.yml file to give depth analyses of netflow. After adding that lines the filebeat started to throw errors even after I removed that lines.
Config for the netflow in the filebeat, Highlighted in block are the config codes have been added.
#
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ['^DBG']
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ['^ERR', '^WARN']

- type: netflow
max_message_size: 10KiB
host: "0.0.0.0:2055"
protocols: [ v5, v9, ipfix ]
expiration_timeout: 30m
queue_size: 8192

# This requires a Kibana endpoint configuration.
setup.kibana:
host: http://localhost:5601

# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:

output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]

# Protocol - either `http` (default) or `https`.
protocol: "http"

# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"

==================
error

2021-06-02T12:36:50.155+0530 ERROR instance/beat.go:971 Exiting: No outputs are defined. Please define one under the output section.
Exiting: No outputs are defined. Please define one under the output section.
===============

Please give your suggestion help to get of this error. What ever the line added or removed no matter only output it gives is
No outputs are defined. Please define one under the output section.


Thanks
Muruganandan.C

 

[dos2unix]

I have configured ELK,Kibana and filebeat to collect netflow data from the router.

You have packetbeat or filebeat installed on the router?

The "L" in ELK typically means logstash, I don't see logstash anywhere here.
But you can go straight to ES. Make sure your firewall ports are open.

Which version of everything are you on?
My netflow module config looks more like this.

- module: netflow
log:
enabled: true
var:
netflow_host: 0.0.0.0
netflow_port: 2055

But that is an input. Your error is saying no output.

Can you post your elasticsearch.conf ?

 

[muruganandan]

Hi

Thanks for the reply,

Here is my Elasticserach config from elasticsearch.,yml and from filebeat,
But the same configuration it was working and stated to show data. Then I added neflow codecs codecs to define data.

id:
- default length in bytes
- :name
id:
- :uintN or :intN: or :ip4_addr or :ip6_addr or :mac_addr or :string
- :name
id:
- :skip

Then filebeat started to show structure error and I removed the above line, after I giving me " no output defined error. I could not identify where the output definitions would be.Even I disabled the Elasticsearch config but still it shows same error.

error:
filebeat", "name": "filebeat", "pid": 1285272, "ppid": 1285225, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-06-07T16:06:02.640+0530"}}}
2021-06-07T16:06:03.213+0530 INFO instance/beat.go:304 Setup Beat: filebeat; Version: 7.12.1
2021-06-07T16:06:03.213+0530 INFO instance/beat.go:337 No outputs are defined. Please define one under the output section.
2021-06-07T16:06:03.213+0530 INFO instance/beat.go:437 filebeat stopped.
2021-06-07T16:06:03.213+0530 ERROR instance/beat.go:971 Exiting: No outputs are defined. Please define one under the output section.
Exiting: No outputs are defined. Please define one under the output section.
bxxxx~$


I disabled the logstash and I’m using Elastic search for log analysis. The following is the elasticsearch.yml file configuration
http.port: 9200
node.name: MIKROTIK
path.data: /var/lib/elasticsearch
# Path to log files:
path.logs: /var/log/elasticsearch
network.host: localhost
http.port: 9200
------------------------------------ Node ------------------------------------
# Use a descriptive name for the node:
#node.name: node-1
node.name: MIKROTIK
# ----------------------------------- Paths ------------------------------------
# Path to directory where to store the data (separate multiple locations by comma):
path.data: /var/lib/elasticsearch
# Path to log files

path.logs: /var/log/elasticsearch
---------------------------------- Network -----------------------------------
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
network.host: localhost
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
http.port: 9200

filebeat elasticsearch config
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.#
hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
protocol: "http"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
username: "elastic"
password: "xxxx"

Thanks
Muruganandan.C

 

[muruganandan]

error:

ilebeat", "name": "filebeat", "pid": 1285272, "ppid": 1285225, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-06-07T16:06:02.640+0530"}}}
2021-06-07T16:06:03.213+0530 INFO instance/beat.go:304 Setup Beat: filebeat; Version: 7.12.1
2021-06-07T16:06:03.213+0530 INFO instance/beat.go:337 No outputs are defined. Please define one under the output section.
2021-06-07T16:06:03.213+0530 INFO instance/beat.go:437 filebeat stopped.
2021-06-07T16:06:03.213+0530 ERROR instance/beat.go:971 Exiting: No outputs are defined. Please define one under the output section.
Exiting: No outputs are defined. Please define one under the output section.
bluenet@bluenet:~$