Tag: logstash

Part 1: http://msprogrammer.serviciipeweb.ro/2016/12/05/logstash-on-windowsinstallation-and-io-examples/ 

Part 2 : http://msprogrammer.serviciipeweb.ro/2016/12/12/logstash-on-windows-transformation-of-data/

Now we want to use logstash for transforming data. For this , we use filter plugins to modify the data.

The process is like this:  Logstash receive the data(input plugin) , then apply a filter plugin( to parse and make new fields of data) and then sends data to output ( with an output plugin)

Let’s say we have this data that comes in a csv form,  like this:

PCName, RAM

AndreiPC, 10

OtherPC, 5

But we want to collect also from local pc ( let’s say console ) and do not put the PC name. The configuration is

input {
tcp {
    port => 9000
    type => "tcpLog"
  }
  stdin {
    type=> "console"
  }
}
filter{
    if [type] == "tcpLog" {   
        csv {
            columns => [       
                "PCName",
                "RAM"
                         ]
            add_field=>{
                "Source" => "tcp"
                }
        }
    }
    if [type] == "console" {   
        csv {
            columns => [                           
                "RAM"
                         ]
            add_field=>{
                "PCName" => "%{host}%"
                }
        }
    }
    mutate {
         convert => { "RAM" => "integer" }
        }
   
   
}
  output {
stdout {codec => rubydebug}

}

I find the configuration easy to understand – the output is a detailed json( rubydebug) and the input can be either tcp, either console.

If type is console, than a field will be add ( PCName ) .

And , at the final of the filter , the RAM field will be mutated into integer.

You can find filter plugins at https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

Part 1: http://msprogrammer.serviciipeweb.ro/2016/12/05/logstash-on-windowsinstallation-and-io-examples/ 

Part 2 : http://msprogrammer.serviciipeweb.ro/2016/12/12/logstash-on-windows-transformation-of-data/

I was very impressed by the declaration of logstash :”Centralize, Transform & Stash Your Data”  . What I think it does it receives, transforms  and outputs data- and it does very configurables.

I will start with some easy examples .

Install

Download the logstash from https://www.elastic.co/downloads/logstash  -  there is a zip file. Download and unblock , then extract all contents .A folder will be created ( the name of my folder is logstash-5.0.0 ). Open a command prompt and run

logstash-5.0.0\bin\logstash

The answer could be several :

1. PC is missing java – go , download, unblock, execute

2. It says something about missing “ server “ folder on java installation – goto your Java installation folder and copy “client” folder to “server”

3. It says “ ERROR: No configuration file was specified “ -  Ok, it is perfect!

Execution

Logstash need input and output to be configured – because it transforms any “input” into any “output” that he knows( via plugins)

Ok, now let’s do a working example – reading and writing to console :

logstash-5.0.0\bin\logstash -e ‘input { stdin { } } output { stdout {} }’

Now when you write something such as

asdasdasd

,the answer will be

2026-11-05T20:11:30.883Z ANDREIPC asdasdasd

Well, this is the first transformation – console to console.

Let’s make something more complicated – now I want to read from tcp port 9000 and output to console – but to see the whole message. For this we will create a file, named tcp.txt, with the following content:

input {
  tcp {
    port => 9000
    type => "tcpLog"
  }
}
  output {
stdout {codec => rubydebug}

}

( the code is for seeing more details about the message  ) And we will run

logstash-5.0.0\bin\logstash -f tcp.txt

Somewhere logstash should say:

Starting tcp input listener {:address=>"0.0.0.0:9000"}

In a separate window, I will start

telnet 127.0.0.1 9000

and enter the same text

asdasd

The answer will be:

{
    "@timestamp" => 2026-11-05T20:27:18.047Z,
          "port" => 51037,
      "@version" => "1",
           "host" => "127.0.0.1",
       "message" => "asdasd\r",
          "type" => "tcpLog"
}

And it is more clear now -  we jave telnet=> console.

Let’s say that now we want to write the output a file. I will modify tcp.txt to add to output the file plugin :

input {
  tcp {
    port => 9000
    type => "tcpLog"
  }
}
  output {
stdout {codec => rubydebug}
file {    path => "a.txt"     }
}

We start again logstash with

logstash-5.0.0\bin\logstash -f tcp.txt

and the telnet console with

telnet 127.0.0.1 9000

and enter the same text

asdasd

The answer will be now:

{
    "@timestamp" => 2026-11-05T20:31:47.639Z,
          "port" => 51213,
      "@version" => "1",
          "host" => "127.0.0.1",
       "message" => "asdasdad\r",
          "type" => "tcpLog"
}
[2026-11-05T22:31:48,534][INFO ][logstash.outputs.file    ] Opening file {:path=>"a.txt"}

For more outputs(such as csv , http. mongodb and others) , please see https://www.elastic.co/guide/en/logstash/current/output-plugins.html

For more inputs(such as file, http, github and others) please see  https://www.elastic.co/guide/en/logstash/current/input-plugins.html