Category: elk

Logstash on Windows– transformation of data

Part 1: http://msprogrammer.serviciipeweb.ro/2016/12/05/logstash-on-windowsinstallation-and-io-examples/ 

Part 2 : http://msprogrammer.serviciipeweb.ro/2016/12/12/logstash-on-windows-transformation-of-data/

Now we want to use logstash for transforming data. For this , we use filter plugins to modify the data.

The process is like this:  Logstash receive the data(input plugin) , then apply a filter plugin( to parse and make new fields of data) and then sends data to output ( with an output plugin)

Let’s say we have this data that comes in a csv form,  like this:

PCName, RAM

AndreiPC, 10

OtherPC, 5

But we want to collect also from local pc ( let’s say console ) and do not put the PC name. The configuration is

input {
tcp {
    port => 9000
    type => "tcpLog"
  }
  stdin {
    type=> "console"
  }
}
filter{
    if [type] == "tcpLog" {   
        csv {
            columns => [       
                "PCName",
                "RAM"
                         ]
            add_field=>{
                "Source" => "tcp"
                }
        }
    }
    if [type] == "console" {   
        csv {
            columns => [                           
                "RAM"
                         ]
            add_field=>{
                "PCName" => "%{host}%"
                }
        }
    }
    mutate {
         convert => { "RAM" => "integer" }
        }
   
   
}
  output {
stdout {codec => rubydebug}

}

I find the configuration easy to understand – the output is a detailed json( rubydebug) and the input can be either tcp, either console.

If type is console, than a field will be add ( PCName ) .

And , at the final of the filter , the RAM field will be mutated into integer.

You can find filter plugins at https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

Logstash on Windows–installation and I/O examples

Part 1: http://msprogrammer.serviciipeweb.ro/2016/12/05/logstash-on-windowsinstallation-and-io-examples/ 

Part 2 : http://msprogrammer.serviciipeweb.ro/2016/12/12/logstash-on-windows-transformation-of-data/

I was very impressed by the declaration of logstash :”Centralize, Transform & Stash Your Data”  . What I think it does it receives, transforms  and outputs data- and it does very configurables.

I will start with some easy examples .

Install

Download the logstash from https://www.elastic.co/downloads/logstash  -  there is a zip file. Download and unblock , then extract all contents .A folder will be created ( the name of my folder is logstash-5.0.0 ). Open a command prompt and run

logstash-5.0.0\bin\logstash

The answer could be several :

1. PC is missing java – go , download, unblock, execute

2. It says something about missing “ server “ folder on java installation – goto your Java installation folder and copy “client” folder to “server”

3. It says “ ERROR: No configuration file was specified “ -  Ok, it is perfect!

Execution

Logstash need input and output to be configured – because it transforms any “input” into any “output” that he knows( via plugins)

Ok, now let’s do a working example – reading and writing to console :

logstash-5.0.0\bin\logstash -e ‘input { stdin { } } output { stdout {} }’

Now when you write something such as

asdasdasd

,the answer will be

2026-11-05T20:11:30.883Z ANDREIPC asdasdasd

Well, this is the first transformation – console to console.

Let’s make something more complicated – now I want to read from tcp port 9000 and output to console – but to see the whole message. For this we will create a file, named tcp.txt, with the following content:

input {
  tcp {
    port => 9000
    type => "tcpLog"
  }
}
  output {
stdout {codec => rubydebug}

}

( the code is for seeing more details about the message  ) And we will run

logstash-5.0.0\bin\logstash -f tcp.txt

Somewhere logstash should say:

Starting tcp input listener {:address=>"0.0.0.0:9000"}

In a separate window, I will start

telnet 127.0.0.1 9000

and enter the same text

asdasd

The answer will be:

{
    "@timestamp" => 2026-11-05T20:27:18.047Z,
          "port" => 51037,
      "@version" => "1",
           "host" => "127.0.0.1",
       "message" => "asdasd\r",
          "type" => "tcpLog"
}

And it is more clear now -  we jave telnet=> console.

Let’s say that now we want to write the output a file. I will modify tcp.txt to add to output the file plugin :

input {
  tcp {
    port => 9000
    type => "tcpLog"
  }
}
  output {
stdout {codec => rubydebug}
file {    path => "a.txt"     }
}

We start again logstash with

logstash-5.0.0\bin\logstash -f tcp.txt

and the telnet console with

telnet 127.0.0.1 9000

and enter the same text

asdasd

The answer will be now:

{
    "@timestamp" => 2026-11-05T20:31:47.639Z,
          "port" => 51213,
      "@version" => "1",
          "host" => "127.0.0.1",
       "message" => "asdasdad\r",
          "type" => "tcpLog"
}
[2026-11-05T22:31:48,534][INFO ][logstash.outputs.file    ] Opening file {:path=>"a.txt"}

For more outputs(such as csv , http. mongodb and others) , please see https://www.elastic.co/guide/en/logstash/current/output-plugins.html

For more inputs(such as file, http, github and others) please see  https://www.elastic.co/guide/en/logstash/current/input-plugins.html

Andrei Ignat weekly software news(mostly .NET)

* indicates required

Please select all the ways you would like to hear from me:

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please visit our website.

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.