Category: mini tools list

Logparser quick and dirty

Sometimes you must find information in text files. Many,many text files, like IIS logs or other custom non-regular formats.

I have a bot from – and I log the messages with log4net in text files, with another messages.

An entry looks like that :

System.ArgumentException: ;channel=private;botkey=<guid>;userkey=<guid>;;network=Yahoo;msg=hello;step=1;value0=hello;to=asdasd

And there are multiple log files that I want to parse and find the email adresses to collect feedback from those persons that use my bot.

LogParser to the rescue! Download from and use this command line

LOGPARSER “Select Text into a.csv from current* where Text like ‘%@%'” -i:TEXTLINE

Explanation of command :

Select Text into a.csv from current* where Text like ‘%@%’ –means find in files that begin with current(current*) all text that contains emails ( ‘%@%’) and put in file a.csv  the results.

-i:TEXTLINE – means the format is text

What can be more simple ?

(Ok, for finding the user name I had to resort to excel, to remove duplicates … )

More I think it is fast enough : for parsing 114 files with 58.8 MB (PC with a 2GB RAM + 7200 RPM ) the results are :

Elements processed: 487176
Elements output:    1044
Execution time:     7.69 seconds

Also logparser can be used for more than text files :

More, it can be as a COM DLL in every .NET project, making it a usefull tool . See

Next time I will show the using Powershell in combination with LogParser.

file helpers

More than one time you need to let users / application import bulk data in your system .

I have had many cases  – data is old data from old software, that is exported in some text files, usually csv.

In order to import fast the data, I used FileHelpers from

Let’s say you have the following data for an car evidence :

Date, StartMiles, Destination, EndMiles

And the data comes from an Excel  that has those columns. The requirements is the user can copy/paste from Excel data

When they copy the data comes in this form

01/01/2010     1000 Washington 1550

02/01/2010     1550 Dallas 2550

and so on.

It is clear that you :

  1. have a class with Date, StartMiles, Destination, EndMiles
  2. accomodate for space – but how to perform various data separator ( we suppose that first is the day, then comes the month).

Now the code in logical steps :

1. Accomodate for dates :

internal class ConvertDate : ConverterBase

        /// <summary>
        /// different forms for date separator : . or / or space
        /// </summary>
        /// <param name="from">the string format of date - first the day</param>
        /// <returns></returns>

        public override object StringToField(string from)
            DateTime dt;

            if (DateTime.TryParseExact(from, "dd.MM.yyyy", null, DateTimeStyles.None, out dt))
                return dt;

            if (DateTime.TryParseExact(from, "dd/MM/yyyy", null, DateTimeStyles.None, out dt))
                return dt;

            if (DateTime.TryParseExact(from, "dd MM yyyy", null, DateTimeStyles.None, out dt))
                return dt;

            throw new ArgumentException("can not make a date from " + from, "from");


2. Create the class that will hold one record:

    internal class DestinationReader
        public DateTime Date;
        public int StartMiles;

        public string Destination;

        public int  EndMiles;

3. Now read the entire string:

            string Text = text that comes from the user
            string TextDelim = Text.Substring(10, 1);// the date has 10 chars - so the eleven is the separator
            while (Text.IndexOf(TextDelim + TextDelim) > 0)//consolidate for 2 delimiters
                Text = Text.Replace(TextDelim + TextDelim, TextDelim);
            DelimitedFileEngine<DestinationReader> flh=new DelimitedFileEngine<DestinationReader>();
            flh.Options.Delimiter = TextDelim;

            var data =flh.ReadString(Text);

In data you have a list of DestinationReader
So for any structured import of data use FileHelpers from

Selenium and testing WebInterfaces

On some cases you need to test the whole web interface. Why ? Suppose you have some Ajax call. You can test the call on server, but how do you ensure that user experience is OK ?

There are several testing projects for Web – for example selenium and Watin

I will show how to test with Selenium + NUNIT

  1. Download selenium from and download Firefox add-on from
  2. Start selenium with java -jar selenium-server.jar
  3. Record the test with FF.
  4. Create a usual NUnit test project and add a reference to ThoughtWorks.Selenium.Core.dll
  5. The code can be like this :
  6. using System;
    using System.Text;
    using System.Text.RegularExpressions;
    using System.Threading;
    using NUnit.Framework;
    using Selenium;
    namespace InvoiceTest
    public class TestWeb
    private ISelenium selenium;
    private StringBuilder verificationErrors;
    public void SetupTest()
    //java -jar selenium-server.jar
    //selenium = new DefaultSelenium("localhost", 4444, @"*firefox d:\Program Files\Mozilla Firefox\firefox.exe","<a href="http://localhost/&quot;);">http://localhost/");</a>
    selenium = new DefaultSelenium("localhost", 4444, @"*iexplore", "<a href="http://localhost/&quot;);">http://localhost/");</a>
    //selenium = new DefaultSelenium("localhost", 4444, @"*iexploreproxy", "<a href="http://localhost/&quot;);">http://localhost/");</a>
    verificationErrors = new StringBuilder();
    public void TeardownTest()
    catch (Exception)
    // Ignore errors if unable to close the browser
    Assert.AreEqual("", verificationErrors.ToString());
    public void FindTextAfterClickOnButton()
    selenium.Open("/<strong>your application</strong>");
    selenium.Type("the textbox", "the text");
    selenium.Click("button ");
    Assert.IsTrue(selenium.IsTextPresent("<strong>new text from your application</strong>"));
    catch (AssertionException e)

Entity Framework profiler

Many times I’ve had problem with the following error when inserting objects with dates with Entity Framework :

System.Data.UpdateException An error occurred while updating the entries. See the inner exception for details.SqlDateTime overflow. Must be between 1/1/1753 12:00:00 AM and 12/31/9999 11:59:59 PM.

Ok, it’s my faute – but to remember each one date is too much for me…

The usual method was to start SqlProfiler, monitor the database and see how the sql is constructed. However, the database being used by all developers, it was not so simple to differentiate between all sql’s.

Other alternative was to log the entity framework generated sql’s . I have discovered Ayende Rahien Entity Framework profiler . Simple to use , as is wrote here in 2 simple steps

  1. add reference to HibernatingRhinos.Profiler.Appender.dll
  2. put this

and start the exe. That will be all.

Do not forget to remove it on release version!


Easy to use, valuable information, good!

Cons :

Not free …

Logging and instrumentation


There are at least two reasons for an application to register what tfhe user do.

The first one and the most important is to see what is happening when a user has an error: what have been done, on which path did he go and what additional data(method parameters, usually) do you need for reproducing the bug.

The second one is to provide the application a way to record in-obtrusive messages for same actions that may seems not so right(check the event log for your computer for more examples 😉 )

Requirements for a logging system

The logging system should be:

1. Robust – if it can not write the message, it should not fail the entire app

2. Multi-write-data : it can write easily to multiple systems(such as databases, files on disk, send emails and so on)

3. Flexibility : the configuration of the logging system should be easy to modify. More, the modifications should be applied and “re-loaded” without re-compiling or re-starting the application

An implementation

For the implementation I will use log4net , that it is satisfies all three conditions.

We will do a console application that sends a random error

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

namespace ConsoleDemo1


class Program


static void Main(string[] args)


long ticks = DateTime.Now.Ticks;



static void FirstFunction(long i)


if (i % 2 == 0)


throw new ArgumentException("from first function");



static void SecondFunction(long i)


if (i % 2 == 1)


throw new ArgumentException("from second function");





As you see this application will have an error no matter if the ticks are even or odd. Download project from here :

Now we will see how the log4net writes data and error to a physical storage, in order to can be retrieved later.

Download log4net, add reference to it and make the following 4 steps

1. add a config file (I will name it log4net.config ) and in properties put “Copy to output Directory” to “Copy always” .


Examples of log4net config files you will find everywhere on internet . If you look at the project you will see a bunch of “appenders” : ConsoleAppender, RollingLogFileAppender , SmtpAppender . The appender appends the log to the storage provided (Console, File, Email ). You will find a list of appenders at and, if you do not found some appender you can wrote your own – just look at the source code.

For this application I want to use just ConsoleAppender and RollingFileAppender


<level value=”DEBUG” />

<appender-ref ref=”ConsoleAppender” />

<appender-ref ref=”RollingLogFileAppender” />


2. In order to see the file, put

log4net.Config.XmlConfigurator.ConfigureAndWatch(new System.IO.FileInfo("log4net.config"));

(for application ,put

log4net.Config.XmlConfigurator.ConfigureAndWatch(new System.IO.FileInfo(Server.MapPath("~/log4net.config")));


3. Add a variable to log the errors :

private static readonly log4net.ILog _logger = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

4. Add a try/catch to log errors :





catch (Exception ex)


if (_logger.IsErrorEnabled)


_logger.Error("see an error!", ex);




Now, if you run the ConsoleDemoLog4net project, then you will see how to exception is logged twice and you will find the currentlog.txt in the same folder as your application executable.

Automatically instrument an application(dll, exe,

This was pretty good – but what about putting a log to every method to see where the application flow has been gone and with which arguments?

You can do this with postsharp / log4postsharp. Postsharp is … Log4Postsharp is …

Let’s make the application do this automatically.

We download Postsharp 1.0 (not 1.5!) from (you can found a more good paid version at )

Now we will do the following :

1. Add the log4net.config from the previous project

2. Add log4net , PostSharp.Public and PostSharp reference

3. Add to the main just:

log4net.Config.XmlConfigurator.ConfigureAndWatch(new System.IO.FileInfo("log4net.config"));

Do NOT add the log4net code to intercept!

4. Modify AssemblyInfo.cs and put

using Log4PostSharp;

[assembly: Log(AttributeTargetTypes = "*", EntryLevel = LogLevel. Error, EntryText = "postsharp :entering {signature} {paramvalues}", ExitLevel = LogLevel. Error, ExceptionLevel = LogLevel.Error, ExitText = "postsharp :exit {signature} {paramvalues} =&gt; {returnvalue}", ExceptionText = "postsharp : error in {signature} ")]

5. Add a text file named <yourprojectname>.psproj (in this case, ConsoleDemoPostSharp.psproj ) with the following content:

<?xml version=”1.0″ encoding=”utf-8″ ?>

<Project xmlns=””>

<SearchPath Directory=”bin/{$Configuration}”/>

<SearchPath Directory=”{$SearchPath}” />

<SearchPath Directory=”lib” />


<AutoDetect />

<Compile TargetFile=”{$Output}” IntermediateDirectory=”{$IntermediateDirectory}” CleanIntermediate=”false” />



6. Edit by hand the csproj file and add this :



<PostSharpDirectory> ..\libs\ postsharp\</PostSharpDirectory>



<Import Project="$(PostSharpDirectory)PostSharp.targets" Condition=" Exists('$(PostSharpDirectory)PostSharp.targets') " />

7. If there is a screen to ask you about a project modification, please tell “Load project normally”

So now , when it’s compiling , it says :

“D:\programe\youtube\loggingDemo\libs\postsharp\PostSharp.exe” “D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\ConsoleDemoPostSharp.psproj” “D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\obj\Release\Before-PostSharp\ConsoleDemoPostSharp.exe” “/P:Output=obj\Release\ConsoleDemoPostSharp.exe ” “/P:ReferenceDirectory=D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp ” “/P:Configuration=Release ” “/P:Platform=AnyCPU ” “/P:SearchPath=bin\Release\,obj\Release\, ” “/P:IntermediateDirectory=obj\Release\PostSharp ” “/P:CleanIntermediate=False ” “/P:MSBuildProjectFullPath=D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\ConsoleDemoPostSharp.csproj ” “/P:SignAssembly=False ” “/P:PrivateKeyLocation= ”

PostSharp 1.0 [] – Copyright (c) Gael Fraiteur, 2005-2008.

EXEC : warning PS0064: A new version of PostSharp 1.0 is available. You have currently and you could download the version from

info PS0035: C:\Windows\Microsoft.NET\Framework\v2.0.50727\ilasm.exe “D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\obj\Release\PostSharp\” /QUIET /EXE /PDB “/RESOURCE=D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\obj\Release\PostSharp\ConsoleDemoPostSharp.res” “/OUTPUT=D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\obj\Release\ConsoleDemoPostSharp.exe” /SUBSYSTEM=3 /FLAGS=1 /BASE=19595264 /STACK=1048576 /ALIGNMENT=512 /MDV=v2.0.50727

ConsoleDemoPostSharp -> D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\bin\Release\ConsoleDemoPostSharp.exe

Done building project “ConsoleDemoPostSharp.csproj”.

We then run the program and enjoy the power of PostSharp :

2010-04-17 23:29:12,705 [1] ERROR 2010-04-17 23:28:43,265 [1] ERROR D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\Program.cs ConsoleDemoPostSharp.Program [(null)] – postsharp :entering Void FirstFunction(Int64) “634071437232634478

[more data]

So , as you see it gives you not only the trace , but also the value : 634071437232634478

You can download code from

Additional resources

Tracing :

Enterprise logging :

VS2010 with Dotfuscator :

CCI metadata :

Postsharp 2.0 improvements over 1.0 : one of many examples here :

Html Agility Pack

This is the ultimate reference of reading web pages.

IF you want to do it yourself , you can try with WebRequest , . But not all HTML is an XML – so you must find a method to parse. But am I the only one ?No – so I found the  HTML Agility Pack , , that knows how to transform HTML in XML. The code is easy :

HtmlWeb hw = new HtmlWeb();
hw.AutoDetectEncoding = true;
HtmlDocument  doc = hw.Load(Url);
HtmlNode NodeRoot = doc.DocumentNode;

And from NodeRoot you can start XPATH with SelectNodes . Try it – it is awesome!

My programmer tools in 2009

you can download the PDF from




SharpZipLib Zip files programatically
nunit test
svn source control
hudson continous integration
EntityFramework sql server database to .NET code
filehelpers parse csv and other files
log4net logging
lumisoft parsing email messages
moq mocking in unit test
automapper transferring data between dal and bll
pagedlist use for paging in ASP.NET MVC
log4postsharp logging every method with easy
postsharp see log4postsharp, version 1.5 still free
webdeployment project to deploy web projects;displaylang=en&displaylang=en
IIS SEO Toolkit verify my sites
nbehave testing with words
reflector analyze code, others and mine
fiddler analyze ajax requests
build utilities from apache svn commit if different
selenium testing web interfaces
HtmAgilityPack saving web pages




notepad++ no notepad
paint graphic editor graphic editor
7zip free archiver-un-archiver
sysinternals bunch of windows utilities
windows live writer blog
psr.exe new method to create help
logparser parse fast files
foxitreader pdf reader
yahoo instant messaging
skype instant messaging
dosbox old games redivivus
vlc movie player
dvddecrypter and dvdshrink backup my movies
magicdisk load iso files
freecommander norton commander
winmerge compare files / folders

Firefox addons

yahoo slow

As I was saying, you can download the PDF from

Andrei Ignat weekly software news(mostly .NET)

* indicates required

Please select all the ways you would like to hear from me:

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please visit our website.

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.