file helpers

More than one time you need to let users / application import bulk data in your system .

I have had many cases  – data is old data from old software, that is exported in some text files, usually csv.

In order to import fast the data, I used FileHelpers from http://www.filehelpers.com/

Let’s say you have the following data for an car evidence :

Date, StartMiles, Destination, EndMiles

And the data comes from an Excel  that has those columns. The requirements is the user can copy/paste from Excel data

When they copy the data comes in this form

01/01/2010     1000 Washington 1550

02/01/2010     1550 Dallas 2550

and so on.

It is clear that you :

  1. have a class with Date, StartMiles, Destination, EndMiles
  2. accomodate for space – but how to perform various data separator ( we suppose that first is the day, then comes the month).

Now the code in logical steps :

1. Accomodate for dates :

internal class ConvertDate : ConverterBase
{

        /// <summary>
        /// different forms for date separator : . or / or space
        /// </summary>
        /// <param name="from">the string format of date - first the day</param>
        /// <returns></returns>

        public override object StringToField(string from)
        {
            DateTime dt;

            if (DateTime.TryParseExact(from, "dd.MM.yyyy", null, DateTimeStyles.None, out dt))
                return dt;

            if (DateTime.TryParseExact(from, "dd/MM/yyyy", null, DateTimeStyles.None, out dt))
                return dt;

            if (DateTime.TryParseExact(from, "dd MM yyyy", null, DateTimeStyles.None, out dt))
                return dt;

            throw new ArgumentException("can not make a date from " + from, "from");

        }
}

2. Create the class that will hold one record:

    [IgnoreEmptyLines(true)]
    [DelimitedRecord(",")]
    internal class DestinationReader
    {
        //[FieldConverter(ConverterKind.Date,"dd.MM.yyyy")]
        [FieldConverter(typeof(ConvertDate))]
        public DateTime Date;
        [FieldConverter(ConverterKind.Int32)]
        public int StartMiles;

        [FieldQuoted(QuoteMode.OptionalForBoth)]
        public string Destination;

        [FieldConverter(ConverterKind.Int32)]
        public int  EndMiles;
    }

3. Now read the entire string:

            string Text = text that comes from the user
            string TextDelim = Text.Substring(10, 1);// the date has 10 chars - so the eleven is the separator
            while (Text.IndexOf(TextDelim + TextDelim) > 0)//consolidate for 2 delimiters
            {
                Text = Text.Replace(TextDelim + TextDelim, TextDelim);
            }
            DelimitedFileEngine<DestinationReader> flh=new DelimitedFileEngine<DestinationReader>();
            flh.Options.Delimiter = TextDelim;

            var data =flh.ReadString(Text);

In data you have a list of DestinationReader
So for any structured import of data use FileHelpers from http://www.filehelpers.com/

Selenium and testing WebInterfaces

On some cases you need to test the whole web interface. Why ? Suppose you have some Ajax call. You can test the call on server, but how do you ensure that user experience is OK ?

There are several testing projects for Web – for example selenium and Watin

I will show how to test with Selenium + NUNIT

  1. Download selenium from http://seleniumhq.org/ and download Firefox add-on from http://seleniumhq.org/projects/ide/
  2. Start selenium with java -jar selenium-server.jar
  3. Record the test with FF.
  4. Create a usual NUnit test project and add a reference to ThoughtWorks.Selenium.Core.dll
  5. The code can be like this :
  6. using System;
    using System.Text;
    using System.Text.RegularExpressions;
    using System.Threading;
    using NUnit.Framework;
    using Selenium;
    
    namespace InvoiceTest
    {
    [TestFixture]
    public class TestWeb
    {
    private ISelenium selenium;
    private StringBuilder verificationErrors;
    
    [SetUp]
    public void SetupTest()
    {
    //java -jar selenium-server.jar
    //selenium = new DefaultSelenium("localhost", 4444, @"*firefox d:\Program Files\Mozilla Firefox\firefox.exe","<a href="http://localhost/&quot;);">http://localhost/");</a>
    selenium = new DefaultSelenium("localhost", 4444, @"*iexplore", "<a href="http://localhost/&quot;);">http://localhost/");</a>
    //selenium = new DefaultSelenium("localhost", 4444, @"*iexploreproxy", "<a href="http://localhost/&quot;);">http://localhost/");</a>
    selenium.Start();
    verificationErrors = new StringBuilder();
    }
    
    [TearDown]
    public void TeardownTest()
    {
    try
    {
    selenium.Stop();
    }
    catch (Exception)
    {
    // Ignore errors if unable to close the browser
    }
    Console.WriteLine(verificationErrors.ToString());
    Assert.AreEqual("", verificationErrors.ToString());
    }
    
    [Test]
    public void FindTextAfterClickOnButton()
    {
    
    selenium.Open("/<strong>your application</strong>");
    selenium.Focus("");
    selenium.WindowMaximize();
    selenium.Type("the textbox", "the text");
    selenium.Click("button ");
    selenium.WaitForPageToLoad("30000");
    try
    {
    Assert.IsTrue(selenium.IsTextPresent("<strong>new text from your application</strong>"));
    }
    catch (AssertionException e)
    {
    verificationErrors.Append(e.Message);
    Console.WriteLine(e.Message);
    }
    }
    
    }
    }
    

Entity Framework profiler

Many times I’ve had problem with the following error when inserting objects with dates with Entity Framework :

System.Data.UpdateException An error occurred while updating the entries. See the inner exception for details.SqlDateTime overflow. Must be between 1/1/1753 12:00:00 AM and 12/31/9999 11:59:59 PM.

Ok, it’s my faute – but to remember each one date is too much for me…

The usual method was to start SqlProfiler, monitor the database and see how the sql is constructed. However, the database being used by all developers, it was not so simple to differentiate between all sql’s.

Other alternative was to log the entity framework generated sql’s . I have discovered Ayende Rahien Entity Framework profiler . Simple to use , as is wrote here in 2 simple steps

  1. add reference to HibernatingRhinos.Profiler.Appender.dll
  2. put this
HibernatingRhinos.Profiler.Appender.EntityFramework.EntityFrameworkProfiler.Initialize();

and start the exe. That will be all.

Do not forget to remove it on release version!

Pros:

Easy to use, valuable information, good!

Cons :

Not free …

Caching in .NET

In every application you have some data that is more read-more, write-once or twice. For example you can have the list of Cities of a country, the list of Countries of the world or list of exchange currency. This data is modified rarely. Also, you can have data that is not very sensitive to be real-time , such as the list of invoices for the day.
In .NET 3.5 you have several options
1. ASP.NET caching – and implementing in other applications with HttpRuntime  ( even if MS says “The Cache class is not intended for use outside of ASP.NET applications”)

2. Enterprise caching block : hard to configure

3. memcached , velocity, sharedcache and other third party providers – that comes with their options and configuration

In .NET 4.0 you have a new kid : objectcache and , more important , an implementation : MemoryCache

http://msdn.microsoft.com/en-us/library/system.runtime.caching.memorycache%28v=VS.100%29.aspx

What is very good is that now you can cache in Memory what do you want – and apply easily to your Business Layer. More, the object is a singleton for the application – that is even better (see the test on the final of the post)

What it is missing is an easy implementation for List and an implementation to remove data after a defined time.

So I decided to do my implementation for that (ok, it is wrong to have both implementations in a single class – but you can separate easily )

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Runtime.Caching;

namespace CachingData{

    /// <summary>
    /// List<int> i = new List<int>() { 1, 10, 100 };
            //CacheData_List<List<int>, int> data = new CacheData_List<List<int>, int>(2);
            //data.Add(i);
            //Assert.AreEqual(3, data.Items().Count, "must be 3");
            //data = new CacheData_List<List<int>, int>();
            //Assert.AreEqual(3, data.Items().Count, "must be 3");
            //Assert.IsTrue(data.ContainsData, "must have data");
            //Thread.Sleep(1000 * 3);
            //Assert.IsFalse(data.ContainsData, "must not have data");
    /// </summary>
    /// <typeparam name="T">and generic ILIST </typeparam>
    /// <typeparam name="U"></typeparam>
    public class CacheData_List<T, U>
        where T:class,IList<U>, new()

    {
        /// <summary>
        /// use this for adding in cache
        /// </summary>
        public event EventHandler RemovedCacheItem;

        private System.Timers.Timer timerEvent;
        private MemoryCache buffer;
        private int TimeToLiveSeconds;

        private DateTimeOffset dto;
        private string Key;
        /// <summary>
        /// default constructor - cache 600 seconds = 10 minutes
        /// </summary>
        public CacheData_List()
            : this(600)
        {
        }
        /// <summary>
        /// constructor cache the mentioned TimeSeconds time
        /// </summary>
        /// <param name="TimeSeconds">value of time for cache</param>
        public CacheData_List(int TimeSeconds)
        {
            TimeToLiveSeconds = TimeSeconds;
            timerEvent=new System.Timers.Timer(TimeToLiveSeconds * 1000);
            timerEvent.AutoReset = true;
            timerEvent.Elapsed += new System.Timers.ElapsedEventHandler(timerEvent_Elapsed);
            dto = new DateTimeOffset(DateTime.UtcNow.AddSeconds(TimeToLiveSeconds));
            Key = typeof(T).ToString();
            buffer = MemoryCache.Default;
        }

        void timerEvent_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
        {
            if (RemovedCacheItem != null)
            {
                RemovedCacheItem(this, EventArgs.Empty);
            }
        }
        /// <summary>
        /// remove item from cache
        /// </summary>
        public void Remove()
        {
            if (buffer.Contains(Key))
            {
                buffer.Remove(Key);

            }
            dto=new DateTimeOffset(DateTime.UtcNow.AddSeconds(TimeToLiveSeconds));
        }
        /// <summary>
        /// add multiple items to cache
        /// </summary>
        /// <param name="items">items to add to the cache</param>
        public void Add(T items)
        {

            if (buffer.Contains(Key))
            {
                T data = Items();
                foreach (var t in data)
                {
                    items.Add(t);
                }
                buffer.Remove(Key);
            }
            buffer.Add(Key, items, dto);
        }
        /// <summary>
        /// add a item to the IList of the cache
        /// </summary>
        /// <param name="item">an item to add</param>
        public void AddItem(U item)
        {

            T data=new T();
            if (buffer.Contains(Key))
            {
                data = buffer.Get(Key) as T;
                buffer.Remove(Key);
            }

            data.Add(item);
            buffer.Add(Key, data,dto);
        }
        /// <summary>
        /// usefull if you do not intercept the removed event
        /// </summary>
        public bool ContainsData
        {
            get
            {
                return buffer.Contains(Key);
            }

        }
        /// <summary>
        /// retrieve items
        /// </summary>
        /// <returns></returns>
        public T Items()
        {
            if (!buffer.Contains(Key))
                return null;

            return buffer.Get(Key) as T;
        }
    }
}

Please note that the test for usage is in the summary :

</pre>
List i = new List() { 1, 10, 100 };
CacheData_List <List<int>, int> data = new CacheData_List<List<int>, int>(2);
data.Add(i);
Assert.AreEqual(3, data.Items().Count, "must be 3");
data = new CacheData_List<List<int>, int>();
Assert.AreEqual(3, data.Items().Count, "must be 3");
Assert.IsTrue(data.ContainsData, "must have data");
Thread.Sleep(1000 * 3);
Assert.IsFalse(data.ContainsData, "must not have data");

You can download the file from

http://msprogrammer.serviciipeweb.ro/wp-content/uploads/2010/05/CachingData.zip

(Last Note : for synchonization maybe it was better to lock on readerwriterlockslim

Intercept errors

Errors interception and rising

Description

We will speak here about the errors raised by external conditions (bugs) and the errors rose by the application.

There is not application without bugs. The bugs are not desired for an application – but programmers are human beings and, by consequence, they cannot verify every possible path that go an application to the failure. Maybe the space on hard is too little, other time the network between database and application is broken – there are many conditions to bring failure to an application. And you have to manage priority between

· repairing bugs

· evolving the application

· Resolving failure cases.

The errors raised by the application are usually logical errors, like the fact that a birthday date for a person cannot be tomorrow. Sometimes you will intercept errors from other components and raise your own error (like the fact that a component should have a unique name – you will intercept the index failure raised by your database and then raise your own error).

How to intercept error in applications

We will discuss here how and where to intercept errors. There are two principles that I guide my interception of errors:

1. Occam’s razor : in components(business layer, repository) intercept only the errors that you are sure to know what to do with (such as an ID is missing from a database). Usually here you can raise your own exception and let the GUI handle it. In

2. In GUI layer intercept all exception and , if possible, suggest to the user an alternative way to do.You must also log the error in order to can be retrieved later.

Please read the logging and instrumentation to see a common way to describe the path that a error has been flow through the application

Examples

First example is how to intercept an error

Second will be how to intercept and raise own errors

Third will be about intercepting errors in a Console application

1)We will first intercept an error and write a custom message for this error .

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Data.SqlClient;

namespace exceptionintercept

{

class Program

{

static void Main(string[] args)

{

using (SqlConnection sc = new SqlConnection())

{

sc.ConnectionString = "Data Source=(local);Initial Catalog=adaad!@#$%^;Trusted_Connection=true";

try

{

sc.Open();

}

catch (SqlException ex)

{

if (ex.Number == 2)

{

Console.WriteLine("no database connection : “+ ex.Message);

return;

}

throw;

}

}

}

}

}

As you see we will intercept only the SqlException and put the custom error message only if error message number for the SqlException is 2 – no network access. Usually it is not a good idea to show your connection string in the output shown to the user.

Download project from http://msprogrammer.serviciipeweb.ro/wp-content/uploads/exception.zip

2) Intercept and raise error

We will do now a more advanced interception of connection : we will see if the machine that has the database answers to the ping. If yes, probably only the database is down , not the PC or the connection.

public void VerifyConnection(string MyConnection)

{

using (SqlConnection sc = new SqlConnection())

{

sc.ConnectionString = MyConnection;

try

{

sc.Open();

}

catch (SqlException ex)

{

//TODO : log the exception

Ping p = new Ping();

try

{

PingReply pr = p.Send(sc.DataSource);

}

catch (PingException)

{

//TODO : log the exception

throw new PCException(sc.DataSource, ex);

}

throw new BasicDatabaseException(sc.DataSource,ex);

}

}

}

3) Any application should intercept globally errors, in order to log / save state if there is something that works not well and it is not handled.

static void Main(string[] args)

{

System.AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);

int i = 1;

i = 1 / (i - 1);

}

static void CurrentDomain_UnhandledException(object sender, UnhandledExceptionEventArgs e)

{

//TODO : log

Console.WriteLine("from program : unexpected error occured " + e.ExceptionObject);

}

Download project from http://msprogrammer.serviciipeweb.ro/wp-content/uploads/exception.zip

TODO

intercept global errors in WinForms, ASP.NET , WPF, Silverlight application

Other resources

ELMAH – intercept ASP.NET errors , http://code.google.com/p/elmah/

Logging : http://msprogrammer.serviciipeweb.ro/2010/04/19/logging-and-instrumentation/

Logging and instrumentation

Why

There are at least two reasons for an application to register what tfhe user do.

The first one and the most important is to see what is happening when a user has an error: what have been done, on which path did he go and what additional data(method parameters, usually) do you need for reproducing the bug.

The second one is to provide the application a way to record in-obtrusive messages for same actions that may seems not so right(check the event log for your computer for more examples 😉 )

Requirements for a logging system

The logging system should be:

1. Robust – if it can not write the message, it should not fail the entire app

2. Multi-write-data : it can write easily to multiple systems(such as databases, files on disk, send emails and so on)

3. Flexibility : the configuration of the logging system should be easy to modify. More, the modifications should be applied and “re-loaded” without re-compiling or re-starting the application

An implementation

For the implementation I will use log4net , http://logging.apache.org/log4net/ that it is satisfies all three conditions.

We will do a console application that sends a random error

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

namespace ConsoleDemo1

{

class Program

{

static void Main(string[] args)

{

long ticks = DateTime.Now.Ticks;

FirstFunction(ticks);

}

static void FirstFunction(long i)

{

if (i % 2 == 0)

{

throw new ArgumentException("from first function");

SecondFunction(i);

}

static void SecondFunction(long i)

{

if (i % 2 == 1)

{

throw new ArgumentException("from second function");

}

}

}

}

As you see this application will have an error no matter if the ticks are even or odd. Download project from here : http://msprogrammer.serviciipeweb.ro/wp-content/uploads/loggingDemo.zip

Now we will see how the log4net writes data and error to a physical storage, in order to can be retrieved later.

Download log4net, add reference to it and make the following 4 steps

1. add a config file (I will name it log4net.config ) and in properties put “Copy to output Directory” to “Copy always” .

clip_image002[4]

Examples of log4net config files you will find everywhere on internet . If you look at the project you will see a bunch of “appenders” : ConsoleAppender, RollingLogFileAppender , SmtpAppender . The appender appends the log to the storage provided (Console, File, Email ). You will find a list of appenders at http://logging.apache.org/log4net/release/sdk/log4net.Appender.html and, if you do not found some appender you can wrote your own – just look at the source code.

For this application I want to use just ConsoleAppender and RollingFileAppender

<root>

<level value=”DEBUG” />

<appender-ref ref=”ConsoleAppender” />

<appender-ref ref=”RollingLogFileAppender” />

</root>

2. In order to see the file, put


log4net.Config.XmlConfigurator.ConfigureAndWatch(new System.IO.FileInfo("log4net.config"));

(for asp.net application ,put


log4net.Config.XmlConfigurator.ConfigureAndWatch(new System.IO.FileInfo(Server.MapPath("~/log4net.config")));

)

3. Add a variable to log the errors :


private static readonly log4net.ILog _logger = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

4. Add a try/catch to log errors :

try

{

FirstFunction(ticks);

}

catch (Exception ex)

{

if (_logger.IsErrorEnabled)

{

_logger.Error("see an error!", ex);

}

throw;

}

Now, if you run the ConsoleDemoLog4net project, then you will see how to exception is logged twice and you will find the currentlog.txt in the same folder as your application executable.

Automatically instrument an application(dll, exe, asp.net)

This was pretty good – but what about putting a log to every method to see where the application flow has been gone and with which arguments?

You can do this with postsharp / log4postsharp. Postsharp is … Log4Postsharp is …

Let’s make the application do this automatically.

We download Postsharp 1.0 (not 1.5!) from http://www.softpedia.com/progDownload/PostSharp-Download-114994.html (you can found a more good paid version at http://www.sharpcrafters.com/downloads/ )

Now we will do the following :

1. Add the log4net.config from the previous project

2. Add log4net , PostSharp.Public and PostSharp reference

3. Add to the main just:

log4net.Config.XmlConfigurator.ConfigureAndWatch(new System.IO.FileInfo("log4net.config"));

Do NOT add the log4net code to intercept!

4. Modify AssemblyInfo.cs and put

using Log4PostSharp;

[assembly: Log(AttributeTargetTypes = "*", EntryLevel = LogLevel. Error, EntryText = "postsharp :entering {signature} {paramvalues}", ExitLevel = LogLevel. Error, ExceptionLevel = LogLevel.Error, ExitText = "postsharp :exit {signature} {paramvalues} =&gt; {returnvalue}", ExceptionText = "postsharp : error in {signature} ")]

5. Add a text file named <yourprojectname>.psproj (in this case, ConsoleDemoPostSharp.psproj ) with the following content:

<?xml version=”1.0″ encoding=”utf-8″ ?>

<Project xmlns=”http://schemas.postsharp.org/1.0/configuration”>

<SearchPath Directory=”bin/{$Configuration}”/>

<SearchPath Directory=”{$SearchPath}” />

<SearchPath Directory=”lib” />

<Tasks>

<AutoDetect />

<Compile TargetFile=”{$Output}” IntermediateDirectory=”{$IntermediateDirectory}” CleanIntermediate=”false” />

</Tasks>

</Project>

6. Edit by hand the csproj file and add this :

<PropertyGroup>

<DontImportPostSharp>True</DontImportPostSharp>

<PostSharpDirectory> ..\libs\ postsharp\</PostSharpDirectory>

<PostSharpUseCommandLine>True</PostSharpUseCommandLine>

</PropertyGroup>

<Import Project="$(PostSharpDirectory)PostSharp.targets" Condition=" Exists('$(PostSharpDirectory)PostSharp.targets') " />

7. If there is a screen to ask you about a project modification, please tell “Load project normally”

So now , when it’s compiling , it says :

“D:\programe\youtube\loggingDemo\libs\postsharp\PostSharp.exe” “D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\ConsoleDemoPostSharp.psproj” “D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\obj\Release\Before-PostSharp\ConsoleDemoPostSharp.exe” “/P:Output=obj\Release\ConsoleDemoPostSharp.exe ” “/P:ReferenceDirectory=D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp ” “/P:Configuration=Release ” “/P:Platform=AnyCPU ” “/P:SearchPath=bin\Release\,obj\Release\, ” “/P:IntermediateDirectory=obj\Release\PostSharp ” “/P:CleanIntermediate=False ” “/P:MSBuildProjectFullPath=D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\ConsoleDemoPostSharp.csproj ” “/P:SignAssembly=False ” “/P:PrivateKeyLocation= ”

PostSharp 1.0 [1.0.12.469] – Copyright (c) Gael Fraiteur, 2005-2008.

EXEC : warning PS0064: A new version of PostSharp 1.0 is available. You have currently 1.0.12.469 and you could download the version 1.0.13.630 from http://www.sharpcrafters.com/postsharp/download.

info PS0035: C:\Windows\Microsoft.NET\Framework\v2.0.50727\ilasm.exe “D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\obj\Release\PostSharp\ConsoleDemoPostSharp.il” /QUIET /EXE /PDB “/RESOURCE=D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\obj\Release\PostSharp\ConsoleDemoPostSharp.res” “/OUTPUT=D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\obj\Release\ConsoleDemoPostSharp.exe” /SUBSYSTEM=3 /FLAGS=1 /BASE=19595264 /STACK=1048576 /ALIGNMENT=512 /MDV=v2.0.50727

ConsoleDemoPostSharp -> D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\bin\Release\ConsoleDemoPostSharp.exe

Done building project “ConsoleDemoPostSharp.csproj”.

We then run the program and enjoy the power of PostSharp :

2010-04-17 23:29:12,705 [1] ERROR 2010-04-17 23:28:43,265 [1] ERROR D:\programe\youtube\loggingDemo\ConsoleDemoPostSharp\Program.cs ConsoleDemoPostSharp.Program [(null)] – postsharp :entering Void FirstFunction(Int64) “634071437232634478

[more data]

So , as you see it gives you not only the trace , but also the value : 634071437232634478

You can download code from http://msprogrammer.serviciipeweb.ro/wp-content/uploads/loggingDemo.zip

Additional resources

Tracing : http://www.15seconds.com/Issue/020910.htm

Enterprise logging : http://msdn.microsoft.com/en-us/library/cc309257%28v=MSDN.10%29.aspx

VS2010 with Dotfuscator : http://msdn.microsoft.com/en-us/library/ff460257.aspx

CCI metadata : http://ccisamples.codeplex.com

Postsharp 2.0 improvements over 1.0 : one of many examples here : http://www.sharpcrafters.com/blog/post/introducing-postsharp-2-0-1-notifypropertychanged.aspx