Tag: mini tools list

Programmer tools 2011

List of programmer tools

I have re-installed the PC and I have been taken notice of what tools I have on the system now:

  1. Magic Disc – can mount .iso files ( for  2 )
  2. Visual Studio – must have for a easy developing path
  3. Sql Server 2008 – primary database for me
  4. 7-zip  – archiver
  5. Firefox ( plus addons, see below)
  6. Smtp4dev – to see messages
  7. AspNetMVC3ToolsUpdateSetup
  8. FreeCommander –  dual panels for windows explorer
  9. Foxit reader – pdf viewer
  10. Notepad ++ ( with hex addon)
  11. Winmerge – files/ folders difference
  12. Clean project – archive solution
  13. SqlSearch from RedGate – fast search after names
  14. SSMS tools – record every operation you do in sql server
  15. EntityFramework41  – code first development
  16. XUnit – automated testing + samples  to do BDD style
  17. SSCERuntime_x86-ENU –SqlCompact provider. Works with EF4.1
  18. SqlCe40Toolbox – SqlCompact viewer
  19. NuGet.Tools.vsix – Nuget is awesome!
  20. ImgBurn  – burn cd-s
  21. consolas font  – see  1 and l ?
  22. LogParser  – never know when you need to parse some files
  23. Moq  – mocking tool
  24. PreviewHandlerPack  -see c# code in preview window
  25. Regulator and Regulazy – regular expression helpers
  26. StringTemplate.NET  – templating generator. Maybe replaced by razor ?
  27. NLog –  logging tool. Log4Net was pretty unreliable in .NET 4
  28. FileHelpers – reading writing text data.
  29. Tcmdwincearm – Total Commander for mobile. Free.
  30. Jquery and Jquery UI.
  31. DataTables – html tables supports sorting , filtering, others.
  32. Windows Live Essential – blogging fast.
  33. InsertFilePlugin – Live writer extension to insert files to upload.
  34. OfficeOpenXMLPart4-MarkupLanguageReference  – markup for Office XML. Used with StringTemplate
  35. SharpZipLib – knows how to zip multiple files.
  36. AutoFixture – generating sample data
  37. HtmlAgilityPack – parsing web pages
  38. T4MVC – get rid of magic names for controllers, actions
  39. Itextsharp – save as pdf
  40. Ninject – DI provider
  41. Hudson – continous integration
  42. AutoMapper  – transferring data between DAL and BLL
  43. Selenium – testing web interfaces
  44. Svn  -source control
  45. MVC Contrib – pages list and more
  46. Msbuildtask from tigris – build make it easy
  47. Psr – help made easy in Windows 7
  48. Firefox addons here

https://addons.mozilla.org/en-US/firefox/collections/ignatandrei/ignatandrei/

For every tool search for it . The first link will give you all details.

You can have as pdf here:List of programmer tools 2011

Optimizing EF4 and EDMGen2

Just reading (from Adrian Florea links ) how the EF4 could be optimized : http://www.codeproject.com/KB/database/PerfEntityFramework.aspx

First modification is to “Pre-generated Your View”. For this you must have .ssdl, .csdl, and .msl files – so you change the “Metadata Artifact Processing property to: Copy to Output Directory.”. Then you process the .ssdl, .csdl, and .msl with edmgen in order to can have the  views.

Until here , all ok.

But, in the next advice, is to keep “Metadata Artifact Processing property to: Embed in Output Assembly.”

One solution is to put “Metadata Artifact Processing property to: Copy to Output Directory.” , compile, put again “Metadata Artifact Processing property to: Embed in Output Assembly.” and compile again. But , if you change the edmx (add fields or tables ) you must redo the operation  – so you will have more things to do(if you remember)

A solution is to build them on the pre-build step .But how to generate the .ssdl, .csdl, and .msl files  ?

Edmgen2 , http://code.msdn.microsoft.com/EdmGen2 , to the rescue. Download, put into a lib folder under your solution folder and put a pre-build like this :

$(SolutionDir)lib\edmgen2\edmgen2 /FromEdmx $(ProjectDir)prod.edmx”

“%windir%\Microsoft.NET\Framework\v4.0.30319\EdmGen.exe” /mode:ViewGeneration /language:CSharp /nologo “/inssdl:prod.ssdl” “/incsdl:prod.csdl”   “/inmsl:prod.msl” “/outviews:$(ProjectDir)prod.Views.cs”

What you must change on your project?

1. IF you are on 64 bit, change Framework to Framework64

2. change the prod with your edmx name.

What is the performance?

Tested by loading a table with 301 rows by doing the steps :

1. open connection, load all table in objects(POCO), closing connection

2. open connection , find object with PK = 1, closing connection

3. open connection ,  loading 1 tables with 2 related (include ) , closing connection

The results are in milliseconds:

Without pre-compiled views

LoadTable LoadID LoadMultiple Total Time
579 800 172 1551
563 755 171 1489
559 754 169 1482
568 762 240 1570

With pre-compiled views:

LoadTable LoadID LoadMultiple Total Time
606 807 183 1596
509 706 177 1392
852 137 192 1181
530 733 221 1484
523 722 183 1428

The average / min / max results:

average max min
without 1523 1570 1482
with 1413.25 1596 1181

In the next picture the smaller the duration(milliseconds), is the better :

image

Conclusions:

1.  For the average and min the difference is 7%, respectively 20%. Please remember we are using only 3 queries.

For the max, it is vey curious : the with is more than without. The penalty is 1%, I think that is a measuring error ? – or maybe not. However , the penalty is small comparing with others.

2. Very curious, find after ID, with a table with 301 rows, took longer than loading the whole table.However, did not take into accound finding in the list the object( it is in memory also)

3. It may worth to add the pre-build step shown before to pre-compile views.

Links :

http://www.codeproject.com/KB/database/PerfEntityFramework.aspx

http://msdn.microsoft.com/en-us/library/bb896240.aspx

LogParser, PowerShell and Quick and dirty parsing of IIS files

For a local enterprise IIS system you do not have to resort to Google analytics or other beasts that interprets IIs logs. After all, users are identified through ActiveDirectory, does not matter from what city do they come, and so on. But it will help to have some details of wjhat happened on the system this day( or the day before , if you want to send an email about the previous day totals)

So logparser to help  – he knows already to read IIS logs with  -i:IISW3C

So I have come up with the syntax  :

C:\LogParser -e:10 -i:IISW3C “SELECT cs-uri-stem as url, DIV(SUM(time-taken),1000) as Seconds, Count(time-taken) as Requests, DIV(Seconds ,Requests) as TimeExecuting   INTO C:\newfile FROM   C:\Windows\System32\LogFiles\W3SVC1\ex100909.log GROUP BY cs-uri-stem Having SUM(time-taken)>0 and Seconds>0 order by Seconds   desc” -o:TPL -tpl:%2\iistime.tpl

Basically , this will do this report about statuses of URL requested  :

Status Requests
200 1541
302 89
401 11
403 61

The problem is that C:\newfile and  C:\Windows\System32\LogFiles\W3SVC1\ex100909.log are hard-coded – we need to modify every time… So PowerShell to the rescue (Ok, I could do a C# Console program – but

1. it is more fun this way – fun meaning I want to learn something new

2. the script could be modified easily

)

So the same command is written this way with arguments , in order to can be executed each time :

%2\LogParser -e:10 -i:IISW3C "SELECT cs-uri-stem as url, DIV(SUM(time-taken),1000) as Seconds, Count(time-taken) as Requests, DIV(Seconds ,Requests) as TimeExecuting   INTO %2\%4 FROM   %5\*%1  GROUP BY cs-uri-stem Having SUM(time-taken)>0 and Seconds>0 order by Seconds   desc" -o:TPL -tpl:%2\iistime.tpl

But who will give arguments ( such as the system date ) ?Now powershell  to the rescue :

$namepc = (gc env:computername)
$a = get-date
$a = (get-date).AddDays(-1)
$allpath= Split-Path -Parent $MyInvocation.MyCommand.Path;
$logfolders = $env:WINDIR +"\system32\Logfiles\W3SVC*"
foreach($logfolder in  Get-ChildItem $logfolders)
{
$logfiles= $logfolder.FullName
Write-Host "parsing"  $logfiles
$log =  $a.ToString("yyMMdd") + ".log"
$process = [Diagnostics.Process]::Start($allpath + "\iis.bat" , $log + " "+ $allpath + " "+ $log + ".html" + " TIME" +$log + ".html" + " " + $logfiles)
$process.WaitForExit()
$content = "<h1>IIS REPORT " + $namepc  + "</H1>"
$content += (get-content ($allpath  + "\" + $log + ".html"))
$content += (get-content ($allpath  + "\TIME" + $log + ".html"))

$SmtpClient = new-object system.net.mail.smtpClient
$SmtpServer = "your server"
$SmtpClient.host = $SmtpServer

$mm = new-Object System.Net.Mail.MailMessage(“<a href="mailto:from@yourcompany.com">from@yourcompany.com</a>”,"<a href="mailto:to@yourcompany.com">to@yourcompany.com</a>")
$mm.Subject = "Report IIS " + $namepc
$mm.Body = $content
$mm.Body=$mm.Body.Replace("&lt;cmp&gt;",$namepc  )
$mm.IsBodyHtml = 1
$SmtpClient.Send($mm)
}

Explanation of code :
line 1: I take the computer name to put in the report
line 2 : take the date ( if you want the current date , just comment the line 3)
line 5 : I go to usual path to logfiles (did I say quick and dirty ?)
line 6 : get all W3SVC folders and iterate to send report
line 11 : launching the bat (that contains logparser command ) in order to parse  arguments
line 12 :waiting for the process to exit – in order to can send files.
line 13 to 15: get the output
line 17 to 27 : send the output by email

Homework :
1. Execute script on a system and modify if it does not work
2. Clean up the temporary files after sending email
3. Instead of sending an email, write into a database with the current date.

You can execute sc.bat at regular times ( such as 1:00 AM)

Here is the zip file with sources logiisparser

LogParser download : http://www.microsoft.com/downloads/en/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en

Powershell scripts : http://gallery.technet.microsoft.com/ScriptCenter/en-us/site/search?f[1].Type=SearchText&f[1].Value=internet&f[0].Value=applications&f[0].Type=RootCategory&f[0].Text=Applications&x=0&y=0

Logparser quick and dirty

Sometimes you must find information in text files. Many,many text files, like IIS logs or other custom non-regular formats.

I have a bot from http://www.imified.com/ – and I log the messages with log4net in text files, with another messages.

An entry looks like that :

System.ArgumentException: ;channel=private;botkey=<guid>;userkey=<guid>;user=name@yhaoo.com;network=Yahoo;msg=hello;step=1;value0=hello;to=asdasd

And there are multiple log files that I want to parse and find the email adresses to collect feedback from those persons that use my bot.

LogParser to the rescue! Download from http://www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en and use this command line

LOGPARSER “Select Text into a.csv from current* where Text like ‘%@%'” -i:TEXTLINE

Explanation of command :

Select Text into a.csv from current* where Text like ‘%@%’ –means find in files that begin with current(current*) all text that contains emails ( ‘%@%’) and put in file a.csv  the results.

-i:TEXTLINE – means the format is text

What can be more simple ?

(Ok, for finding the user name I had to resort to excel, to remove duplicates … )

More I think it is fast enough : for parsing 114 files with 58.8 MB (PC with a 2GB RAM + 7200 RPM ) the results are :

Statistics:
———–
Elements processed: 487176
Elements output:    1044
Execution time:     7.69 seconds

Also logparser can be used for more than text files :

http://support.microsoft.com/kb/910447

http://www.stevebunting.org/udpd4n6/forensics/logparser.htm

More, it can be as a COM DLL in every .NET project, making it a usefull tool . See

http://www.codeproject.com/KB/recipes/SimpleLogParse.aspx

Next time I will show the using Powershell in combination with LogParser.

file helpers

More than one time you need to let users / application import bulk data in your system .

I have had many cases  – data is old data from old software, that is exported in some text files, usually csv.

In order to import fast the data, I used FileHelpers from http://www.filehelpers.com/

Let’s say you have the following data for an car evidence :

Date, StartMiles, Destination, EndMiles

And the data comes from an Excel  that has those columns. The requirements is the user can copy/paste from Excel data

When they copy the data comes in this form

01/01/2010     1000 Washington 1550

02/01/2010     1550 Dallas 2550

and so on.

It is clear that you :

  1. have a class with Date, StartMiles, Destination, EndMiles
  2. accomodate for space – but how to perform various data separator ( we suppose that first is the day, then comes the month).

Now the code in logical steps :

1. Accomodate for dates :

internal class ConvertDate : ConverterBase
{

        /// <summary>
        /// different forms for date separator : . or / or space
        /// </summary>
        /// <param name="from">the string format of date - first the day</param>
        /// <returns></returns>

        public override object StringToField(string from)
        {
            DateTime dt;

            if (DateTime.TryParseExact(from, "dd.MM.yyyy", null, DateTimeStyles.None, out dt))
                return dt;

            if (DateTime.TryParseExact(from, "dd/MM/yyyy", null, DateTimeStyles.None, out dt))
                return dt;

            if (DateTime.TryParseExact(from, "dd MM yyyy", null, DateTimeStyles.None, out dt))
                return dt;

            throw new ArgumentException("can not make a date from " + from, "from");

        }
}

2. Create the class that will hold one record:

    [IgnoreEmptyLines(true)]
    [DelimitedRecord(",")]
    internal class DestinationReader
    {
        //[FieldConverter(ConverterKind.Date,"dd.MM.yyyy")]
        [FieldConverter(typeof(ConvertDate))]
        public DateTime Date;
        [FieldConverter(ConverterKind.Int32)]
        public int StartMiles;

        [FieldQuoted(QuoteMode.OptionalForBoth)]
        public string Destination;

        [FieldConverter(ConverterKind.Int32)]
        public int  EndMiles;
    }

3. Now read the entire string:

            string Text = text that comes from the user
            string TextDelim = Text.Substring(10, 1);// the date has 10 chars - so the eleven is the separator
            while (Text.IndexOf(TextDelim + TextDelim) > 0)//consolidate for 2 delimiters
            {
                Text = Text.Replace(TextDelim + TextDelim, TextDelim);
            }
            DelimitedFileEngine<DestinationReader> flh=new DelimitedFileEngine<DestinationReader>();
            flh.Options.Delimiter = TextDelim;

            var data =flh.ReadString(Text);

In data you have a list of DestinationReader
So for any structured import of data use FileHelpers from http://www.filehelpers.com/

Html Agility Pack

This is the ultimate reference of reading web pages.

IF you want to do it yourself , you can try with WebRequest , http://msdn.microsoft.com/en-us/library/system.net.webrequest.aspx . But not all HTML is an XML – so you must find a method to parse. But am I the only one ?No – so I found the  HTML Agility Pack , http://www.codeplex.com/htmlagilitypack , that knows how to transform HTML in XML. The code is easy :

HtmlWeb hw = new HtmlWeb();
hw.AutoDetectEncoding = true;
HtmlDocument  doc = hw.Load(Url);
HtmlNode NodeRoot = doc.DocumentNode;

And from NodeRoot you can start XPATH with SelectNodes . Try it – it is awesome!