Optimizing EF4 and EDMGen2

Just reading (from Adrian Florea links ) how the EF4 could be optimized : http://www.codeproject.com/KB/database/PerfEntityFramework.aspx

First modification is to “Pre-generated Your View”. For this you must have .ssdl, .csdl, and .msl files – so you change the “Metadata Artifact Processing property to: Copy to Output Directory.”. Then you process the .ssdl, .csdl, and .msl with edmgen in order to can have the  views.

Until here , all ok.

But, in the next advice, is to keep “Metadata Artifact Processing property to: Embed in Output Assembly.”

One solution is to put “Metadata Artifact Processing property to: Copy to Output Directory.” , compile, put again “Metadata Artifact Processing property to: Embed in Output Assembly.” and compile again. But , if you change the edmx (add fields or tables ) you must redo the operation  – so you will have more things to do(if you remember)

A solution is to build them on the pre-build step .But how to generate the .ssdl, .csdl, and .msl files  ?

Edmgen2 , http://code.msdn.microsoft.com/EdmGen2 , to the rescue. Download, put into a lib folder under your solution folder and put a pre-build like this :

$(SolutionDir)lib\edmgen2\edmgen2 /FromEdmx $(ProjectDir)prod.edmx”

“%windir%\Microsoft.NET\Framework\v4.0.30319\EdmGen.exe” /mode:ViewGeneration /language:CSharp /nologo “/inssdl:prod.ssdl” “/incsdl:prod.csdl”   “/inmsl:prod.msl” “/outviews:$(ProjectDir)prod.Views.cs”

What you must change on your project?

1. IF you are on 64 bit, change Framework to Framework64

2. change the prod with your edmx name.

What is the performance?

Tested by loading a table with 301 rows by doing the steps :

1. open connection, load all table in objects(POCO), closing connection

2. open connection , find object with PK = 1, closing connection

3. open connection ,  loading 1 tables with 2 related (include ) , closing connection

The results are in milliseconds:

Without pre-compiled views

LoadTable LoadID LoadMultiple Total Time
579 800 172 1551
563 755 171 1489
559 754 169 1482
568 762 240 1570

With pre-compiled views:

LoadTable LoadID LoadMultiple Total Time
606 807 183 1596
509 706 177 1392
852 137 192 1181
530 733 221 1484
523 722 183 1428

The average / min / max results:

average max min
without 1523 1570 1482
with 1413.25 1596 1181

In the next picture the smaller the duration(milliseconds), is the better :

image

Conclusions:

1.  For the average and min the difference is 7%, respectively 20%. Please remember we are using only 3 queries.

For the max, it is vey curious : the with is more than without. The penalty is 1%, I think that is a measuring error ? – or maybe not. However , the penalty is small comparing with others.

2. Very curious, find after ID, with a table with 301 rows, took longer than loading the whole table.However, did not take into accound finding in the list the object( it is in memory also)

3. It may worth to add the pre-build step shown before to pre-compile views.

Links :

http://www.codeproject.com/KB/database/PerfEntityFramework.aspx

http://msdn.microsoft.com/en-us/library/bb896240.aspx

EF Profiler and investigating object context

Summary : EF Profiler worth his money !

Long Description :

I have finished a application with Entity Framework and POCO – and custom generated (.tt) files.

Now it was time to investigate performance  – and it was a terrific problem.

How EF Prof helped me :

1.  Displaying number of ObjectContext created and disposed in a Unit of Work and / or web page.

image

2. Displaying problem with the ObjectCOntext : using same context from multiple threads, alerts about code like  here :

image

The red dot is about querying the database from the view. The gray one is about selecting a whole table without selecting top 1000 rows(OK, it was a dictionary table, like the list of countries)

3. When you want caching on your site , you can fast see this by seeing the difference between the number of queries and/or number of ObjectContext for the first time and the second one ( first time : 10, second time:2 or 0 😉 )

4. See most expensive queries as time – usually this is a good option to put an index on the “where” columns.More , you can see also non-unique queries (that you perform more than 1 time)

5. Investigate easily all queries. Know what you want to optimize and where.

Improvements  :

1. Say where is the ObjectContext that is not closing.

Alternatives:

How to: View the Store Commands : http://msdn.microsoft.com/en-us/library/bb896348.aspx

Sql Profiler : http://msdn.microsoft.com/en-us/library/aa173918%28SQL.80%29.aspx

(free ones :

http://sites.google.com/site/sqlprofiler/

http://code.google.com/p/sqlexpressprofiler/

)

But the job will be done by yourself!

Error intercepting in MVC when saving data

I have seen many times in MVC the following code , when doing a post and saving data :

try
{
        //code to save to database
}
catch
{
         ModelState.AddRuleViolations(TheModel.GetRuleViolations());
}

Why is not good ?
I will do a simple example : if some error occurs on the database level ( such as simple unique index on a name column ) or even some error occurs whenestablishing the database connection – you will never see what’s happening. And you will have no chance to repair the problem
A slighty better code is :

try
{
     ModelState.AddRuleViolations(TheModel.GetRuleViolations());
     if(ModelState.Count == 0)// no error in the model
     {
     //code to save to database
     }
}
catch(Exception ex)
{
      ModelState.AddModelError("", ex.Message);
}

Please see also how to do logging to the exception at
http://msprogrammer.serviciipeweb.ro/2010/04/19/logging-and-instrumentation/

path.combine idiosyncrasies

I have this small program :

var s = Path.Combine(@"E:\andrei", UnknownVar);
Console.WriteLine(Directory.GetFiles(s).FirstOrDefault());

In what conditions , if I have an “C:\andrei” folder with a single file , the output will be the name of the file ?
And where you want to be aware of this behaviour ?
Hint : The output is : \andrei\New Text Document.txt

A programmer day with “That assembly does not allow partially trusted callers”

I make a website, works on local IIS. Performs user registration  + uploading some zip files + generating custom feeds for those zip files.

Using SharpZipLib_0860_Bin ( to unzip file ) , StringTemplate.dll ( to perform custom feed generation ) and NUnit-2.5.7.10213 ( to perform tests).

So far ,so good. Moving into production . User registration works, upload works, trying feeds

“That assembly does not allow partially trusted callers”
Feeds were generated with StringTemplate- it was custom feeds ;-).

Searching , talking with hosting – seeing that this happens if your asp.net does not run under full trust , but under “Medium trust”.

Normally the provider does not want to change and send me advice to put AllowPartiallyTrustedCallersAttribute (APTCA) on the class:
http://msdn.microsoft.com/en-us/library/system.security.allowpartiallytrustedcallersattribute.aspx

This does not work without signing with a strong name – so I generate the snk file , sign mine assemblies, re-deploy. Same error:

“That assembly does not allow partially trusted callers”

That is normally and I have suspected – because I have to sign the stringtemplate, not mine dll.

Trying to see if the stringtemplate is signed  -it is! When put AllowPartiallyTrustedCallersAttribute  and trying to re-build – failed because I do not have his snk

Then, in disperation, I remove the snk from stringtemplate and re-make my project.

And now –

System.Security.SecurityException: Request failed.

Antlr.StringTemplate.FileSystemTemplateLoader.InternalLoadTemplateContents(String templateName) +0

That means that the files to read contents to generate feeds are not accesible to read. But, oh my dear .NET framework :could you not tell from the beginning ?

So, possible solutions to identify causes :  identify the assembly that  causes the harm and either

1. put a snk to this assembly and for all assemblies in superior chaining

2. remove the snk from this assembly

In either case, you should see what the error is.
So how I managed to solve ?
1. Put Reflector and dissasemble the source.
2. Re-compile without snk.
3. Put in Web.Config 4. Compile and run with debug
Now it stops at :
string templateText = InternalLoadTemplateContents(templateName);

This , in StringTemplate class, was

  protected override string InternalLoadTemplateContents(string templateName)
        {
            string templateText = null;
            string templateLocation = null;

            try
            {
                //templateLocation = Path.Combine(LocationRoot, GetLocationFromTemplateName(templateName));
                templateLocation = string.Format("{0}/{1}", LocationRoot, GetLocationFromTemplateName(templateName)).Replace('\\', '/');
                StreamReader br;
                try
                {
                    br = new StreamReader(templateLocation, encoding);
                }
                catch(FileNotFoundException)
                {
                    return null;
                }
                catch(DirectoryNotFoundException)
                {
                    return null;
                }
                catch(Exception ex)
                {
                    throw new TemplateLoadException("Cannot open template file: " + templateLocation, ex);
                }

                try 
                {
                    templateText = br.ReadToEnd();
                    if ((templateText != null) && (templateText.Length > 0))
                    {
                        //templateText = templateText.Trim();

                        if (filesWatcher == null)
                        {
                            filesWatcher = new FileSystemWatcher(LocationRoot, "*.st");
                            //filesWatcher.InternalBufferSize *= 2;
                            filesWatcher.NotifyFilter = 
                                NotifyFilters.LastWrite 
                                | NotifyFilters.Attributes
                                | NotifyFilters.Security 
                                | NotifyFilters.Size 
                                | NotifyFilters.CreationTime 
                                | NotifyFilters.DirectoryName 
                                | NotifyFilters.FileName;
                            filesWatcher.IncludeSubdirectories = true;
                            filesWatcher.Changed += new FileSystemEventHandler(OnChanged);
                            filesWatcher.Deleted += new FileSystemEventHandler(OnChanged);
                            filesWatcher.Created += new FileSystemEventHandler(OnChanged);
                            filesWatcher.Renamed += new RenamedEventHandler(OnRenamed);
                            filesWatcher.EnableRaisingEvents = true;
                        }
                    }
                    fileSet.Remove(templateLocation);
                }
                finally
                {
                    if (br != null) ((IDisposable)br).Dispose();
                    br = null;
                }
            }
            catch (ArgumentException ex) 
            {
                string message;
                if (templateText == null)
                    message = string.Format("Invalid file character encoding: {0}", encoding);
                else
                    message = string.Format("The location root '{0}' and/or the template name '{1}' is invalid.", LocationRoot, templateName);

                throw new TemplateLoadException(message, ex);
            }
            catch (IOException ex) 
            {
                throw new TemplateLoadException("Cannot close template file: " + templateLocation, ex);
            }
            return templateText;
        }

Does something ring a bell to you ?
Wait for an answer
Wait for an answer
Wait for an answer
Wait for an answer
Wait for an answer
Wait for an answer
Wait for an answer
Wait for an answer

Yes – FileSystemWatcher .
http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx
[IODescriptionAttribute(“FileSystemWatcherDesc”)]
[PermissionSetAttribute(SecurityAction.LinkDemand, Name = “FullTrust”)]
[PermissionSetAttribute(SecurityAction.InheritanceDemand, Name = “FullTrust”)]
public class FileSystemWatcher : Component,

Solution : Once defined my FileSystemTemplateLoader_MT – without FileSystemWatcher – all works well
So – the idea is : reflector, sources, find something strange. Remove, rebuild, re-test
(Thanks to OrcsWeb team for helping me on this problem !)

Find the Error

This code was written by me, in a late night moment, when copy paste seems the best option
It started this way :

public string ImportFeed(string URL)
        {
            string ret = "";
            XmlDocument xd = new XmlDocument();
            xd.Load(URL);            
            for(int i=0;i<xd.ChildNodes.Count;i++)
            {
                ret += xd.ChildNodes[i].Name;
            }
            return ret;
            
        }

So far , nothing special. Just open the url, read into an XML , return first nodes. Then I wanted more – next nodes. Nothing easier – why bother with recursion or proper variable names ? Just copy and paste :

public string ImportFeed(string URL)
        {
            string ret = "";
            XmlDocument xd = new XmlDocument();
            xd.Load(URL);            
            for(int i=0;i<xd.ChildNodes.Count;i++)
            {
                ret += xd.ChildNodes[i].Name;
                for (int j = 0; i < xd.ChildNodes[i].ChildNodes.Count; j++)
                {
                    ret += xd.ChildNodes[i].ChildNodes[j].Name;
                }
            }
            return ret;
            
        }

Could you spot the error ?
Hint : I have discovered after 10 seconds …but not looking at the code…

Five common mistakes for ASP.NET (MVC) accesing resources : css, js, images, ajax

To have once for all the link to show to people, because too much makes the same error again and again.

(From here – you can use ResolveUrl or Url.Content – it’s the same for me. I use ResolveUrl because I used first …)

Case 1 The image does not display

Please check you have the following :

<img src='<%= ResolveUrl("~/Content/images/YOURIMAGE.jpg" )%>'  alt="image" style="border:0" />

Case 2 The css does not show

Please check you have the following :

<%= string.Format("<link href='{0}' type='text/css' rel='stylesheet' />", ResolveUrl("~/Content/your.css")) %>

or

<style type="text/css">
@import '<%= ResolveUrl("~/Content/your.css") %>';

</style>

Case 3 The js does not execute (undefined error)

Please check you have the following :

<script type="text/javascript" src='<%= ResolveUrl("~/Scripts/yourjs.js")%>'></script>

Case 4 The js does execute in aspx page, but did not execute under js file

Please check you DO NOT have the following

<%

in the js file. The js file is not interpreted by the same engine as aspx, so can not interpret asp.net tags. Instead , add a parameter to your function : the path.

Simple example : Let’s say in aspx you have :

<script type=”text/javascript”>

function Generate(){

window.open(‘<% Url.Action(“About”)%>’);
}

</script>

and you call Generate();

When you put in .js file, please put this :

function Generate(url){

window.open(url);
}

and call like this :

Generate('<% Url.Action(“About”)%>');

Case 5 The ajax request gives you a 404 error.

Please ensure that you have

<%= ResolveUrl("~/path”)%>

and not

/path

Bonus 1: T4MVC , http://mvccontrib.codeplex.com/releases

Bonus 2: Edit the project file and put <MvcBuildViews>true</MvcBuildViews>

( short url: http://bit.ly/asp5Mistakes)

About programming – 7 points

So many times I see people saying : “I can not have time to follow the tutorials – but give me some fast acces to some tutorials that solve my problem”.

I have

  1. Programming is not about code – programming is about decisions that you do about what code to write.
  2. Programming can not be learned by one tutorial – find more , skip it the obvious and read more
  3. Programming is not an easy job to do : you must have  knowledge (about at least the language used), it takes divide et impera and many more (testing being the obvious forgotten one …)
  4. Programming is a self-continuing learning : the products that you are using change – and I am more productive today with VS 2010 (and .NET4 ) than I was with VB  3 ( although VB 3 was EXCELLENT for that times )
  5. Programming is about experience : you can learn from others , but from yours mistakes , things that worked and right things ( my least : error to not de-normalize tables…)
  6. Programming is about keeping happy your technical leader ( or, if you are the technical leader, the project manager) . And I advise you to being one if you can, to see their problems.
  7. Finally  – there is not a royal way to programming  – take your time and enjoy!

Generating history trigger with EF , edmx and TT files

I have wrote in an older post ( http://msprogrammer.serviciipeweb.ro/2010/06/28/ef-automatic-history-of-table-and-t4-files-tt-files/ ) how to generate history code for tables . The easy solution was to create a tt file that track for the ObjectContext the SaveChanges for each table that has a “history” in name. the limitation is that , when you raise an sql command such as “update table ” you must load from database a lot of rows for this….

Now I want to show the same thing, but generating triggers in database for that ! I start also from edmx file and with a template stealed from http://forums.asp.net/p/1599616/4083198.aspx ( to have type of fields in the database ) and another stealed and modified from StackOverflow(to generate trigger for after insert , update, delete) I manage to have a solution to generate sql tables and trigger code.

When is that good ?At the beginning stages of a project when the table structure changes by adding a new parameter.

Sample code generated for table Notes(ID, TextNote, PersonID)

print 'Create table Notes_History ';
Create Table Notes_History(
		ID_Notes_History BIGINT IDENTITY NOT NULL
		,History_Action varchar(50)
		,History_From varchar(100) default HOST_NAME()
		,History_User varchar(50) default SYSTEM_USER
		,History_Date varchar(50) default  getdate()
	   ,Id  int   NOT NULL
	   ,Textnote  nvarchar (255)  NULL
	   ,PersonId  int   NULL
	)
print 'end Create table Notes_History ';
GO
print 'create trigger for Notes'
GO
CREATE TRIGGER dbo.TR_IUP_Notes
   ON  Notes
   AFTER INSERT, UPDATE, DELETE
AS
BEGIN
    SET NOCOUNT ON;
	DECLARE @Ins int

DECLARE @Del int

SELECT @Ins = Count(*) FROM inserted

SELECT @Del = Count(*) FROM deleted
if(@Ins  + @Del = 0)
	return;

declare @operation varchar(50)
	set @operation ='update';
	if(@ins < @del)
		set @operation ='delete';
	if(@ins > @del)
		set @operation ='insert';

	if(@ins <= @del)
	begin
		INSERT INTO Notes_History(History_Action ,Id,Textnote,PersonId)
			select @operation ,Id,Textnote,PersonId from deleted
	end
    else
	begin
    INSERT INTO Notes_History(History_Action ,Id,Textnote,PersonId)
			select @operation ,Id,Textnote,PersonId from inserted
	end
END

The drawback of the code : the user that executes is not always the logged sql server user…
Anyway, what you have to do to use this automatically generated history/audit for tables ?

download historysql and modify

string inputFile = @”Model1.edmx”;

from historysql.tt to your edmx name. Then take the generated code and execute in sql server.