Task and generic list

Could you spot the problem in transforming one code that is sequential into a Task ( threaded ) one?

First , this is the sequential one:

 

            var ret = new List<Indicator>();
            var jsonData = await data.JsonData();
            var jo = JArray.Parse(jsonData);
            var page = jo[0].ToObject<Pagination>();
            var array = jo[1].ToObject<Indicator[]>();
            ret.AddRange(array);
            var currentPage = 1;
            while (currentPage < page.pages)
            {
                currentPage++;
                jsonData = await data.JsonData(currentPage);
                jo = JArray.Parse(jsonData);
                array = jo[1].ToObject<Indicator[]>();
                ret.AddRange(array);

            }
            Debug.Assert(ret.Count == page.total, $"{nameof(ret.Count)} : {ret.Count} should be equal {nameof(page.total)} : {page.total}");
            return ret.ToArray();


 

This is the modified with task:

 

            var jsonData = await data.JsonData();
            var jo = JArray.Parse(jsonData);
            var page = jo[0].ToObject<Pagination>();
            var array = jo[1].ToObject<Indicator[]>();
            var ret = new List<Indicator>(array);

            var currentPage = 1;
            var downloads = new List<Task>();
            while (currentPage < page.pages)
            {
                currentPage++;
                var itemPage = currentPage;
                var task = data.JsonData(itemPage)
                    .ContinueWith(it =>
                    {
                        var data = JArray.Parse(it.Result);
                        var pageNr = data[0].ToObject<Pagination>();
                        var indicators = data[1].ToObject<Indicator[]>();
                       ret.AddRange(indicators);

                    }
                );

                downloads.Add(task);
            }
            await Task.WhenAll(downloads);
            Debug.Assert(ret.Count == page.total, $"{nameof(ret.Count)} : {ret.Count} should be equal {nameof(page.total)} : {page.total}");
            return ret.ToArray();

Ok. If you do not know , here is a helper:
Sometimes, at line

 ret.AddRange(indicators)

it gives the error:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. —> System.ArgumentException: Source array was not long enough. Check srcIndex and length, and the array’s lower bounds.
at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
at System.Collections.Generic.List`1.set_Capacity(Int32 value)
at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)

Ok. I let you think.
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..

The answer is simple : List < T > is not Thread safe. If you have small quantities of data, does not matter. I was having 16174 records – and sometimes , when multiple threads have come to .AddRange , the internal re-dimensioning of array it does not fulfill the task ( imagine one thread redim to 100, then other 3 threads coming with request of 50+ data)

Solution 1.

Use ConcurrentBag< T > instead of List < T > . This is thread safe and does the trick

Solution 2.
Add to the array in the final of data ( not use ContinueWith) . The code is slightly low performant

           var jsonData = await data.JsonData();
            var jo = JArray.Parse(jsonData);
            var page = jo[0].ToObject<Pagination>();
            var array = jo[1].ToObject<Indicator[]>();
            var ret = new List<Indicator>(array);

            var currentPage = 1;
            var downloads = new List<Task<string>>();
            while (currentPage < page.pages)
            {
                currentPage++;
                var itemPage = currentPage;
                var task = data.JsonData(itemPage);
                downloads.Add(task);
            }
            await Task.WhenAll(downloads);

            foreach (var t in downloads)
            {
                var data = JArray.Parse(t.Result);
                //var pageNr = data[0].ToObject<Pagination>();
                var indicators = data[1].ToObject<Indicator[]>();
                //Console.WriteLine($"reading {pageNr.page} with {item.Length}");
                ret.AddRange(indicators);

            }
            //Console.WriteLine($"total records {ret.Count}");
            Debug.Assert(ret.Count == page.total, $"{nameof(ret.Count)} : {ret.Count} should be equal {nameof(page.total)} : {page.total}");
            return ret.ToArray();

You will find the code at IndicatorRepository in https://github.com/ignatandrei/WorldBankAPi

InternalsVisibleTo usage

I do not know if you know  InternalsVisibleToAttribute

https://msdn.microsoft.com/en-us/library/system.runtime.compilerservices.internalsvisibletoattribute%28v=vs.110%29.aspx?f=255&MSPPError=-2147217396  .

I have used on a project https://github.com/ignatandrei/WorldBankAPi , that deals with World Bank API to get information

Let’s take an example: Countries: http://api.worldbank.org/countries?format=json

I have had a class CountriesRepository, that deals with transforming countries Json from WorldBank to Country classes. How can I test it WITHOUT relying on reading data from the internet ?

Simple:

1. Make an interface to grab JSON , IJSonData

    public interface IJsonData
    {
        Task<string> JsonData(int page = 1);
    }

2. Make a class JSONCountry that reads JSON from HTTP and implements IJSonData

3.  Make the CountryRepository class with 2 constructors  – one with IJSonData and the default calling JSONCountry

[assembly: InternalsVisibleTo("WorldBank.Test")]
namespace WorldBank.Repository
{
    public class CountriesRepository
    {
        private IJsonData data;
        public CountriesRepository():this(new JsonCountries())
        {

        }
//this is internal - could not see by outside projects with exception of WorldBank.Test
//see InternalsVisibleTo above
        internal CountriesRepository(IJsonData data)
        {
            this.data = data;
        }
    }
}

Optional: when testing manually, save the json into same files on the hard disk – you will need for point 5

4. The code that will use CountryRepository in official way will use the default – will not see the one with IJSonData

var c = new CountriesRepository();
var data = c.GetCountries().Result;

5. The testing code will define another class JSONFromHard that implements IJSonData and reads data from Hard

class JsonFromHard : IJsonData
{
//code to read files from disk 
// you have saved those when testing manually 
//see point 2 the optional part 
}

6. Because I have defined InternalsVisibleToAttribute to the WorldBank.Test, the Test dll can see the constructor with IJSonData  and then it class with JSONFromHard   – and I have no dependecy on the Http

 

public void GetAndInterpretData()
        {
            //uses [assembly: InternalsVisibleTo("WorldBank.Test")]
            var c = new CountriesRepository(new JsonFromHard("Countries"));
            var data = c.GetCountries().Result;
            Assert.Equal(218, data.Length);
        }

You can download the project from https://github.com/ignatandrei/WorldBankAPi  to see working.

Maintaining history of data

 

What is this about?

I tend to quote Fowler: “Usually when we see properties on a class, they represent questions we can ask of an object now. However there are times when we don’t want to just ask questions about a property of an object now, we also want to ask these questions about some point in the past when things may have changed. “

For example , we want to track every change that occurs at an Employee ( changing name, salary) and see the previous versions( what was the salary 2 years ago ?)

So what are the software solutions to do an audit log ? First, we should have a history table for each table that we want to maintain an audit log. E.g. Employee => EmployeeHstory

Solution 1 : Coding / Event Sourcing

That means code should maintain history himself – i.e. adding to the history table every modifications.

For Entity Framework this can be done easy, by overiding SaveChanges.

Demo at https://youtu.be/YZ9GDjFApns 

Solution 2 : Table Triggers

This can be done from database . There are triggers for insert , update, delete – and those triggers insert the data in the history table

Demo at https://youtu.be/9xsvCJ-1f7Q

Solution 3, Temporal Tables, Microsot  Sql Server 2016 :

Documentation at https://docs.microsoft.com/en-us/sql/relational-databases/tables/temporal-tables 

Limitations at https://docs.microsoft.com/en-us/sql/relational-databases/tables/temporal-table-considerations-and-limitations 

Demo at https://youtu.be/1Q3DmXJI5h8

Powershell vs console application

I have decided to make the utilities that I use in form of Powershell applications instead of Windows / Console applications

Pros:

There are simpler to use  – just run the powershell

The parsing of command line is powerfull in powershell

There are simpler to maintain ( just text scripts files)

There are no big dependencies( .NET Core/ .NET 4.x  to be installed)

This is a good tool under  my toolbelt

Everything you do in .NET you can do in Powershell ( sometimes it is much harder in .NET rather then powershell)

 

The repository is https://github.com/ignatandrei/PowershellUtils and the first item is to move the big files from C: into another drive to free space by making a junction dir ( e.g. C:\ProgramData\Package Cache and C:\Windows\Installer )