Could you spot the problem in transforming one code that is sequential into a Task ( threaded ) one?
First , this is the sequential one:
var ret = new List<Indicator>();
var jsonData = await data.JsonData();
var jo = JArray.Parse(jsonData);
var page = jo[0].ToObject<Pagination>();
var array = jo[1].ToObject<Indicator[]>();
ret.AddRange(array);
var currentPage = 1;
while (currentPage < page.pages)
{
currentPage++;
jsonData = await data.JsonData(currentPage);
jo = JArray.Parse(jsonData);
array = jo[1].ToObject<Indicator[]>();
ret.AddRange(array);
}
Debug.Assert(ret.Count == page.total, $"{nameof(ret.Count)} : {ret.Count} should be equal {nameof(page.total)} : {page.total}");
return ret.ToArray();
This is the modified with task:
var jsonData = await data.JsonData();
var jo = JArray.Parse(jsonData);
var page = jo[0].ToObject<Pagination>();
var array = jo[1].ToObject<Indicator[]>();
var ret = new List<Indicator>(array);
var currentPage = 1;
var downloads = new List<Task>();
while (currentPage < page.pages)
{
currentPage++;
var itemPage = currentPage;
var task = data.JsonData(itemPage)
.ContinueWith(it =>
{
var data = JArray.Parse(it.Result);
var pageNr = data[0].ToObject<Pagination>();
var indicators = data[1].ToObject<Indicator[]>();
ret.AddRange(indicators);
}
);
downloads.Add(task);
}
await Task.WhenAll(downloads);
Debug.Assert(ret.Count == page.total, $"{nameof(ret.Count)} : {ret.Count} should be equal {nameof(page.total)} : {page.total}");
return ret.ToArray();
Ok. If you do not know , here is a helper:
Sometimes, at line
ret.AddRange(indicators)
it gives the error:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. —> System.ArgumentException: Source array was not long enough. Check srcIndex and length, and the array’s lower bounds.
at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
at System.Collections.Generic.List`1.set_Capacity(Int32 value)
at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
Ok. I let you think.
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
The answer is simple : List < T > is not Thread safe. If you have small quantities of data, does not matter. I was having 16174 records – and sometimes , when multiple threads have come to .AddRange , the internal re-dimensioning of array it does not fulfill the task ( imagine one thread redim to 100, then other 3 threads coming with request of 50+ data)
Solution 1.
Use ConcurrentBag< T > instead of List < T > . This is thread safe and does the trick
Solution 2.
Add to the array in the final of data ( not use ContinueWith) . The code is slightly low performant
var jsonData = await data.JsonData();
var jo = JArray.Parse(jsonData);
var page = jo[0].ToObject<Pagination>();
var array = jo[1].ToObject<Indicator[]>();
var ret = new List<Indicator>(array);
var currentPage = 1;
var downloads = new List<Task<string>>();
while (currentPage < page.pages)
{
currentPage++;
var itemPage = currentPage;
var task = data.JsonData(itemPage);
downloads.Add(task);
}
await Task.WhenAll(downloads);
foreach (var t in downloads)
{
var data = JArray.Parse(t.Result);
//var pageNr = data[0].ToObject<Pagination>();
var indicators = data[1].ToObject<Indicator[]>();
//Console.WriteLine($"reading {pageNr.page} with {item.Length}");
ret.AddRange(indicators);
}
//Console.WriteLine($"total records {ret.Count}");
Debug.Assert(ret.Count == page.total, $"{nameof(ret.Count)} : {ret.Count} should be equal {nameof(page.total)} : {page.total}");
return ret.ToArray();
You will find the code at IndicatorRepository in https://github.com/ignatandrei/WorldBankAPi
