When tansmitting an array of data to an application , usually the application transfers all data – or in chunks – if it is a large numbers ( page 1, page 2 and so on). Both are having drawbacks
- all data from start means that the time to see the data increases with the number of data
- paging the data means to show the user the number of pages and an action from the user to go further to the next page.
What if I can use IASyncEnumerable to push data one by one to the GUI ?
So – those are the steps for the backend (.NET ) and frontend( Angular ) to pass the transfer from array to one by one ( IASyncEnumerable)
Backend
I have a function that loads the data from database and transforms into an array . Also caches the data
private async Task<TILT_Note_Table[]?> LatestTILTs(string urlPart, int numberTILTS){
if (cache.TryGetValue<TILT_Note_Table[]>(urlPart, out var result))
{
return result;
}
//find id from urlPart - if not found , return null
//caches the data and returns the array
}
Also a controller that sends the data
public async Task<ActionResult<TILT_Note_Table[]?>> LatestTILTs(string urlPart, int numberTILTS, [FromServices] ISearchDataTILT_Note searchNotes)
{
var data = await publicTILTS.LatestTILTs(urlPart,numberTILTS);
if (data== null)
{
return new NotFoundObjectResult($"cannot find {urlPart}");
}
return data;
}
Also some tests that verifies that when I post a new tilt, the numbers is 1
Step 1: transform from array to IASyncEnumerable
transformation of the main function
private async Task<IAsyncEnumerable<TILT_Note_Table>?> privateLatestTILTs(string urlPart, int numberTILTS)
{
if (cache.TryGetValue<TILT_Note_Table[]>(urlPart, out var result))
{
return result.ToAsyncEnumerable();// modification here
}
//find id from urlPart - if not found , return null
//caches the data and returns the array.ToAsyncEnumerable();
}
transformation of the controller- just add IAsyncEnumerable
public async Task<ActionResult<IAsyncEnumerable<TILT_Note_Table[]>?>> LatestTILTs(string urlPart, int numberTILTS, [FromServices] ISearchDataTILT_Note searchNotes)
{
var data = await publicTILTS.LatestTILTs(urlPart,numberTILTS);
return new NotFoundObjectResult($"cannot find {urlPart}");
}
return Ok(data);
}
Also for the tests you can add .ToArrayAsync()
Step 2: Get rid of the Task
So now the obtaining of TILTS looks like this
private async IAsyncEnumerable<TILT_Note_Table> privateLatestTILTs(string urlPart, int numberTILTS)
{
if (cache.TryGetValue<TILT_Note_Table[]>(urlPart, out var result))
{
//why I cant return result.ToAsyncEnumerable() ?
await foreach (var item in result.ToAsyncEnumerable())
{
await Task.Delay(1000);
yield return item;
}
}
//same with retrieving data
//when sending back data, we have to send one by one , as for the caching
The controller looks pretty much the same
[HttpGet("{urlPart}/{numberTILTS}")]
public ActionResult<IAsyncEnumerable<TILT_Note_Table>> LatestTILTs(string urlPart, int numberTILTS, [FromServices] ISearchDataTILT_Note searchNotes)
{
var data = publicTILTS.LatestTILTs(urlPart,numberTILTS);
if (data== null)
{
return new NotFoundObjectResult($"cannot find {urlPart}");
}
return Ok(data);
}
And this is the modification for the backend . If you want to see in action , open your browser to https://tiltwebapp.azurewebsites.net/api/PublicTILTs/LatestTILTs/ignatandrei/100000 and see TILTS how they come one by one
Frontend
In Angular I have obtained the whole array at once .
public getTilts(id:string, nr:number): Observable<TILT[]>{
return this.http.get<TILT[]>(this.baseUrl+'PublicTILTs/LatestTILTs/'+id + '/'+nr)
.pipe(
tap(it=>console.log('received',it)),
map(arr=>arr.map(it=>new TILT(it)))
)
;
}
Now we are obtaining one by one – how to let the page knows that it has an array instead of modifying also the page ?
( Yes, the display could be modified to accumulate – but I want minimal modifications to the page)
To obtain one by one we display the fetch
//https://gist.github.com/markotny/d21ef4e1af3d6ea5332b948c9c9987e5
//https://medium.com/@markotny97/streaming-iasyncenumerable-to-rxjs-front-end-8eb5323ca282
public fromFetchStream<T>(input: RequestInfo, init?: RequestInit): Observable<T> {
return new Observable<T>(observer => {
const controller = new AbortController();
fetch(input, { ...init, signal: controller.signal })
.then(async response => {
const reader = response.body?.getReader();
if (!reader) {
throw new Error('Failed to read response');
}
const decoder = new JsonStreamDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) break;
if (!value) continue;
decoder.decodeChunk<T>(value, item => observer.next(item));
}
observer.complete();
reader.releaseLock();
})
.catch(err => observer.error(err));
return () => controller.abort();
});
}
And for modifying the function to have an array, instead of just one, RXJS scan to the rescue
public getTilts(id:string, nr:number): Observable<TILT[]>{
return this.fromFetchStream<TILT>(this.baseUrl+'PublicTILTs/LatestTILTs/'+id + '/'+nr)
.pipe(
tap(it=>console.log('received',it)),
map(it=>new TILT(it)),
scan((acc,value)=>[...acc, value], [] as TILT[])
);
}
You can see the final result ( with 1 sec delay between tilts, to be visible ) here http://tiltwebapp.azurewebsites.net/AngTilt/tilt/public/ignatandrei
