TILT-Passing to IAsyncEnumerable instead of array–part 25

When tansmitting an array of data to an application , usually the application transfers all data – or in chunks – if it is a large numbers ( page 1, page 2 and so on). Both are having drawbacks

• all data from start means that the time to see the data increases with the number of data
• paging the data means to show the user the number of pages and an action from the user to go further to the next page.

What if I can use IASyncEnumerable to push data one by one to the GUI ?

So – those are the steps for the backend (.NET ) and frontend( Angular ) to pass the transfer from array to one by one ( IASyncEnumerable)

Backend

I have a function that loads the data from database and transforms into an array . Also caches the data

private async Task<TILT_Note_Table[]?> LatestTILTs(string urlPart, int numberTILTS){
    if (cache.TryGetValue<TILT_Note_Table[]>(urlPart, out var result))
    {
        return result;
    }
    //find id from urlPart - if not found , return null
    //caches the data and returns the array
}

Also a controller that sends the data

 public async Task<ActionResult<TILT_Note_Table[]?>> LatestTILTs(string urlPart, int numberTILTS, [FromServices] ISearchDataTILT_Note searchNotes)
{
    var data = await publicTILTS.LatestTILTs(urlPart,numberTILTS);
    if (data== null)
    {
        return new NotFoundObjectResult($"cannot find {urlPart}");
    }
    return data;
}

Also some tests that verifies that when I post a new tilt, the numbers is 1

Step 1: transform from array to IASyncEnumerable

transformation of the main function

 private async Task<IAsyncEnumerable<TILT_Note_Table>?> privateLatestTILTs(string urlPart, int numberTILTS)
        {
            if (cache.TryGetValue<TILT_Note_Table[]>(urlPart, out var result))
            {
                return result.ToAsyncEnumerable();// modification here
            }
    //find id from urlPart - if not found , return null
    //caches the data and returns the array.ToAsyncEnumerable();
        }

transformation of the controller- just add IAsyncEnumerable

 public async Task<ActionResult<IAsyncEnumerable<TILT_Note_Table[]>?>> LatestTILTs(string urlPart, int numberTILTS, [FromServices] ISearchDataTILT_Note searchNotes)
    {
        var data = await publicTILTS.LatestTILTs(urlPart,numberTILTS);
            return new NotFoundObjectResult($"cannot find {urlPart}");
        }
        return Ok(data);
    }

Also for the tests you can add .ToArrayAsync()

Step 2: Get rid of the Task

So now the obtaining of TILTS looks like this

private async IAsyncEnumerable<TILT_Note_Table> privateLatestTILTs(string urlPart, int numberTILTS)
{
    if (cache.TryGetValue<TILT_Note_Table[]>(urlPart, out var result))
    {
        //why I cant return result.ToAsyncEnumerable() ?
        await foreach (var item in result.ToAsyncEnumerable())
        {
            await Task.Delay(1000);
            yield return item;
        }
    }
    //same with retrieving data
    //when sending back data, we have to send one by one , as for the caching

The controller looks pretty much the same

[HttpGet("{urlPart}/{numberTILTS}")]
public ActionResult<IAsyncEnumerable<TILT_Note_Table>> LatestTILTs(string urlPart, int numberTILTS, [FromServices] ISearchDataTILT_Note searchNotes)
{
    var data =  publicTILTS.LatestTILTs(urlPart,numberTILTS);
    
    if (data== null)
    {
        return new NotFoundObjectResult($"cannot find {urlPart}");
    }

    return Ok(data);
}

And this is the modification for the backend . If you want to see in action , open your browser to https://tiltwebapp.azurewebsites.net/api/PublicTILTs/LatestTILTs/ignatandrei/100000 and see TILTS how they come one by one

Frontend

In Angular I have obtained the whole array at once .

 public getTilts(id:string, nr:number): Observable<TILT[]>{
    return this.http.get<TILT[]>(this.baseUrl+'PublicTILTs/LatestTILTs/'+id + '/'+nr)
    .pipe(
      tap(it=>console.log('received',it)),
      map(arr=>arr.map(it=>new TILT(it)))
    )
    ;
 }

Now we are obtaining one by one – how to let the page knows that it has an array instead of modifying also the page ?

( Yes, the display could be modified to accumulate – but I want minimal modifications to the page)

To obtain one by one we display the fetch

 //https://gist.github.com/markotny/d21ef4e1af3d6ea5332b948c9c9987e5
  //https://medium.com/@markotny97/streaming-iasyncenumerable-to-rxjs-front-end-8eb5323ca282
  public fromFetchStream<T>(input: RequestInfo, init?: RequestInit): Observable<T> {
    return new Observable<T>(observer => {
      const controller = new AbortController();
      fetch(input, { ...init, signal: controller.signal })
        .then(async response => {
          const reader = response.body?.getReader();
          if (!reader) {
            throw new Error('Failed to read response');
          }
          const decoder = new JsonStreamDecoder();
          while (true) {
            const { done, value } = await reader.read();
            if (done) break;
            if (!value) continue;
            decoder.decodeChunk<T>(value, item => observer.next(item));
          }
          observer.complete();
          reader.releaseLock();
        })
        .catch(err => observer.error(err));
      return () => controller.abort();
    });
  }

And for modifying the function to have an array, instead of just one, RXJS scan to the rescue

 public getTilts(id:string, nr:number): Observable<TILT[]>{
    return this.fromFetchStream<TILT>(this.baseUrl+'PublicTILTs/LatestTILTs/'+id + '/'+nr)
    .pipe(
      tap(it=>console.log('received',it)),
      map(it=>new TILT(it)),
      scan((acc,value)=>[...acc, value], [] as TILT[])
    );  
}

You can see the final result ( with 1 sec delay between tilts, to be visible ) here http://tiltwebapp.azurewebsites.net/AngTilt/tilt/public/ignatandrei