Category: WebAPI2CLI

Zip the whole ASP.NET Core application

WebAPI2CLI

This is a part of the series where about how I made the WebAPI2CLI - Execute ASP.NET Core WebAPI from Command Line
Source code on https://github.com/ignatandrei/webAPI2CLI/
1WebAPI2CLI - Description
2WebAPI2CLI- Organization
3WebAPI2CLI - implementing
4WebAPI2CLI - tests
5WebAPI2CLI - Devops and CI/CD
6WebAPI2CLI - documentation
7WebAPI2CLI - Conclusions
8WebAPI2CLI - Zip application

I realized that I should have somehow let the user play with the application from his PC. How can I do that , if the application is on some site  ? Answer: download the whole site as a zip file.

It should be relatively easy  for the user– so an endpoint routing should be used – something like “/zip”

It should be relatively easy  for the programmer – so I figured

app.UseEndpoints(endpoints =>
{
endpoints.MapControllers();

endpoints.MapZip(app); // IApplicationBuilder app
});

The code is also relativeley simple: enumerating recursive file and folders , creating the zip archive in memory, sending to the user

public static void MapZip(this IEndpointRouteBuilder endpoints, IApplicationBuilder app)
{
//see more at
//https://andrewlock.net/converting-a-terminal-middleware-to-endpoint-routing-in-aspnetcore-3/
endpoints.Map(“/zip”, async context =>
{
var response = context.Response;
var name = Assembly.GetEntryAssembly().GetName().Name + “.zip”;
response.ContentType = “application/octet-stream”;
var b = GetZip(app.ApplicationServices.GetService<IWebHostEnvironment>());
//https://github.com/dotnet/aspnetcore/blob/master/src/Mvc/Mvc.Core/src/Infrastructure/FileResultExecutorBase.cs
var contentDisposition = new Microsoft.Net.Http.Headers.ContentDispositionHeaderValue(“attachment”);
contentDisposition.SetHttpFileName(name);
response.Headers[HeaderNames.ContentDisposition] = contentDisposition.ToString();
await response.Body.WriteAsync(b);

});

}

private static Memory<byte> GetZip(IWebHostEnvironment env)
{
//var b = new Memory<byte>(Encoding.ASCII.GetBytes($”{env.ContentRootPath}”));
var firstDir = new DirectoryInfo(env.ContentRootPath);
var nameLength = firstDir.FullName.Length + 1;
using var memoryStream = new MemoryStream();
using var zipToOpen = new ZipArchive(memoryStream, ZipArchiveMode.Create, true);

 

foreach (FileInfo file in firstDir.RecursiveFilesAndFolders().Where(o => o is FileInfo).Cast<FileInfo>())
{
var relPath = file.FullName.Substring(nameLength);
var readmeEntry = zipToOpen.CreateEntryFromFile(file.FullName, relPath);
}
zipToOpen.Dispose();
var b = new Memory<byte>(memoryStream.ToArray());
return b;

}

private static IEnumerable<FileSystemInfo> RecursiveFilesAndFolders(this DirectoryInfo dir)
{
foreach (var f in dir.GetFiles())
yield return f;
foreach (var d in dir.GetDirectories())
{
yield return d;
foreach (var o in RecursiveFilesAndFolders(d))
yield return o;
}
}

 

Some points to be noted:

1. It will work only if the site and user have the same operating system( assuming you have did self contained the .NET Core application)

2. ContentDispositionHeaderValue – exists in 2 libraries

3. I have shameless copy code from StackOverflow and github aspnet

4. I did not put in a separate library, nor I do automatic tests. Sorry, too tired

5. It works only if you put

endpoints.MapZip(app); // IApplicationBuilder app

If not, it does not zip!

6. IEndpointRouteBuilder – it is defined only you have the library .net core  3, not .net standard 2

<TargetFramework>netcoreapp3.1</TargetFramework>

<ItemGroup>
<FrameworkReference Include=”Microsoft.AspNetCore.App” />
</ItemGroup>

Post Mortem -part 7

WebAPI2CLI

This is a part of the series where about how I made the WebAPI2CLI - Execute ASP.NET Core WebAPI from Command Line
Source code on https://github.com/ignatandrei/webAPI2CLI/
1WebAPI2CLI - Description
2WebAPI2CLI- Organization
3WebAPI2CLI - implementing
4WebAPI2CLI - tests
5WebAPI2CLI - Devops and CI/CD
6WebAPI2CLI - documentation
7WebAPI2CLI - Conclusions
8WebAPI2CLI - Zip application

It was fun to create https://github.com/ignatandrei/webAPI2CLI/ and to deal with various technical problems.

It was less fun to make the devops working – however , the previous experiences helped a lot. But even so, waiting to compile the project , making the docs … was not fun.

It was cumbersome writing the documentation – however, I know that , without documentation, the project is useless.

It was daunting writing those blog posts – remembering the phases of the project.

I hope that you enjoy using it!

Documentation–part 6

WebAPI2CLI

This is a part of the series where about how I made the WebAPI2CLI - Execute ASP.NET Core WebAPI from Command Line
Source code on https://github.com/ignatandrei/webAPI2CLI/
1WebAPI2CLI - Description
2WebAPI2CLI- Organization
3WebAPI2CLI - implementing
4WebAPI2CLI - tests
5WebAPI2CLI - Devops and CI/CD
6WebAPI2CLI - documentation
7WebAPI2CLI - Conclusions
8WebAPI2CLI - Zip application

I need 2 types of documentation:

1. For let people know how to use

2. For the programmers – to understand what the classes are doing

 

For the first point, documentation to use

a. Github has already the readme.md file that shows details for the repository . What I need is something more- i.e. detailed instruction, FAQ, Author.

b. Github has made easy to make an online site  -by having the docs folder public with his own address. For example, from  https://github.com/ignatandrei/webAPI2CLI/ I can have https://ignatandrei.github.io/WebAPI2CLI/ . Now , the problem is how to concatenate those and put back to site

c. GitHub has also the concept of Actions –i.e.  DevOps integrated.

 

So –what I have done is to use pandoc , compile all the .md files into html ( and pdf ) and push modifications back to the site.

See https://github.com/ignatandrei/WebAPI2CLI/blob/master/.github/workflows/main.yml

– uses: actions/checkout@v2

– run: |

echo generate help file

mkdir output

git pull

– name: generate html

uses: docker://pandoc/latex:2.9

with: # needs a README in your repo root!

#args: “–standalone –output=output/README.html README.md”

args: “README.md docs/Demo.md docs/FAQ.md docs/Author.md –standalone -f gfm -t html  –toc -o output/output.html –metadata title=WebAPI2CLI”

– name: generate pdf

uses: docker://pandoc/latex:2.9

with: # needs a README in your repo root!

#args: “–standalone –output=output/README.html README.md”

args: “README.md docs/Demo.md docs/FAQ.md docs/Author.md –standalone -f gfm -t pdf  –toc -o output/output.pdf –metadata title=WebAPI2CLI”

– run: |

cp ./output/output.html ./docs/index.html

cp ./output/output.pdf ./docs/Web2CLI.pdf

rm -rf ./output

 

You can see the results at https://ignatandrei.github.io/WebAPI2CLI/ and the sources at https://github.com/ignatandrei/WebAPI2CLI/tree/master/docs

For the second point, generating documentation from XML Comments in C#

I need some free .NET tool to do this. I did not found. I have hesitated also between https://github.com/EWSoftware/SHFB ( an old friend, but cumbersome ) and https://github.com/dotnet/docfx/ ( a new friend, but still cumbersome)

What I have done is to download docfx, add to the site, follow instructions , modify some of the templates , resist temptation to put everything into docfx ( he kinda want to be all inclusive)

docs_csharp:

# The type of runner that the job will run on

runs-on: windows-latest

steps:

# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it

– uses: actions/checkout@v2

– name: run documentation

run: |

DocumentationSources.bat

rem git config –local user.email “action@github.com”

rem git config –local user.name “GitHub Action”

rem git commit -m “generate documentation sources” -a –allow-empty

shell: cmd

– name: Push changes

uses: ad-m/github-push-action@master

with:

github_token: ${{ secrets.GITHUB_TOKEN }}

You can see the results at https://ignatandrei.github.io/WebAPI2CLI/sitedocs/api/index.html  and sources at https://github.com/ignatandrei/WebAPI2CLI/tree/master/docs/sitedocs

Devops + CI/CD-part 5

WebAPI2CLI

This is a part of the series where about how I made the WebAPI2CLI - Execute ASP.NET Core WebAPI from Command Line
Source code on https://github.com/ignatandrei/webAPI2CLI/
1WebAPI2CLI - Description
2WebAPI2CLI- Organization
3WebAPI2CLI - implementing
4WebAPI2CLI - tests
5WebAPI2CLI - Devops and CI/CD
6WebAPI2CLI - documentation
7WebAPI2CLI - Conclusions
8WebAPI2CLI - Zip application

What I need for the devops:

1. Building the solution

2. Running tests

3. Deploying packages to Nuget

Being part of the Microsoft stack, it is normal that I have choose for CI / CD the Azure Devops.

You can see the AzureDevops at https://dev.azure.com/ignatandrei0674/WebAPI2CLI/_build?definitionId=7&_a=summary and how it is done by reading https://github.com/ignatandrei/WebAPI2CLI/blob/master/azure-pipelines.yml

Seeing code coverage in AzureDevops

I need not only to know that tests have runned with success, but also to see the code coverage

For this I use the .NET local tools, coverlet and report generator

dotnet coverlet bin\$(buildConfiguration)\netcoreapp3.1\CLITests.dll –target “dotnet” –targetargs “test –no-build –configuration $(buildConfiguration)” –exclude ‘[*Test*]*’ –format opencover  –output $(Build.ArtifactStagingDirectory)\testResults\coverlet.xml

dotnet reportgenerator “-reports:$(Build.ArtifactStagingDirectory)\testResults\coverlet.xml” “-targetdir:$(Build.ArtifactStagingDirectory)\testResults” “-reporttypes:Cobertura;HtmlSummary;Badges;HtmlInline_AzurePipelines”

– task: PublishTestResults@2

inputs:

testResultsFormat: ‘VSTest’

testResultsFiles: ‘**/*.trx’

searchFolder: ‘$(Build.ArtifactStagingDirectory)\trx’

displayName: publish tests

– task: PublishCodeCoverageResults@1

displayName: ‘Publish code coverage’

inputs:

codeCoverageTool: Cobertura

summaryFileLocation: ‘$(Build.ArtifactStagingDirectory)\testResults\Cobertura.xml’

reportDirectory: ‘$(Build.ArtifactStagingDirectory)\testResults’

In this way , you can see the

Modify version 

For a good CD I need also a way to modify the version automatically.

 

The .Net local Tools provides a way to install some tools on the local system to be run within dotnet.

So I install pwsh  -and run a setversion.ps1

 

$TimeNow = Get-Date

$d = $TimeNow.ToUniversalTime()

$year = $TimeNow.Year

$startOfYear = Get-Date -Year $year -Month 1 -Day 1 -Hour 0 -Minute 0 -Second 0 -Millisecond 0

$diff = NEW-TIMESPAN -Start $startOfYear -End $TimeNow

#$diff.TotalSeconds -as [int]

$assemblyVersion=$d.ToString(“1.yyyy.1MMdd.1HHmm”)

dotnet-property “**/*.csproj” AssemblyVersion:”$assemblyVersion”

dotnet dotnet-property “**/*.csproj” AssemblyVersion:”$assemblyVersion”

$version=$d.ToString(“1.0.yyyy.”) + ($diff.TotalSeconds -as [int]).ToString()

dotnet-property “**/*.csproj” Version:”$version”

dotnet dotnet-property “**/*.csproj” Version:”$version”

$releaseNotes = “BuildNumber $env:BUILD_BUILDNUMBER”

$releaseNotes += “;author $env:BUILD_SOURCEVERSIONAUTHOR”

$releaseNotes += “;message $env:BUILD_SOURCEVERSIONMESSAGE”

$releaseNotes +=”;source for this release github.com/ignatandrei/webAPI2CLI/commit/$env:BUILD_SOURCEVERSION”

$releaseNotes

dotnet-property “**/*.csproj” PackageReleaseNotes:”$releaseNotes”

dotnet dotnet-property “**/*.csproj” PackageReleaseNotes:”$releaseNotes”

I think that it is self explanatory.

Be a good nuget citizen –do not deploy every time a modification is made

For this I use devops variable :  deployNuget: ‘0’

– task: NuGetCommand@2

condition: and(succeeded(), eq(variables[‘deployNuget’], ‘1’))

inputs:

command: push

nuGetFeedType: external

packagesToPush: ‘$(Build.ArtifactStagingDirectory)/**/*.symbols.nupkg’

publishFeedCredentials: ‘nugetAndrei’

displayName: ‘dotnet nuget push’

Be a good AzureDevops citizen – do not run automation every time

I have some documentation on .md files – no need to rebuild everything when just the documentation is done

This is done in AzureDevOps by the trigger:

trigger:

branches:

include:

– master

paths:

exclude:

– docs/*

– README.md

Test your application

I also can test in AzureDevops what I have done by runnning

TestWebAPISite.exe  –CLI_ENABLED=1 –CLI_Commands=”GetMathId_Http,MathPOST”

It is just nice – maybe I can find a way to do something practical …

Create and running tests- part 4

WebAPI2CLI

This is a part of the series where about how I made the WebAPI2CLI - Execute ASP.NET Core WebAPI from Command Line
Source code on https://github.com/ignatandrei/webAPI2CLI/
1WebAPI2CLI - Description
2WebAPI2CLI- Organization
3WebAPI2CLI - implementing
4WebAPI2CLI - tests
5WebAPI2CLI - Devops and CI/CD
6WebAPI2CLI - documentation
7WebAPI2CLI - Conclusions
8WebAPI2CLI - Zip application

I have made a TestWebSite , in .NET Core 3.1, I was also thinking that WebApplicationFactory exists and tutorial like https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-3.1   it will be easy to test the site.

However, I have found the hard way that is not a real server : it does not have the adresses / ports open. Read https://github.com/dotnet/aspnetcore/issues/4892 .

So I have made a hack – see public class LocalServerFactory in the source code.

Also, WithWebHostBuilder helps create a new WebApplicationFactory  -used to test with / or without /  CLI_ENABLED command line

 

Second point was testing methods like

configuration.GetValue<int?>(“CLI_HELP”, null);

Because it is an extension function, I cannot mock. So I refactored the code to

public bool ShouldShowHelp()
{
//deleted for easy of automating testing
//var showHelp = configuration.GetValue<int?>(“CLI_HELP”, null);
var cliHelp = configuration[“CLI_HELP”];
var showHelp = int.TryParse(cliHelp, out var val) && val == 1;
return showHelp;
}

Also, I have resorted to XBehave, to have tests written in easy form. And to Moq to mock various properties.

Implementing–part 3

WebAPI2CLI

This is a part of the series where about how I made the WebAPI2CLI - Execute ASP.NET Core WebAPI from Command Line
Source code on https://github.com/ignatandrei/webAPI2CLI/
1WebAPI2CLI - Description
2WebAPI2CLI- Organization
3WebAPI2CLI - implementing
4WebAPI2CLI - tests
5WebAPI2CLI - Devops and CI/CD
6WebAPI2CLI - documentation
7WebAPI2CLI - Conclusions
8WebAPI2CLI - Zip application

First implementation was just a bunch a code thrown in an API project. All self contained, all in one WebAPI project.

Some problems that I have faced:

 

1. The class that executes( the hosted service )  should have access to the address of the server . However, the class is injected at the moment of creating services – and at this time the IServerAddressesFeature do not exists yet.

Solution: Inject at useCLi the IApplicationBuilder to get the services and verify if the adresses are ok:

// in private async void DoWork(object state)

serverAddresses = app.ServerFeatures.Get<IServerAddressesFeature>();

if (serverAddresses == null)

 

2. But how to get the instance of the Hosted service  ? Simple,with Singletons:

// in public static IServiceCollection AddCLI(this IServiceCollection serviceCollection)

serviceCollection.AddSingleton<CLIAPIHostedService>();

serviceCollection.AddHostedService<CLIAPIHostedService>(p => p.GetService<CLIAPIHostedService>());

 

and later on

//in public static IApplicationBuilder UseCLI(this IApplicationBuilder app)

var service = app.ApplicationServices.GetService<CLIAPIHostedService>();
service.app = app;

 

3.  How to serialize the headers, url, parameters, body and others in a file  – as to pass easy json ?

The idea is that I wrote Json( for sending body data)  in a json file it will become faster complicated. So I was thinking : What other serialized formats I know ?

So I was attracted by YAMLhttps://en.wikipedia.org/wiki/YAML . And there is an implementation https://github.com/aaubry/YamlDotNet . You can find a sample file at https://github.com/ignatandrei/WebAPI2CLI/blob/master/src/TestWebAPISite/cli.txt

4. What to do with errors ?

For example, if some HTTP returns an error message , what can I do  ? The obvious idea is to return on the console the error too…

So I implemented this class to emulate answer

public class ReturnValue_v1
{
public ReturnValue_v1(HttpStatusCode statusCode):this(statusCode,null)
{

}
public ReturnValue_v1(HttpStatusCode statusCode,string result)
{
StatusCode = statusCode;
Result = result;
}
public ICLICommand Command { get; private set; }

public HttpStatusCode StatusCode { get; private set; }
public string Result { get; private set; }

public ReturnValue_v1 FromCommand(ICLICommand cmd)
{
this.Command = cmd;
return this;
}

}

to can return to the console the JSON of this object.

5. How to exit the application ?

This is simple –  Environment.Exit(0);

However , maybe could be a feature to exit if we encounter an error in calling the API ? And if you want to help, https://github.com/ignatandrei/WebAPI2CLI/issues/10

6. How to figure the WebAPI endpoints ?

Simple – with IApiDescriptionGroupCollectionProvider

7. How to help the user to not change the port of adresses every time the .NET Core server change the adress ?

There are so many ways to configure the address of ASP .NET Core  – see https://josephwoodward.co.uk/2017/02/many-different-ways-specifying-host-port-asp-net-core . The CLI.txt file should work the same , no matter the port . So I decided to put just the beginning of the http and then find the right address ( and handle also the case with full URL)

In this file put just –
Host: http://

or

Host: https://

( Assumption : just one http and/or just 1 https when asp.net core will start )

WebAPI2CLI will find the adress and the port comparing

Alternatively, you can put the full URI ( without RelativeRequestUrl ! )

Host: http://localhost:5000/

8. How to make easier for the developer to integrate ?

Create extension – AddCLI and UseCLI .

Thinking of the project–part 2

WebAPI2CLI

This is a part of the series where about how I made the WebAPI2CLI - Execute ASP.NET Core WebAPI from Command Line
Source code on https://github.com/ignatandrei/webAPI2CLI/
1WebAPI2CLI - Description
2WebAPI2CLI- Organization
3WebAPI2CLI - implementing
4WebAPI2CLI - tests
5WebAPI2CLI - Devops and CI/CD
6WebAPI2CLI - documentation
7WebAPI2CLI - Conclusions
8WebAPI2CLI - Zip application

The questions that I have to solve is :

1.  How to run something in an ASP.NET Core project, when we have just WEBAPI controllers ?

The answer was pretty simple: Hosted Services: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-3.1&tabs=visual-studio . It runs unattended and can be started by the project

2. Which version of .NET Core should I use ?

Another simple answer, looking at https://dotnet.microsoft.com/platform/support/policy/dotnet-core – the only one that has LTS now is .NET Core 3.1 . Although, if someone requires , I can do a .NET Core 2.1

3. How can I serialize the HTTP arguments to send to the  commands?

There are 2 means of a program to read arguments: by Command Line arguments and by file on disk.

Now, let’s see how many options we have to configure a HTTP call : Verb, Arguments, ContentType, URL, Headers…. hard to fit into a command line. Not mentions that I want to have the option to run many HTTP calls / commands . So , as usual , it will be a mix: comand line for activating the file with the commands.

4. What do I do with the result of HTTP calls ?

The output of the commands will be written to the console. It is the easy way  – and you can redirect this output to a file to persist.

5. Where do I store the sources ?

Easy – GitHub – https://github.com/ignatandrei/WebAPI2CLI/

6. How do I find the the project works ?

Azure DevOps to the rescue – it can build the project and run tests. And it is free for Github

7. Where and how I deploy it ?

Where : easy – it will be hosted on NuGet https://www.nuget.org/packages/ExtensionNetCore3

How –easy  -Azure DevOps again.  https://dev.azure.com/ignatandrei0674/WebAPI2CLI/_build?definitionId=7&_a=summary

8. Where and how I can make the help for this project ?

Again –easy – GitHub has docs – https://ignatandrei.github.io/WebAPI2CLI/

 

Now let’s do the implementation!

 

New side project- WebAPI To CLI–part 1

WebAPI2CLI

This is a part of the series where about how I made the WebAPI2CLI - Execute ASP.NET Core WebAPI from Command Line
Source code on https://github.com/ignatandrei/webAPI2CLI/
1WebAPI2CLI - Description
2WebAPI2CLI- Organization
3WebAPI2CLI - implementing
4WebAPI2CLI - tests
5WebAPI2CLI - Devops and CI/CD
6WebAPI2CLI - documentation
7WebAPI2CLI - Conclusions
8WebAPI2CLI - Zip application

I have had an idea; What if the WebAPI application can be called from the console application itself  ?

This will permit just having the DI running  – and all the stuff that WebAPI can work with when calling a controller.

Will be good in a CI / CD workflow – testing the full API instead of relying to test just the dll ( Yes, I know about skinny controllers )

And will be good in Jenkins / any other tool / in order to execute directly the WebAPI instead of recurring to curl / powershell

 

So – seems to have a good piece of software to start – how we do this ?

First, how all the commands should be passed ? If we pass via command line, there is pretty much possible for someone to miss things.

Second , if the user does not pass via command line, the WebAPI should work as usual.

So , 5 days and 88 commits later , this is the result: https://github.com/ignatandrei/webAPI2CLI/ .

It has the documentation about how to install and use it. Enjoy !

 

Andrei Ignat weekly software news(mostly .NET)

* indicates required

Please select all the ways you would like to hear from me:

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please visit our website.

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.