Covid Data–part 1

In this days of Covid I needed 2 things:

  1. Put somewhere the list of free resources
  2. Gather the data about Covid from different countries, in order to understand the evolution of the disease

Because I am a programmer the easy way to have a database is on GitHub  – and I figure that with the docs folder I can have a static site .

You can find the free resources here :  https://ignatandrei.github.io/WFH_Resources/resources

And you can find the situation here: https://ignatandrei.github.io/WFH_Resources/covidData/Italy-Romania

How I constructed the site-  in the next posts.

BitBucket pipelines vs AzureDevOps pipelines

TL;DR; : Choose Azure – it can integrate also with BitBucket repository

For my  previous experience with AzureDevOps, please see http://msprogrammer.serviciipeweb.ro/category/azure-devops/ ,

I  have had the opportunity to play with BitBucket CI this weekend. Nothing fance, just a CI + test + artifacts for a .NET solution.

Both have  yaml files for CI /CD

Both have support for Docker

However  there were some things in BitBucket that were unpleaseant

– for the artifacts in BitBucket, I cannot find how to name it differently (https://confluence.atlassian.com/bitbucket/using-artifacts-in-steps-935389074.html  ) .  For Azure DevOps, you can put a name : https://docs.microsoft.com/en-us/azure/devops/pipelines/artifacts/build-artifacts?view=azure-devops&tabs=yaml

– artifacts in BitBucket and a .tar and a .gs file . That means, for a regular Windows user, 2 operations to get the sources. For AzureDevops, it is zip.

– both have test concepts. However, AzureDevOps let you see the CodeCoverage and test details( see a run at https://dev.azure.com/ignatandrei0674/WebAPI2CLI/_build?definitionId=7&_a=summary ) – BitBucket  just list the number of tests passed.

– For a repository, BitBucket is giving 50 minutes per month free (https://bitbucket.org/blog/everything-you-need-to-know-about-build-minutes-in-bitbucket-pipelines )  -that means something like 2 build per day . Azure Devops is giving 1800 minutes free https://azure.microsoft.com/en-us/pricing/details/devops/azure-devops-services/

Post Mortem -part 7

WebAPI2CLI

This is a part of the series where about how I made the WebAPI2CLI - Execute ASP.NET Core WebAPI from Command Line
Source code on https://github.com/ignatandrei/webAPI2CLI/
1WebAPI2CLI - Description
2WebAPI2CLI- Organization
3WebAPI2CLI - implementing
4WebAPI2CLI - tests
5WebAPI2CLI - Devops and CI/CD
6WebAPI2CLI - documentation
7WebAPI2CLI - Conclusions
8WebAPI2CLI - Zip application

It was fun to create https://github.com/ignatandrei/webAPI2CLI/ and to deal with various technical problems.

It was less fun to make the devops working – however , the previous experiences helped a lot. But even so, waiting to compile the project , making the docs … was not fun.

It was cumbersome writing the documentation – however, I know that , without documentation, the project is useless.

It was daunting writing those blog posts – remembering the phases of the project.

I hope that you enjoy using it!

Documentation–part 6

WebAPI2CLI

This is a part of the series where about how I made the WebAPI2CLI - Execute ASP.NET Core WebAPI from Command Line
Source code on https://github.com/ignatandrei/webAPI2CLI/
1WebAPI2CLI - Description
2WebAPI2CLI- Organization
3WebAPI2CLI - implementing
4WebAPI2CLI - tests
5WebAPI2CLI - Devops and CI/CD
6WebAPI2CLI - documentation
7WebAPI2CLI - Conclusions
8WebAPI2CLI - Zip application

I need 2 types of documentation:

1. For let people know how to use

2. For the programmers – to understand what the classes are doing

 

For the first point, documentation to use

a. Github has already the readme.md file that shows details for the repository . What I need is something more- i.e. detailed instruction, FAQ, Author.

b. Github has made easy to make an online site  -by having the docs folder public with his own address. For example, from  https://github.com/ignatandrei/webAPI2CLI/ I can have https://ignatandrei.github.io/WebAPI2CLI/ . Now , the problem is how to concatenate those and put back to site

c. GitHub has also the concept of Actions –i.e.  DevOps integrated.

 

So –what I have done is to use pandoc , compile all the .md files into html ( and pdf ) and push modifications back to the site.

See https://github.com/ignatandrei/WebAPI2CLI/blob/master/.github/workflows/main.yml

– uses: actions/checkout@v2

– run: |

echo generate help file

mkdir output

git pull

– name: generate html

uses: docker://pandoc/latex:2.9

with: # needs a README in your repo root!

#args: “–standalone –output=output/README.html README.md”

args: “README.md docs/Demo.md docs/FAQ.md docs/Author.md –standalone -f gfm -t html  –toc -o output/output.html –metadata title=WebAPI2CLI”

– name: generate pdf

uses: docker://pandoc/latex:2.9

with: # needs a README in your repo root!

#args: “–standalone –output=output/README.html README.md”

args: “README.md docs/Demo.md docs/FAQ.md docs/Author.md –standalone -f gfm -t pdf  –toc -o output/output.pdf –metadata title=WebAPI2CLI”

– run: |

cp ./output/output.html ./docs/index.html

cp ./output/output.pdf ./docs/Web2CLI.pdf

rm -rf ./output

 

You can see the results at https://ignatandrei.github.io/WebAPI2CLI/ and the sources at https://github.com/ignatandrei/WebAPI2CLI/tree/master/docs

For the second point, generating documentation from XML Comments in C#

I need some free .NET tool to do this. I did not found. I have hesitated also between https://github.com/EWSoftware/SHFB ( an old friend, but cumbersome ) and https://github.com/dotnet/docfx/ ( a new friend, but still cumbersome)

What I have done is to download docfx, add to the site, follow instructions , modify some of the templates , resist temptation to put everything into docfx ( he kinda want to be all inclusive)

docs_csharp:

# The type of runner that the job will run on

runs-on: windows-latest

steps:

# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it

– uses: actions/checkout@v2

– name: run documentation

run: |

DocumentationSources.bat

rem git config –local user.email “action@github.com”

rem git config –local user.name “GitHub Action”

rem git commit -m “generate documentation sources” -a –allow-empty

shell: cmd

– name: Push changes

uses: ad-m/github-push-action@master

with:

github_token: ${{ secrets.GITHUB_TOKEN }}

You can see the results at https://ignatandrei.github.io/WebAPI2CLI/sitedocs/api/index.html  and sources at https://github.com/ignatandrei/WebAPI2CLI/tree/master/docs/sitedocs

Devops + CI/CD-part 5

WebAPI2CLI

This is a part of the series where about how I made the WebAPI2CLI - Execute ASP.NET Core WebAPI from Command Line
Source code on https://github.com/ignatandrei/webAPI2CLI/
1WebAPI2CLI - Description
2WebAPI2CLI- Organization
3WebAPI2CLI - implementing
4WebAPI2CLI - tests
5WebAPI2CLI - Devops and CI/CD
6WebAPI2CLI - documentation
7WebAPI2CLI - Conclusions
8WebAPI2CLI - Zip application

What I need for the devops:

1. Building the solution

2. Running tests

3. Deploying packages to Nuget

Being part of the Microsoft stack, it is normal that I have choose for CI / CD the Azure Devops.

You can see the AzureDevops at https://dev.azure.com/ignatandrei0674/WebAPI2CLI/_build?definitionId=7&_a=summary and how it is done by reading https://github.com/ignatandrei/WebAPI2CLI/blob/master/azure-pipelines.yml

Seeing code coverage in AzureDevops

I need not only to know that tests have runned with success, but also to see the code coverage

For this I use the .NET local tools, coverlet and report generator

dotnet coverlet bin\$(buildConfiguration)\netcoreapp3.1\CLITests.dll –target “dotnet” –targetargs “test –no-build –configuration $(buildConfiguration)” –exclude ‘[*Test*]*’ –format opencover  –output $(Build.ArtifactStagingDirectory)\testResults\coverlet.xml

dotnet reportgenerator “-reports:$(Build.ArtifactStagingDirectory)\testResults\coverlet.xml” “-targetdir:$(Build.ArtifactStagingDirectory)\testResults” “-reporttypes:Cobertura;HtmlSummary;Badges;HtmlInline_AzurePipelines”

– task: PublishTestResults@2

inputs:

testResultsFormat: ‘VSTest’

testResultsFiles: ‘**/*.trx’

searchFolder: ‘$(Build.ArtifactStagingDirectory)\trx’

displayName: publish tests

– task: PublishCodeCoverageResults@1

displayName: ‘Publish code coverage’

inputs:

codeCoverageTool: Cobertura

summaryFileLocation: ‘$(Build.ArtifactStagingDirectory)\testResults\Cobertura.xml’

reportDirectory: ‘$(Build.ArtifactStagingDirectory)\testResults’

In this way , you can see the

Modify version 

For a good CD I need also a way to modify the version automatically.

 

The .Net local Tools provides a way to install some tools on the local system to be run within dotnet.

So I install pwsh  -and run a setversion.ps1

 

$TimeNow = Get-Date

$d = $TimeNow.ToUniversalTime()

$year = $TimeNow.Year

$startOfYear = Get-Date -Year $year -Month 1 -Day 1 -Hour 0 -Minute 0 -Second 0 -Millisecond 0

$diff = NEW-TIMESPAN -Start $startOfYear -End $TimeNow

#$diff.TotalSeconds -as [int]

$assemblyVersion=$d.ToString(“1.yyyy.1MMdd.1HHmm”)

dotnet-property “**/*.csproj” AssemblyVersion:”$assemblyVersion”

dotnet dotnet-property “**/*.csproj” AssemblyVersion:”$assemblyVersion”

$version=$d.ToString(“1.0.yyyy.”) + ($diff.TotalSeconds -as [int]).ToString()

dotnet-property “**/*.csproj” Version:”$version”

dotnet dotnet-property “**/*.csproj” Version:”$version”

$releaseNotes = “BuildNumber $env:BUILD_BUILDNUMBER”

$releaseNotes += “;author $env:BUILD_SOURCEVERSIONAUTHOR”

$releaseNotes += “;message $env:BUILD_SOURCEVERSIONMESSAGE”

$releaseNotes +=”;source for this release github.com/ignatandrei/webAPI2CLI/commit/$env:BUILD_SOURCEVERSION”

$releaseNotes

dotnet-property “**/*.csproj” PackageReleaseNotes:”$releaseNotes”

dotnet dotnet-property “**/*.csproj” PackageReleaseNotes:”$releaseNotes”

I think that it is self explanatory.

Be a good nuget citizen –do not deploy every time a modification is made

For this I use devops variable :  deployNuget: ‘0’

– task: NuGetCommand@2

condition: and(succeeded(), eq(variables[‘deployNuget’], ‘1’))

inputs:

command: push

nuGetFeedType: external

packagesToPush: ‘$(Build.ArtifactStagingDirectory)/**/*.symbols.nupkg’

publishFeedCredentials: ‘nugetAndrei’

displayName: ‘dotnet nuget push’

Be a good AzureDevops citizen – do not run automation every time

I have some documentation on .md files – no need to rebuild everything when just the documentation is done

This is done in AzureDevOps by the trigger:

trigger:

branches:

include:

– master

paths:

exclude:

– docs/*

– README.md

Test your application

I also can test in AzureDevops what I have done by runnning

TestWebAPISite.exe  –CLI_ENABLED=1 –CLI_Commands=”GetMathId_Http,MathPOST”

It is just nice – maybe I can find a way to do something practical …

Create and running tests- part 4

WebAPI2CLI

This is a part of the series where about how I made the WebAPI2CLI - Execute ASP.NET Core WebAPI from Command Line
Source code on https://github.com/ignatandrei/webAPI2CLI/
1WebAPI2CLI - Description
2WebAPI2CLI- Organization
3WebAPI2CLI - implementing
4WebAPI2CLI - tests
5WebAPI2CLI - Devops and CI/CD
6WebAPI2CLI - documentation
7WebAPI2CLI - Conclusions
8WebAPI2CLI - Zip application

I have made a TestWebSite , in .NET Core 3.1, I was also thinking that WebApplicationFactory exists and tutorial like https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-3.1   it will be easy to test the site.

However, I have found the hard way that is not a real server : it does not have the adresses / ports open. Read https://github.com/dotnet/aspnetcore/issues/4892 .

So I have made a hack – see public class LocalServerFactory in the source code.

Also, WithWebHostBuilder helps create a new WebApplicationFactory  -used to test with / or without /  CLI_ENABLED command line

 

Second point was testing methods like

configuration.GetValue<int?>(“CLI_HELP”, null);

Because it is an extension function, I cannot mock. So I refactored the code to

public bool ShouldShowHelp()
{
//deleted for easy of automating testing
//var showHelp = configuration.GetValue<int?>(“CLI_HELP”, null);
var cliHelp = configuration[“CLI_HELP”];
var showHelp = int.TryParse(cliHelp, out var val) && val == 1;
return showHelp;
}

Also, I have resorted to XBehave, to have tests written in easy form. And to Moq to mock various properties.

Implementing–part 3

WebAPI2CLI

This is a part of the series where about how I made the WebAPI2CLI - Execute ASP.NET Core WebAPI from Command Line
Source code on https://github.com/ignatandrei/webAPI2CLI/
1WebAPI2CLI - Description
2WebAPI2CLI- Organization
3WebAPI2CLI - implementing
4WebAPI2CLI - tests
5WebAPI2CLI - Devops and CI/CD
6WebAPI2CLI - documentation
7WebAPI2CLI - Conclusions
8WebAPI2CLI - Zip application

First implementation was just a bunch a code thrown in an API project. All self contained, all in one WebAPI project.

Some problems that I have faced:

 

1. The class that executes( the hosted service )  should have access to the address of the server . However, the class is injected at the moment of creating services – and at this time the IServerAddressesFeature do not exists yet.

Solution: Inject at useCLi the IApplicationBuilder to get the services and verify if the adresses are ok:

// in private async void DoWork(object state)

serverAddresses = app.ServerFeatures.Get<IServerAddressesFeature>();

if (serverAddresses == null)

 

2. But how to get the instance of the Hosted service  ? Simple,with Singletons:

// in public static IServiceCollection AddCLI(this IServiceCollection serviceCollection)

serviceCollection.AddSingleton<CLIAPIHostedService>();

serviceCollection.AddHostedService<CLIAPIHostedService>(p => p.GetService<CLIAPIHostedService>());

 

and later on

//in public static IApplicationBuilder UseCLI(this IApplicationBuilder app)

var service = app.ApplicationServices.GetService<CLIAPIHostedService>();
service.app = app;

 

3.  How to serialize the headers, url, parameters, body and others in a file  – as to pass easy json ?

The idea is that I wrote Json( for sending body data)  in a json file it will become faster complicated. So I was thinking : What other serialized formats I know ?

So I was attracted by YAMLhttps://en.wikipedia.org/wiki/YAML . And there is an implementation https://github.com/aaubry/YamlDotNet . You can find a sample file at https://github.com/ignatandrei/WebAPI2CLI/blob/master/src/TestWebAPISite/cli.txt

4. What to do with errors ?

For example, if some HTTP returns an error message , what can I do  ? The obvious idea is to return on the console the error too…

So I implemented this class to emulate answer

public class ReturnValue_v1
{
public ReturnValue_v1(HttpStatusCode statusCode):this(statusCode,null)
{

}
public ReturnValue_v1(HttpStatusCode statusCode,string result)
{
StatusCode = statusCode;
Result = result;
}
public ICLICommand Command { get; private set; }

public HttpStatusCode StatusCode { get; private set; }
public string Result { get; private set; }

public ReturnValue_v1 FromCommand(ICLICommand cmd)
{
this.Command = cmd;
return this;
}

}

to can return to the console the JSON of this object.

5. How to exit the application ?

This is simple –  Environment.Exit(0);

However , maybe could be a feature to exit if we encounter an error in calling the API ? And if you want to help, https://github.com/ignatandrei/WebAPI2CLI/issues/10

6. How to figure the WebAPI endpoints ?

Simple – with IApiDescriptionGroupCollectionProvider

7. How to help the user to not change the port of adresses every time the .NET Core server change the adress ?

There are so many ways to configure the address of ASP .NET Core  – see https://josephwoodward.co.uk/2017/02/many-different-ways-specifying-host-port-asp-net-core . The CLI.txt file should work the same , no matter the port . So I decided to put just the beginning of the http and then find the right address ( and handle also the case with full URL)

In this file put just –
Host: http://

or

Host: https://

( Assumption : just one http and/or just 1 https when asp.net core will start )

WebAPI2CLI will find the adress and the port comparing

Alternatively, you can put the full URI ( without RelativeRequestUrl ! )

Host: http://localhost:5000/

8. How to make easier for the developer to integrate ?

Create extension – AddCLI and UseCLI .

Andrei Ignat weekly software news(mostly .NET)

* indicates required

Please select all the ways you would like to hear from me:

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please visit our website.

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.