Category: github

Covid Data -CI/ CD–part 5

Now I need that every data ( either from the github .md file, either gathered from https://github.com/CSSEGISandData/COVID-19/ ) be updated into the site.

Github Actions to the rescue!

I have created 2 different workflows : one for gathering the data from the https://systems.jhu.edu/research/public-health/ncov/ , the other for compiling the .md files and the Angular app. Thinking more, I should have had the Angular compiled into another workflow that was called after each one – but this is more AzureDevOps , than GitHub Actions.

1. Workflow for .md and for Angular

You can find the source at https://github.com/ignatandrei/WFH_Resources/blob/master/.github/workflows/blank.yml . It uses a Docker file to compile the Angular: https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/compile.txt and a Docker batch to run the docker file and gather the results : https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/compile.bat

Uses all the previous js and TypeScript files to transform and load data. Copies the data into the docs folder ( to be public available ) and then commits the data back to Github

It is run on both push at a cron job

on:

push:

pull_request:

schedule:

– cron: ‘5 4 * * *’

2. Workflow for gather last country data for Covid

You can find the source at https://github.com/ignatandrei/WFH_Resources/blob/master/.github/workflows/allnewdata.yml .It uses also a Docker file to compile data and run https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/addNew.txt 

And a docker compile file https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/addNew.bat to gather the data from docker.

Also the workflow commits data to github.  It is run at a cron job:

on:

schedule:

– cron: ‘5 2 * * *’

Covid Data–transforming – part 3

I figured out that from data ( .md files and .js files) I need to have a proper data to be processed by the application. Since the application will be a static web app, the easy way is to be a generated js file to be imported into the application.

1.   MD to js

So now I must figure a way to transforme .md files into .js files. I was having dreams about Stankins , but it is not yet ready. So Js to the rescue. You can find the .js file here: https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/server.js

It reads  the folders files

var folders = [“Country”,”Entertainment”,”FreeSoftware”, “Kids”,”Learn”,”MapsAndData”,”NotParsed”];

//…

for (let folder of folders) {

const directoryPath = path.join(__dirname + “/..”, folder);

const list = await rra.list(directoryPath);

then parses a file( e.g. https://github.com/ignatandrei/WFH_Resources/blob/master/Learn/DigitalLibrary.md )  with a lexer

const tokens = marked.lexer(fileContents);

for(let iToken=0;iToken<tokens.length;iToken++){

and writes to a file

fs.writeFileSync(directoryPathWrite,content);

2. Importing  Country data  for Covid

Now I need every data for Covid. I figure out the the only one that have the data is https://github.com/CSSEGISandData/COVID-19/ . However , the data is not always in the same form.

Again, because t Stankins is not yet ready, I recurse to TypeScript. It is better that JS for me – it keeps tracking the errors that you may have . You can find the file at https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/importData.ts

Some pain points :

or there are not always countries – see const NotCountry where  “Diamond Princess” and “MS Zaandam” are countries ( there are cruise ships)

  • I should verify if there is always data for this day, if it exists for the previous ( yes, I know, Covid will eventually disappear or not be registered otherwise than influenza –but, for the moment ,it is ok) :  see async function Verify in https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/importData.ts
  • I have had problems finding a CSV parser to TypeScript – I have found papaparse

3. Importing other data

I need to have the country population for each country, to display the relative percentage of the people . For this, I have the data from https://en.wikipedia.org/wiki/List_of_sovereign_states_and_dependent_territories_by_mortality_rate – but how to import into already existing data https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/countryList.ts ?

Simple: Import data into Excel. Make a case instruction with each country, iterate via countries, ( find discrepancies in names – use alternate names for this) and put the property as js , save the file. See https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/test.ts

.

Covid Data- making a database–part 2

First, I need to understand how to store the data. For the resources , I was thinking to have a simple way for the people to edit data and to add resources ( yes, I know – I am counting too much on the kindness of people)

So I figure out the a .md file will be enough clear to do it. As an example, see : https://github.com/ignatandrei/WFH_Resources/blob/master/Learn/DigitalLibrary.md

or , in the raw form, https://raw.githubusercontent.com/ignatandrei/WFH_Resources/master/Learn/DigitalLibrary.md 

For the data about how many cases of covid are, I was thinking about JSON – see https://github.com/ignatandrei/WFH_Resources/blob/master/obj/allData.js

Now , about having  the data updated :

For the resources, it is enough to someone to edit the .md file and make a pull request.

For the data about Covid per country, it will be presented later in the CI / CD deployment

GitHub Actions and docker for writing a book

I am trying to wrote a book  ( with Daniel Tila ) about what are the challenges to transfer an application from console to SAAS.

We have decided to put the code on GitHub – and , because of this, the text will be still on GitHub .Each chapter will be in the form of the readme.md file – usual for GitHub.

Now – how we can make a PDF / HTML out of this ?

So, first challenge: how to make the PDF ? In these days, all is node-js – and I have found https://www.npmjs.com/package/markdown-pdf .

So I have created a simple package.json file

{

“name”: “print”,

“version”: “1.0.0”,

“description”: “print on Docker”,

“author”: “Andrei Ignat”,

“scripts”: {

“start”: “node server.js”

},

“dependencies”: {

“markdown-pdf”: “10.0.0”

}

}

and a javascript file

‘use strict’;

console.log(`start print`);

var markdownpdf = require(“markdown-pdf”)

var mdDocs = [

“README.md”,”Chapter01/readme.md”, “Chapter02/readme.md”,”Chapter03/readme.md”,”Chapter04/readme.md”

]

, bookPath = “book.pdf”

markdownpdf().concat.from(mdDocs).to(bookPath, function () {

console.log(“Created”, bookPath)

})

And how I can execute ? Docker seems to be the answer ; run the javascript inside  docker ( in order to not install on my pc all kind of tools)

The docker file is simple

FROM node:10

WORKDIR /usr/src/

COPY print/package.json ./print/

RUN npm install ./print/

COPY . .

RUN ls -l print/*.*

RUN node print/server.js

CMD tail -f /dev/null

and to obtain the pdf book:

docker build -t ignatandrei/printsaas .. -f ./exportPDF.txt

docker run -d –name printsaas ignatandrei/printsaas

docker cp printsaas:/usr/src/book.pdf .

docker container kill printsaas

docker container rm printsaas

You can find all those files on https://github.com/ignatandrei/console_to_saas/tree/master/print .

Where should I put the PDF ? Turns out the GitHub has the docs folder – where you can put any kind of static data . Ad this is available for any project – by switching the name of the project from source code url https://github.com/ignatandrei/console_to_saas to https://ignatandrei.github.io/console_to_saas/

Now , the problem is how to run this every time ? Remember that GitHub ( like any other source control) has Actions – that can run every time something is pushed on the repository. So this is the yml file:

name: ‘CD for generate book and docs folder’

on:

push:

paths:

– ‘**.md’

jobs:

build:

runs-on: ubuntu-latest

steps:

– uses: actions/checkout@v2

– name: print pdf

run: |

        chmod +x ./print/getBook.bat

        cd print       

        ./getBook.bat

        cp ./book.pdf ../docs/ConsoleToSaas.pdf

– uses: actions/upload-artifact@v1

with:

name: ConsoleToSaas.pdf

path: ./print/book.pdf

– name: Commit files

run: |

        git config –local user.email “action@github.com”

        git config –local user.name “GitHub Action”

        git commit -m “generate pdf” -a –allow-empty

– name: Push changes

uses: ad-m/github-push-action@master

with:

github_token: ${{ secrets.GITHUB_TOKEN }}

( see https://github.com/ignatandrei/console_to_saas/blob/master/.github/workflows/main.yml )

So now I obtain the PDF automatically generated by docker, I put into docs folder( and into artifact ), and I commit the PDF back to the repository( and yes, GitHub is enough smart to not start the cycle again)

Now I wanted to have also HTML – or for any other format  . So I figure I need a general converte – and I have found PanDoc.

The code is more simple and just in the yml

convert_via_pandoc:

runs-on: ubuntu-18.04

needs: build

steps:

– uses: actions/checkout@v2

– run: |

          git pull

          mkdir output

– uses: docker://pandoc/latex:2.9

with: # needs a README in your repo root!

#args: “–standalone –output=output/README.html README.md”

args: “README.md Chapter01/readme.md Chapter02/readme.md  Chapter03/readme.md Chapter04/readme.md –standalone -f gfm -t html –number-sections –toc -o output/output.html”

– uses: actions/upload-artifact@master

with:

name: output

path: output

– run: |

          cp ./output/output.html ./docs/index.html

          rm -rf ./output

– name: Commit files

run: |

          git config –local user.email “action@github.com”

          git config –local user.name “GitHub Action”

          git commit -m “generate html” -a –allow-empty

– name: Push changes

uses: ad-m/github-push-action@master

with:

github_token: ${{ secrets.GITHUB_TOKEN }}

( see https://github.com/ignatandrei/console_to_saas/blob/master/.github/workflows/main.yml )- 

Some explanations:

  1. I need to serialize the 2 jobs – they cannot commit in parallel
  2. I need a git pull because the previous jobs commits  – and , somehow, the sources are just once read – not at run time
  3. I delete the output folder to not commit to the repository
  4. To commit , the file must exists and will be overwritten by the cp linux command ( that overwrites by default)

You can find the end result at https://ignatandrei.github.io/console_to_saas/ (make sure you see the download section)  and the source code at https://github.com/ignatandrei/console_to_saas

BingoMeetings–build and deploy Angular WebSite to GitHub–part 21

Bingo is a small project, written in TypeScript , and developed with Alexandru Badita in launch break (one hour - more or less). You can find sources at https://github.com/alexandru360/PresentationBingoCards/ . Those are my blog posts for Bingo : ( scroll below for the post)
NrLink
1Create meeting
2Create Tests
3Finalize Create meeting
4Sharing meeting
5Keep Score
6Add obsolete
7Finalizing obsolete
8End meeting
9Dockerize tests
10Azure CI tests
11Yarn workspaces
12CLI
13Intermezzo - CLI improvements
14typescript compile run with node
15NestJS ,swagger and create a meeting
16Finalizing API
17Intermezzo - jest vs jasmine error
18Refactor WebAPI and test service
19Heroku Deploy NestJs
20Angular
21Deploy Angular to GitHub
22WebAPI and Web
23Documentation
24Documentation of the code
25Conclusions

(Now the actual blog post for Bingo Meetings project)

What  we need is to deploy somewhere the WebSite application – the Angular. One version is to compile and deploy to the WebApi (http://bingo-meeting-api.herokuapp.com/api/ ), however  , I do prefer keeping separate.

One simple idea is to deploy to GitHub site . For each repository , GitHub lets you have a website correlated . For example , for our project, https://github.com/alexandru360/PresentationBingoCards/ , github lets you have the site https://alexandru360.github.io/PresentationBingoCards/ ( see username and repository changing position?) . What is there could come form different GitHub project settings – I choose to be in the docs folder.

So we need to compile the Angular Application and put the result on the docs folder on  Github

If we do this on a CI on Azure, it could provide an endless loop ( push code oin GitHub, compoiling on Azure, push on GitHub folder docs, compile again…)

So it should come from Developer PC. But how to do this in a repeatable manner ? Simple – Docker!

So we build inside Docker the Angular site, copy the compile from Docker on the local system, then we manually push the modifications on Github. The system has the advantage that it can be reproducible also on Azure.

So the bat looks like this

call build_web.bat
del /S /Q ..\docs\*.*
copy dist\bingo-cards-ui\*.* ..\docs\

See https://github.com/alexandru360/PresentationBingoCards/tree/master/docs  for more details about the docker files.

One small modification also: the base href from angular should be this, to accept https://alexandru360.github.io/PresentationBingoCards/ that is not rooted:

<!– <base href=”/”> –>

<script>

var isCordovaApp = !!window.cordova;

var hrefApp = “.”;

if (!isCordovaApp) {

var baseHref = window.location.href.split(‘/’);

baseHref.pop();

hrefApp = baseHref.join(‘/’) + ‘/’;

}

document.write(‘<base href=”‘ + hrefApp + ‘” />’);

</script>

create docker to deploy to docs

build_docs.bat

GitHub as a central Hub

I do not know how, but GitHub is now a hub for all things.

I , personnally, used GitHub as a

– source control ( see https://github.com/ignatandrei/Exporter )

– documentation ( see https://github.com/ignatandrei/Exporter/wiki )

– Angular application deployment (https://ignatandrei.github.io/Decl230   – done with  gh-pages –  see build.bat from https://github.com/ignatandrei/Decl230 )

– read-only database ( Sqlite download from github in the project  https://github.com/ignatandrei/IsThisTaxiLegal )

I have seen many other uses

–  like Docker files generator ( for example, for docker phone gap  see https://github.com/webratio/docker )

– blog

and may others ( https://readwrite.com/2013/11/08/seven-ways-to-use-github-that-arent-coding/)

I need just a server hosting service affordable….

YouTube playlist exporter

 

The point here is about how simple is in our days to make a simple script as a programmer.

I wanted to export my playlists ( 5 minutes .NET , Export Word Exce l PDF  , EF 6 , Traceability ) videos into a simple HTML file that I could put into my website.

I searched first on internet – however, I found just you tube google api .

So I read the documentation (e.g. https://developers.google.com/youtube/v3/docs/videos/list)  , grab a developer key from https://console.developers.google.com/ and made a simple project.

Used also a NuGet package for JSON( Newtonsoft.Json) and one for export in Excel /Word /HTML / PDF /ODS (ExporterWordExcelPDF).

Total time : 2 hours. You can find the project here https://github.com/ignatandrei/YouTube . It has only console application  – but it will have soon a GUI .

The point is that in our days it is easy to use third party application – and , for a programmer, easy as pie( eating, not making ; – ) )

Entity Framework 6 Record and play use : Unit Testing ( part 2 of 5)

 

Part 1 : What is EF record and play : http://msprogrammer.serviciipeweb.ro/2014/11/29/entity-framework-6-record-and-play-1-of-5/ 

Part 2: EF Record and play use: Testing : http://msprogrammer.serviciipeweb.ro/2014/12/08/entity-framework-6-record-and-play-use-unit-testing-part-2-of-5/

Part 3: EF Record and play use: Make demo: http://msprogrammer.serviciipeweb.ro/2014/12/14/entity-framework-6-record-and-play-use-making-demos-part-3-of-5/ 

Part 4: EF Record and play use: Record user Sql when a bug occurs: http://msprogrammer.serviciipeweb.ro/2014/12/26/ef-record-and-play-use-recording-user-sql-when-a-bug-occurred-part-4-of-5/

Part 5: EF record and play: conclusions: http://msprogrammer.serviciipeweb.ro/2015/01/05/ef-record-and-play-conclusions/

 

Let’s suppose that we have a program that have Departments and Employees.

And we want to make sure that, when we add an employee, the department must exists.

We can ensure this from database ( by foreign key) but we can pro-actively search for the department and throw a more meaningful validation .

More, I like more validation than errors.

So, let’s suppose that in the Validation for the Employee we must check in the database for the IdDepartment to see if there is such a department.

How could we make a test for that runs without a database ?

With some trick:  we first Record with a database  – then we can Play the file – and we do not need anymore the database. The test is self contained. 

 

Let’s see in action here

 

Database.SetInitializer<ContextForDatabase>(null);
            #region set record EF
            var record = new InterceptionRecordOrPlay(@"VerifyIValidatableWorks.zip", ModeInterception.Play);

            DbInterception.Add(record);
            #endregion
            var e= new Employee();
            e.ValidateEmployee = true;
            e.IDDepartment = 60000;
            var err= e.Validate(null).ToArray();
            Assert.IsNotNull(err);
            Assert.AreEqual(1, err.Length);

Source code is available at  https://github.com/ignatandrei/EFRecordAndPlay/wiki/

 

There is also a NuGet package at https://www.nuget.org/packages/EFRecordAndPlay/

Entity Framework 6 Record and play – 1 of 5

 

Part 1 : What is EF record and play : http://msprogrammer.serviciipeweb.ro/2014/11/29/entity-framework-6-record-and-play-1-of-5/ 

Part 2: EF Record and play use: Testing : http://msprogrammer.serviciipeweb.ro/2014/12/08/entity-framework-6-record-and-play-use-unit-testing-part-2-of-5/

Part 3: EF Record and play use: Make demo: http://msprogrammer.serviciipeweb.ro/2014/12/14/entity-framework-6-record-and-play-use-making-demos-part-3-of-5/ 

Part 4: EF Record and play use: Record user Sql when a bug occurs: http://msprogrammer.serviciipeweb.ro/2014/12/26/ef-record-and-play-use-recording-user-sql-when-a-bug-occurred-part-4-of-5/

Part 5: EF record and play: conclusions: http://msprogrammer.serviciipeweb.ro/2015/01/05/ef-record-and-play-conclusions/

 

Entity Framework Record And Play

With this helper you can record and then play the actions in Entity Framework(>= 6).

For recording actions just reference the dll and use

DbInterception.Add(new InterceptionRecordOrPlay(@"a.zip", ModeInterception.Record)); 

(Note: For ASP.NET you will use Server.MapPath("~/a folder that supports write/namefile.zip")

For replay use

DbInterception.Add(new InterceptionRecordOrPlay(@"a.zip", ModeInterception.Play));

This can be use for

  1. Unit Testing
  2. Making demos
  3. Recording user actions when a bug occurred

 

Source code is available at  https://github.com/ignatandrei/EFRecordAndPlay/wiki/

and has also a test 😉

There is also a NuGet package at https://www.nuget.org/packages/EFRecordAndPlay/ 

 

image

Andrei Ignat weekly software news(mostly .NET)

* indicates required

Please select all the ways you would like to hear from me:

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please visit our website.

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.