Tag: github;github actions;

Covid Data -CI/ CD–part 5

Now I need that every data ( either from the github .md file, either gathered from https://github.com/CSSEGISandData/COVID-19/ ) be updated into the site.

Github Actions to the rescue!

I have created 2 different workflows : one for gathering the data from the https://systems.jhu.edu/research/public-health/ncov/ , the other for compiling the .md files and the Angular app. Thinking more, I should have had the Angular compiled into another workflow that was called after each one – but this is more AzureDevOps , than GitHub Actions.

1. Workflow for .md and for Angular

You can find the source at https://github.com/ignatandrei/WFH_Resources/blob/master/.github/workflows/blank.yml . It uses a Docker file to compile the Angular: https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/compile.txt and a Docker batch to run the docker file and gather the results : https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/compile.bat

Uses all the previous js and TypeScript files to transform and load data. Copies the data into the docs folder ( to be public available ) and then commits the data back to Github

It is run on both push at a cron job

on:

push:

pull_request:

schedule:

– cron: ‘5 4 * * *’

2. Workflow for gather last country data for Covid

You can find the source at https://github.com/ignatandrei/WFH_Resources/blob/master/.github/workflows/allnewdata.yml .It uses also a Docker file to compile data and run https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/addNew.txt 

And a docker compile file https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/addNew.bat to gather the data from docker.

Also the workflow commits data to github.  It is run at a cron job:

on:

schedule:

– cron: ‘5 2 * * *’

Covid Data–transforming – part 3

I figured out that from data ( .md files and .js files) I need to have a proper data to be processed by the application. Since the application will be a static web app, the easy way is to be a generated js file to be imported into the application.

1.   MD to js

So now I must figure a way to transforme .md files into .js files. I was having dreams about Stankins , but it is not yet ready. So Js to the rescue. You can find the .js file here: https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/server.js

It reads  the folders files

var folders = [“Country”,”Entertainment”,”FreeSoftware”, “Kids”,”Learn”,”MapsAndData”,”NotParsed”];

//…

for (let folder of folders) {

const directoryPath = path.join(__dirname + “/..”, folder);

const list = await rra.list(directoryPath);

then parses a file( e.g. https://github.com/ignatandrei/WFH_Resources/blob/master/Learn/DigitalLibrary.md )  with a lexer

const tokens = marked.lexer(fileContents);

for(let iToken=0;iToken<tokens.length;iToken++){

and writes to a file

fs.writeFileSync(directoryPathWrite,content);

2. Importing  Country data  for Covid

Now I need every data for Covid. I figure out the the only one that have the data is https://github.com/CSSEGISandData/COVID-19/ . However , the data is not always in the same form.

Again, because t Stankins is not yet ready, I recurse to TypeScript. It is better that JS for me – it keeps tracking the errors that you may have . You can find the file at https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/importData.ts

Some pain points :

or there are not always countries – see const NotCountry where  “Diamond Princess” and “MS Zaandam” are countries ( there are cruise ships)

  • I should verify if there is always data for this day, if it exists for the previous ( yes, I know, Covid will eventually disappear or not be registered otherwise than influenza –but, for the moment ,it is ok) :  see async function Verify in https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/importData.ts
  • I have had problems finding a CSV parser to TypeScript – I have found papaparse

3. Importing other data

I need to have the country population for each country, to display the relative percentage of the people . For this, I have the data from https://en.wikipedia.org/wiki/List_of_sovereign_states_and_dependent_territories_by_mortality_rate – but how to import into already existing data https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/countryList.ts ?

Simple: Import data into Excel. Make a case instruction with each country, iterate via countries, ( find discrepancies in names – use alternate names for this) and put the property as js , save the file. See https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/test.ts

.

Andrei Ignat weekly software news(mostly .NET)

* indicates required

Please select all the ways you would like to hear from me:

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please visit our website.

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.