Tag: github;github actions;

Covid Data -CI/ CD–part 5

Now I need that every data ( either from the github .md file, either gathered from https://github.com/CSSEGISandData/COVID-19/ ) be updated into the site.

Github Actions to the rescue!

I have created 2 different workflows : one for gathering the data from the https://systems.jhu.edu/research/public-health/ncov/ , the other for compiling the .md files and the Angular app. Thinking more, I should have had the Angular compiled into another workflow that was called after each one – but this is more AzureDevOps , than GitHub Actions.

1. Workflow for .md and for Angular

You can find the source at https://github.com/ignatandrei/WFH_Resources/blob/master/.github/workflows/blank.yml . It uses a Docker file to compile the Angular: https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/compile.txt and a Docker batch to run the docker file and gather the results : https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/compile.bat

Uses all the previous js and TypeScript files to transform and load data. Copies the data into the docs folder ( to be public available ) and then commits the data back to Github

It is run on both push at a cron job

on:
push:
pull_request:
schedule:
– cron: ‘5 4 * * *’

2. Workflow for gather last country data for Covid

You can find the source at https://github.com/ignatandrei/WFH_Resources/blob/master/.github/workflows/allnewdata.yml .It uses also a Docker file to compile data and run https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/addNew.txt

And a docker compile file https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/addNew.bat to gather the data from docker.

Also the workflow commits data to github. It is run at a cron job:

on:
schedule:
– cron: ‘5 2 * * *’

Covid Data–transforming – part 3

I figured out that from data ( .md files and .js files) I need to have a proper data to be processed by the application. Since the application will be a static web app, the easy way is to be a generated js file to be imported into the application.

1. MD to js

So now I must figure a way to transforme .md files into .js files. I was having dreams about Stankins , but it is not yet ready. So Js to the rescue. You can find the .js file here: https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/server.js

It reads the folders files

var folders = [“Country”,”Entertainment”,”FreeSoftware”, “Kids”,”Learn”,”MapsAndData”,”NotParsed”];

//…

for (let folder of folders) {
const directoryPath = path.join(__dirname + “/..”, folder);
const list = await rra.list(directoryPath);

then parses a file( e.g. https://github.com/ignatandrei/WFH_Resources/blob/master/Learn/DigitalLibrary.md ) with a lexer

const tokens = marked.lexer(fileContents);
for(let iToken=0;iToken<tokens.length;iToken++){

and writes to a file

fs.writeFileSync(directoryPathWrite,content);

2. Importing Country data for Covid

Now I need every data for Covid. I figure out the the only one that have the data is https://github.com/CSSEGISandData/COVID-19/ . However , the data is not always in the same form.

Again, because t Stankins is not yet ready, I recurse to TypeScript. It is better that JS for me – it keeps tracking the errors that you may have . You can find the file at https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/importData.ts

Some pain points :

The names of the countries are not always the same. See alternate names for https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/countryList.ts

or there are not always countries – see const NotCountry where “Diamond Princess” and “MS Zaandam” are countries ( there are cruise ships)

I should verify if there is always data for this day, if it exists for the previous ( yes, I know, Covid will eventually disappear or not be registered otherwise than influenza –but, for the moment ,it is ok) : see async function Verify in https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/importData.ts
I have had problems finding a CSV parser to TypeScript – I have found papaparse

3. Importing other data

I need to have the country population for each country, to display the relative percentage of the people . For this, I have the data from https://en.wikipedia.org/wiki/List_of_sovereign_states_and_dependent_territories_by_mortality_rate – but how to import into already existing data https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/countryList.ts ?

Simple: Import data into Excel. Make a case instruction with each country, iterate via countries, ( find discrepancies in names – use alternate names for this) and put the property as js , save the file. See https://github.com/ignatandrei/WFH_Resources/blob/master/makeData/test.ts