28 July 2018

# Automatically Update README.md with Travis-CI

Update: As of May 2020, a better solution is to switch from Travis Ci to the recently launched GitHub Actions

While working on content for the UBC OpenChemE Initiative, I got tired of manually updating the README.md file with new notebooks and decided to look for a more elegant solution.

The goal is to update README.md automatically with links to new Jupyter notebooks whenever we make a commit on GitHub. The readme file feeds into Jekyll/GitHub Pages and generates the website that we see here.

We’ll do this with a bash script that collects the names of all .ipynb files in the Notebooks folder and dump the contents into README.md with the right markdown formatting and nbviewer links. We’ll use Travis CI to run the script every time we make a commit on GitHub.

# Bash script

We’ll start by writing a bash script to grab all directories (*) in the Notebooks folder. Save this script as deploy.sh in a folder called scripts in your main project folder.

for d in ./Notebooks/* ; do
echo "$d" # Do something with files in this folder done  If you run that, you’ll see that $d prints out the entire path and not just the directory name. After some Googling, I found out how to split the base paths and the name using parameter expansion.

for d in ./Notebooks/* ; do
xpath=${d%/*} xbase=${d##*/}
xfext=${xbase##*.} xpref=${xbase%.*}

# Do something with files in this folder
done


Now, we want not only the directory, but also all notebook files in a particular directory, so we’ll do something like this to loop through the files:

for d in ./Notebooks/* ; do
xpath=${d%/*} xbase=${d##*/}
xfext=${xbase##*.} xpref=${xbase%.*}

echo "$d" for f in "$d"/*.ipynb; do
fpath=${f%/*} fbase=${f##*/}
ffext=${fbase##*.} fpref=${fbase%.*}

echo "$f" done done  If you run the script and look at the echo outputs, you’ll see that we are getting close. We want to output the directories as Markdown h1 headings and the files as bullet points. We also want a space between each heading for readibility. Finally, we want to write all this to the README.md file in the root directory. So here’s what we’ll do:  readme_path="README.md" for d in ./Notebooks/* ; do xpath=${d%/*}
xbase=${d##*/} xfext=${xbase##*.}
xpref=${xbase%.*} echo "#$xbase" >> "$readme_path" for f in "$d"/*.ipynb; do
fpath=${f%/*} fbase=${f##*/}
ffext=${fbase##*.} fpref=${fbase%.*}

echo "* $fpref" >> "$readme_path"
done

echo -e "\n" >> "$readme_path" done  For the individual files, we want to link to nbviewer directly, and convert spaces to %20 in the URL. Here’s how to do it. The // in ${xbase// /%20} means change all spaces   to %20.


nbviewer_path="http://nbviewer.jupyter.org/github/OpenChemE/CHBE356/blob/master/Notebooks"

for d in ./Notebooks/* ; do
xpath=${d%/*} xbase=${d##*/}
xfext=${xbase##*.} xpref=${xbase%.*}

echo "# $xbase" >> "$readme_path"

for f in "$d"/*.ipynb; do fpath=${f%/*}
fbase=${f##*/} ffext=${fbase##*.}
fpref=${fbase%.*} echo "* [$fpref]($nbviewer_path/${xbase// /%20}/${fbase// /%20})" >> "$readme_path"
done

echo -e "\n" >> "$readme_path" done  # Final bash script Almost there. We want to also create a separate ‘header’ file for our README.md and append the links below it. We will also add shopt -s nullglob in case we have empty folders. Here’s the final file: #!/bin/bash # a pattern that matches nothing "disappears", rather than treated as a literal string: # https://unix.stackexchange.com/questions/239772/bash-iterate-file-list-except-when-empty shopt -s nullglob # Our paths for the readme file header_path="header.md" readme_path="README.md" nbviewer_path="http://nbviewer.jupyter.org/github/OpenChemE/CHBE356/blob/master/Notebooks" # Copy the header over and add a blank line cat "$header_path" > "$readme_path" echo -e "\n" >> "$readme_path"

# https://stackoverflow.com/questions/3362920/get-just-the-filename-from-a-path-in-a-bash-script
for d in ./Notebooks/* ; do
xpath=${d%/*} xbase=${d##*/}
xfext=${xbase##*.} xpref=${xbase%.*}

echo "# $xbase" >> "$readme_path"

for f in "$d"/*.ipynb; do fpath=${f%/*}
fbase=${f##*/} ffext=${fbase##*.}
fpref=${fbase%.*} echo "* [$fpref]($nbviewer_path/${xbase// /%20}/${fbase// /%20})" >> "$readme_path"
done

echo -e "\n" >> "$readme_path" done  # Travis CI We’re all done with the bash script at this point. Now for the Travis CI part. Travis CI is integrated with GitHub. The service allows us to run scripts every time we make a commit on GitHub. These scripts could be unit tests or any bash script. I didn’t really understand the value of CI services like Travis CI or Circle CI until I had to implement unit tests for the UBC Envision site to prevent our team members from unintentionally(?) breaking it with bad commits (to be covered in a separate post). ## Set up a Travis CI account Goto http://travis-ci.org and register for an account. You’ll then need to link it to your GitHub repository by syncing your account and clicking Activate on the repository that you want. ## Set up your travis.yml file You’ll need to create a travis.yml file in your project’s root folder first. Let’s put this in for now, and we’ll get into the details later: language: ruby script: - bash ./scripts/deploy.sh branches: only: - master  ## Set up tokens We’ll need an authentication token from GitHub to allow Travis to make changes to the repository. Run this bash command and save the value of the token key on your screen: $ curl -u csianglim -d '{"scopes":["public_repo"],"note":"CI"}' https://api.github.com/authorizations


Note: If you need to redo the token setup process, go to your GitHub’s profile settings, Developer Settings, Personal Acceess Tokens and then remove the CI token first. After that, you can run the curl command again.

Navigate to your project folder and install the travis gem

$gem install travis  Make sure you’re in your project folder, then run this command to encrypt the token. Travis will automatically add it to your .travis.yml file: $ travis encrypt GIT_NAME="Travis CI" --add
$travis encrypt GIT_EMAIL="[email protected]" --add$ travis encrypt GH_TOKEN=<token> --add


Note: Make sure there is no space between your and the equal sign.

Here’s what my .travis.yml file looks like:

language: ruby
script:
- bash ./scripts/deploy.sh
branches:
only:
- master
env:
global:
- secure: <encrypted_token>
- secure: <encrypted_token>
- secure: <encrypted_token>


Now we just need add a few more lines to our deploy.sh script and tell Travis to set up git and push to the repo:

# Setup and push to master
git config --global user.email ${GIT_EMAIL} git config --global user.name${GIT_NAME}
git checkout master
git commit --message "Travis $TRAVIS_BUILD_NUMBER:$TRAVIS_COMMIT_MESSAGE"
git remote set-url origin https://${GH_TOKEN}@github.com/OpenChemE/CHBE356.git git push origin master  The ${GIT_EMAIL} and \${GIT_NAME} will be set to whatever you put previously as your tokens.