In a previous post we wrote an article about an RShiny app for exploring traffic accidents in Scotland. Unfortunately, if we wanted to run this application today, we would face compatibility issues.
Imagine, if as an organisation we decided to deploy this application somewhere else or a colleague, client or any user wanted to run it on their own environment. Would it work? Can we ensure that it would work? How can we be certain that it would work especially in a production setting?
The problem we currently face is that every time that something does not work, since we have not taken any precautions against this, more time has to be invested in fixing it than extending it. This inadvertently adds to the technical debt of our application.
This problem becomes more prevalent when we have multiple tools, commonly in different codebases that have to be managed and maintained. Having different processes, especially for DS/ML applications, adds another layer in the hidden technical debt that we should be aiming to reduce (discussed in detail in the paper Hidden Technical Debt in Machine Learning Systems).
To alleviate this, one solution is to create reproducible workflows by managing our programming environments correctly. The objective is to use a single process in order to make an application easily reproducible, extensible and deployable with minimal or no friction at all.
In R’s ecosystem, there are several tools that help us with environment management and reproducibility, with renv, R-Studio Package Manager and checkpoint being a few of them. In this post we will focus on renv, an R library, and mamba, a library that streamlines a developer’s workflow for both R and python environments.
Running the R-Shiny App
Before running the application locally for the first time after it was written, we have to install all of the dependencies manually:
install.packages(c("leaflet", "shinyWidgets", "shinydashboard", "shinydashboardPlus", "shinyBS", "shinyalert", "shinycssloaders", "RColorBrewer", "htmltools", "htmlwidgets", "scales", "lattice", "dplyr", "xlsx", "ggplot2", "e1071", "ranger"))
Let’s run it. What happens?
If we don’t have Java installed we’ll get an error about Java required by the xlsx package (if you don’t have any issues, consider yourself lucky!). This is also not a good sign if we want to make a lightweight and portable application.
To fix this, we can use a library that does not introduce a transitive dependency of Java to our application. A good alternative is to use openxlsx. Let’s make the appropriate changes and verify them in the following diff:
-library(xlsx)
+library(openxlsx)
-bankHolidayList<-read.xlsx("data/BankHolidaysScotland.xlsx", sheetIndex = 1)
+bankHolidayList<-read.xlsx("data/BankHolidaysScotland.xlsx", sheet = 1)
Let’s install it with: install.packages("openxlsx").
Now we should be good to go! Let’s run the application again.
Alas! What happened again? This time it looks like the library shinydashboardPlus we are using, has broken backwards compatibility when it introduced changes to its API. This is worrying because we don’t know when the change was introduced and gives us one more reason to keep track of the library versions we use.
The latest version at the time of writing is 2.0.4 and that’s what was installed in the beginning. Fortunately, by searching we can find that version 0.7.5 will work with the current code version we have. Let’s install the older library version. We’ll use the remotes library that provides the install_version function and allows us to install an older version.
install.packages(“remotes”) remotes::install_version(“shinydashboardPlus”, “0.7.5”)
And that’s it! If we run the application once again, it should work as intended.
What we could have done differently: The case for renv
renv is a package that helps with the creation of reproducible R environments. It works by automatically pinning the versions of the packages we used to a specific version. Simply install it and type renv::init()
This will initialize renv and setup the environment for our application. We can see that now we have an renv.lock file in the project’s directory. The renv.lock file saves the packages and more importantly the versions of the libraries used so we can reproduce the application and run it anywhere.
At this point we should have a working app and the renv.lock file will have pinned the right versions of the libraries we used from the previous step. If we change or update any library we can regenerate a new lockfile by typing renv::snapshot()
If we wanted to run the application again in a new environment, all we need is the renv.lock file. To test this, let’s restart R and type:
renv::restore() shiny::runApp()
The application should run without errors.
What we could have done more: The case for mamba
Mamba is a new and faster dependency solver than conda (one of python’s most popular dependency solvers). Mamba is built on top of libmamba, and works as a replacement for conda. In this case, we are going to use it as the default solver to make a reproducible application and manage our R environment.
What makes it different from renv is that it’s not an R library bound by R’s ecosystem. This means that it makes an ideal candidate to address the technical debt problem as it enables us to standardise our workflow across both R and python without having to use different tools for each.
First, we have to install anaconda and then install mamba in the base conda environment using the conda-forge channel.
After installing, open anaconda prompt and type:
conda install mamba -n base -c conda-forge
Now that we have mamba installed, we can create a new environment called traffic-app, set it to use the conda-forge channel by default and activate it.
mamba create -n traffic-app -c conda-forge conda activate traffic-app
Since we didn’t use an environment the first time, we have to install the libraries manually. Notice, that we want to install the same older version as in the previous step for the r-shinydashboardplus library. In addition, r-conda packages have the same name as on CRAN but with the addition of an r- prefix.
mamba install -y r-base r-e1071 r-leaflet r-shinywidgets r-shinydashboard r-shinycssloaders r-shinyalert r-shinydashboardplus==0.7.5 r-shinybs r-dplyr r-caret r-ranger r-openxlsx
To make the process reproducible for later use, we have to export the libraries used in our traffic-app environment and pipe the output to a file. Mamba uses a YAML file to maintain the state of the environment, that includes the libraries, versions and all the relevant metadata that we need to re-build the same environment.
mamba env export -n traffic-app > requirements.yml
Now, if we wanted to run the application anywhere else all we have to do is to create a new environment and use the yaml file to install the libraries, which can be done in a single command:
mamba env create -n new-env -f requirements.yml
The new environment should have all the dependencies installed and will be ready to run the RShiny application. The downside of this approach is that there are not as many libraries in conda-forge as on CRAN. Nevertheless, there is a convenient skeleton that can easily convert an R-package to r-conda.
Final thoughts
We investigated two ways for making a reproducible Shiny App in R. We also saw how easy it is to add to the technical debt when we don’t: a) use a reproducible environment and b) pin the libraries’ versions. Whatever your choice of tools, make sure that at the very least you don’t make your future self’s life more difficult without adhering to some of coding’s best practices. The updated RShiny application and files can be found on our Github page.
Leave a Reply