shiny is my goto package for building interactive dashboard. I have built more than fifty shiny apps so far and have found the entire package infrastructure around it to be extremely useful in showcasing data, algos, metrics - you name it. This book by Hadley Wickham came out in 2021 but I never had a chance to go over it, until now. It was wonderful to see so many code patterns/hacks that I have learned over years appearing in the book. Needless to say, I have learned so many fascinating aspects of shiny from this book. This blogpost summarizes some of my learnings/relearnings from the book:

Your first Shiny app

This chapter walks through the basic components of a Shiny app and helps the reader to create something a web app very quickly. Mainly I think it serves as a motivation to read the rest of the book.

Basic UI

This chapter gives a quick tour of all the input and output functions that make up any shinyapp. It is meant to be a quick refresher to all the shiny functions if you are a bit rusty with the package. If you are a beginner, of course reading through all the functions at once is a massive cognitive overload that can only be reduced by enough practice.

Basic Reactivity

  • ui object in a shinyapp contains the HTML presented to every user of your app. The ui is simple because every user gets the same HTML. The ui is simple because every user gets the same HTML. The server is more complicated because every user needs to get an independent version of the app. Shiny invokes your server() function each time a new session starts. When a server function, it creates a new local environment that is independent of every other invocation of the function. This allows each session to have a unique state, as well as isolating the variables created inside the function. Hence all the reactive programming you’ll do in Shiny will be inside the server function
  • server function takes in three inputs, input, output and session
  • input objects are read-only
  • input objects: it’s selective about who is allowed to read it .To read from an input, you must be in reactive context created by a function like renderText() or reactive()
  • The render function does two things
    • It sets up a special reactive context that automatically tracks what inputs the output uses
    • It converts the output of your R code into HTML suitable for display for a web page
  • The programming style of Shiny is declarative. You express higher-level goals or describe important constraints, and rely on someone else to decide how and/or when to translate that into action
  • A Shiny app will only ever do the minimal amount of work needed to update the output controls that you can currently see
  • The reactive graph contains one symbol for every input and output, and we connect an input to an output whenever the output accesses the input
  • Order the code is run is solely determined by reactive graph. This is different from most R code where the execution order is determined by the order of lines
  • producers - reactive inputs and expressions
  • consumers - reactive expressions and outputs
  • Why do we need reactive expressions ? Because in normal functions, you cannot access input variables
  • eventReactive has two arguments: the first argument specifies what to take a dependency on, and the second argument specifies what to compute
  • observeEvent has two important arguments, eventExpr and handlerExpr. The first argument is the input or expression to take a dependency on, the second argument is the code that will be run
  • There are two important differences between observeEvent and eventReactive
    • You don’t assign the result of observeEvent
    • You can’t refer to it from other reactive consumers

Workflow

  • Came across a link to a wonderful talk by Jennifer Bryan during RStudio conference(2022)
  • In R, every error is accompanies by a traceback or call stack, which literally traces back through the sequence of calls that lead to the error
  • Add a call to browser() in your source code to launch interactive debugger
  • To get the most useful help as quickly as possible, you need to create a reprex or reproducible example. The goal of reprex is to provide the smallest possible snippet of R code that illustrates the problem and can easily be run on another computer
  • Most of the learnings from the chapter is from the link to Jennifer Bryan’s talk. There are four strategies to debug your code
    1. Fresh start
      • Restart your R-console(Cleans workspace, resets options and environment variables, clears search path)
    2. Reproducible
      • Create a reprex so that you can recreate the bug with a simplified code so that others can help you
    3. Debug
      • Use traceback(), options(error=recover), browser() options
      • Comparison to Certificate of Death, Autopsy, War of Crafts Analogy
      • Use debug, debugonce functions
    4. Deter
      • Use libraries such as testthat packages to write unit tests
      • R CMD check
      • testthat::test_check()
      • Write errors for humans
      • Leave access panels
      • Automate your checks

Layout, Themes, HTML

  • Layout functions provide the high-level visual structure of an app. Layouts are created by a hierarchy of function calls, where the hierarchy in R matches the hierarchy in the generated HTML
  • To make complex layouts, you’ll need all layout functions inside of fluidPage(). To make a two-column layout with inputs on the left and outputs on the right, you can use sidebarLayout()
  • Break up page into pieces using tabsetPanel() and tabPanel().

I am familiar and have worked with most of the functions mentioned in the chapter and hence did not learn anything particularly interesting from this chapter

Graphics

  • A plot can respond to four different mouse events : click, dblclick, hover and brush
  • One can modify the size of the plot, one can pass width and height arguments to renderPlot
  • One can supply a string to the corresponding plotOutput arguments such as click, brush, dbclick, to create corresponding shiny input objects
  • There are functions built in shiny such as nearPoints, brushedPoints
  • The basic data flow in interactive plots in order to understand their limitations. The basic flow is something like this:
    • JavaScript captures the mouse event
    • Shiny sends the mouse event data back to R, telling the app that the input is now out of date
    • All the downstream reactive consumers are recomputed
    • plotOutput() generates a new PNG and sends it to the browser

User Feedback

  • req: This is a function that I should have used in my apps. I have always resorted to somewhat convoluted way of checking whether the user has provided an input or clicked a specific button before launching the reactive engine. This nice little function seems to be doing exactly that. This function is used to pause reactives so that nothing happens until some condition is true. req() checks for required values before allowing a reactive producer to continue.
    • It works by signaling a special condition that causes all downstream reactives and outputs to stop executing.
  • shinyFeedback= package can be used to display messages related to problems relating to a single input
  • validate() can be used to stop execution of the rest of the code and instead displays the message in any downstream outputs. My hack has always been to use reactive values and then check the values for displaying messages or data. However using validate() is definitely a better way of handling feedback and invalidation at the same time
  • By default, the message shown by showNotification will disappear after 5 seconds
  • withProgress() and incProgress() can be used to create progress bars
  • If you want more visual options, one can use waiter package. One can use use_waitress() function to show a visually appealing progress bar on a specific HTML element
  • waiter package provides a lot of functionality to show progress bars
  • I had never known about waiter package. The first version of the package came out in 2021. Here is what I learned from reading the manual
    • The function autoWaiter is there for convenience to easily add waiters to dynamically rendered Shiny content where ‘dynamic’ means render* and *output function pair. By default autoWaiter will be applied to all dynamically-rendered elements
    • One can show a loading screen on app launch
    • There are various functions in the package such as Waiter, Waitress, Hostess package

Uploads and downloads

This chapter is mainly about transferring files to and from a shinyapp. In the past, I have always provided pre-configured flat files for convenience. I don’t think I will deviate from the process for any future apps. So, for now, I have skimmed through the chapter and will revisit at a later date.

  • fileInput() variable returns a data frame with four columns, name, size, type and datapath
  • If the user is uploading a dataset, there are two details that one needs to be aware of
    • input$upload is initialized to NULL on page load
    • The accept argument allows you to limit the possible inputs
  • Unlike other outputs, downloadButton() is not paired with a render function. Instead, you use downloadHandler()

Dynamic UI

  • There are three key techniques for creating dynamic user interfaces
    • Using the update family of functions to modify parameters of input controls
    • Using tabsetPanel to conditionally show and hide parts of the user interface
    • Using uiOutput and renderUI to generate selected parts of the user interface with code
  • ObserveEvent and updateSliderInput usually go together to dynamically change UI elements
  • freezeReactiveValue ensures that any reactives or outputs that use the input won’t be updated until the next full round of invalidation
  • One can also use the tabsetPanel that has varies tabPabelBody but they are hidden. You can programatically control what gets shown by using selected parameter

Have used most of the techniques mentioned in this chapter in one of my shinyapps.

Bookmarking

This chapter talks about the way to store a specific state of shinyapp and revisit later. Skimmed this chapter and will revisit later. For all the 50 or more apps I have built, I have never found the use of bookmarking any of the demos. So, in all likelihood, I would never use this feature of Shiny

Tidy Evaluation

  • There are two types of variables env-variable and data-variable. The former is a programming variable that you create with <-. The latter is a statistical variable that lies inside a data frame
  • Data-masking functions allow you to use variables in the “current” data frame without any extra syntax. It’s used in many dplyr functions.
  • In base R, one can switch between data-variable and environment variable by switching from $ to [[
  • Inside data-masking functions, you can use .data and .env, if you want to be explicit about whether you’re talking about a data variable or an environment variable
1
2

diamonds %>% filter(.data$carat > .env$min)
diamonds %>% filter(.data[[var]] > .env$min)
  • Try to avoid paste() + parse() + eval() pattern as you can easily introduce bugs
  • Working with multiple variables is trivial when you are working with a function that uses tidy-selection: you can just pass a character vector of variable names in to any_of() or all_of(). Wouldn’t it be nice if we could do that in data-masking functions too .That’s the idea of the across() function. it allows you to use tidy-selection inside data-masking functions

I found this chapter very interesting and useful as I had paid attention to the fact that functions in tidyverse use data-variables and environment-variables. I had always hacked around and got this working. But reading this chapter makes it very clear on the way to handle the two types of variables in data masking and data selection functions. More over, this chapter also gives the motivation behind introducing across() function

Why reactivity?

  • Reactivity is important for shinyapps because they’re interactive: users change input controls which causes logic to run on the server ultimately resulting in outputs updating.
  • For shinyapps to be maximally useful, we need reactive expressions and outputs to update if and only if their inputs change. We want outputs to stay in sync with inputs while ensuring that we never do more work than necessary.
  • Event-driven programming solves the problem of unnecessary computation, but it creates a new problem: you have to carefully track which inputs affect which computations
  • A reactive expression has two important properties
    • it’s lazy
    • it’s cached
  • While the ideas of reactivity have bbeen around for a long time, it wasn’t until the late 1990s; that they were seriously studied in academic computer science. Research in reactive programming was kicked off by FRAN, a novel system for incorporating changes over time and user input into a functional programming language. It wasn’t until the 2010s that reactive programming roared into the programming mainstream through the fast-paced world of JavaScript UI frameworks.
  • Pioneering frameworks like Knockout, Ember, and Meteor demonstrated that reactive programming could make UI programming dramatically easier.

The reactive graph

  • As soon as the app is started and the server function has been executed for the first time, all reactive consumers and producers are instantiated in the reactive graph with no connections. All reactive expressions and outputs are in their starting state(invalidated grey)
  • The reactive inputs are ready(green) indicating that their values are available for computation
  • In the execution phase, Shiny picks an invalidated output and starts executing it. It traverses the reactive graph establishing connections in such a way that output value is filled. This happens for all the output nodes and session goes to rest
  • The most interesting part is what happens when an input changes. The system invalidates the relevant input, traverses through the dependencies and invalidates all the producers and consumers dependent on the changed input mode. It also removes all the relationships between those nodes. But why erase relationships ? That is the key part of Shiny’s reactive programming model: though these particular arrows were important, they are now out of date. The only way to ensure that our graph stays accurate is to erase arrows when they become stale, and let Shiny rediscover the relationships around these nodes as they re-execute
  • Learned about reactlog package that shows how the reactive graph evolves over time

Reactive building blocks

  • There are two types of reactive values. A single reactive value is created by reactiveVal(). A list of reactive values is created by reactiveValues
  • Reactive expressions cache errors in exactly the same way that they cache values
  • Errors are also treated the same way as values when it comes to the reactive graph: errors propagate through the reactive graph exactly the same way as regular values. The only difference is what happens when an error hits an output or observer:
    • An error in an output will be displayed in the app.
    • An error in an observer will cause the current session to terminate. If you don’t want this to happen, you’ll need to wrap the code in try() or tryCatch()
  • Observers and outputs are terminal nodes in a reactive graph. They differ from reactive expressions in two important was
    • They are eager and forgetful
    • The value returned by an observer is ignored because they are designed to work with functions called for their side-effects
    • Observers are often coupled with reactive values in order to track state changes over time
    • isolate() allows you to access the current value of a reactive or expression without taking a dependency on it
  • Difference between observeEvent() and eventReactive()
    • It’s like the difference between observe and reactive. One is intended to be run when some reactive variable is “triggered” and is meant to have side effects (observeEvent), and the other returns a reactive value and is meant to be used as a variable (eventReactive). Even in the documentation for those functions, the former is shown without being assigned to a variable (because it is intended to just produce a side effect), and the latter is shown to be assigned into a variable and used later on.
  • An eventReactive creates an object that you define like reactive does, but with out usual chain-reaction behavior you get from reactive. However it is lazily evaluated and cached like the other reactives.
  • An observeEvent can not create an object that you define (it creates something else). It is immediately evaluated and not cached. It is for causing side-effects.

Escaping the graph

  • Reactive dependency is not created between the reactive value and the observer in the following cases
    • You call an update function setting the value argument. This sends a message to the browser to change the value of an input, which then notifies R that the input value has been changed
    • You modify the value of a reactive value
  • All the examples mentioned in the chapter are centered around one message:
    • observeEvent in an app creates connections that are captured by the reactive graph and one can us invalidateLater() to manually control the invalidation of an observer

General guidelines

Skimmed this chapter as it was more about software development skills and not specific to Shiny as such

Functions

This chapter resonated with my work a lot as I have mostly followed the material mentioned in this book by trial and error. After building more than a dozen apps, I realized that there must be a way to cut down repetitive tasks in building ui and server components. That’s when I learned about namespaces in R that has completely changed the way I build a shiny app.

  • Why write separate functions ?
    • In the UI, you have components that are repeated in multiple places with minor variation. Pulling out repeated code into a function reduces duplication
    • In the server, complex reactives are hard to debug. Pulling out a reactive into a separate function, makes it substantially easier to debug
  • Put large functions in their own R/{function-name}.R
  • When extracting out helpers, avoid taking reactives as input or returning outputs. Instead, pass values into arguments and assume the caller will turn the result in to a reactive if needed.
  • Let’s say you have a complicated reactive function, it is better to put in req and then outsource the heavy duty to a non-reactive function

It was fantastic to see all the hacks that I have used in my work, formalized and expressed as concepts and rules that can be followed in a shinyapp development. Frankly I had never thought about the structure behind decoupling reactives and non-reactives. Feels so nice that all the patterns that I have been unconsciously following does have a structure to it.

Shiny Modules

Despite building over 50 shiny apps, I had never delved in to Shiny Modules and I must say this has been the biggest learning experience from the book. Before going through this chapter, I went through some of the online webinars on the topic and got familiarized with the topic. Here are some of the webinars/online videos that I went through:

After going through the above videos, I was in a much better shape to understand this chapter. The following are some of the points mentioned in this chapter:

  • A module is very similar to an app. Like an app, it’s composed of two pieces
    • module UI function that generates ui specification
    • module server function that runs code inside the server specification
  • Namespacing is different in module UI and server
    • It is explicit in module UI
    • It is implicit in moduleServer

Packages

  • The core idea of a package is that it’s a set of conventions for organising your code and related artefacts: if you follow those conventions, you get a bunch of tools for free.
  • Converting an app to a package
    • Create an R directory and move app.R into it
    • Transform your app into a standalone function
    • Call usethis::use_description() to create a description file
    • Remove any calls to source(), since load_all() automatically sources all .R files in R/
    • If you are loading datasets using read.csv you can instead use usethis::use_data(mydataset) to save the data in the data/ directory. load_all automatically loads the data for you
  • There are two common extra steps you might take beyond the basics: making it easy to deploy your app-package, and turning it into a “real” package
    • You will need an app.R that tells the deployment server how to run your app
    • With a DESCRIPTION file, you need to explicitly specify all the packages in it. One can do so using usethis::use_package("shiny") commands
  • A minimal package contains R/ directory, a DESCRIPTION file, and a function to run your app. This is very useful because it unlocks some useful workflows to speed up app development

Rest of the book

The last three chapters talk about testing, security and performance. By the time I had managed to come to these sections, I was a bit exhausted and reserved these chapters for a read at a later date.

Takeaway

This is probably one of the first books that I am reading, AFTER putting in a considerable amount of effort in building aroundy shiny apps. The advantage of reading AFTER putting in the work is that I was able spot patterns that I had figured out from trial and error. Seeing the patterns codified as general rules makes it easier to remember than trusting your subconscious mind to fill in, whenever it is needed. Shiny has a ton of features and it is only by building apps that one can learn and internalize. This book definitely helps you get a great comprehensive view of reactive framework as well as practical guidelines to developing shinyapps. My guess is that it will take at least a few years of developing apps to fully internalize the guidelines mentioned in this book and become really good at Shinyapp development.