Thomas Williams blog

R Markdown and the DBA: charting average full database backup durations

2026-08-18T02:00:00+00:00

DBAs spend time dealing with SQL Server performance, capacity, monitoring, troubleshooting, provisioning, etcetera; I’ve previously mentioned that R is a powerful, open-source language with a great ecosystem of libraries for analysis and visualisation, so it’s no surprise that I think mixing SQL Server and R Markdown for reporting goes together like a Vegemite and cheese sandwich…lunch perfection!

Here’s a couple of real-world examples of how I’ve used R Markdown connected to SQL Server (for a recap of how to do this from a technical perspective, see my earlier blog post “Connecting to a SQL Server database from R Markdown”):

SSIS error report from log records in the SSISDB database
Reporting Services reports from the ReportServer database - reports with schedules, execution history, folders & reports with non-parent permissions etc.
database data & log file size history, backup size history, SQL Agent job history, database inventory (really anything from the list in my automated DBA monitoring post that can supplement other monitoring)
data from third-party scripts such as sp_Blitz, sp_WhoIsActive, or Ola Hallengren’s backup and integrity checks

I’m going to look at the last one today, charting average full database backup durations from Ola Hallengren’s CommandLog table to see if outliers could be identified that may warrant further investigation:

To run R Markdown, you’ll need R and RStudio, or a Shiny Server, or Docker as I wrote about at “Dockerizing R Markdown/Shiny”.

As a DBA, why consider R Markdown for reports and dashboards? In addition to the wide variety of ready made - and free - libraries, one of the key advantages of Markdown is it’s just code i.e. text files. This makes sharing and versioning a lot simpler. Compared to the highly capable visualisation libraries like ggplot, Plotly, and Highcharter in R, the charts in Reporting Services are severely lacking. What’s more, developer mindshare has clearly left - there’s no cool new blog articles, chart types are old and static, and people have moved on to alternatives like Power BI, Python, R, Tableau etc.

It’s not going to be possible to demonstrate connecting to a “live” SQL Server, so instead I’ll simulate database full backup times similar to those recorded in Ola Hallengren’s CommandLog table. I’ll chart average backup times by day which might identify slower days, showing details for the day when clicking on a chart point. All this in an interactive single R Markdown file.

Why would I be interested in how long full database backups take? It might help point to contention with storage, unusual database growth, adding a new database to a server, or other activity. I’d want to know if this happens repeatedly or is a once-off. I also don’t want to hard-code a threshold to investigate (e.g. “backups were 5 minutes longer than usual” or “5% longer than usual”), so I’ll look for a way to tackle this.

Setting up the environment

Before starting, make sure of an up-to-date version of R and RStudio installed, along with following R packages: DT, dplyr, highcharter, glue, lubridate, and purrr (plus DBI and odbc for database connectivity).

You can get the complete code from my repo at https://github.com/thomasswilliams/r-markdown-DBA/blob/main/DBA-backup-durations.Rmd.

The first section fakes 10 days worth of full database backup times for a server in a data frame df (I won’t go over that here).

The next section takes the data frame, calculates some stats, and charts the average full backup duration per day as a psuedo-statistical control chart with median and upper/lower limits. Here’s the code:

# create a new, summary, dataset based on the average elapsed seconds per day
# calculate the median, sigma and upper and lower "control" limits
# display in a Highchart chart
# adapted in part from https://norbertocioffi.medium.com/shewhart-control-charts-with-r-f62a3359a4a1

# placeholder for highchart
highcharter::highchartOutput("summary_chart", height = "500px")

# render highchart
output$summary_chart <- highcharter::renderHighchart({

  # create new summary data frame from df: date (no time), average_elapsed_seconds
  df_summary <- df %>%
    # get date, no time
    dplyr::mutate(date = as.Date(StartTime)) %>%
    # group by date
    dplyr::group_by(date) %>%
    # average elapsed seconds per day (ignore DatabaseName)
    dplyr::summarise(
      average_elapsed_seconds = mean(elapsed_seconds),
      # drop fields not explicitly named here
      .groups = "drop"
    ) %>%
    # order by date (necessary for chart)
    dplyr::arrange(date)

  # calculate limits using Median Absolute Deviation (MAD) method
  center <- median(df_summary$average_elapsed_seconds)
  # calculate sigma using MAD, multiplied by constant to make comparable to standard deviation for normal distribution
  sigma <- mad(df_summary$average_elapsed_seconds, constant = 1.4826)
  # Upper Control Limit, rounded to nearest whole second three sigma above the center/median
  ucl <- round(center + 3L * sigma)
  # Lower Control Limit, rounded to nearest whole second
  lcl <- round(center - 3L * sigma)
  # reset LCL to zero if lower than zero (duration in seconds cannot be *less* than zero)
  lcl <- max(lcl, 0L)

  # add coloring for if plot points are outliers (only for above upper range)
  # red for out of range, transparent for in range
  df_summary <- df_summary %>%
    dplyr::mutate(
      point_color = ifelse(average_elapsed_seconds > ucl, "red", "transparent")
    )
  # new field "ts", convert date to JS timestamp
  df_summary <- df_summary %>%
    dplyr::mutate(ts = highcharter::datetime_to_timestamp(date))

  # output the chart
  highcharter::highchart() %>%
    # line chart
    highcharter::hc_chart(type = "line") %>%
    # title
    highcharter::hc_title(
      text = "Database average backup duration by day",
      align = "left",
      x = 55L
    ) %>%
    highcharter::hc_subtitle(
      text = "Click a point to see backup details by database for that day",
      align = "left",
      x = 55L
    ) %>%
    # axis labels
    # x axis is date, formatted as day/month (e.g. 1/Jan)
    highcharter::hc_xAxis(
      type = "datetime",
      labels = list(format = "{value:%e/%b}")
    ) %>%
    # y axis is average duration in seconds
    highcharter::hc_yAxis(
      title = list(text = "Avg. duration (seconds)"),
      # format with thousands separator and no decimal places
      labels = list(format = "{value:,.0f}"),
      # on y axis, add dotted green lines for UCL and LCL, and dashed purple line for center/median
      # left-aligned labels in same colors
      plotLines = list(
        list(value = ucl, color = "#3EA57B", width = 1L, dashStyle = "Dot", opacity = 0.6, label = list(text = paste0("Upper control limit = ", ucl, " secs"), align = "left", x = 0L, style = list(color = "#3EA57B"))),
        list(value = lcl, color = "#3EA57B", width = 1L, dashStyle = "Dot", opacity = 0.6, label = list(text = paste0("Lower control limit = ", lcl, " secs"), align = "left", x = 0L, style = list(color = "#3EA57B"))),
        list(value = center, color = "#9481CC", width = 1L, dashStyle = "Dash", opacity = 0.6, label = list(text = paste0("Median/centerline = ", round(center, 2L), " secs"), align = "left", x = 0L, style = list(color = "#9481CC")))
      ),
      # set y axis to start at zero, and hide the zero label (as it's obvious from the axis)
      # start at zero
      min = 0L,
      # don't display the zero label
      showFirstLabel = FALSE
    ) %>%
    # custom tooltip
    highcharter::hc_tooltip(
      useHTML = TRUE,
      formatter = JS("
        function() {
          return '<b>Date:</b> ' + Highcharts.dateFormat('%e/%b/%Y', this.x) + '<br>' +
                 '<b>Avg. duration:</b> ' + Highcharts.numberFormat(this.y, 0) + ' seconds';
        }
      ")
    ) %>%
    # line series
    highcharter::hc_add_series(
      name = "Average duration",
      data = purrr::map2(
        # map timestamp to x, and elapsed seconds to y
        df_summary$ts,
        df_summary$average_elapsed_seconds,
        # output a list
        ~list(x = .x, y = .y)
      ),
      # heavy line
      lineWidth = 4L,
      color = "#8f8f8f",
      opacity = 0.8
    ) %>%
    # points
    highcharter::hc_add_series(
      name = "Points",
      type = "scatter",
      data = purrr::pmap(
        # map timestamp to x, elapsed seconds to y, and color
        list(df_summary$ts, df_summary$average_elapsed_seconds, df_summary$point_color),
        # set color based on if above UCL (red) or not
        function(x, y, col) {
          # output a list
          list(x = x, y = y, color = col)
        }
      ),
      # slightly larger point
      marker = list(radius = 3L)
    ) %>%
    # disable the legend (as only one series, points are colored by if above UCL or not, legend not needed)
    highcharter::hc_legend(
      enabled = FALSE
    ) %>%
    # for clicks on a point, set "selected_date" to the date of the clicked point
    # (converted from JS timestamp to R date)
    highcharter::hc_plotOptions(
      series = list(
        cursor = "pointer",
        point = list(
          events = list(
            # set selected date on click, convert from JS timestamp to R date in format YYYY-MM-DD
            # set the input value input$selected_date, which triggers a change to selected_date_outer$date
            click = JS("function() { Shiny.setInputValue('selected_date', new Date(this.x).toISOString().slice(0,10), { priority: 'event' }); }") # nolint: line_len
          )
        )
      )
    )
})

I’ve commented the code liberally, but if this is your first time looking at R Markdown, here’s a couple of hints to reading:

”#” starts a comment
”%>%” pipes one statement to another
”<-“ is assignment
passing a list (needed for some highcharter calls) is explicit with “list” e.g. list(x = x, y = y, color = col)
functions can be prefaced by their library for disambiguation e.g. highcharter::highchartOutput
fields in a data frame retrieved from a database with odbc are strongly-typed e.g. dates are POSIXct, strings are characters, numbers are numbers etc.
R can operate efficiently on sets of data without looping - the following sets a variable “center” to the median value of the field “average_elapsed_seconds” in the data frame: center <- median(df_summary$average_elapsed_seconds)

The last section in the R Markdown file is the detail table, shown only when clicking on a date in the chart. See the R Markdown file for code - it’s pretty straightforward, filtering the data frame df for the selected_date_outer$date date, and displaying full database backups for that date.

Running the R Markdown file gets interactive HTML output - tooltip hovering on points in the chart, clicking a point in the chart, table sorting. It’s worth noting how little R code it takes to get a data frame to an interactive table (I embellish with extra formatting).

This is a start on what’s possible using R Markdown. I’ve left some suggestions for “production-ready” upgrades in the R Markdown file. You can find the full code in my new DBA R Markdown repository on GitHub at https://github.com/thomasswilliams/r-markdown-DBA.

More improving inputs in Shiny & R Markdown

2026-07-27T02:00:00+00:00

In my last post on improving inputs in R Markdown/Shiny, I covered four improvements to quality of life in interactive reports and dashboards. In this blog post I have three more, all approaches I’ve used to add professionalism to self-service R Markdown files.

In yet another plug for R Markdown: most of these techniques can be used in a single *.Rmd file, or can be included in many by putting them in a common CSS or javascript file and linking to it.

Newer Bootstrap

For a start, let’s use bslib defined in the YAML front-matter, which bumps the Bootstrap version from 3 (circa 2015) to 5 (circa 2021):

output:
  html_document:
    theme:
      # bslib Bootstrap version 5 theming
      # required by bslib components like "tooltip"
      version: 5
      bslib: true

With bslib comes a simpler dark mode switch too, with a nice animated icon. Now, a working dark mode switcher is as simple as:

# R
bslib::input_dark_mode(id = "dark_mode")

CSS to visually improve inputs

I’ve noticed with some themes, Shiny inputs don’t visually differentiate from a plain text box. The CSS below displays background icons from SVGs for date inputs and dropdowns, but could be extended for other inputs:

/* CSS */
/* dropdown/select caret */
select.shiny-input-select {
  padding-right: 2rem;
  /* Bootstrap down caret; color ("fill") works well with dark or light theme */
  background-image:
    url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='16' height='16' fill='%236c757d' viewBox='0 0 16 16'%3E%3Cpath fill-rule='evenodd' d='M1.646 5.646a.5.5 0 0 1 .708 0L8 11.293l5.646-5.647a.5.5 0 0 1 .708.708l-6 6a.5.5 0 0 1-.708 0l-6-6a.5.5 0 0 1 0-.708z'/%3E%3C/svg%3E");
  background-repeat: no-repeat;
  background-position: right 0.5rem center;
  background-size: 1rem;
}

/* date parameter (remember to use your own identifer) - style with calendar icon at right of input */
#date_input > .form-control {
  padding-right: 2rem;
  /* calendar icon from SVG, translated from Bootstrap "calendar3" icon; color ("fill") works well with dark or light theme */
  background-image:
    url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='16' height='16' fill='%236c757d' viewBox='0 0 16 16'%3E%3Cpath d='M11 0a1 1 0 0 1 1 1v1h1.5A1.5 1.5 0 0 1 15 3.5v11A1.5 1.5 0 0 1 13.5 16h-11A1.5 1.5 0 0 1 1 14.5v-11A1.5 1.5 0 0 1 2.5 2H4V1a1 1 0 0 1 2 0v1h4V1a1 1 0 0 1 1-1zM2 6v8.5a.5.5 0 0 0 .5.5h11a.5.5 0 0 0 .5-.5V6H2z'/%3E%3C/svg%3E");
  background-repeat: no-repeat;
  background-position: right 0.5rem center;
  background-size: 1rem;
}

See the source code at my repo https://github.com/thomasswilliams/r-markdown-snippets/blob/main/improve-inputs-2.Rmd which can be run using R and RStudio.

On a side note, the date input has a lot of options including disabling future dates, setting earliest selectable date, formatting the date, setting the start day of the week - it might be worth taking the time to read the docs at https://shiny.posit.co/r/reference/shiny/latest/dateinput.html.

CSS styling could be taken further by using a background icon that has meaning for users - either at right (like demonstrated above) or the left side - and helps visually differentiate the purpose of the input. For example, a flag icon for countries, a map icon if selecting localities, phone icon for phone numbers, currency indicator for money etc.

Taking it to the next stage: https://dreamrs.github.io/shinyWidgets/ has a lot more input types.

Hiding complex parameters behind links

This technique is about conserving space and hiding complexity. If a report has a few well-used combinations of parameters, a link could be used to quickly set them. Below is the code for links which call javascript to set the month and “type”, though every report is different. This technique works best with larger numbers of parameters, or even seldom-used parameters you may not want to make visible if they add clutter.

First, the R snippet which generates the links and respective javascript code:

# R
# get current month so we can set previous month
current_month_date <- Sys.Date()
# use lubridate to get last and next month names which we can pass to javascript
last_month_name <- format(current_month_date %m-% months(1), "%B")
next_month_name <- format(current_month_date %m+% months(1), "%B")
# need "results = 'asis'" to output HTML directly, otherwise will be escaped
# these are the quick links, in this case calling custom functions to set one or more parameters
# call custom javascript functions to set the month and type inputs
# need to escape single quotes with leading backslash
cat(sprintf(
  '<br><a href="#" onclick="setMonthInput(\'%s\'); setTypeInput(\'Sales\'); return false;">&bullet; Last month\'s sales</a><br><a href="#" onclick="setMonthInput(\'%s\'); setTypeInput(\'Projected\'); return false;">&bullet; Next month\'s projected</a>',
  last_month_name, next_month_name
))

Then the javascript functions themselves, called by the links above like setMonthInput('June'); setTypeInput('Sales');:

// javascript
// custom javascript functions to set Shiny inputs
// set month input to passed full month name
function setMonthInput(monthName) {
  // get the Selectize instance for the month input
  // can substitute your own Shiny input name here
  var selectizeInstance = $('#month_input')[0].selectize;
  // if we got the Selectize instance, use its API to set the value
  if (selectizeInstance) {
    // use Selectize API to set value rather than Shiny
    // don't suppress change event (second argument), so Shiny will react to change
    selectizeInstance.setValue(monthName, false);
  } else {
    // fallback if Selectize hasn't initialised yet and just set raw HTML select value
    // using jQuery
    $('#month_input').val(monthName).trigger('change');
  }
}
// set "type" dropdown to passed value
function setTypeInput(typeName) {
  // set the value, trigger change event
  $('#type_input').val(typeName).trigger('change');
}

#tsql2sday July 2026: Query red flags

2026-07-06T14:00:00+00:00

Brent Ozar asks what makers us groan on reading in an SQL query, for this month’s T-SQL Tuesday.

Nowadays I look after 3rd-party databases more than internally-developed ones, so I accept there’s a whole lot of ex-best practices, vendor preferences, and possibly shortcuts in queries I might come across - whether it’s a poorly-performing query, a blocker, or an error.

(Although, when I developed software more frequently, I was guilty of all the gripes below. My start in SQL, last century, was poring over a big yellow “For Dummies” book. I was the dummy.)

SELECT *

In my experience, this is a shortcut that doesn’t save time in the long run. Often just the opposite - create a table, create a view with SELECT * FROM table1, update the table to add/remove columns, and the view breaks!
Instead: explicitly specify the column names. Yes, you may have to do it multiple times if you use common table expressions or subqueries.

Hungarian notation like tbl_Something, vw_SomethingElse

There was possibly a time where this helped, but I believe the art of writing software has moved on and Hungarian notation is no longer needed or desired.
My approach is: lower-case, snake-case table names that reflects the data stored, often in plurals e.g. users, addresses, categories etc. I don’t often use views, but once again I’d name based on data returned user_locations, addresses_no_deliveries etc.

Spaces in returned column names

I find this can cause problems consuming the query.
I favor: lower snake-case column names, I can always change the names in the viewer (R Markdown, Excel, Power BI etc.), and lower snake-case is what JSON uses.

Big long queries with no formatting or line breaks

Perhaps the return key was broken?
To fix: I format things how I want anyway, helps my readability. An LLM can do this easily.

Old school joins like …FROM table1, table2, table3 WHERE table1.x = table2.y…

I understand why people might write joins this way, but I’ve found that the join clauses may get missed or inadvertently commented out.
The modern alternative: is ...FROM table1 INNER JOIN table2 ON table1.x = table2.y..., which gives a natural flow to remembering the ON bits.

NTEXT and TIMESTAMP data types

These can appear in ooold databases and will cause headaches.
Strong recommendation: convert to modern data types to do anything with them.

Wrapping up

These are a couple of my query red flags. Of course there’s other things I don’t like seeing - index hints and user-defined types for every field come to mind - but I admit they may not harm the query or give me a headache when investigating.

I’ve been fan of Brent’s writing and work for some time (BlitzFirst is amazing, will hopefully have more to write on this), and I look forward to reading others’ takes on this month’s T-SQL Tuesday query nasties!

Improving inputs in Shiny & R Markdown

2026-06-24T02:00:00+00:00

Shiny - and R Markdown output to HTML - have a range of input controls available for data-driven interactive reports & dashboards: text, dropdowns, date, checkboxes, range sliders, and more.

I’ve got a few posts on R Markdown and a whole repo with tips and tricks at https://github.com/thomasswilliams/r-markdown-snippets. Here’s why I keep plugging R Markdown and Shiny: it’s a free, open-source language; using a declarative format (Markdown) with very little boilerplate; incorporating amazing third-party libraries for data analysis & display; that can be deployed using Docker; resulting in interactive, powerful, self-serve reporting. With a few lines of code I can display the results of SQL query in a paged, sortable table - or interactive chart. For a time-poor DBA, having a way to easily get data out is key.

In this post I go over some low-effort ways to improve controls to save space, offer missing functionality, and better help end users. See the file https://github.com/thomasswilliams/r-markdown-snippets/blob/main/improve-inputs-1.Rmd for the full source code in a ready-to-run R Markdown file (all you need is R and RStudio).

Shortcuts for checkbox groups

Thoughtful parameters can make complicated reports & dashboards simpler. Rendering R Markdown to a web page means you can use CSS and javascript enhancements, like I’ve done to enhance a group of checkboxes with jQuery to select/deselect items in a single click:

# R
shiny::checkboxGroupInput(
  "checkbox_group_input",
  # output HTML into the label
  label = htmltools::HTML("Checkbox group with quick links <a class='quick-link' id='checkbox_group_input_select_all'>All</a><span style='opacity:0.3'>|</span><a class='quick-link' id='checkbox_group_input_select_none'>None</a><span style='opacity:0.3'>|</span><a class='quick-link' id='checkbox_group_input_select_default'>Defaults</a>"),
  choices = c("Kermit", "Fozzie", "Miss Piggy", "Gonzo"),
  selected = c("Kermit", "Fozzie"),
  width = "400px"
)

/* CSS */
/* "select all"/"none"/"defaults" quick links for check box group */
.quick-link {
  /* smaller text */
  font-weight: normal;
  font-size: 0.8em;
  /* underline */
  text-decoration: underline dotted;
  /* style like other links */
  color: #007bc2;
  text-decoration-color: #007bc2dd;
  /* cursor */
  cursor: pointer;
}

// javascript
// "select all" quick link click
$(document).on("click", "#checkbox_group_input_select_all", function() {
  // check all options in the checkbox group
  $('#checkbox_group_input input[type="checkbox"]').prop('checked', true).trigger('change');
});
// "select none" quick link click
$(document).on("click", "#checkbox_group_input_select_none", function() {
  // uncheck all checkboxes
  $('#checkbox_group_input input[type="checkbox"]').prop('checked', false).trigger('change');
});
// "select defaults" quick link click
$(document).on("click", "#checkbox_group_input_select_default", function() {
  // uncheck all checkboxes
  $('#checkbox_group_input input[type="checkbox"]').prop('checked', false);
  // loop through all checkbox inputs
  $('#checkbox_group_input input[type="checkbox"]').filter(function() {
    // get the value of the checkbox
    var val = this.value;
    // return true if value matches with case-sensitive, hard-coded defaults
    return val == "Kermit" || val == "Fozzie";
  }).prop('checked', true).trigger('change');
});

A multi-select dropdown may be preferable to a checkbox group, specially when vertical space is a premium. One consideration is losing a little visibility into available items with a multi-select dropdown. An enhancement for the multi-select dropdown is styling items - I use this technique to match color-coding in a report:

# R
# multi-select Selectize input, with colors thanks to custom CSS
# see CSS below
shiny::selectizeInput(
  "multi_selectize_input",
  label = "Multi-select dropdown with item styling",
  choices = c("Green", "Blue", "Red"),
  # defaults from YAML front-matter
  selected = params$default_colors,
  # allow multiple
  multiple = TRUE,
  # wider
  width = "400px"
)

/* CSS */
/* styles for multi-select Selectize input items
   need to match the "data-value" to case-sensitive choices in the input */
#multi_selectize_input + .shiny-input-select.multi div.item[data-value="Green"] {
  color: green;
  border: 1px solid green;
}
#multi_selectize_input + .shiny-input-select.multi div.item[data-value="Blue"] {
  color: blue;
  border: 1px solid blue;
}
#multi_selectize_input + .shiny-input-select.multi div.item[data-value="Red"] {
  background-color: red;
  /* white text for better contrast */
  color: white !important;
}

Tooltips and placeholder text

Placeholder text is great, but when there’s real text in the input, the placeholder is hidden. One approach I’ve utilised is tooltips from bslib - wrapping the control in a tooltip means the tooltip is visible when the user is typing, so can provide context-sensitive help:

# R
# text input surrounded by tooltip, tooltip will be visible on focus
# and hover; requires bslib
bslib::tooltip(
  # this is the original input
  shiny::textInput(
    "text_input",
    label = "Tooltip for text input",
    placeholder = "Placeholder for text input"
  ),
  # tooltip text
  "Tooltip visible on focus and hover",
  # tooltip position
  placement = "right"
)

Columns vs. single line

Inputs in R Markdown can often take up a whole line. Here’s one method, using Pandoc fenced divs, that allows for columns to locate inputs next to each other:

<!-- a div with more colons, can contain one or more divs with less colons -->
:::: {.row}
<!-- Bootstrap styles -->
::: {.col-md-3}
```{r, echo = FALSE}
# create a normal dropdown (not Selectize)
# if no selected, defaults to first option - there is no "empty" option
shiny::selectInput("select_input", label = "Single option dropdown", choices = c("Small", "Medium", "Large"), selectize = FALSE, width = "200px")
```
:::
<!-- increase top padding to line up with input -->
::: {.col-md-9 style="margin-top: 38px;"}
```{r, echo = FALSE}
# bslib switch
bslib::input_switch("switch_input", label = "Switch input")
```
:::
::::

(See full docs at https://pkg.yihui.org/rmarkdown-cookbook/multi-column.)

Strava activity heatmap in R Markdown

2026-06-12T02:00:00+00:00

Recently I came across Sam Wilson’s Python heatmap map of Strava runs, and wanted to see if I could get a similar result using R Markdown. Check out the original for a great write-up of the ins and outs and mapping frequency of visits, pace and gradient at Sam’s GitHub repo https://github.com/moresamwilson/running-heatmap.

Instead of using the Strava API (like I did in April for the 2026 chart challenge), Sam uses a download of Strava activities which makes the whole process a lot faster. I’ve adopted the same approach, with some help from Claude.ai.

My version keeps just the frequency. The hard work of producing the “heatmap” is via Leaflet (which I’ve blogged about before) and leaflet.extras::addHeatmap - basically, the more times a GPS point appears in the Strava activities, the brighter it’s displayed on the map.

Here’s the heatmap generated from my runs around beautiful Hobart, Tasmania…looking (unintentionally) a little like a drawing of a falling runner:

The R Markdown file, so you can generate your own heatmap, can be found at https://github.com/thomasswilliams/r-strava-heatmap-2026.

To use the R Markdown file

You’ll need:

If you don’t want to install R and RStudio, an alternative is to run the code in Docker. I go through the necessary steps for another R Markdown file at https://github.com/thomasswilliams/r-markdown-docker-test.

R and RStudio
packages shiny, flexdashboard, leaflet, leaflet.extras and others (see the strava_heatmap.Rmd R Markdown file in my GitHub repo at https://github.com/thomasswilliams/r-strava-heatmap-2026/blob/main/strava_heatmap.Rmd)
a download of your Strava data; Strava calls this “bulk export” and documents it at https://support.strava.com/hc/en-us/articles/216918437-Exporting-your-Data-and-Bulk-Export#h_01KGG64N4RTPX2SC9VV9JJZR3R
unzip the download into a directory called strava_export in the same directory as strava_heatmap.Rmd - the directory structure will look like:

┣ 📂 strava_export
┃ ┣ 📂 activities
┃ ┃ ┣ (.gpx, .fit, .gpx.gz files)
┃ ┃ ┗ ...
┃ ┗ activities.csv
┗ strava_heatmap.Rmd

You only need the activities directory and activities.csv from the Strava download.

With all this in place, run the R Markdown file in RStudio, and click the “Load data” button.

To get the best results, center the Leaflet map using latitude and longitude, and set an appropriate zoom level - controlled in strava_heatmap.Rmd at the top of the file on lines 55-59:

# map centre — Hobart, Tasmania defaults for me!
DEFAULT_LAT <- -42.8821
DEFAULT_LON <- 147.3272
# initial zoom level of Leaflet map
DEFAULT_ZOOM <- 13L

Other options at the top of the file include: ACTIVITY_TYPES, which controls the type of Strava activities included (though only those with GPS points display on the map); and DEBUG helps with understanding why certain files don’t load.

Parallel SSH

2026-05-18T02:00:00+00:00

In my work I support Linux servers, which means focusing on the command line; that starts with the SSH command.

(I’ve previously blogged about personalising the Linux command line, and different SSH keys - this post continues this theme.)

Parallel SSH (install with sudo apt install pssh) is faster to run commands against multiple remote hosts from the command line, while using SSH under the hood. The output is a host name header followed by the command output, for instance running the date command on SERVER1 and SERVER2 with parallel SSH would result in:

[1] 09:14:18 [SUCCESS] person@SERVER1.local
Mon 18 May 18 2026 09:14:18 AEST
[2] 09:14:18 [SUCCESS] person@SERVER2.local
Mon 18 May 18 2026 09:14:18 AEST

You can see from the results I connected as person, and the command was run on both servers at once.

While parallel SSH isn’t going to beat a full-blown automation tool, it’s great for a small amount of servers, or a small number of changes - and it doesn’t require afore-mentioned tool. Nothing needs to be installed on the remote hosts.

Anatomy of a parallel-ssh command

Breaking down the above command to connect to two remote hosts, and run a command on each: parallel-ssh -i -H 'person@SERVER1.local person@SERVER2.local' -A -x '-i ~/.ssh/key_file_name' -- date:

interactive (e.g. run command): -i
list of hosts follows: -H
passes SSH key in extra options to SSH command set the key file name: -x
if key has passphrase (it should!), parallel-ssh prompts once for the passphrase (using the same key for all the hosts): -A
command to run is after double dashes
fails if you haven’t accepted server fingerprint (so, need to connect first time manually)
if username is the same for all connections (like person), can specify once with -l: parallel-ssh -i -l person -H 'SERVER1.local SERVER2.local' -A -x '-i ~/.ssh/key_file_name ' -- w
can specify file instead of list of hosts (-H) with -h linux_hosts.txt (you can use any name you like, I chose “linux_hosts.txt”)
- newline separated
- can contain comments starting with hash
- in list of hosts, can also specify user to connect with per connection, or not e.g. person2@SERVER1.local SERVER2.local
I haven’t needed it yet, but appears difficult to be prompted for sudo password when running a command that requires sudo on multiple remote hosts…though possibly a start at https://groups.google.com/g/parallel-ssh/c/hN3qf5zQ7gI/m/KdmGe8woCgAJ

Full documentation is at https://manpages.ubuntu.com/manpages/jammy/man1/parallel-ssh.1.html.

Here’s some of the ways I’ve used parallel SSH, given a file linux_hosts.txt with a list of remote hosts, separated by newlines:

# what version of OS is running
parallel-ssh -i -l person -h ~/linux_hosts.txt -A -x '-i ~/.ssh/key_file_name' 'hostnamectl | grep Operating'

# what SSH keys do I have deployed
parallel-ssh -i -l person -h ~/linux_hosts.txt -A -x '-i ~/.ssh/key_file_name' 'cat ~/.ssh/authorized_keys'

# getting a hash of a file, to compare to see if it's the same file
# e.g. with config files such as .inputrc or .bashrc
parallel-ssh -i -l person -h ~/linux_hosts.txt -A -x '-i ~/.ssh/key_file_name' 'sha256sum ~/.ssh/authorized_keys'

# it's possible to get a hash of a whole directory too! Claude tells me the command is something like
parallel-ssh -i -l person -h ~/linux_hosts.txt -A -x '-i ~/.ssh/key_file_name' "find ~ -maxdepth 1 -type f -printf '%f|%s|%TY-%Tm-%Td %TH:%TM:%TS|%m|%u|%g\n' | sort | sha256sum"

There’s also parallel SCP, which you can use to copy files from your computer to multiple hosts at the same time:

parallel-scp -l person -h ~/linux_hosts.txt -A -x '-i ~/.ssh/key_file_name' local.file remote.file

(And parallel-slurp for the reverse.)

References:

Mermaid diagrams for Jekyll

2026-05-10T02:00:00+00:00

Mermaid https://mermaid.ai/ is “Markdown inspired” diagrams as code. With the diagram defined as text, special/proprietary software isn’t needed to create or edit diagrams. The text defining the diagrams can be edited by anyone - so can be kept up-to-date. Diagrams can also be version controlled, like any other code.

This matters because better diagrams improve documentation, and communication.

Mermaid supports a lot of different diagram types (flowcharts, Gantt charts, mindmaps, and org charts, among others). Mermaid syntax is supported in Confluence, GitHub, VS Code, MediaWiki, and more.

If you don’t want to fiddle with text, there’s online, free tools to visually create diagrams, which can then be exported to Mermaid syntax (though be aware, there’s perhaps not the level of fine-grained control that might be expected using afore-mentioned proprietary software like Visio or https://app.diagrams.net/, which is a trade-off for the simplicity of text).

Here’s a simple example of a flowchart diagram in Mermaid. The code block below results in the diagram underneath:

```mermaid
flowchart TD
  A(["Normal text and unicode characters 👤 ★"]) --> B("<strong>HTML</strong><br><em>format</em>")
  B -.-> C["`Or **Markdown** _format_`"]
  D@{ shape: text, label: "<small>See full syntax at <a href=https://mermaid.ai/open-source/syntax/flowchart.html>https://mermaid.ai/open-source/syntax/flowchart.html<a></small>" }
  style A stroke-width:2px
  style C fill:#FFE97F,stroke:#fc0,stroke-dasharray:3,color:#333
```

flowchart TD
  A(["Normal text and unicode characters 👤 ★"]) --> B("<strong>HTML</strong><br><em>format</em>")
  B -.-> C["`Or **Markdown** _format_`"]
  D@{ shape: text, label: "<small>See full syntax at <a href=https://mermaid.ai/open-source/syntax/flowchart.html>https://mermaid.ai/open-source/syntax/flowchart.html<a></small>" }
  style A stroke-width:2px
  style C fill:#FFE97F,stroke:#fc0,stroke-dasharray:3,color:#333

I use these sort of diagrams for quick, high-level system architecture. Rendered in a browser, text in the diagram is selectable and links can be clicked.

Of course, there’s a lot more than can be done with regards to styling & font, layout, dark mode etc. In the flowchart above, I use single upper-case letters to refer to “nodes”, with the displayed text separate (which makes connecting, referencing and styling nodes easier).

Some other useful Mermaid flowchart styling:

dotted lines on nodes: style <node> stroke-dasharray:3 (or even dots and dashes, see https://developer.mozilla.org/en-US/docs/Web/SVG/Reference/Attribute/stroke-dasharray)
node text color: style <node> color:<hex color code>
node background color: style <node> fill:<hex color code>
node border width: style <node> stroke-width:<number><unit e.g. "px", "pt">

Here’s how to add Mermaid to a Jekyll blog, based on https://stuff-things.net/2025/01/19/mermaid-diagramming-in-jekyll-in-2025/:

create a new file mermaid.html in the _includes directory
put the following code into mermaid.html:

<script type="module">
  /* latest version of Mermaid diagrams
     adapted from https://stuff-things.net/2025/01/19/mermaid-diagramming-in-jekyll-in-2025/ */
  import mermaid from 'https://cdn.jsdelivr.net/npm/mermaid/dist/mermaid.esm.min.mjs';
  /* initialise Mermaid */
  mermaid.initialize({ startOnLoad: true, theme: 'default' });
  /* run Mermaid on the page, looking for code blocks with the class 'language-mermaid' */
  await mermaid.run({
    /* specify the CSS selector for Mermaid code blocks
       by default, Mermaid looks for elements with the class 'mermaid', but thanks to Jekyll there's
       "language-" in front when using triple-backtick code blocks */
    querySelector: '.language-mermaid'
  });
</script>

add the following lines to your Jekyll HTML head template (mine is in a file called head.html):

<!-- if "mermaid" flag is true, include snippet -->
{% if page.mermaid %}{% include mermaid.html %}{% endif %}

in the front-matter for a post, add the following to enable Mermaid:

mermaid: true

With the above in place, you can define a Mermaid code block in a blog post, and it will be rendered as a diagram.

Ten years of blogging

2026-05-03T02:00:00+00:00

Ten years ago, I started this blog.

It’s my third or fourth blog - in 2010, I retired my personal blog https://thomasswilliams.blogspot.com as well as my second technical blog on abandoned site “TheRuntime”, now only on archive.org (side note: 2010 was a busy year, I started studying a Masters degree).

In 2016 I started this blog with the aim to write at least one post a month.

Ten years on, I’ve written nearly 100 posts. I took a break between 2023 and 2025 as work was super-busy. But I always planned on coming back and documenting what I’d learned, in the hope that it could be a reference point for me, outlasting specific jobs, as well as help others.

The posts I’m most proud of are a series on learning R Markdown (Markdown and R code for data visualisation and interactive reports), a series on learning Leaflet (web-based map tiles with an Antarctic twist), my home networking evolution, and random developer & IT bits and pieces that reflect me - a curious learner.

There’s also tons of DBA-related posts from the day-to-day operational level, to the long-term view and product ownership level, that mirror my career journey.

I’m a firm believer that the act of writing is the best way to prove and improve writing quality. Writing as a skill is critical, and technology and computing are more exciting and dynamic than ever.

I hope to continue - as I have over the past 25 years of working in IT - documenting/clarifying/reflecting on whatever platform, for another 10 years.

#30DayChartChallenge: Trend change detection in R, from Strava runs

2026-04-25T02:00:00+00:00

This is my entry for the 2026 #30DayChartChallenge https://bsky.app/profile/30daychartchall.bsky.social; day 26 is all about trends.

I wanted to understand: when do trends change? So, based on a hunch and using Strava data, I fired up RStudio and set to work (Claude and ChatGPT helped with the code, especially the Strava API stuff).

The resulting R Markdown file and instructions are on GitHub, ready to plug in your Strava client ID and secret: https://github.com/thomasswilliams/r-strava-2026

TLDR: the R package “RegimeChange” provides detection of trend changes using a variety of approaches. I used the Pruned Exact Linear Time (PELT) default algorithm. Read on for the full story, or take a sneak peek at the chart, generated using Plotly (the red horizontal lines are the trend changes):

I don’t record all my runs in Strava, but runs over the last few months have felt faster and I hoped the data would reflect this. After a slow patch with an injury in mid-2025, I upped my running regularity to twice weekly (now, three times) and didn’t have much of a break during the Aussie summer in January 2026.

Back to the chart challenge: inspired by a Nicola Rennie post https://nrennie.rbind.io/blog/mapping-a-marathon-with-rstrava/index.html, my first step was gathering and filtering the data from the Strava API for the last 2 years. The 2-year timeframe should give enough data points to identify trends. I’m using just the first 5KMs of runs which are at least 5KM long, marked as a “run” in Strava, and have a 5KM time not slower than 50 minutes (e.g. faster than a walk). Variables are defined at the top of the code; to follow along (with RStudio and a Strava client ID and secret, easy to obtain even for free Strava users), open the R Markdown file in RStudio, install any missing libraries, enter your Strava client ID and secret on lines 54 and 55, then click Knit.

On the first knit you’ll be prompted to open a browser window and authenticate with Strava. Depending on your settings, the knitted HTML file will appear in RStudio’s viewer. It can then be opened in a web browser too.

I found the “RegimeChange” package simple to use and well documented, following the example from https://cran.r-project.org/web/packages/RegimeChange/vignettes/introduction.html, with the default algorithm. The raw analysis from detecting trend changes is included in the HTML output. In my case, two distinct trend “segments” were detected - a mean of 8 minutes, 53 seconds per KM up to March 2026, and a mean of 7:51 per KM since then. Exciting progress! And matches with how I thought my pace had potentially improved.

Over a few more iterations, I tidied up the chart, adding a moving average and tooltips, and also cached data from the Strava API to avoid hitting limits and speed up knitting (probably still needs work).

I had fun participating this year’s #30DayChartChallenge. I’m interested in how trend detection might work for other datasets. Hopefully others can use my code to detect trend changes in their own Strava data.

Improving R linting (2026 update)

2026-04-15T01:00:00+00:00

Linting helps catch potential issues in code before they cause problems, and helps keep coding style consistent, whether for yourself or as part of a team or project. Linting uncovers surprising issues: recently, I added lints which caught that I was passing the same parameter by name twice in a call - meaning one of the parameter values was silently ignored.

Previously, using lintr for R, I specified only lint rules I wanted (whitelist). Now, I include all lint rules, and then tune (or disable) rules I don’t care about (blacklist).

This has the advantage that, if new rules are added, running lintr would automatically includes the new rules.

My lint configuration is below. It suits my style when coding in R: explicitly scoped function calls preceded by name of library (e.g. “DT::renderDataTable” over “renderDataTable”), libraries loaded at top of each script for portability and reproducibility, spaces instead of tabs, double quotes for strings, line lengths less than 120 characters for readability, and more.

lintr is pretty particular about lint config file formatting, and will error if the config file isn’t correct. Don’t forget to leave a blank line at the end of the file.

To lint your R code from RStudio, install the lintr package, create a config file .lintr and add rules (or copy-paste my lint rules below), then run from the R console either lintr::lint(filename = "<file name>.Rmd") or lint the whole directory with lintr::lint_dir(). See the full lintr documentation at https://lintr.r-lib.org/reference/.

My .lintr file as at 2026:

linters: all_linters(
    # use all lint rules - customise some, and disable others (see below)
    # see linter details at https://lintr.r-lib.org/reference/index.html
    # --------- customised linters ---------
    # line lengths should be less than 120 characters
    line_length_linter(120L),
    # check for unused imports
    # allow packages referenced for namespace (e.g. pkg::function)
    # always allow dplyr
    unused_import_linter(
      allow_ns_usage = TRUE,
      except_packages = c("dplyr"),
      interpret_glue = TRUE
    ),
    # variables should be snake case
    object_name_linter(c("snake_case", "SNAKE_CASE")),
    # assignment should use specific operators (not equal sign)
    assignment_linter(
      operator = c("<-", "<<-", "%<>%")
    ),
    # consistent pipes
    pipe_consistency_linter(pipe = "auto"),
    # ---------- disabled linters ----------
    # lint rules are disabled by setting them to NULL
    # allow "stop" calls, used in Rmd database connection errors
    condition_call_linter = NULL,
    # not cyclomatic complexity
    cyclocomp_linter = NULL,
    # allow relative paths (expected by some of the javascript libraries)
    nonportable_path_linter = NULL,
    # allow "paste" and "paste0"
    paste_linter = NULL,
    # allow comparisons to empty strings for readability
    nzchar_linter = NULL,
    # allow one pipe commands
    one_call_pipe_linter = NULL,
    # allow redundant calls to "== TRUE" for readability
    redundant_equals_linter = NULL,
    # turn off "trailing blank lines"
    trailing_blank_lines_linter = NULL,
    # allow calls to "library" (helps with portability)
    undesirable_function_linter = NULL
  )
encoding: "UTF-8"

Linting can be ignored for specific lines of code. For example, if there’s a line of code longer than 12 characters, add # nolint: line_len to the end to have it ignored by lintr.

Going further: linting for R has room to improve. By comparison, linting in Python is a little more mature, and some issues can be automatically fixed.

There’s the possibility of better linting with LLMs, such as assessing metrics for code like complexity & readability, and perhaps new metrics that don’t exist yet. LLMs could (potentially) suggest better, newer ways to achieve the same outcomes, while still keeping consistency to a preferred style or team norms.

In any case - good luck with linting and finding those tricky issues!