François Michonneau, PhD

Advent of SQL 2025 with DuckDB and R

2025-12-11T00:00:00+00:00

Intro

This year, the Advent of SQL is hosted by the Database School. I don’t know anything about them except that they took over the Advent of SQL from last year. There will be only 10 challenges this year (with 25 challenges last year, it felt a little long, so this is a welcome change). The spirit of the challenges seem to remain the same: using SQL to solve Christmas theme puzzles. The delivery format is however different as it uses the Database School platform and format. You need to create an account, and log in to access the challenges and their associated data. Each challenge is in the form of a video tutorial with an associated playground.

I’m going to use these challenges as an opportunity to brush up my SQL skills, using DuckDB. I’m going to work from R (just in case I need to do any additional data manipulation or visualization), but my goal this year is to do everything using DuckDB SQL (and not use LLMs for help, just searching and reading the docs the old fashion way). I might use LLMs to propose more elegant/alternative solutions once I have a working solution.

I’ll post my solutions daily (or as often as I can manage) below. The data can be downloaded from the Database School website once you created an account.

Day 1

It’s a single table, containing messy wish list data. The goal is to find the most common wishes ordered in descending order.

# Replace `BIGSERIAL` with `INTEGER` in `wish_list` table definition.

# Create DuckDB database with:
#  duckdb ./data_duckdb/advent_day_01.duckdb < ./data_sql/day1-wish-list.sql
# Be patient, these single inserts take a while to run in duckdb (about 90s)

con <- DBI::dbConnect(duckdb::duckdb(), "data_duckdb/advent_day_01.duckdb")
DBI::dbGetQuery(
  con,
  "
  SELECT wish, count(wish) AS n
    FROM (SELECT lower(trim(raw_wish)) AS wish FROM 'wish_list')
    GROUP BY wish
    ORDER BY n DESC;
"
)

Day 2

With day 2, we have two tables: snowball_inventory and snowball_categories. The goal is to find the total quantity of items in inventory for each category, ordered by total quantity ascending. Only items with quantity > 0 should be included. You need to watch the video to understand the challenge as some information is not included in the challenge description itself.

# Create DuckDB database with (no need to edit the file):
# duckdb ./data_duckdb/advent_day_02.duckdb < ./data_sql/day2-inserts.sql

con <- DBI::dbConnect(duckdb::duckdb(), "data_duckdb/advent_day_02.duckdb")
DBI::dbGetQuery(
  con,
  "
  SELECT category_name, SUM(quantity) AS total_quantity
  FROM (
       SELECT i.category_name, i.status, i.quantity, o.*
       FROM snowball_inventory i
       JOIN snowball_categories o
      ON (i.category_name = o.official_category AND quantity > 0)
    )
    GROUP BY category_name
    ORDER BY total_quantity ASC;
  "
)

Day 3

Copy and paste from the challenge:

Using the hotline_messages table, update any record that has “sorry” (case insensitive) in the transcript and doesn’t currently have a status assigned to have a status of “approved”. Then delete any records where the tag is “penguin prank”, “time-loop advisory”, “possible dragon”, or “nonsense alert” or if the caller’s name is “Test Caller”. After updating and deleting the records as described, write a final query that returns how many messages currently have a status of “approved” and how many still need to be reviewed (i.e., status is NULL).

# Create DuckDB database with (no need to edit the file):
# duckdb ./data_duckdb/advent_day_03.duckdb < ./data_sql/day3-inserts.sql

con <- DBI::dbConnect(duckdb::duckdb(), "data_duckdb/advent_day_03.duckdb")

DBI::dbExecute(
  con,
  "
  UPDATE hotline_messages
  SET status = 'approved' 
  WHERE LOWER(transcript) LIKE '%sorry%'
    AND status IS NULL;
  "
)

DBI::dbExecute(
  con,
  "
  DELETE FROM hotline_messages
  WHERE tag IN (
    'penguin prank',
    'time-loop advisory',
    'possible dragon',
    'nonsense alert'
    )
    OR caller_name = 'Test Caller';
  "
)

DBI::dbGetQuery(
  con,
  "
  SELECT clean_status, COUNT(clean_status)
  FROM (
    SELECT
    status, 
      CASE 
        WHEN status IS NULL THEN 'TBD'
        ELSE 'approved'
      END as clean_status
    FROM hotline_messages
    )
  GROUP BY clean_status;
"
)

Day 4

Copy and paste from the challenge:

Using the official_shifts and last_minute_signups tables, create a combined de-duplicated volunteer list. Ensure the list has standardized role labels of Stage Setup, Cocoa Station, Parking Support, Choir Assistant, Snow Shoveling, Handwarmer Handout. Make sure that the timeslot formats follow John’s official shifts format.

I used the snake case format for the role, it looks like the challenge actually asked for title case. I left the ‘ELSE TBD’ clauses in there as I used them when building the queries to make sure I caught all the cases. I had also checked for unique values in the time slots and given there were just a few I went for a CASE WHEN approach rather than something more sophisticated.

# Create DuckDB database with (no need to edit the file):
# duckdb ./data_duckdb/advent_day_04.duckdb < ./data_sql/day4-inserts.sql

con <- DBI::dbConnect(duckdb::duckdb(), "data_duckdb/advent_day_04.duckdb")

DBI::dbGetQuery(
  con,
  "SELECT * FROM official_shifts"
)

DBI::dbGetQuery(
  con,
  "SELECT
  volunteer_name,
  CASE
    WHEN assigned_task ILIKE '%choir%' THEN 'choir_assistant'
    WHEN assigned_task ILIKE '%stage%' THEN 'stage_setup'
    WHEN assigned_task ILIKE '%cocoa%' THEN 'cocoa_station'
    WHEN assigned_task ILIKE '%parking%' THEN 'parking_support'
    WHEN assigned_task ILIKE '%shovel%' THEN 'snow_shoveling'
    WHEN assigned_task ILIKE '%hand%' THEN 'handwarmer_handout'
    ELSE 'TBD'
  END as role,
  CASE
    WHEN (time_slot='10AM' OR time_slot ='10 am') THEN '10:00 AM'
    WHEN (time_slot='2 PM' OR time_slot='2 pm') THEN '2:00 PM'
    WHEN time_slot = 'noon' THEN '12:00 PM'
    ELSE 'TBD'
  END as shift_time
  FROM last_minute_signups
 
  UNION

  SELECT volunteer_name,
         role,
         shift_time
  FROM official_shifts
  ORDER BY volunteer_name;
"
)

Day 5

Copy and paste from the challenge:

Challenge: Write a query that returns the top 3 artists per user. Order the results by the most played.

## Create DuckDB database with (no need to edit the file):
# duckdb ./data_duckdb/advent_day_05.duckdb < ./data_sql/day5-inserts.sql

con <- DBI::dbConnect(duckdb::duckdb(), "data_duckdb/advent_day_05.duckdb")

dbGetQuery(
  con,
  "
  SELECT * FROM(
    SELECT 
      user_name,
      artist,
      COUNT(artist) AS n,
      row_number() OVER (PARTITION BY user_name ORDER BY n DESC) as top
    FROM listening_logs
    GROUP BY user_name, artist
    ORDER BY user_name, n DESC
  )
  WHERE top <= 3;
  "
)

Day 6

Challenge: Generate a report that returns the dates and families that have no delivery assigned after December 14th, using the families and deliveries_assigned. Each row in the report should be a date and family name that represents the dates in which families don’t have a delivery assigned yet. Label the columns as unassigned_date and name. Order the results by unassigned_date and name, respectively, both in ascending order.

## Create DuckDB database with (no need to edit the file):
# duckdb ./data_duckdb/advent_day_06.duckdb < ./data_sql/day6-inserts.sql

con <- dbConnect(duckdb::duckdb(), "data_duckdb/advent_day_06.duckdb")

dbGetQuery(
  con,
  "WITH december_2025 AS
     (SELECT date::DATE date
      FROM generate_series(
        DATE '2025-12-15',
        DATE '2025-12-31',
        INTERVAL '1 day'
      ) AS t(date)
      ),

  full_info AS (
    SELECT december_2025.date,
      families.id AS family_id,
      families.family_name
    FROM families
    CROSS JOIN december_2025
  )

  SELECT
    full_info.family_id AS full_fid,
    full_info.family_name,
    full_info.date AS full_date,
    deliveries_assigned.*
  FROM full_info
  LEFT JOIN deliveries_assigned ON (
     full_info.date = deliveries_assigned.gift_date AND
     deliveries_assigned.family_id = full_info.family_id
  )
  WHERE deliveries_assigned.gift_name IS NULL
  ORDER BY date ASC, family_name ASC 
  ;
 "
)

Day 7

Challenge: Get the stewards a list of all the passengers and the cocoa car(s) they can be served from that has at least one of their favorite mixins. Remember only the top three most-stocked cocoa cars remained operational, so the passengers must be served from one of those cars.

# Create DuckDB database with (no need to edit the file):
# duckdb ./data_duckdb/advent_day_07.duckdb < ./data_sql/day7-inserts.sql

con <- dbConnect(duckdb::duckdb(), "data_duckdb/advent_day_07.duckdb")

dbGetQuery(
  con,
  "
  WITH available_mixins AS (
    SELECT
      car_id AS mixins_car_id,
      available_mixins
    FROM cocoa_cars
    ORDER BY total_stock DESC
    LIMIT 3
  )

  SELECT 
    passenger_name,
    string_agg(mixins_car_id) AS available_cars
  FROM passengers
  JOIN available_mixins ON (list_has_any(passengers.favorite_mixins, available_mixins.available_mixins))
  GROUP BY passenger_name
  ORDER BY passenger_name
  "
)

Day 8

Generate a report, using the products and price_changes tables for leadership that returns the product_name, current_price, previous_price, and the difference between the current and previous prices.

I took a (maybe?) unconventional approach by using the list functions to solve this challenge. I was focused on getting the price difference first. Using the lag would have reduced the redundancy or the list(... ORDER BY rn).

# Create DuckDB database with (no need to edit the file):
# duckdb ./data_duckdb/advent_day_08.duckdb < ./data_sql/day8-inserts.sql

con <- dbConnect(duckdb::duckdb(), "data_duckdb/advent_day_08.duckdb")

dbGetQuery(
  con,
  "
  WITH sub_prices AS (SELECT 
    product_id,
    price,
    effective_timestamp,
    row_number() OVER (PARTITION BY product_id ORDER BY effective_timestamp DESC) AS rn
  FROM price_changes)

  SELECT
    product_name,
    list(price ORDER BY rn)[2] AS current_price,
    list(price ORDER by rn)[1] AS previous_price,
    list_reduce(list(price ORDER by rn), lambda x,y : x - y) AS price_change
  FROM  sub_prices
  JOIN products USING (product_id)
  WHERE rn < 3
  GROUP BY product_id, product_name
  ORDER BY product_id;
  "
)

Day 9

Build a report using the orders table that shows the latest order for each customer, along with their requested shipping method, gift wrap choice (as true or false), and the risk flag in separate columns. Order the report by the most recent order first so Evergreen Market can reach out to them ASAP.

# Edit the orders table definition to replace `JSONB` with `JSON`.

# Create DuckDB database with :
# duckdb ./data_duckdb/advent_day_09.duckdb < ./data_sql/day9-inserts.sql

library(duckdb)
con <- dbConnect(duckdb::duckdb(), "data_duckdb/advent_day_09.duckdb")
dbExecute(con, "INSTALL json; LOAD JSON;")
dbGetQuery(
  con,
  "
  WITH customer_orders AS (
    SELECT *,
      row_number() OVER (PARTITION BY customer_id ORDER BY created_at DESC) AS rn
    FROM orders
    ORDER BY customer_id, rn
  )

  SELECT
    customer_id,
    json_extract_string(order_data, '$.shipping.method') AS shipping_method,
    json_extract_string(order_data, '$.gift.wrapped')::BOOL AS gift_wrap,
    json_extract_string(order_data, '$.risk.flag') AS risk_flag
  FROM customer_orders 
  WHERE rn = 1
  ORDER BY created_at DESC;
  "
)

Day 10

Challenge: Clean-up the deliveries table to remove any records where the delivery_location is ‘Volcano Rim’, ‘Drifting Igloo’, ‘Abandoned Lighthouse’, ‘The Vibes’. Move those records to the misdelivered_presents with all the same columns as deliveries plus a flagged_at column with the current time and a reason column with “Invalid delivery location” listed as the reason for each moved record. Make sure your final step shows the misdelivered_presents records that you just moved (i.e. don’t include any existing records from the misdelivered_presents table).

I first solved the challenge by using CTEs to return the appropriate sets of rows. After watching the solution, I discovered RETURNING. I don’t think I have ever used operations in SQL that were destructive, so that was new to me. It seems (and don’t quote me on that) that it’s not possible to do a CTE using DELETE in duckDB. Instead, I relied on a temporary table, first combined with an anti-join to delete the records from the deliveries table, and then combined with an INSERT to add these records to the misdelivered_presents. I still got to use RETURNING to only see the insert rows at the end of the query.

The last few queries at the end validated that the tables were modified correctly.

# Create DuckDB database with (no need to edit the file):
# duckdb ./data_duckdb/advent_day_10.duckdb < ./data_sql/day10-inserts.sql

con <- dbConnect(duckdb::duckdb(), "data_duckdb/advent_day_10.duckdb")

dbGetQuery(
  con,
  "
    CREATE TEMPORARY TABLE deliveries_to_remove AS (
      SELECT * FROM deliveries
      WHERE delivery_location IN
        ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes')
    );

    CREATE OR REPLACE TABLE deliveries AS (
      SELECT * FROM deliveries
      ANTI JOIN deliveries_to_remove USING (id)
    );

    INSERT INTO misdelivered_presents
    SELECT
      id, child_name, delivery_location, gift_name, scheduled_at, NOW(), 'Invalid Delivery Location'
    FROM deliveries_to_remove
    RETURNING *
  "
)

dbGetQuery(con, "SELECT * FROM deliveries;")
dbGetQuery(
  con,
  "SELECT * FROM deliveries
      WHERE delivery_location IN
        ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes')"
)
dbGetQuery(con, "SELECT * FROM misdelivered_presents;")

Using Air to reformat code with Emacs ESS

2025-02-24T00:00:00+00:00

Air is an R formatter and language server written in Rust.

It is very fast and opiniated. It integrates with VSCode and Positron, RStudio (and soon with Zed).

Maybe there is a better way of integrating it with Emacs and ESS but for the time being, I wrote this short snippet that uses its command line interface to reformat the current buffer on save:

;; use Air to format the content of the file
(defun run-air-on-r-save ()
  "Run Air after saving .R files and refresh buffer."
  (when (and (stringp buffer-file-name)
             (string-match "\\.R$" buffer-file-name))
    (let ((current-buffer (current-buffer)))
      (shell-command (concat "air format " buffer-file-name))
      ;; Refresh buffer from disk
      (with-current-buffer current-buffer
        (revert-buffer nil t t)))))

(add-hook 'after-save-hook 'run-air-on-r-save)

From my limited testing, it works well enough for now.

Advent of SQL with DuckDB and R

2024-12-01T00:00:00+00:00

Quick Overview

Advent of Code is a popular advent calendar of programming puzzles. I have attempted to do it in the past using R but I always gave up after a few days because it was taking too much of my time, and prefer programming puzzles that work with data. Last year, I had fun going through the challenges of the Hanukkah of Data. This year, Bob Rudis, via his excellent Daily Drop Newsletter, pointed to Advent of SQL. I solved the challenges using DuckDB and/or {dplyr}. I appreciated that I could solve all the challenges relatively quickly.

My answers to the challenges and some annotations are in this post.

Data import

The Advent of SQL provides data for each challenge as a SQL file. They use Postgres, and while the compatibility between Postgres and DuckDB is pretty good, some features are not available in DuckDB, and the SQL dump files need to be modified to be able to import data with DuckDB.

To create a DuckDB database from a SQL file:

duckdb  <

Once the database file is created, you can work with it from R.

Day 1

Create the DuckDB database:

duckdb advent_day_01.duckdb < advent_of_sql_day_1.sql

library(duckdb)
library(dplyr)

## Connect to the Database
con_day01 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_01.duckdb")

## Check the content
dbListTables(con_day01)
dbGetQuery(con_day01, "SELECT wishes FROM wish_lists limit 10;")

## Install and load the json extension to work with the JSON data
dbExecute(con_day01, "INSTALL json; LOAD json;")

## Create tidy version of the data
dbExecute(
  con_day01,
  "
  CREATE OR REPLACE VIEW tidy_wishlist AS
  SELECT
    list_id,
    child_id,
    trim(wishes.first_choice::VARCHAR, '\"') AS primary_wish,
    trim(wishes.second_choice::VARCHAR, '\"') as backup_wish,
    trim(wishes.colors[0]::VARCHAR, '\"') AS favorite_color,
    json_array_length(wishes.colors) AS color_count
  FROM wish_lists;
  "
)

## Inspect newly created VIEW
tbl(con_day01, "tidy_wishlist")

## Build answer
left_join(
  tbl(con_day01, "children") |>
    select(child_id, name),
  tbl(con_day01, "tidy_wishlist"),
  by = "child_id"
) |>
  left_join(
    tbl(con_day01, "toy_catalogue"),
    by = c("primary_wish" = "toy_name")
  ) |>
  mutate(
    gift_complexity = case_when(
      difficulty_to_make == 1 ~ "Simple Gift",
      difficulty_to_make == 2 ~ "Moderate Gift",
      difficulty_to_make >= 3 ~ "Complex Gift",
      TRUE ~ NA_character_
    ),
    workshop_assignment = case_when(
      category == "outdoor" ~ "Outside workshop",
      category == "educational" ~ "Learning workshop",
      TRUE ~ "General workshop"
    )
  ) |>
  select(name, primary_wish, backup_wish, favorite_color, color_count, gift_complexity, workshop_assignment) |>
  arrange(name) |>
  collect(n = 5) |>
  rowwise() |>
  mutate(
    answer = glue::glue_collapse(c(name, primary_wish, backup_wish, favorite_color, color_count, gift_complexity, workshop_assignment), sep = ",")
  ) |>
  pull(answer)

dbDisconnect(con_day01, shutdown = TRUE)

The JSON extension in DuckDB allowed me to extract the required data to create a tidy version to solve the problem.

Day 2

The SQL dump for this challenge used the SERIAL data type which is not supported by DuckDB. SERIAL is a convenience to create unique, auto-incrementing, ids. The way around it in DuckDB, is to create a sequence and using in the table definition. I edited the SQL dump so the beginning of the file now looks like:

DROP TABLE IF EXISTS letters_a CASCADE;
DROP TABLE IF EXISTS letters_b CASCADE;

CREATE SEQUENCE laid START 1;
CREATE TABLE letters_a (
  id INTEGER PRIMARY KEY DEFAULT nextval('laid'),
  value INTEGER
);

CREATE SEQUENCE lbid START 1;
CREATE TABLE letters_b (
  id INTEGER PRIMARY KEY DEFAULT nextval('lbid'),
  value INTEGER
);

con_day02 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_02.duckdb")

## Create a single table combining `letters_a` and `letters_b` with `UNION`
## Use function `chr()` to convert ASCII codes into letters
dbExecute(
  con_day02,
  "CREATE OR REPLACE VIEW letters_decoded AS
   SELECT  *, chr(value) AS character FROM letters_a
   UNION
   SELECT  *, chr(value) AS character FROM letters_b
  "
)

## Define list of valid characters
valid_characters <- "[A-Za-z !\"\'(),-.:;?]"

## Filter data to only keep valid_characters
## and collapse results to extract message
tbl(con_day02, "letters_decoded") |>
  filter(grepl(valid_characters, character)) |>
  arrange(id) |>
  pull(character) |>
  paste(x =_, collapse = "")

dbDisconnect(con_day02, shutdown = TRUE)

Day 3

Again, the SQL dump used SERIAL. Additionally, DuckDB does not support the XML data type, so I switched to use VARCHAR and used R to work with the XML data. I edited the beginning of the dump file to look like:

DROP TABLE IF EXISTS christmas_menus CASCADE;

CREATE SEQUENCE cmid START 1;
CREATE TABLE christmas_menus (
  id INTEGER PRIMARY KEY DEFAULT nextval('cmid'),
  menu_data VARCHAR
);

I really didn’t use DuckDB’s engine for this challenge. I only relied on R to work with the XML data:

con_day03 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_03.duckdb")

menus <- tbl(con_day03, "christmas_menus") %>%
  collect()


## Figure out how many XML schemas are being used in the data
get_menu_version <- function(menu_data) {
  xml2::read_xml(menu_data) |>
    xml2::xml_find_all(".//@version") |>
    xml2::xml_text()
}

menu_versions <- purrr::map_chr(menus$menu_data, get_menu_version)

## There are 3 different versions
unique(menu_versions)

## Extract the number of guests based on the XML schema
get_guest_number <- function(menu_data, xml_version) {
  element <- switch(
    xml_version,
    "3.0" = ".//headcount/total_present",
    "2.0" = ".//total_guests",
    "1.0" = ".//total_count"
  )

  xml2::read_xml(menu_data) |>
    xml2::xml_find_all(element) |>
    xml2::xml_text() |>
    as.numeric()
}

n_guests <- purrr::map2_dbl(menus$menu_data, menu_versions, get_guest_number)

## Extract the food ids (only for events with the right number of guests)
food_ids <- purrr::map2(menus$menu_data, n_guests, function(.x, .g) {
  if (.g < 78) return(NULL)
  xml2::read_xml(.x) |>
    xml2::xml_find_all(".//food_item_id") |>
    xml2::xml_text()
})

## And count them to find the most common one
unlist(food_ids) |>
  as_tibble() |>
  count(value, sort = TRUE)

dbDisconnect(con_day03, shutdown = TRUE)

Day 4

Again, the original dump used SERIAL, for this challenge, I simply replaced it with INTEGER, so the beginning of the file looks like:

DROP TABLE IF EXISTS toy_production CASCADE;

CREATE TABLE toy_production (
  toy_id INTEGER PRIMARY KEY,
  toy_name VARCHAR(100),
  previous_tags TEXT[],
  new_tags TEXT[]
  );

This challenge required to dive into DuckDB’s functions to work with lists. While there is a list_intersect() function, there does not seem to be a list_setdiff() so instead I combined list_where() with list_transform() and list_contains() to get there. I would be happy to hear alternative approaches!

con <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_04.duckdb")

dbGetQuery(
  con,
  "SELECT
     toy_id,
     list_where(new_tags, list_transform(new_tags, x -> NOT list_contains(previous_tags, x))) AS added_tags,
     list_intersect(previous_tags, new_tags) AS unchanged_tags,
     list_where(previous_tags, list_transform(previous_tags, x -> NOT list_contains(new_tags, x))) AS removed_tags,
     len(added_tags) AS added_tags_length,
     len(unchanged_tags) AS unchanged_tags_length,
     len(removed_tags) AS removed_tags_length
   FROM toy_production;"
) |>
  slice_max(added_tags_length) |>
  select(toy_id, ends_with("length"))

dbDisconnect(con_day04, shutdown = TRUE)

Day 5

The data could be imported directly from the SQL dump.

Solving this challenge required using the lead() function from the tidyverse to calculate the change in production and its percentage. I then used slice_max() to extract the row with the largest percentage change.

con_day05 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_05.duckdb")

tbl(con_day05, "toy_production") |>
  mutate(previous_day_production = lead(toys_produced)) |>
  mutate(production_change = toys_produced - previous_day_production) |>
  mutate(production_change_percentage = production_change / toys_produced * 100) |>
  slice_max(production_change_percentage)

dbDisconnect(con_day05, shutdown = TRUE)

Day 6

The original dump used SERIAL, so I updated the beginning of the file to look like:

DROP TABLE IF EXISTS children CASCADE;
DROP TABLE IF EXISTS gifts CASCADE;

CREATE TABLE children (
    child_id INTEGER PRIMARY KEY,
    name VARCHAR(100),
    age INTEGER,
    city VARCHAR(100)
);

CREATE TABLE gifts (
    gift_id INTEGER PRIMARY KEY,
    name VARCHAR(100),
    price DECIMAL(10,2),
    child_id INTEGER REFERENCES children(child_id)
);

con_day06 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_06.duckdb")

## First calculate average  price
avg_price <- tbl(con_day06, "gifts") |>
  summarize(mean_price = mean(price, na.rm = TRUE)) |>
  pull(mean_price)

## Join the gifts and children table and filter out results based on average
## price. Finally arrange by price.
tbl(con, "children") |>
  left_join(tbl(con, "gifts"), by = join_by(child_id)) |>
  filter(price >= avg_price) |>
  arrange(price)

dbDisconnect(con_day06, shutdown = TRUE)

Day 7

The original dump used SERIAL again, but provided all the elf_id so I updated the beginning of the file to use INTEGER instead and it looks like:

DROP TABLE IF EXISTS workshop_elves CASCADE;
CREATE TABLE workshop_elves (
    elf_id INTEGER PRIMARY KEY,
    elf_name VARCHAR(100) NOT NULL,
    primary_skill VARCHAR(50) NOT NULL,
    years_experience INTEGER NOT NULL
);

con_day07 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_07.duckdb")

tbl(con_day07, "workshop_elves") |>
  filter(
    years_experience == min(years_experience) | years_experience == max(years_experience),
    .by = primary_skill
  ) |>
  filter(
    elf_id == min(elf_id),
    .by = c(primary_skill, years_experience)
  ) |>
  arrange(
    primary_skill,
    desc(years_experience)
  ) |>
  collect() |> 
  summarize(
    result = paste(elf_id, collapse = ","),
    .by = primary_skill
  ) |>
  summarize(
    result = paste(result, primary_skill, sep = ","),
    .by = primary_skill
  )

dbDisconnect(con_day07, shutdown = TRUE)

Day 8

Again, the original data dump used SERIAL which I substituted for INTEGER to be able to import the data into DuckDB, so the beginning of the file looks like:

DROP TABLE IF EXISTS staff CASCADE;
CREATE TABLE staff (
    staff_id INTEGER PRIMARY KEY,
    staff_name VARCHAR(100) NOT NULL,
    manager_id INTEGER
);

I was in the rush that day, and the solution I came up with is quite hacky and slow. All the computation takes place in R using a recursive function. This challenge is a good opportunity to learn recursive CTEs but I’ll need to come back to it. (See Day 18 for the resurvie CTE approach).

con_day08 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_08.duckdb")

## make sure there is a single NA
tbl(con_day08, "staff") |>
  pull(manager_id) |>
  is.na() |>
  sum()

find_boss <- function(.data, idx) {
  if (is.na(idx)) {
    return(1)
  }
  
  res <- .data |>
    filter(staff_id == idx[1]) |>
    pull(manager_id)

  c(find_boss(.data, res), idx)
}

staff <- tbl(con_day08, "staff") |>
  collect()

staff |>
  rowwise() |> 
  mutate(path = list(find_boss(staff, .data$manager_id))) |>
  mutate(level = length(path)) |>
  ungroup() |>
  slice_max(level)

dbDisconnect(con_day08, shutdown = TRUE)

Day 9

To replace SERIAL, I used SEQUENCE for both the reindeer_id and the session_id so the beginning of the dump file looks like:

DROP TABLE IF EXISTS training_sessions CASCADE;
DROP TABLE IF EXISTS reindeers CASCADE;

CREATE SEQUENCE r_id START 1;
CREATE TABLE reindeers (
    reindeer_id INTEGER PRIMARY KEY DEFAULT nextval('r_id'),
    reindeer_name VARCHAR(50) NOT NULL,
    years_of_service INTEGER NOT NULL,
    speciality VARCHAR(100)
);

CREATE SEQUENCE s_id START 1;
CREATE TABLE training_sessions (
    session_id INTEGER PRIMARY KEY DEFAULT nextval('s_id'),
    reindeer_id INTEGER,
    exercise_name VARCHAR(100) NOT NULL,
    speed_record DECIMAL(5,2) NOT NULL,
    session_date DATE NOT NULL,
    weather_conditions VARCHAR(50),
    FOREIGN KEY (reindeer_id) REFERENCES reindeers(reindeer_id)
);

con_day09 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_09.duckdb")

tbl(con, "training_sessions") |>
  left_join(
    tbl(con, "reindeers"),
    by = join_by(reindeer_id)
  ) |>
  filter(reindeer_name != "Rudolf") |>
  summarize(
    avg_speed = mean(speed_record, na.rm = TRUE),
    .by = c(reindeer_name, exercise_name)
  ) |>
  slice_max(avg_speed, by = reindeer_name) |>
  slice_max(avg_speed, n = 3) |>
  collect() |>
  glue::glue_data("{reindeer_name},{round(avg_speed, 2)}")

dbDisconnect(con_day09, shutdown = TRUE)

Day 10

I again replaced SERIAL with using a SEQUENCE in the data dump. The beginning of the file looks like:

DROP TABLE IF EXISTS Drinks CASCADE;

CREATE SEQUENCE d_id START 1;
CREATE TABLE Drinks (
    drink_id INTEGER PRIMARY KEY DEFAULT nextval('d_id'),
    drink_name VARCHAR(50) NOT NULL,
    date DATE NOT NULL,
    quantity INTEGER NOT NULL
    );

con_day10 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_10.duckdb")

tbl(con_day10, "Drinks") |>
  summarize(
    quantity = sum(quantity, na.rm = TRUE), .by = c(date, drink_name)
  ) |>
  tidyr::pivot_wider(names_from = drink_name, values_from = quantity) |>
  filter(
    `Hot Cocoa` == 38,
    `Peppermint Schnapps` == 298,
    `Eggnog` == 198
  )

dbDisconnect(con_day10, shutdown = TRUE)

The magic here is that it all happens within DuckDB even with the call to pivot_wider().

Day 11

The data could be imported directly from DuckDB.

I first wrote the solution to this challenge using the {slider} package to get the moving average. But the data has to be pulled in R’s memory to make this work. I then tried to solve it using just DuckDB to practice window functions, but the query returns a second result. I have not investigated why this is the case yet.

con_day11 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_11.duckdb")

## R solution
tbl(con_day11, "TreeHarvests") |>
  collect() |>
  mutate(
    avg_yield = slider::slide_dbl(trees_harvested, mean, .before = 2, .complete = FALSE),
    .by = c(field_name, harvest_year)
  ) |>
  slice_max(avg_yield)

## DuckDB SQL solution
dbGetQuery(con_day11,
  "
  WITH results AS (
  SELECT field_name, harvest_year, season,
     CASE WHEN season = 'Spring' THEN 1 WHEN season = 'Summer' THEN 2
          WHEN season = 'Fall' THEN 3 WHEN season = 'Winter' THEN 4 END AS season_order,
     trees_harvested,
     avg(trees_harvested) OVER
       (PARTITION BY 'field_name', 'harvest_year' ORDER BY 'season_order'
        ROWS 2 PRECEDING) AS avg_yield
  FROM TreeHarvests
  )
  SELECT * FROM results WHERE avg_yield = (SELECT max(avg_yield) FROM results)
  "
)

dbDisconnect(con_day11, shutdown = TRUE)

Day 12

The data dump needed again to use SEQUENCE to replace SERIAL, so I edited the beginning of the file to look like:

CREATE OR REPLACE SEQUENCE g_id START 1;
CREATE OR REPLACE SEQUENCE r_id START 1;

CREATE OR REPLACE TABLE gifts (
    gift_id INTEGER PRIMARY KEY DEFAULT nextval('g_id'),
    gift_name VARCHAR(100) NOT NULL,
    price DECIMAL(10,2)
);

CREATE OR REPLACE TABLE gift_requests (
    request_id INTEGER PRIMARY KEY DEFAULT nextval('r_id'),
    gift_id INT,
    request_date DATE,
    FOREIGN KEY (gift_id) REFERENCES gifts(gift_id)
);

This challenge could be solve using only {dplyr} functions. There were 10 gifts that tied for first place, so I relied on the n argument from the print function to display more data to be able to see the second most popular item.

con_day12 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_12.duckdb")

tbl(con_day12, "gift_requests") |>
  count(gift_id) |>
  mutate(overall_rank = percent_rank(n)) |>
  left_join(tbl(con_day12, "gifts"), by = join_by(gift_id)) |>
  arrange(desc(overall_rank), gift_name) |>
  print(n = 20)

dbDisconnect(con_day12, shutdown = TRUE)

Day 13

To be able to import the data into DuckDB, I once again edited the beginning of the dump file to use SEQUENCE to replace SERIAL:

CREATE OR REPLACE SEQUENCE cl_id START 1;

DROP TABLE IF EXISTS contact_list;
CREATE TABLE contact_list (
    id INTEGER PRIMARY KEY DEFAULT nextval('cl_id'),
    name VARCHAR(100) NOT NULL,
    email_addresses TEXT[] NOT NULL
);

con_day13 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_13.duckdb")

dbGetQuery(
  con_day13,
  "
  WITH all_domains AS
  (SELECT
   id, name, unnest(email_addresses) AS addresses,
   regexp_extract(addresses, '@(.+)$', 1) AS domains
  FROM contact_list
  )
  SELECT domains, COUNT(domains) AS n_users FROM all_domains
  GROUP BY domains
  ORDER BY n_users DESC
  "
)

dbDisconnect(con_day13, shutdown = TRUE)

Day 14

I replaced SERIAL with SEQUENCE once again:

DROP TABLE IF EXISTS SantaRecords CASCADE;

CREATE OR REPLACE SEQUENCE rid START 1;

CREATE TABLE SantaRecords (
    record_id INTEGER PRIMARY KEY DEFAULT nextval('rid'),
    record_date DATE NOT NULL,
    cleaning_receipts JSON NOT NULL
);

con_day14 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_14.duckdb")

dbExecute(con, "LOAD json;")

dbGetQuery(
  con_day14,
  "WITH extracted AS (
   SELECT
     record_date,
     cleaning_receipts->>'$..garment' AS garment,
     cleaning_receipts->>'$..color' AS color,
     cleaning_receipts->>'$..drop_off' AS drop_off,
     cleaning_receipts->>'$..receipt_id' AS receipt_id
   FROM SantaRecords
  ),
  tidy AS (
    SELECT
      record_date,
      unnest(garment) AS tidy_garment,
      unnest(color) AS tidy_color,
      unnest(drop_off) AS dropoff,
      unnest(receipt_id) AS tidy_receipt_id
    FROM extracted
  )
  SELECT * FROM tidy
  WHERE tidy_garment = 'suit' AND tidy_color = 'green'
  ORDER BY dropoff DESC;"
)

dbDisconnect(con_day14, shutdown = TRUE)

Day 15

This was the first challenge of the series that dealt with spatial data. The data required a little more preparation. I updated the dump file to:

use SEQUENCE instead of SERIAL
replace GEOGRAPHY(POINT) and GEOGRAPHY(POLYGON) with GEOMETRY
for each spatial feature, I removed ST_setSRID(..., 4326) given that it’s the default in DuckDB.

The beginning of the file looked like:

CREATE OR REPLACE SEQUENCE sid START 1;
CREATE TABLE sleigh_locations (
id INTEGER PRIMARY KEY DEFAULT nextval('sid'),
timestamp TIMESTAMP WITH TIME ZONE NOT NULL,
coordinate GEOMETRY NOT NULL
);


CREATE OR REPLACE SEQUENCE aid START 1;
CREATE TABLE areas (
    id INTEGER PRIMARY KEY DEFAULT nextval('aid'),
    place_name VARCHAR(255) NOT NULL,
    polygon GEOMETRY NOT NULL
);

and the sleigh location table data looked like:

INSERT INTO sleigh_locations (timestamp, coordinate) VALUES
('2024-12-24 22:00:00+00', ST_Point(37.717634, 55.805825));

I edited the areas table the same way, for instance the first area looked like:

('New_York', ST_GeomFromText('POLYGON((-74.25909 40.477399, -73.700272 40.477399, -73.700272 40.917577, -74.25909 40.917577, -74.25909 40.477399))')),

con_day15 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_15.duckdb")

dbExecute(
  con_day15, "INSTALL spatial; LOAD spatial;"
)

dbGetQuery(
  con_day15,
  "
 SELECT areas.place_name
 FROM areas
 JOIN sleigh_locations on ST_Within(sleigh_locations.coordinate, areas.polygon)
  "
)

Day 16

Day 16 required the same data preparation as for day 15.

con_day16 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_16.duckdb")

dbExecute(
  con_day16, "INSTALL spatial; LOAD spatial;"
)

dbGetQuery(
  con_day16,
  "
  SELECT sleigh_locations.timestamp, areas.place_name
  FROM sleigh_locations
  JOIN areas on ST_Within(sleigh_locations.coordinate, areas.polygon)
  "
) |>
  summarize(time_spent = max(timestamp) - min(timestamp), .by = "place_name") |>
  slice_max(time_spent)

dbDisconnect(con_day16, shutdown = TRUE)

Day 17

con_day17 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_17.duckdb")
dbExecute(con_day17, "INSTALL icu; LOAD icu;")
dbGetQuery(
  con_day17,
  "SELECT
     *,
     ('2024-12-24' || ' ' || business_start_time || ' ' || timezone)::TIMESTAMPTZ AS start_time_utc,
     ('2024-12-24' || ' ' || business_end_time || ' ' || timezone)::TIMESTAMPTZ AS end_time_utc
   From Workshops,
 "
) |> filter(
  ## meeting cannot start before the earliest open workshop
  start_time_utc >= max(start_time_utc)
) |>
  pull(start_time_utc)

Day 18

The data for this challenge is the same as for day 8, and requires the same preparation: replacing SERIAL with INTEGER. Instead of using the same (inefficient) recursive function in R as for day 8, here I learned how to use them in DuckDB to compute the managerial paths. The wording of this challenge seems confusing and it seemed uncessary to compute the number of peers with the same manager to find the answer.

con_day18 <- dbConnect(
  duckdb(),
  "2024-advent-of-sql-data/advent_day_18.duckdb"
)

dbGetQuery(
  con_day18,
  "
  WITH RECURSIVE path_tbl(staff_id, path) AS (
      SELECT staff_id, [manager_id] AS path
      FROM staff
      WHERE manager_id IS NULL
    UNION ALL
      SELECT staff.staff_id, list_prepend(staff.manager_id, path_tbl.path)
      FROM staff, path_tbl
      WHERE staff.manager_id = path_tbl.staff_id
  )
  SELECT path_tbl.staff_id, staff.manager_id, len(path) AS level
  FROM path_tbl
  JOIN staff ON staff.staff_id = path_tbl.staff_id
  ORDER BY path_tbl.staff_id, level DESC
  "
) |>
  mutate(
    total_peers_same_level = n(),
    .by = c(level)
  ) |> 
  arrange(desc(total_peers_same_level), level, staff_id)

dbDisconnect(con_day18, shutdown = TRUE)

Day 19

Replace SERIAL with a SEQUENCE.

DROP TABLE IF EXISTS employees CASCADE;

CREATE OR REPLACE SEQUENCE eid START 1;
CREATE TABLE employees (
employee_id INTEGER PRIMARY KEY DEFAULT nextval('eid'),
name VARCHAR(100) NOT NULL,
salary DECIMAL(10, 2) NOT NULL,
year_end_performance_scores INTEGER[] NOT NULL
);

The main challenge here, is that the result is a large number, and by default R does not print enough significant digits to get the correct answer. There are several ways to get the correct number displayed but, in the end, used tibble::num() which was new to me.

con_day19 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_19.duckdb")

dbExecute(
  con_day19,
  "CREATE OR REPLACE VIEW average_score AS
    (SELECT
    *,
    year_end_performance_scores[len(year_end_performance_scores)] AS last_score
   FROM employees)
")

avg_score <- tbl(con_day19, "average_score") |>
  summarize(avg_score = mean(last_score)) |>
  pull(avg_score)

res <- tbl(con_day19, "average_score") |>
  mutate(gets_bonus = last_score >= avg_score) |>
  mutate(total_comp = case_when(
    gets_bonus ~ round(salary * 1.15, 2),
    TRUE ~ salary
  )) |> 
  summarize(total = sum(total_comp)) |>
  pull(total)

tibble::num(res, digits = 2)

dbDisconnect(con_day19, shutdown = TRUE)

Day 20

Replace SERIAL with SEQUENCE, so the beginning of the file looks like:

DROP TABLE IF EXISTS web_requests CASCADE;
CREATE OR REPLACE SEQUENCE id START 1;
CREATE TABLE web_requests (
  request_id INTEGER PRIMARY KEY DEFAULT nextval('id'),
  url TEXT NOT NULL
  );

con_day20 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_20.duckdb")

dbGetQuery(
  con_day20,
  "SELECT
    *,
    string_split(regexp_extract(url, '\\?(.+)', 1), '&') AS query
   FROM web_requests
   WHERE contains(url, 'utm_source=advent-of-sql')
   ORDER BY len(list_distinct(list_transform(query, p -> p.split('=')[1]))) DESC, url
   LIMIT 1
  ") |>
  pull(url)

dbDisconnect(con_day20, shutdown = TRUE)

Day 21

Replace SERIAL with SEQUENCE, so the beginning of the file looks like:

DROP TABLE IF EXISTS sales CASCADE;

CREATE OR REPLACE SEQUENCE id START 1;
CREATE TABLE sales (
  id INTEGER PRIMARY KEY DEFAULT nextval('id'),
  sale_date DATE NOT NULL,
  amount DECIMAL(10, 2) NOT NULL
);

con_day21 <- dbConnect(duckdb(), "2024-advent-of-sql-data/advent_day_21.duckdb")

dbGetQuery(
  con_day21,
  "
  SELECT *
  FROM (
    SELECT
      year(sale_date) AS year,
      quarter(sale_date) AS quarter,
      sum(amount) AS total_sale,
      lag(total_sale, 1) OVER (ORDER BY year, quarter) AS prev_sale,
      (total_sale-prev_sale)/prev_sale AS growth
    FROM sales
    GROUP BY year, quarter
    ORDER BY year, quarter
  )
  ORDER BY growth DESC
")

dbDisconnect(con_day21, shutdown = TRUE)

Day 22

Once again, I used SEQUENCE to replace SERIAL:

DROP TABLE IF EXISTS elves CASCADE;
CREATE OR REPLACE SEQUENCE id START 1;
CREATE TABLE elves (
  id INTEGER PRIMARY KEY DEFAULT nextval('id'),
  elf_name VARCHAR(255) NOT NULL,
  skills TEXT NOT NULL
);

con_day22 <- dbConnect(
  duckdb(),
  "2024-advent-of-sql-data/advent_day_22.duckdb"
)

dbGetQuery(
  con_day22,
  "
  SELECT
     count(id)
   FROM elves
   WHERE str_split(skills, ',').list_contains('SQL')
   "
)

dbDisconnect(con_day22, shutdown = TRUE)

Day 23

The data could be imported as provided, and chose to solve the challenge with dplyr.

con_day23 <- dbConnect(
  duckdb(),
  "2024-advent-of-sql-data/advent_day_23.duckdb"
)

seq_id <- tbl(con_day23, "sequence_table") |>
  collect()

full_seq <- tibble(id = seq(min(seq_id$id), max(seq_id$id)))

## join complete and provided sequence and keep both
left_join(full_seq, seq_id, keep = TRUE) |>
  ## the NAs are the gaps
  filter(is.na(id.y)) |>
  ## identify groups
  mutate(next_id = id.x - lag(id.x, default = 1)) |>
  mutate(next_id = cumsum(next_id != 1)) |>
  ## format as expected
  nest_by(next_id) |>
  mutate(res = paste(data$id.x, collapse = ","))

dbDisconnect(con_day23, shutdown = TRUE)

Day 24

The data could be imported directly into DuckDB. For this challenge, manipulating the data with the dplyr verbs felt like the most efficient way to get to the solution.

con_day24 <- dbConnect(
  duckdb(),
  "2024-advent-of-sql-data/advent_day_24.duckdb"
)

tbl(con_day24, "user_plays") |>
  left_join(tbl(con_day24, "songs")) |>
  mutate(has_skip = as.integer(duration < song_duration)) |>
  summarize(
    n_plays = n(),
    n_skips = sum(has_skip),
    .by = song_title
  ) |>
  arrange(desc(n_plays), n_skips)

dbDisconnect(con_day24, shutdown = TRUE)

How to work with remote Parquet files with the duckdb R package?

2023-06-19T00:00:00+00:00

For large datasets, it is sometimes convenient to explore them without downloading them locally. With Arrow, you can work with these remotes files if they are stored in AWS S3 or Google Cloud Storage. It is however not yet possible for files stored over HTTPS (it is on the roadmap). On the other hand, with the “httpfs” extension, DuckDB allows you to query over the wire these Parquet files.

You can even set things up so you can use dplyr verbs to work with these remote files. I will demonstrate this using a Parquet version of the penguins dataset hosted on my site.

Let’s start by loading the required packages:

library(DBI)
library(duckdb)
library(dplyr)

We are creating a con object to hold our DuckDB connection:

con <- duckdb::duckdb()

Let’s install (only needed once) and load the httpfs extension:

dbExecute(con, "INSTALL httpfs;")
dbExecute(con, "LOAD httpfs;")

At this point, we could use DuckDB’s SQL syntax to work with our remote dataset:

dbGetQuery(con,
  "SELECT species,
          AVG(bill_length_mm) AS avg_bill_length,
          AVG(bill_depth_mm) AS avg_bill_depth
   FROM PARQUET_SCAN('https://francoismichonneau.net/assets/data/penguins.parquet')
   GROUP BY species;")

# A tibble: 3 × 3
  species   avg_bill_length avg_bill_depth
                           
1 Adelie               38.8           18.3
2 Gentoo               47.5           15.0
3 Chinstrap            48.8           18.4

However, you can create a view using this remote file, which in turn, will allow you to use dplyr to query your file:

dbExecute(con,
  "CREATE VIEW penguins AS
   SELECT * FROM PARQUET_SCAN('https://francoismichonneau.net/assets/data/penguins.parquet');
")

You can check it worked by running:

dbListTables(con)

[1] "penguins"

Now you can work with this remote data with dplyr:

tbl(con, "penguins") |>
  group_by(species) |>
  summarize(
    avg_bill_length = mean(bill_length_mm),
    avg_bill_depth = mean(bill_depth_mm)
  )

# Source:   SQL [3 x 3]
# Database: DuckDB 0.8.1 [francois@Linux 6.2.0-20-generic:R 4.3.0/:memory:]
  species   avg_bill_length avg_bill_depth
                           
1 Adelie               38.8           18.3
2 Gentoo               47.5           15.0
3 Chinstrap            48.8           18.4

How to use Arrow to work with large CSV files?

2022-10-13T00:00:00+00:00

Some background

Lucky you! You just got hold of a largish CSV file (let’s say 15 GB, about 140 million rows). How do you handle this file to be able to work with it using Apache Arrow?

Going through the documentation of Arrow, you might notice that several ways are mentioned to import data. They fall into two families:

one that I will refer to as the Single file API¹;
the other is the Dataset API.

The Single file API contains functions for each supported file format (CSV, JSON, Parquet, Feather/Arrow, ORC). They work on one file at a time, and they load the data in memory. So depending on the size of your file and the amount of memory you have available on your system, it might not be possible to load the dataset this way. If you can load the dataset in memory queries will run faster because the data will be readily accessible to the query engine.

The Dataset API is very flexible. It can read multiple file formats, you can point to a folder with multiple files and create a dataset from them, and it can read datasets from multiple sources (even combining remote and local sources). This API can also be used to read single files that are too large to fit in memory. This works because the files are not actually loaded in memory. The functions scan the content so they know where to look for the data and what the schema is (the data types and names of each column). When you query the data, there is some overhead because the query engine needs to first read the data before it can operate on it. (If you want to see some examples of what the Dataset API can do, check out the two previous posts on datasets with Arrow: Part 1, and Part 2)

In this post, we will explore how to convert a large CSV file to the Apache Parquet format using the Single file and the Dataset APIs with code examples in R and Python. We do the conversion from CSV to Parquet, because in a previous post we found that the Parquet format provided the best compromise between disk space usage and query performance. Having the content of this file in the Apache Parquet format will ensure that we can read and operate on this data quickly.

The Single file API in R

The functions in the Single file API in R start with read_ or write_ followed by the name of the file format. For instance, read_csv_arrow(), read_parquet(), and read_feather() belong to what I refer here as the Single file API.

To read the data with our 15 GB CSV file, we would use:

library(arrow)

data <- read_csv_arrow(
  "~/dataset/path_to_file.csv",
  as_data_frame = FALSE
)

Using as_data_frame = FALSE keeps the result as an Arrow table which is a better representation for a file of this size. Attempting to convert it into a data frame will take longer to load, and you will most likely run out of memory.

This step takes about 15 seconds on my system. As far as I can tell, the arrow R package is the only way to load a file of this size in memory. Both readr/vroom and data.table ran out of memory after several minutes and before being able to finish reading the file.

At this point, you have an Arrow formatted table loaded in memory that is ready for you to work with.

To convert this file into the Apache Parquet format using the Single file API, you would use:

write_parquet(data, "~/dataset/data.parquet")

Creating this file takes about 85 seconds on my system. The resulting file is about 9.5 GB, reducing the amount of hard drive space needed to store the data by about 60%.

The read_parquet() function will load this dataset the next time you need to work with it:

data <- read_parquet("~/dataset/data.parquet", as_data_frame = FALSE)

Let’s count the number of unique values in one of the columns of this dataset:

data %>%
  count(variable) %>%
  collect()

This query takes only half a second on my laptop. Half a second to summarize the content of 140 million rows: this is fast! Very fast!

Whether you use read_csv_arrow() or read_parquet(), the dataset is loaded in memory using the same representation: an Arrow table. The performance of queries would therefore be the same regardless of the format used to store the data. In this case, the decision to storing the data as a CSV or a Parquet file will be based on the amount of storage, how fast reading from CSV or Parquet compares to the overhead associated with the conversion from one format to the other.

Let’s now use the Dataset API.

The Dataset API in R

We will read the large CSV file with open_dataset(). This function can be pointed to a folder with several files but it can also be used to read a single file.

data <- open_dataset("~/dataset/path_to_file.csv")

With our 15 GB file, it takes 0.05 seconds to “read” the file. It is fast because the data does not get loaded in memory. open_dataset() scans the content of the file to identify the name of the columns and their data types.

Running the same query as above, which counts the number of unique values in a column, takes 18 seconds compared to the 0.5 seconds when the data is loaded in memory. It is slower because the query engine needs to read the data. It is the same result that we had found in a previous post: running queries directly on a CSV file is slow. In that post, we also found that storing the data in the Parquet format sped things up. Let’s now convert this dataset to Parquet using the Dataset API.

Instead of using a single Parquet file as we did above when we looked at the Single file API, we will also partition the Parquet dataset to see how it could help with query performance. The particular dataset I have on hand does not have any obvious variable we can use to partition the data. If you are dealing with a dataset that has timestamps for data collected at regular intervals, partitioning on a temporal dimension could make sense (that’s what the NYC taxi dataset does by partitioning by year and month). Instead, here, we can use the max_rows_per_file argument of the write_dataset() function to limit how large each Parquet file is. At least for this dataset, I found that limiting the number of rows to 10 million per file seemed like a good compromise. Each file is about 720 MB which is close to the file sizes in the NYC taxi dataset. The PyArrow documentation has a good overview of strategies for partitioning a dataset. The general recommendation is to avoid individual Parquet files smaller than 20 MB and larger than 2 GB while avoiding a partition layout that would create more than 10,000 partitions.

write_dataset(
  data,
  format = "parquet",
  path = "~/datasets/my-data/",
  max_rows_per_file = 1e7
)

Writing these files on my system takes about 50 seconds. We end up with 14 Parquet files totaling 9.9 GB.

Next time we want to work with this data, we can load these files with:

data <- open_dataset(
  "~/datasets/my-data"
  )

It takes about the same amount of time as scanning the CSV files. It is almost instantaneous taking only 0.02 seconds. Again, this is fast because the data is not loaded in memory. We saw that with this approach it took almost 20 seconds to run this query on our CSV file. So what is the performance of a query on this dataset split into multiple Parquet files?

Counting the unique values in a column takes just 1 second. You read that correctly. One second to summarize 140 million rows. It is a little slower than doing it when the entire dataset is loaded in memory but scanning the files is faster. And because the dataset is not loaded in memory, you are not limited by the amount of memory you have available. With the Single File API, a file of 15 GB is the upper limit of what my laptop with 32 GB of RAM can handle.

One of the advantages of the Arrow ecosystem is that it is polyglot. The approach we described with R also works with Python. And because both languages use the same C++ backend, the code looks very similar.

Single file API in Python

There are two functions in the PyArrow Single API to read CSV files: read_csv() and open_csv(). While read_csv() loads all the data in memory and does it fast by using multiple threads to read different parts of the files, open_csv() reads the data in batches and uses a single thread.

If the CSV file is small enough, you should use read_csv(). The code to read the CSV file and write it to a Parquet file would then look like this:

import pyarrow as pa
import pyarrow.csv
import pyarrow.parquet as pq

in_path = '~/datasets/data.csv'
out_path = '~/datasets/data.parquet'

data =  pa.csv.read_csv(in_path)

pq.write_table(data, out_path)

In our case, the file is too large to fit in memory². So instead of using read_csv(), we need to use open_csv(). Because, the CSV file is read in chunks, the code is a little more complex. We need to loop through each chunk, read it, and write it to the Parquet file. This uses little memory but is not as fast as using read_csv(). While read_csv() is multi-threaded, open_csv() uses a single thread. When using open_csv(), the data types need to be consistent in your columns. The function infers the data types on the first chunk of data read, and if the type changes halfway through your dataset in one of your columns, you will run into errors. You can avoid this by specifying the data types manually.

# from 
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.csv

in_path = '~/datasets/data.csv'
out_path = '~/datasets/data.parquet'

writer = None
with pyarrow.csv.open_csv(in_path) as reader:
    for next_chunk in reader:
        if next_chunk is None:
            break
        if writer is None:
            writer = pq.ParquetWriter(out_path, next_chunk.schema)
        next_table = pa.Table.from_batches([next_chunk])
        writer.write_table(next_table)
writer.close()

On my system, the conversion from file to Parquet takes about 190 seconds. Reading the Parquet file can be done with:

data = pq.ParquetDataset(out_path).read()

With this approach, the dataset is in memory, just like when we were using R. Again with 32 GB of RAM in my laptop, I need to be careful with what is running on my system to be able to load this dataset without running out of memory and crashing my Python session.

The Dataset API in Python

To load the CSV file with the Dataset API, we use the dataset() function:

import pyarrow.dataset as ds

in_path = "~/datasets/data.csv"
out_path = "~/datasets/my-data/"

data = ds.dataset(in_path)

Just like with R, importing this file takes about 0.02 seconds.

To convert it to a collection of Parquet files, you use the write_dataset() function. This function takes the same max_rows_per_file argument to control the size of the Parquet file in each partition.

ds.write_dataset(data, out_path, format = "parquet",
                 max_rows_per_file = 1e7)

Reading this collection of Parquet files can also be done with the dataset() function, just like when we used the function to read the single CSV file above. The dataset() function is very flexible and can be used to import data in a variety of formats, and structures, and even combines files from local and remote locations. The format argument is optional as the function detects automatically the file type.

data = ds.dataset(out_path, format = "parquet")

Given the current functionalities implemented PyArrow, querying datasets of this size is possible but it is neither blazing fast nor convenient. A good alternative is to use Ibis with DuckDB as a backend. Ibis provides a single interface to work with data stored in memory or in databases. DuckDB is a self-contained database designed for data analytics. These tools deserve a lot more than a one-sentence summary but this is beyond the scope of this post.

To count the number of unique values, you could use the following approach:

import ibis

ibis.options.interactive = True

con = ibis.duckdb.connect()
data = con.register("parquet:///home/user/datasets/my-data/*.parquet", table_name = "table")

con.table("table").variable.value_counts()

Just like with using R, this takes about a second to count the unique values in one column of our 140 million row dataset.

What this post didn’t mention

I focused on the reading of a CSV file and its conversion to Parquet. I didn’t talk about all the options that both the Single file and the Dataset APIs have to customize the format of the files that are being imported. For instance, both APIs can be used to specify a different column-separator, and cell content that should be treated as missing data.

Conclusion

For a 15 GB data file, the dataset API is better suited to read, convert, and query the data. There is an overhead associated with not having the data in memory but it is greatly reduced if the data is stored as Parquet files. Another advantage is that the approach developed here would scale to much larger datasets where the Single file API would not be able to serialize the data in memory.

With the dataset in this example, the Single file API did not have an opportunity to shine given the hardware constraints of a modern laptop. However, if you are dealing with datasets that fit easily in memory, working with data directly in memory will lead to better query performance.

To summarize what we learned in this post, this brief decision guide to help you choose the correct API to import your data.

Decision tree to help you choose the appropriate Apache Arrow API for your dataset.

Acknowledgments

Thank you to Kae Suarez and Danielle Navarro for reviewing this post and providing feedback that improved its content.

Footnotes

This is not an official name for it but found it is helpful to group these functions that work on a file type at a time. ↩
I am not sure why it fit in memory when I was loading it in R but not with Python. ↩

Creating an Arrow dataset (part 2)

2022-09-06T00:00:00+00:00

Background

In this follow-up post (see part 1 if you missed it), we will explore what happens to the query performance if we read the files straight into Arrow instead of downloading them locally first.

Reading remote CSV files

In the first part, we first downloaded the compressed CSV files locally (using the download.file() function) and then we used the open_dataset() function on this set of files to make it available to Arrow.

However, it is possible to bypass the local download. We can import the files directly over an Internet connection using the read_csv_arrow() function and providing the file URL as the first argument. Once the file is loaded in memory, we can then write it to disk in the parquet format (given that we learned in part 1 that this format provided the best compromise of disk space usage and query performance).

We can then modify the code from the download_daily_package_logs_csv() function from part 1 to the following (lines changed have comments indicated by # <--- at the end of the line).

library(tidyverse)
library(arrow)

## Download the data set for a given date from the RStudio CRAN log website.
## `date` is a single date for which we want the data
## `path` is where we want the data to live
download_daily_package_logs_parquet <- function(date,
                                                path = "~/datasets/cran-logs-parquet-by-day") {

  ## build the URL for the download
  date <- parse_date(date)
  url <- paste0(
    'https://cran-logs.rstudio.com/', date$year, '/', date$date_chr, '.csv.gz'
  )

  ## build the path for the destination of the download
  file <- file.path(
    path,
    paste0("year=", date$year),
    paste0("month=", date$month),
    paste0(date$date_chr, ".parquet")   # <--- change extension to .parquet
  )

  ## create the folder if it doesn't exist
  if (!dir.exists(dirname(file))) {
    dir.create(dirname(file), recursive = TRUE)
  }

  ## download the file
  message("Downloading data for ", date$date_chr, " ... ", appendLF = FALSE)
    arrow::read_csv_arrow(url) %>%      # <--- read directly from URL
      arrow::write_parquet(sink = file) # <--- convert to parquet on disk
  message("done.")

  ## quick check to make sure that the file was created
  if (!file.exists(file)) {
    stop("Download failed for ", date$date_chr, call. = FALSE)
  }

  ## return the path
  file
}

## This function is unchanged from part 1
## and extract the year and month from it
parse_date <- function(date) {
  stopifnot(
    "`date` must be a date" = inherits(date, "Date"),
    "provide only one date" = identical(length(date), 1L),
    "date must be in the past" = date < Sys.Date()
  )
  list(
    date_chr = as.character(date),
    year = as.POSIXlt(date)$year + 1900L, 
    month = as.POSIXlt(date)$mon + 1L
  )
}

Now that we are set up, we can create the file system the same way we did, in part 1.

dates_to_get <- seq(
  as.Date("2022-06-01"),
  as.Date("2022-08-15"),
  by = "day"
)

purrr::walk(dates_to_get, download_daily_package_logs_parquet)

The result is similar to what we achieved in part 1. We have one file for each day placed in a folder corresponding to their month. Except that this time, instead of having compressed CSV files, we have parquet files:

~/datasets/cran-logs-parquet-by-day/
└── year=2022
    ├── month=6
    │   ├── 2022-06-01.parquet
    │   ├── 2022-06-02.parquet
    │   ├── 2022-06-03.parquet
    │   ├── ...
    │   └── 2022-06-30.parquet
    ├── month=7
    │   ├── 2022-07-01.parquet
    │   ├── 2022-07-02.parquet
    │   ├── 2022-07-03.parquet
    │   ├── ...
    │   └── 2022-07-31.parquet
    └── month=8
        ├── 2022-08-01.parquet
        ├── 2022-08-02.parquet
        ├── 2022-08-03.parquet
        ├── ...
        └── 2022-08-15.parquet

Let’s check how large this data is compared to the datasets we created in part 1:

dataset_size <- function(path) {
  fs::dir_info(path, recurse = TRUE) %>%
    filter(type == "file") %>%
    pull(size) %>%
    sum()
}

tribble(
  ~ Format, ~ size,
  "Compressed CSV", dataset_size("~/datasets/cran-logs-csv/"),
  "Arrow", dataset_size("~/datasets/cran-logs-arrow/"),
  "Parquet", dataset_size("~/datasets/cran-logs-parquet/"),
  "Parquet by day",  dataset_size("~/datasets/cran-logs-parquet-by-day/")
) 

# A tibble: 4 × 2
  Format                size
            
1 Compressed CSV       5.01G
2 Arrow               29.67G
3 Parquet              5.06G
4 Parquet by day       4.63G

The dataset with one parquet file per day, is slightly smaller than when we let write_dataset() do its own partitioning which led to one file per month.

We can now compare how quickly Arrow can read these datasets.

bench::mark(
  parquet = open_dataset("~/datasets/cran-logs-parquet", format = "parquet"),
  parquet_by_day = open_dataset("~/datasets/cran-logs-parquet-by-day", format = "parquet"),
  check = FALSE
)

# A tibble: 2 × 6
  expression          min   median `itr/sec` mem_alloc `gc/sec`
                  
1 parquet        139.43ms 143.66ms      6.62    7.91MB     0   
2 parquet_by_day   3.52ms   3.82ms    254.      4.28KB     6.45

Even though there are more files to parse (76 vs. 3), loading the dataset with a parquet file per day is a bit faster.

cran_logs_parquet <- open_dataset("~/datasets/cran-logs-parquet",  format = "parquet")
cran_logs_parquet_by_day <- open_dataset("~/datasets/cran-logs-parquet-by-day",  format = "parquet")

Let’s now explore the performance of a few queries on these datasets.

First, how long does it take to compute the number of rows in these datasets:

bench::mark(
  parquet = nrow(cran_logs_parquet),
  parquet_by_day = nrow(cran_logs_parquet)
)

# A tibble: 2 × 6
  expression          min   median `itr/sec` mem_alloc `gc/sec`
                  
1 parquet           743µs    773µs     1267.    4.74KB     8.48
2 parquet_by_day    745µs    773µs     1273.    1.97KB    10.7 

Not much of a difference.

Let’s now compare the performance of the query we ran in part 1, where we computed the 10 most downloaded packages in the period covered by our dataset.

top_10_packages <- function(data) {
  data %>%
    count(package, sort = TRUE) %>%
    head(10) %>%
    mutate(n_million_downloads = n/1e6) %>%
    select( - n) %>% 
    collect()
}

bench::mark(
  top_10_packages(cran_logs_parquet),
  top_10_packages(cran_logs_parquet_by_day)
)

Warning: Some expressions had a GC in every iteration; so filtering is disabled.

# A tibble: 2 × 6
  expression                                     min   median `itr/sec` mem_al…¹
                                         
1 top_10_packages(cran_logs_parquet)           3.58s    3.58s     0.279   7.19MB
2 top_10_packages(cran_logs_parquet_by_day)    5.76s    5.76s     0.174 165.36KB
# … with 1 more variable: `gc/sec` , and abbreviated variable name
#   ¹​mem_alloc
# ℹ Use `colnames()` to see all variable names

This query runs 1.5 seconds faster on the dataset with one parquet file per month compared to the dataset with one parquet file per day.

The way a dataset is partitioned has an impact on the performance of queries. If you are filtering your dataset along a variable used in the partitioning, some of the files can be skipped. Arrow can directly and only read the file(s) with the relevant information for your query. For instance, if you are performing a query that only touches the month of July, Arrow does not need to look at the files for June or August, leading to potential speed-ups.

Would the partitioning by day help us run our query faster if we were to compute the 10 most downloaded packages for a single day? After all, in this case, we would only need to look at one of the files in our folder of parquet files, and the file in question would be smaller than one that has all the data for the month. Let’s compare the performance of this query for August 1st, 2022:

top_10_packages_by_day <- function(data) {
  data %>%
    filter(date == as.Date("2022-08-01")) %>%
    count(package, sort = TRUE) %>%
    head(10) %>%
    collect()
}

bench::mark(
  top_10_packages_by_day(cran_logs_parquet),
  top_10_packages_by_day(cran_logs_parquet_by_day)
)

# A tibble: 2 × 6
  expression                                          min median itr/s…¹ mem_a…²
                                              
1 top_10_packages_by_day(cran_logs_parquet)         304ms  348ms    2.87   222KB
2 top_10_packages_by_day(cran_logs_parquet_by_day)  354ms  354ms    2.82   167KB
# … with 1 more variable: `gc/sec` , and abbreviated variable names
#   ¹​`itr/sec`, ²​mem_alloc
# ℹ Use `colnames()` to see all variable names

Interestingly, running the query on the monthly parquet file is still faster. It takes about 30% longer to run the queries on the one parquet file per day. The overhead associated with having too many small files in this situation does not compensate for having to look inside a single file to perform this operation. For the benefits of partitioning to be visible, we would need to have more data in each parquet file.

We don’t see a performance benefit of having many small files even when we try to get the result on a single day. But how does this partitioning impact the performance of a query that needs to access multiple random rows? Let’s compare how a query that looks at the number of downloads per day for a given package.

package_downloads_by_day <- function(data, pkg = "arrow") {
  data %>%
    filter(package == pkg) %>%
    count(date) %>%
    arrange(date) %>%
    collect()
}

bench::mark(
  package_downloads_by_day(cran_logs_parquet),
  package_downloads_by_day(cran_logs_parquet_by_day)
)

Warning: Some expressions had a GC in every iteration; so filtering is disabled.

# A tibble: 2 × 6
  expression                                              min   median `itr/sec`
                                                 
1 package_downloads_by_day(cran_logs_parquet)           3.31s    3.31s     0.302
2 package_downloads_by_day(cran_logs_parquet_by_day)    4.46s    4.46s     0.224
# … with 2 more variables: mem_alloc , `gc/sec` 
# ℹ Use `colnames()` to see all variable names

In this case, it takes about 45% longer to perform this query. In this situation, the performance is affected by having to look inside many more files in the dataset with one parquet file per day.

Conclusion

This small example illustrates that it might be worth exploring how best to partition your dataset to benefit the most from the speed that Arrow brings to your queries. In this example, the partitioning that seemed the most “natural” based on the format the data is provided (one parquet file per day) is not the best to make queries run fast.

The variables you include in your queries have also a role to play when deciding how to partition your dataset. It might be best to partition your dataset according to variables you use most often in your queries.

The useR!2022 Arrow tutorial has a convincing demonstration that taking advantage of partitioning for your queries makes them run much faster.

Expand for Session Info

Creating an Arrow dataset

2022-08-22T00:00:00+00:00

Background

While getting started with Apache Arrow, I was intrigued by the variety of formats Arrow supports. Arrow tutorials tend to start with already prepared datasets ready to be ingested by open_dataset(). I wanted to explore what it takes to create your dataset aimed to be analyzed with Arrow and understand the respective benefits of the different file formats it supports.

Arrow can read in a variety of formats: parquet, arrow (also known as ipc and feather)¹, and text-based formats like csv (as well as tsv). Additionally, Arrow provides tools to convert between these formats.

Having the possibility to import datasets in a variety of formats is helpful as you are less constrained by the type of data you can start your analysis on. However, if you are building a dataset from scratch, which one should you choose?

To try to answer this question, we will be using the {arrow} R package to compare the amount of hard drive space these file formats use and the performance of a query in a multi-file dataset using these different formats. This is not a formal evaluation of the performance of Arrow or how best to optimize the partitioning of a dataset, rather it is a brief exploration of the tradeoffs that come with using the different datasets supported by Arrow. I also don’t explain the differences in the data structure of these different formats.

The dataset

We will be using data from https://cran-logs.rstudio.com/. This site gives you access to the log files for all hits to the CRAN² mirror hosted by RStudio. For each day since October 1st, 2012, there is a compressed CSV file (file with the extension .csv.gz) that records the downloaded packages. Each row contains the date, the time, the name of the R package downloaded, the R version used, the architecture (32-bit or 64-bit), the operating system, the country inferred from the IP address, and a daily unique identifier assigned to each IP address. This website has also similar data for the daily downloads of R itself but I will not be using this data in this post.

For this exploration, we are going to limit ourselves to a couple of months of data which will be providing enough data for our purpose. We will download the data for the period from June 1st, 2022 to August 15th, 2022.

Arrow is designed to read data that is split across multiple files. So, you can point open_dataset() to a directory that contains all the files that make up your dataset. There is no need to loop over each file to build your dataset in memory. Splitting your datasets across multiple files can even make queries on your dataset faster, as only some of the files might need to be accessed to get the results needed. Depending on the type of queries you perform most often on your dataset, it can be worth considering how best to partition your files to accelerate your analyses (but this is beyond the scope of this post). Here, the files are provided by date, and we will keep a time-based file organization.

We will use a Hive-style partitioning by year and month. We will have a directory for each year (there is only one year in our example), and within it, a directory for each month. The directory are named according to the convention =. So we will want to organize the files as illustrated below:

└── year=2022
    ├── month=6
    │   └── 
    ├── month=7
    │   └── 
    └── month=8
        └── 

Import the data as it is provided

library(arrow)
library(tidyverse)
library(fs)
library(bench)

The open_dataset() function in the {arrow} package can directly read compressed CSV files³ (with the extension .csv.gz) as they are provided on the RStudio CRAN logs website.

As a first step, we can download the files from the site and organize them using the Hive-style directory structure as shown above.

## Check that the date is really a date,
## and extract the year and month from it
parse_date <- function(date) {
  stopifnot(
    "`date` must be a date" = inherits(date, "Date"),
    "provide only one date" = identical(length(date), 1L),
    "date must be in the past" = date < Sys.Date()
  )
  list(
    date_chr = as.character(date),
    year = as.POSIXlt(date)$year + 1900L, 
    month = as.POSIXlt(date)$mon + 1L
  )
}

## Download the data set for a given date from the RStudio CRAN log website.
## `date` is a single date for which we want the data
## `path` is where we want the data to live
download_daily_package_logs_csv <- function(date,
                                            path = "~/datasets/cran-logs-csv") {

  ## build the URL for the download
  date <- parse_date(date)
  url <- paste0(
    'https://cran-logs.rstudio.com/', date$year, '/', date$date_chr, '.csv.gz'
  )

  ## build the path for the destination of the download
  file <- file.path(
    path,
    paste0("year=", date$year),
    paste0("month=", date$month),
    paste0(date$date_chr, ".csv.gz")
  )

  ## create the folder if it doesn't exist
  if (!dir.exists(dirname(file))) {
    dir.create(dirname(file), recursive = TRUE)
  }

  ## download the file
  message("Downloading data for ", date$date_chr, " ... ", appendLF = FALSE)
  download.file(
    url = url,
    destfile = file,
    method = "libcurl",
    quiet = TRUE,
    mode = "wb"
  )
  message("done.")

  ## quick check to make sure that the file was created
  if (!file.exists(file)) {
    stop("Download failed for ", date$date_chr, call. = FALSE)
  }

  ## return the path
  file
}

## build sequence of dates for which we want the data
dates_to_get <- seq(
  as.Date("2022-06-01"),
  as.Date("2022-08-15"),
  by = "day"
)

## download the data
walk(dates_to_get, download_daily_package_logs_csv)

Let’s check the content of the folder that holds the data we downloaded:

~/datasets/cran-logs-csv/
└── year=2022
    ├── month=6
    │   ├── 2022-06-01.csv.gz
    │   ├── 2022-06-02.csv.gz
    │   ├── 2022-06-03.csv.gz
    │   ├── ...
    │   └── 2022-06-30.csv.gz
    ├── month=7
    │   ├── 2022-07-01.csv.gz
    │   ├── 2022-07-02.csv.gz
    │   ├── 2022-07-03.csv.gz
    │   ├── ...
    │   └── 2022-07-31.csv.gz
    └── month=8
        ├── 2022-08-01.csv.gz
        ├── 2022-08-02.csv.gz
        ├── 2022-08-03.csv.gz
        ├── ...
        └── 2022-08-15.csv.gz

We have one file for each day, placed in a folder corresponding to their month. We can now read this data using {arrow}’s open_dataset() function:

cran_logs_csv <- open_dataset(
  "~/datasets/cran-logs-csv/",
  format = "csv",
  partitioning = c("year", "month")
)
cran_logs_csv

FileSystemDataset with 76 csv files
date: date32[day]
time: time32[s]
size: int64
r_version: string
r_arch: string
r_os: string
package: string
version: string
country: string
ip_id: int64
year: int32
month: int32

The partitioning has been taken into consideration as the output shows that the dataset contains the variables year and month which are not part of the data we downloaded. They are coming from the way we organized the downloaded files.

Convert to Arrow and Parquet files

Now that we have the compressed CSV files on disk, and that we opened the dataset with open_dataset(), we can convert it to the other file formats supported by Arrow using {arrow}’s write_dataset() function. We are going to convert our collection of .csv.gz files into the Arrow and Parquet formats.

## Convert the dataset into the Arrow format
write_dataset(
  cran_logs_csv,
  path = "~/datasets/cran-logs-arrow",
  format = "arrow",
  partitioning = c("year", "month")
)

## Convert the dataset into the Parquet format
write_dataset(
  cran_logs_csv,
  path = "~/datasets/cran-logs-parquet",
  format = "parquet",
  partitioning = c("year", "month")
)

Let’s inspect the content of the directories that contain these datasets.

fs::dir_tree("~/datasets/cran-logs-arrow/")

~/datasets/cran-logs-arrow/
└── year=2022
    ├── month=6
    │   └── part-0.arrow
    ├── month=7
    │   └── part-0.arrow
    └── month=8
        └── part-0.arrow

fs::dir_tree("~/datasets/cran-logs-parquet/")

~/datasets/cran-logs-parquet/
└── year=2022
    ├── month=6
    │   └── part-0.parquet
    ├── month=7
    │   └── part-0.parquet
    └── month=8
        └── part-0.parquet

These two directories have the same layout organized by year and month as with our CSV files given that we kept the same partitioning. The files within the directories have an extension that matches their file format. One difference is that there is a single file for each month. We used the default values for write_dataset() and the number of rows for each month is smaller than the threshold this function uses to split the dataset into multiple files.

Comparison of the different formats

Let’s compare how much space these different file formats take on disk:

dataset_size <- function(path) {
  fs::dir_info(path, recurse = TRUE) %>%
    filter(type == "file") %>%
    pull(size) %>%
    sum()
}

tribble(
  ~ Format, ~ size,
  "Compressed CSV", dataset_size("~/datasets/cran-logs-csv/"),
  "Arrow", dataset_size("~/datasets/cran-logs-arrow/"),
  "Parquet", dataset_size("~/datasets/cran-logs-parquet/")
) 

# A tibble: 3 × 2
  Format                size
            
1 Compressed CSV       5.01G
2 Arrow               29.67G
3 Parquet              5.06G

The Arrow format takes the most space with almost 30GB while both the compressed CSV and the Parquet files use about 5GB of hard drive.

We are now set up to compare the performance of doing computation of these different dataset formats.

Let’s open these datasets with the different formats:

cran_logs_csv <- open_dataset("~/datasets/cran-logs-csv/", format = "csv")
cran_logs_arrow <- open_dataset("~/datasets/cran-logs-arrow/", format = "arrow")
cran_logs_parquet <- open_dataset("~/datasets/cran-logs-parquet/", format = "parquet")

We will compare how long it takes for Arrow to compute the 10 most downloaded packages in the time period our dataset covers using each file format.

top_10_packages <- function(data) {
  data %>%
    count(package, sort = TRUE) %>%
    head(10) %>%
    mutate(n_million_downloads = n/1e6) %>%
    select( - n) %>% 
    collect()
}

bench::mark(
  top_10_packages(cran_logs_csv),
  top_10_packages(cran_logs_arrow),
  top_10_packages(cran_logs_parquet)
)

Warning: Some expressions had a GC in every iteration; so filtering is disabled.

# A tibble: 3 × 6
  expression                              min   median itr/se…¹ mem_al…² gc/se…³
                                    
1 top_10_packages(cran_logs_csv)       29.57s   29.57s   0.0338   8.19MB   0    
2 top_10_packages(cran_logs_arrow)       2.1s     2.1s   0.475  165.39KB   0.475
3 top_10_packages(cran_logs_parquet)    3.32s    3.32s   0.301  137.11KB   0    
# … with abbreviated variable names ¹​`itr/sec`, ²​mem_alloc, ³​`gc/sec`

While it takes about 4 seconds to perform this task on the Arrow or Parquet files, it takes more than 30 seconds to do it on the CSV files.

Conclusion

Having Arrow point directly to the folder of compressed CSV file might be the most convenient but it comes with a high-performance cost. Arrow and Parquet have similar performance but the Parquet files take less space on disk and would be more suitable for long-term storage. This is why large datasets like the NYC taxi data is distributed as a series of Parquet files.

In the future, I might explore how using different variables for partitioning or how the number of files in the partitions affects the performance of the queries (EDIT: this post is now available. If you have other ideas of topics that you would me to explore, do not hesitate to leave a comment below.

Going further

If you would like to learn more about the different formats, check out the Arrow workshop (especially Part 3: Data Storage) that Danielle Navarro, Jonathan Keane, and Stephanie Hazlitt taught at useR!2022.

Acknowledgments

Thank you to Kae Suarez and Danielle Navarro for reviewing this post.

Post Scriptum

I wrote a follow-up post that explores the impact of partitioning the dataset on performance.

Expand for Session Info

`foghorn` 1.3.1 released

2020-09-08T00:00:00+00:00

A new version of foghorn (version 1.3.1) was just accepted on CRAN.

foghorn is an R package that allows you to:

browse the results of the CRAN checks on your package (with cran_results() and cran_details());
check where your package stands when submitted to CRAN (with cran_incoming());
and starting with version 1.3.1, check whether your package is in the Win builder queue (with winbuilder_queue()).

The idea of inspecting the Win-builder queue was proposed by Kirill Müller.

If you would like to start using foghorn, check out the vignette that comes with the package.

Feedback and suggestions for foghorn are welcome!

Migrate from Gmail to HelpScout with R

2020-04-17T00:00:00+00:00

Preamble

This is a long and somewhat dense post. Even if you do not have to migrate emails from Gmail to HelpScout, I hope this post will be useful to you, as the general approach could be interesting to other problems that involve working with APIs.
The full code I actually used for the email migration is available at: https://github.com/carpentries/emailmigration and I include links pointing to functions in the GitHub repo¹ throughout the post below to illustrate my points.

The problem and its solution

At The Carpentries, Regional Coordinators help us organize workshops across the globe. In the past, each Regional Coordinator was set up with a Gmail account (through The Carpentries’s GSuite plan). However, as the number of Regional Coordinators grew, and as some geographic areas have more than one Regional Coordinator, the Gmail account model was starting to cause some issues.

The Carpentries Core Team has been using HelpScout for a while and is a much more suitable tool to manage emails and inboxes as a team.

The main challenge with transitioning the Regional Coordinators to using HelpScout was to import the old messages from Gmail to HelpScout. To tackle this problem, I used R and this blog post describes the approach I took.

Technical overview

Before doing anything else, we used the GSuite data migration tool to transfer all emails for each Regional Coordinator account into a single account. Having all the emails to import in the same place makes things easier.

This post goes through the steps I took to perform this migration:

Figure out authentication with the Gmail API, and with the HelpScout API
Get familiar with the HelpScout API and write R functions to perform the tasks needed
Convert Gmail threads into HelpScout conversations
Test migration on 100 Gmail threads
Perform the full migration

Choice of packages and approach:

Working with the Gmail API is made much easier with the wonderful gmailr package.
I didn’t find an already made package to work with the HelpScout web API so I wrote a few functions to interact with the endpoints I needed using the httr package.
The mechanics of converting the data coming from the Gmail web API into the format needed by the HelpScout API to import the conversation was done using the R6 package. The R6 classes and methods made it easier to separate storing each element needed by the HelpScout API as private elements and the actual formatting that was handled with methods.
When working with web APIs a lot can go wrong: there is a weird data format that your code didn’t know how to handle, your internet connection goes down, you reach the rate limit, etc. Therefore, I used the storr package to cache (1) the R6 objects that act as the bridge between the 2 APIs; (2) the responses from the HelpScout API to make sure all the threads were converted correctly.
I organized all the code as a barebone package. It makes code management easier and is a good habit to take. Here it was a one-off task but if it was something that I’d use regularly, it means that I could develop tests, write documentation, and enable continuous testing. I could then write and update my code, and rely on devtools::load_all().

1. Authentication

1.1. Gmail API

The instructions in the gmailr package’s README are clear. You can use the gm_threads() function, for instance, to check that the authentication is working as expected.

1.2. The HelpScout API

The HelpScout API uses the OAuth 2.0 protocol. The httr package handles this well.

Create a new app within HelpScout, and use https://localhost:1410/ for the redict URL. Take note of the key and secret. Use this information to create a new app object in R with httr:

hs_app <- httr::oauth_app(
  "helpscout",
  key = "",
  secret = ""
)

and then use this object to do the authentication online:

hs_token <- httr::oauth2.0_token(
  httr::oauth_endpoint(
    authorize = "https://secure.helpscout.net/authentication/authorizeClientApplication",
    access = "https://api.helpscout.net/v2/oauth2/token"),
  app = hs_app)

htoken <- httr::config(token = hs_token)

We can then use the htoken object across all our calls to the HelpScout web API.

2. Getting started with the HelpScout web API

When working with a new web API, first read the documentation to understand how things are set up. From this initial reading, it became clear that Gmail and HelpScout use different words for related concepts.

HelpScout	Gmail
thread	message
conversation	thread

Keeping this straight in my mind took some time… and because I’m more used to the terms used by Gmail, I used this vocabulary in my function names (for the most part).

Another thing that I needed was HelpScout’s internal identifier for the mailbox into which the emails were being imported. So the first function I wrote against HelpScout’s API was hs_mailbox_id() which returned the internal identifier for the mailbox that was of interest to me.

The second thing I needed to do was to make sure I understood how to use the API to import an actual conversation. I started with fake data I could control to ensure that I had something simple that I knew worked and I could compare against when things didn’t work with real data. Even if the documentation of an API is good, there are, more often than not, small details that are not described that you need to figure out. Having this data as a starting point is useful for these tests.

The actual code to create a new ~~thread~~ conversation in HelpScout ended up being:

hs_create_thread <- function(thread, hstoken) {
  body <- jsonlite::toJSON(thread, auto_unbox = TRUE)

  httr::POST(
    "https://api.helpscout.net",
    path = "/v2/conversations",
    body = body,
    htoken,
    httr::content_type("application/json; charset=UTF-8")
  )
}

This is not the code I would have written if it was part of a package intended for others to use. For instance, I would have wanted to check the response of the API after each request. But for my particular use case, it made it easier to return this response and inspect manually after the fact once I had confirmed that this code was working for most requests.

2. Extracting the content of the emails from Gmail

This was the most time-consuming part as there were lots of unexpected details that came up to get a smooth conversion between the two APIs.

2.1. Things that were easy

The gmailr::gm_subject() worked every time to get the subject of the threads for all the messages.

2.2. Things that were almost easy

Extracting the people involved in the conversation. The gmailr::gm_to() and gmailr::gm_from() worked well to extract the email addresses. The small catch was that some email addresses were formatted as FirstName LastName, others had only email@address.rr, and when multiple people were involved a comma separated them. However, in some cases, people have a comma in their names.
Extracting the date. The gmailr::date() returns the date from the email in Unix time. The anytime package is useful at converting Unix time into other formats, including the iso8601 that was expected by the HelpScout API. I still had to manually add a final Z to the character string.

2.3. Things that were not so easy

Extracting the email attachments. The attachments themselves are not returned by the API. Instead, the API returns an URL that points to the address where the attachments can be retrieved. The HelpScout’s API accepts the attachments as base64-encoded strings. The gmailr package helped to retrieve this data, but the data returned by the Gmail API is base64url encoded. Thankfully, converting to regular base64 is a short regular expression substitution away once you know the difference between the two.
The thing that was the most puzzling was parsing the actual body of the emails. The gmailr::gm_body() worked for only a small fraction of the emails I had to deal with. After many trials and errors, I wrote a function to reliably retrieve the content of the emails². There were many situations to deal with as the messages can be:
- “multipart” the body of the email is provided both in plain text format or in HTML format which allows for email clients that don’t support HTML-formatting to provide the plain text version of the message;
- either only plain text or in HTML format
- provided as attachments (what some email clients do when you forward a message).
Depending on the situation, the location of the body of the email within the deeply nested list that was returned by the Gmail API could vary. I ended up writing a recursive algorithm that traversed the list to find and retrieve the relevant content of the emails.

The last catch was that plain text messages that included an URL were interpreted by the HelpScout API as being HTML-formatted. It meant that the whitespace to indicate the line breaks were ignored making the body of the messages large blocks of texts that were very hard to read and follow. I relied on the commonmark::markdown_html() to convert these plain text messages into HTML that then looked good once they were uploaded onto HelpScout using the API.

3. Conversion between Gmail and HelpScout

Now that I had access to all the relevant information from the emails, I needed to format it so it could be imported by the HelpScout API. For this, I used the R6 object-oriented programming system.

Each element coming from the Gmail API was individually stored as a private field, and an accessor method ($get()) created the list in the format needed to be ingested by HelpScout’s API.

I used 3 classes for this:

one for the HelpScout conversations (the Gmail threads)
one for the HelpScout threads (the Gmail messages)
one for the attachments

This modularity helped debugging and limited the complexity of each class.

Because all the emails are going to be in the same inbox in HelpScout, I wanted an easy way to tag the conversations based on the team of Regional Coordinators that were involved. The R6 system was useful for this because once the email information was stored within the object, I could use a private method called by the accessor to extract all the people involved, and add tags in HelpScout to help Regional Coordinators find past conversations that are relevant to them.

It was one of the first times I used R6³ for a real task and I could see its potential. If the code written here were for public consumption, it would have provided a good framework to add more tests on the data structure of the individual elements that were coming from the Gmail API to ensure that the output from the accessor method was always formatted correctly before trying to convert it in the format required by HelpScout’s API.

4. Caching

My previous experience working with web APIs have taught me that things can go wrong, and it is always a good idea to keep track (on disk and not only on memory) of the requests that have been tried and the ones that have not, and the requests that succeeded and the ones that failed. Especially, when your scripts do thousands of API calls, you don’t want to have to run everything again once your script fails because your internet connection goes down for a short while, or the data is not formatted properly because you are dealing with an edge case.

For this, I use the storr package and its functionality to rely on hooks to retrieve external data. storr is a key-value store. It is not that different than using variable names to store objects in memory as you normally do in your R terminal:

## setting a variable
cat_name <- "Felix"

## getting the content of the variable
cat_name

When using a storr store:

## defining the storr
st <- storr::storr_rds(path = "cache")

## setting a variable
st$set("cat_name", "Felix")

## getting the variable name
st$get("cat_name")

The difference is that storr provides different backends for storing your object and, if like in this example, you use storr_rds, your objects are stored as rds files on your disk and are available beyond your current R session. How does that help with the problem here?

A great feature of storr is that you can set up your store to call a function to create the object instead of providing it directly with $set().

It means that you store the content of a variable, your key into the store, and you can retrieve it:

## the hook function
fetch_hook_random_cat_name <- function(key, namespace) {
  sample(c("Felix", "Garfield", "Tigger", "Mowgli"), 1)
}

## defining the storr
st <- storr::storr_external(
  storr::driver_rds(path = "cache"),
  fetch_hook_random_cat_name
)

## the first time you call a key, it will run the hook function
st$get("cat_name")

## subsenquently, it will return the value stored in the store
st$get("cat_name")

The hook function always takes the two arguments key and namespace but they don’t need to be used in the body of the function as in the example above.

We can extend this approach to store the output of time-consuming computations or the results of API calls⁴. For instance, here, I created a store to keep the output of the function convert_gmail_thread(), and used get_gmail_thread() as a wrapper to access the store.

fetch_hook_gmail_threads <- function(key, namespace) {
  convert_gmail_thread(key)
}

store_gmail_threads <- function(path = "cache/threads") {
  storr::storr_external(
    storr::driver_rds(path),
    fetch_hook_gmail_threads
  )
}

get_gmail_thread <- function(thread_id, namespace) {
  store_gmail_threads()$get(thread_id, namespace)
}

When calling get_gmail_thread(), using a thread_id that had not been retrieved using the Gmail API before, the function convert_gmail_thread() will be called, getting all the information needed for this particular thread, and storing it in an R6-class object. If another part of the script fails, we do not need to redo the calls to the Gmail API, instead the cached copy within the store will be retrieved.

I used a similar approach to store the responses from the HelpScout API, and wrapped at the same time the call to the get_gmail_thread() function above. A slightly simplified version of what I used is:

fetch_hook_hs_response <- function(key, namespace) {
  res <- get_gmail_thread(key, namespace)
  hs_create_thread(res$get(), htoken)
}

store_hs_responses <- function(path = "cache/hs_responses") {
  storr::storr_external(
    storr::driver_rds(path),
    fetch_hook_hs_response
  )
}

get_hs_response <- function(thread_id, namespace) {
  store_hs_responses()$get(thread_id, namespace)
}

So, what’s happening here? I use the Gmail thread ID as a single point of entry for the entire script (retrieve the thread from the Gmail API, convert it to the format expected by the HelpScout API, upload the thread to HelpScout). Depending on whether the queries have already been made and stored in the cache, the script will retrieve the data from the API or the objects stored on disk in the cache.

What does the namespace argument do? Using namespacing in storr allows you to organize your objects in your store. Especially, it allows you to have objects with the same name but with different values. Here, I planned to use namespaces to keep track of my different attempts. If the first attempt would have failed for some threads, I could fix the problem in the code, and re-attempt the HelpScout API calls just for the ones that failed under a different namespace.

5. Putting it all together

Once I had most of the pieces together, I started by testing the code on the first 100 threads (as it’s the default number of threads that gmailr returns). That was a manageable number to see how the script behaved while being large enough that many different types of messages would be encountered. At that time, I didn’t use the caching system.

Once the first 100 threads could be imported successfully in HelpScout, I wrote a function to retrieve the identifiers for all the threads in the inbox that needed to be imported, and iterated on these identifiers to call the get_hs_response function:

get_all_threads <- function() {
  
  first_it <- gm_threads()
  next_token <- first_it[[1]]$nextPageToken
  
  res <- append(list(), first_it)
  
  while (length(next_token) > 0) {
    tmp <- gm_threads(page_token = next_token)
    res <- append(
      res, tmp
    )
    next_token <- tmp[[1]]$nextPageToken
    message("next token: ", next_token)
  }
  
  res
}

threads <- get_all_threads()

threads_ids <- purrr::map(
  threads,
  ~ purrr::map_chr(.$threads, ~ .$id)
) %>%
  unlist()

hs_res <- purrr::walk(
  threads_ids,
  ~ get_hs_response(., namespace = "v2020-04-10.1")
)

As part of the hook function that takes care of uploading conversations to HelpScout, I check whether the upload was successful and based on that I created and assigned a Gmail label to the thread. This was an additional safeguard that I could use to flag threads that didn’t import successfully.

Once the upload completed, I could then inspect the content of the store:

## Retrieve the threads_ids from the store
idx <- store_hs_responses()$list(namespace = "v2020-04-10.1")

## Retrieve the status code for the HelpScout API responses
is_error <- purrr::map_lgl(
  idx,
  ~ httr::status_code(
    store_hs_responses()$get(., namespace = "v2020-04-10.1")
  ) >=  400
)

## How many calls failed?
sum(is_error)

## Which thread_ids failed?
idx[is_error]

and double check that it was the same threads that were labeled with failure- in Gmail.

Lessons learned

As often with using programming to solve problems, what might seem like a simple task: “Transfering emails from one system to an other” is a collection of small problems. Being able to break down the big problems into small ones, and knowing how to address them comes with experience. Experience will help you recognize problems similar to some you have already solved, and reflecting on these past experiences will help you identify the algorithms, packages, and general code organization that are most likely to help you solve your problem.

In The Carpentries Instructor Training, when we teach about expertise, we talk about how the mental model of experts is denser and more connected. These features make it more difficult for experts to teach beginners because they have forgotten what it is like to not know how to break down a large problem into multiple small ones. The problem here is not just “migrate a bunch of emails between two systems”, there is a lot more to it. I wrote this blog post with the intent to demonstrate the approach I took to break down a problem into small ones and, in the process, describe the tools and techniques I chose to address them.

Expertise is subjective and relative, and I certainly do not claim that the approach I chose here is the best, the most efficient or the most elegant. There is certainly room for improvement. For instance, parts of the code could be re-factored to make it more organized, parts could be rewritten to be more defensive, and there is no documentation (besides this blog post) and barely any comments.

I am interested in hearing your perspective and thoughts on how the problem could have been approached differently and the tools you would have chosen to address it. If this post was useful to you to help you solve a different problem, I would also love to hear about it! Leave a comment below or contact me using the info provided on the left of this page.

Footnotes

You may notice that the Git history for the repo includes the key and secret for the HelpScout OAuth authentication. By themselves, these are not enough to access any data, as you also need to authenticate with a valid HelpScout account within our organization. These credentials have also been revoked. ↩
I’ll be submitting a pull request to gmailr soon. ↩
If you are interested in learning more about the object-oriented programming R6 system, the chapter about it in the “Advanced R” book by Hadley Wickham is a great place to start. ↩
If you are interested in learning more about storr, read the documentation for the package and the vignette on external data that initially helped me get started with this amazingly useful package. ↩

Advent of Code 2018

2018-12-01T00:00:00+00:00

I’m going to try to complete the Advent of Code again this year. I’ll put all the exercises I complete in this post.

Links to the puzzles are at https://adventofcode.com/2018

Day 1

library(tidyverse)


## part 1
readr::read_lines("advent-data/2018-12-01-day1.txt") %>%
  as.numeric() %>%
  sum()

## [1] 408

## part 2
input <- readr::read_lines("advent-data/2018-12-01-day1.txt") %>%
  as.numeric()

already_seen <- function(input) {

  i <- 1

  while (TRUE) {
    v_sum <- cumsum(rep(input, i))
    has_dup <- any(duplicated(v_sum))
    if (has_dup) {
      return(v_sum[which(duplicated(v_sum))[1]])
    }
    i <- i + 1
  }
  
}

already_seen(input)

## [1] 55250

Day 2

input <- readr::read_lines("advent-data/2018-12-02-day2.txt") 

## part 1
count_letters <- function(input) {

  n_letters <- strsplit(input, "") %>%
    purrr::map(table)
  
  has_2 <- function(x) {
    as.integer(any(x == 2))
  }
  has_3 <- function(x) {
    as.integer(any(x == 3))
  }

  has_2_vec <- purrr::map_int(n_letters, has_2)
  has_3_vec <- purrr::map_int(n_letters, has_3)

  sum(has_2_vec) * sum(has_3_vec)
}

count_letters(input)

## [1] 6000

## part 2
all_in <- crossing(in1 = input, in2 = input) %>%
  mutate(
    split1 = strsplit(in1, ""),
    split2 = strsplit(in2, ""),     
    n_diff = map2_int(split1, split2, ~ sum(.x != .y))    
  )

all_in %>%
  filter(n_diff == 1) %>%
  slice(1) %>%
  mutate(word = map2_chr(
    split1,
    split2,
    function(x, y) {
      x <- unlist(x)
      y <- unlist(y)
      paste(x[x == y], collapse = "")
    })) %>%
  pull(word)

## [1] "pbykrmjmizwhxlqnasfgtycdv"

Day 3

That’s far from the prettiest code I’ve written! But it gets the job done.

extract_coords <- function(input) {

  readr::read_delim(
    input, delim = " ", col_names = FALSE
  ) %>%
    tidyr::extract(X1, into = "id", regexp = "([[:digit]]+)") %>%
    tidyr::extract(X3, into = c("x_begin", "y_begin"), regex = "([[:digit:]]+),([[:digit:]]+):") %>%
    tidyr::extract(X4, into = c("width", "height"), regex = "([[:digit:]]+)x([[:digit:]]+)") %>%
    dplyr::select(-X2) %>%
    dplyr::mutate_all(as.numeric)
  
}


find_total_dim <- function(coords) {
  res <- coords %>%
    dplyr::mutate(total_width = x_begin + width,
                  total_height = y_begin + height)

  c(total_width = max(res$total_width),
    total_height = max(res$total_height))
  
}


fill_matrix <- function(input) {

  c <- extract_coords(input)
  m_dim <- find_total_dim(c)

  M <- matrix(0,
              nrow = m_dim[1],
              ncol = m_dim[2])


  for (i in seq_len(nrow(c))) {
    i_s <- (c$x_begin[i] + 1):(c$x_begin[i] + c$width[i])
    j_s <- (c$y_begin[i] + 1):(c$y_begin[i] + c$height[i])
    M[i_s, j_s] <- M[i_s, j_s] + 1
  }

  M
  
}


more_two_claims <- function(input) {
  M <- fill_matrix(input)
  sum(M >= 2) 
}

## part 1 answer
more_two_claims("advent-data/2018-12-03-day3.txt")

## Parsed with column specification:
## cols(
##   X1 = col_character(),
##   X2 = col_character(),
##   X3 = col_character(),
##   X4 = col_character()
## )

## [1] 109716

overlaps <- function(x_begin, y_begin, width, height, M) {
  i <- (x_begin + 1):(x_begin + width)
  j <- (y_begin + 1):(y_begin + height)
  all(M[i, j] == 1) 
}

no_overlap <- function(input) {
  M <- fill_matrix(input)

  c <- extract_coords(input)
  res <- logical(nrow(c))
  
  for (i in seq_len(nrow(c))) {
    res[i] <- overlaps(c$x_begin[i], c$y_begin[i],
                       c$width[i], c$height[i], M)
  }

  c$id[res]
  
}

## part 2 answer
no_overlap("advent-data/2018-12-03-day3.txt")

## Parsed with column specification:
## cols(
##   X1 = col_character(),
##   X2 = col_character(),
##   X3 = col_character(),
##   X4 = col_character()
## )
## Parsed with column specification:
## cols(
##   X1 = col_character(),
##   X2 = col_character(),
##   X3 = col_character(),
##   X4 = col_character()
## )

## [1] 124