diff --git a/instructors/01-practical-tutors.qmd b/instructors/01-practical-tutors.qmd index 2497a96d..8ff6144f 100644 --- a/instructors/01-practical-tutors.qmd +++ b/instructors/01-practical-tutors.qmd @@ -35,7 +35,6 @@ knitr::opts_chunk$set( This practical is based in the following tutorial episodes: - -- - @@ -55,44 +54,42 @@ This practical has two activities. **Goal:** -Get a clean and standardized data frame using the following available inputs: +Produce a clean and standardized data frame from the following input: -- Raw messy data frame in CSV format +- Raw, messy CSV file **Steps:** -- Open file `01-practical-activity-1.R` and complete all the lines marked with `#`, following the detailed steps provided within the R file. -- First, complete the argument to read the data. Paste the link as a “string” in `read_csv()`. -- Second, complete the cleaning process. Add functions based on the data needs. Connect them using the pipe `%>%`: - - Standardize column names - - Standardize dates - - Check date sequence - - Convert to numeric - - Replace missing values - - Clean using dictionary - - Remove constants - - Remove duplicates -- Third, complete the cleanepi::timespan() arguments. Access the help manual running `?cleanepi::timespan()` in the console. -- Paste the outputs. Reply to questions. +1. Open `01-practical-activity-1.R` and complete every line marked with `#`, following the instructions in the file. +2. Complete the argument in `read_csv()` by pasting the data URL link as a string. +4. Complete the arguments of cleaning function. Remove functions based on the data needs. +5. Paste screenshots of the data frame before and after cleaning. Answer the questions below. **Questions:** -Within your room, Write your answers to these questions: +1. **Diagnose the raw data.** What cleaning operations are needed? Which ones diagnose issues or perform cleaning actions? +2. **Time unit.** What time unit best describes the time span you need to calculate? +3. **Report.** Print the report. Which features are most useful for you as an analyst to understand or communicate the issues in the data frame? -- Diagnose the raw data. What data cleaning operations need to be performed on the dataset? Write all of them before writing the code. -- What time unit best describes the time span to calculate? -- Print the report: What features do you find useful to communicate with a decision-maker? -- Compare: What differences do you identify from other room outputs? (if available) +Discuss your answers with your group before sharing with the wider room. ### Inputs -| Room | Data | Link | Calculate time span | Categorize time span | -|---|---|---|---|---| -| 1 | Small linelist | | Age as of today | breaks = c(0, 20, 35, 60, 80) | -| 2 | Large linelist | | Delay from onset of symptoms to the time of death | breaks = c(0, 10, 15, 40) | -| 3 | Serology data [^1] | | Time from last exposure to vaccine | breaks = c(0, 30, 100, 600) | +| Room | Data | Link | Calculate time span | +|---|---|---|---| +| 1 | Small linelist | | Age (as of today) | +| 2 | Large linelist | | Time from symptom onset to death | +| 3 | Serology data [^1] | | Time from last vaccine dose to sample collection [^2] | + +[^1]: **Context of Serological data**: Participants of a study are vaccinated against COVID-19 in multiple doses. Researchers collect serum samples some days after each vaccination dose and expose the serum to emerging SARS-CoV-2 variants. They measure the titre of this immunological response. The higher the titre, the higher the antigenic response. Ref: -[^1]: Context of Serological data: Participants of a study are exposed to COVID-19 vaccines, then their serum samples are collected and challenged to emerging SARS-CoV-2 variants. They measure the titer of this immunological response. The higher the titre, the higher the antigenic response. Let's focus on describing the change in the frequency of vaccine categories (last vaccine exposure) through time (last date of exposure). Ref: https://www.thelancet.com/journals/laninf/article/PIIS1473-3099(24)00484-5/fulltext +[^2]: **How to read the repeated-measurements cohort dataset**: +Subject `pid = 1` was `Infection naive` (never exposed to infection). +Was exposed 2 times (`exp_num`) to the vaccine type BNT162b2 (`last_vax_type`). +The last vaccine exposure was on 2021-03-08 (`last_exp_date`). +One serum sample was obtained on 2021-03-10 (`date`, two days after last vaccine exposure). +The antibody titre against `Alpha` and `Delta` SARS-CoV-2 variants (`titre_type`) was 5 units (`value`). +The antibody titre against `Ancestral` SARS-CoV-2 variants was 176 units. ::: {.content-visible when-format="docx"} @@ -167,44 +164,39 @@ Write your answers to the questions above: -## Activity 2: Validate linelist and plot epicurve +## Activity 2: Plot delays and epicurves **Goal:** -Get a validated linelist and incidence plot using the following available inputs: +Using a clean linelist data frame, produce: -- Clean data frame object +- A delay distribution plot showing time between two epidemiological events. +- An incidence plot (epicurve) showing case counts over time. **Steps:** -- Open the file `01-practical-activity-2.R` and complete all the lines marked with `#`, following the detailed steps provided within the R file. -- First, complete linelist::make_linelist() arguments. -- Second, complete the {linelist} function that can validate a linelist. -- Third, complete the arguments of the incidence2::incidence() -- Fourth, keep, drop, or change argument values in function plot() -- Paste the outputs. Reply to questions. +1. Open `01-practical-activity-2.R` and complete every line marked with `#`, following the instructions in the file. +2. Describe the pre-configured epidemiological delay. +3. Complete the arguments of `incidence2::incidence()` to generate an incidence object. +4. Adjust the arguments of `plot()` to get the most informative epicurve. Read the [`plot()` reference manual](https://www.reconverse.org/incidence2/manual.html#sec:man-plot.incidence2) to find available arguments. +5. Paste your plots and reply to the questions below. **Questions:** -Within your room, Write your answers to these questions: +- Which combination of time unit and case categories best captures the outbreak pattern and why? +- What does the shape of your epicurve suggest about how this outbreak spread? +- Which is larger in your delay distribution: the mean or the median, and what does that tell you about its shape? +- How might delays in the data collection process affect your interpretation of the most recent cases? -- In the validation step, Do you need to allow for extra variable names and types for the `Date` and `Categorical` variable? - - _[Read this GitHub issue as a hint](https://github.com/epiverse-trace/linelist/issues/176) to allow for extra variables._ -- What is the most apprioriate time unit to aggregate the incidence plot, based on visual inspection? -- Does keeping or dropping arguments like `fill`, `show_cases`, `angle`, `n_breaks` improve the incidence plot? - - _[Read `plot()` reference manual](https://www.reconverse.org/incidence2/manual.html#sec:man-plot.incidence2) to find its arguments._ -- Interpret: How would you communicate these results to a decision-maker? -- Compare: What differences do you identify from other room outputs? (if available) +Discuss your answers with your group before sharing with the wider room. ### Inputs -Use outputs from activity 1. - -| Room | Date | Categorical variable | +| Room | Disease | Link | |---|---|---| -| 1 | Date reporting | Age category | -| 2 | Date onset | Outcome | -| 3 | Last exposure date | Last vaccine type | +| 1 | COVID | | +| 2 | Ebola | | +| 3 | Unknown | | ::: {.content-visible when-format="docx"} @@ -214,7 +206,7 @@ Room 1 | Data | Paste screenshot below | |---|---| -| validated | | +| epidemiological delay | | | incidence plot | | Write your answers to the questions above: @@ -236,7 +228,7 @@ Room 2 | Data | Paste screenshot below | |---|---| -| validated | | +| epidemiological delay | | | incidence plot | | Write your answers to the questions above: @@ -258,7 +250,7 @@ Room 3 | Data | Paste screenshot below | |---|---| -| validated | | +| epidemiological delay | | | incidence plot | | Write your answers to the questions above: @@ -340,17 +332,17 @@ activity 2 ##### activity 2 -| Room | Output | -|---|---| -| 1 | ![image](https://hackmd.io/_uploads/ry5d6xnA1e.png) | -| 2 | ![image](https://hackmd.io/_uploads/SJ2f0e2Cyx.png) | -| 3 | ![image](https://hackmd.io/_uploads/H1-PRlhA1g.png) | +| Group | Delay | Incidence | +| --- | --- | --- | +| 1 | ![image](https://hackmd.io/_uploads/SkIzvCrMMe.png) | ![image](https://hackmd.io/_uploads/rkIuVRBGMe.png) | +| 2 | ![image](https://hackmd.io/_uploads/rys4wCHGfl.png) | ![image](https://hackmd.io/_uploads/H1F-FAHGfe.png) | +| 3 | ![image](https://hackmd.io/_uploads/BkoSPCBMfg.png) | ![image](https://hackmd.io/_uploads/SJpIICSGzl.png) | -| Room | validation | time unit | incidence plot | -|---|---|---|---| -| 1 | The timespan_category (age categories) must be added as an extra variable. Declare age as “factor”. | An appropriate time unit to aggregated is using interval by month. | The argument `show_cases` can improve the visibility of `fill` categorical variables | -| 2 | It is not required to add an extra variable. | An appropriate time unit to aggregate is using interval by day. | Keeping angle and `n_breaks` works. Dropping `fill` keeps the facets making "died" are move visible. | -| 3 | last_exp_date and last_vax_type must be added as extra variables. Declare last_vax_type as “character”, and last_vax_date as “Date”. | An appropriate time unit to aggregated is using interval by month. | Keeping `fill` by last_vax_type works. | +| Room | Incidence arguments | Plot arguments | +|---|---|---| +| 1 | Date by onset and outcome help identify the low proportion of deaths among cases in time. | Keeping `fill` helps visualizing the delay and peak from cases to deaths in the whole population. | +| 2 | Date by onset and outcome help identify the high proportion of deaths among cases in time. | Dropping `fill` helps identify a higher proportion of deaths in older age groups. | +| 3 | Interval by month help identify a continuous growth and decay pattern. | Using `fill` with age category helps compare the higher proportion of cases in male compared to female. | #### Interpretation @@ -359,13 +351,16 @@ Cleaning - In small data frames, we can diagnose cleaning operations easier than large data frames. - For example, in the large data frame, before cleaning the sex variable with a data dictionary, we need to remove unconsistent missing values. We can use `dplyr::count()` to find this issue. -Validation +Epicurve -- Using the `linelist::tags_df()` output can keep stable downstream analysis. Jointly with `linelist::lost_tags_action(action = "error")` we can improve the capacity to diagnose changes in the input data. This can prevent getting misleading outputs from automatic daily code runs or dashboards updates. +- COVID: Onset cases peak sharply around 2023-W03, dominated by the under-20 age group. Outcome dates are sparse and spread later, reflecting the delay between onset and death. +- Ebola: Across all age groups, symptom onset dates peaks almost as high as outcome dates (2023-W03 to W05). Possibly with a shorter delay from onset to death than the COVID. +- Unknown: Males had substantially higher case counts than females across all age groups, with both sexes peaking around mid-2023. Older age groups (35-60 and 60-100) make up a larger share in males, while females show a more even age distribution throughout. -Epicurve +Delays -- The argument `show_cases` can improve the visibility of `fill` categorical variables when the amount of observed cases is small. +- If the mean exceeds the median, the delay distribution is right-skewed, meaning most samples are collected relatively soon after exposure, but a small number of cases have unusually long delays that pull the average upward. +- Recent cases are likely undercounted. If there is a lag between an event occurring and it being recorded, the most recent time periods will appear artificially low simply because many cases have not been logged yet. This can make it look like there is a recent decline when it is really just a reporting delay (also known as a right-censoring effect). # Continue your learning path diff --git a/instructors/02-practical-tutors.qmd b/instructors/02-practical-tutors.qmd index c239e483..234e882e 100644 --- a/instructors/02-practical-tutors.qmd +++ b/instructors/02-practical-tutors.qmd @@ -33,9 +33,7 @@ knitr::opts_chunk$set( This practical is based in the following tutorial episodes: -- - -- - ::: diff --git a/instructors/README.md b/instructors/README.md index 7b29f668..87965c66 100644 --- a/instructors/README.md +++ b/instructors/README.md @@ -4,7 +4,16 @@ The goal of the `##-practical-tutors.qmd` files in this folder is to host comple ## Usage -One Quarto QMD file generates: +### Inputs: + +- `data/`: practical scaffold scripts +- `fig/`: practical solutions + - `-challenge` for tutorial challenge without solution + - `-miscellanea` for extended sessions + +### Outputs: + +In `files/`, one Quarto QMD file generates: - Tutors MD file: for Tutors to use during practical or propose edits. Saved as `files/##-practical-tutors.md`. - Guide DOCX file: for Learners to complete during the practical. Saved as `files/##-practical-guide.docx`. diff --git a/instructors/data/01-practical-activity-1.R b/instructors/data/01-practical-activity-1.R index fef6730f..1e0084c6 100644 --- a/instructors/data/01-practical-activity-1.R +++ b/instructors/data/01-practical-activity-1.R @@ -7,30 +7,30 @@ room_number <- # replace with 1/2/3/4 # Load packages ---------------------------------------------------------- library(cleanepi) -library(linelist) -library(incidence2) library(tidyverse) # Adapt the data dictionary ---------------------------------------------- -# Replace 'variable_name' when you have the information +# Replace '#' when you have the information dat_dictionary <- tibble::tribble( - ~options, ~values, ~grp, ~orders, - "1", "male", "variable_name", 1L, - "2", "female", "variable_name", 2L, - "M", "male", "variable_name", 3L, - "F", "female", "variable_name", 4L, - "m", "male", "variable_name", 5L, - "f", "female", "variable_name", 6L + ~options, ~values, ~grp, ~orders, + "1", "male", "#", 1L, + "2", "female", "#", 2L, + "M", "male", "#", 3L, + "F", "female", "#", 4L, + "m", "male", "#", 5L, + "f", "female", "#", 6L ) dat_dictionary # Read raw data ---------------------------------------------------------- + +# Replace the string with the URL provided (or location to local file) dat_raw <- readr::read_csv( - # + "paste/complete/URL/#" # ) dat_raw @@ -39,35 +39,51 @@ dat_raw # Clean and standardize data --------------------------------------------- # How many cleanepi functions did you use to get clean data? -dat_clean <- dat_raw %>% - cleanepi::# - -dat_clean +# Remove those functions not needed in the cleaning workflow - -# Create time span variable ---------------------------------------------- - -# What time span unit best describes the 'delay' from 'onset' to 'death'? -dat_timespan <- dat_clean %>% - cleanepi::timespan( - # - # - # - span_column_name = "timespan_variable", - span_remainder_unit = NULL +dat_clean <- dat_raw %>% + cleanepi::standardize_column_names() %>% + cleanepi::standardize_dates( + target_columns = c( + #, + #, + # + ) + ) %>% + cleanepi::check_date_sequence( + target_columns = c( + #, + #, + # + ) + ) %>% + cleanepi::convert_to_numeric( + target_columns = # + ) %>% + # using data_dictionary requires valid missing entries + cleanepi::replace_missing_values( + target_columns = #, + na_strings = # ) %>% - # skimr::skim(timespan_variable) - # Categorize the delay numerical variable - dplyr::mutate( - timespan_category = base::cut( - x = timespan_variable, - breaks = #, - include.lowest = TRUE, - right = FALSE + cleanepi::clean_using_dictionary( + dictionary = # + ) %>% + cleanepi::remove_constants() %>% + cleanepi::remove_duplicates( + target_columns = c( + #, + # ) + ) %>% + cleanepi::timespan( + target_column = #, + end_date = #, + span_unit = #, + span_column_name = #, + span_remainder_unit = # ) -dat_timespan +dat_clean # nolint end \ No newline at end of file diff --git a/instructors/data/01-practical-activity-2.R b/instructors/data/01-practical-activity-2.R index ebb73caf..77a1a269 100644 --- a/instructors/data/01-practical-activity-2.R +++ b/instructors/data/01-practical-activity-2.R @@ -5,31 +5,44 @@ room_number <- # replace with 1/2/3/4 -# Validate linelist ------------------------------------------------------ - -# Activate error message -linelist::lost_tags_action(action = "error") -# linelist::lost_tags_action(action = "warning") - -# Print tag types, names, and data to guide make_linelist -linelist::tags_types() -linelist::tags_names() -dat_timespan - -# Does the categorical variable of interest pass the validation step? -dat_validate <- dat_timespan %>% - # Tag variables - linelist::make_linelist( - #, - # # include one categorical variable - ) %>% - # Validate linelist - linelist::# %>% - # Test safeguard - # dplyr::select(case_id, date_onset, sex) - # INSTEAD - linelist::tags_df() +# Load packages ---------------------------------------------------------- +library(cleanepi) +library(incidence2) +library(tidyverse) +# Read raw data ---------------------------------------------------------- + +# Replace the string with the URL provided (or location to local file) +dat_linelist <- readr::read_rds( + "paste/complete/URL/#" # + ) + +dat_linelist %>% dplyr::glimpse() + +# Describe delays -------------------------------------------------------- + +# Run and describe +dat_delays <- dat_linelist %>% + cleanepi::timespan( + target_column = "date_onset", + end_date = "date_reporting", + span_unit = "days", + span_column_name = "delay_reporting" + ) + +# Run and describe +dat_delays %>% + dplyr::select(id, date_onset, date_reporting, delay_reporting) + +# Run and describe +dat_delays %>% + skimr::skim(delay_reporting) + +# Run and describe +dat_delays %>% + ggplot(aes(delay_reporting)) + + geom_histogram(binwidth = 1) + + xlim(0,30) # Create incidence ------------------------------------------------------- @@ -46,11 +59,13 @@ dat_incidence <- dat_validate %>% # Plot epicurve ---------------------------------------------------------- -# Do arguments like 'fill', 'show_cases', 'angle', 'n_breaks' improve the plot? +# Do arguments like 'fill', 'nrow', 'show_cases', 'angle', 'n_breaks' +# improve the plot? dat_incidence %>% plot( fill = #, # the categorical variable # - show_cases = TRUE, # + #nrow = 1, # 1 or 2 + show_cases = FALSE, # angle = 45, # n_breaks = 5 # ) diff --git a/instructors/fig/01-practical-instructor-1-G1.R b/instructors/fig/01-practical-instructor-1-G1.R index 9d02cbe2..39ca7ec9 100644 --- a/instructors/fig/01-practical-instructor-1-G1.R +++ b/instructors/fig/01-practical-instructor-1-G1.R @@ -7,7 +7,6 @@ room_number <- 1 # Load packages ---------------------------------------------------------- library(cleanepi) -library(linelist) library(incidence2) library(tidyverse) @@ -37,57 +36,43 @@ dat_raw # How many cleanepi functions did you use to get clean data? dat_clean <- dat_raw %>% cleanepi::standardize_column_names() %>% - cleanepi::standardize_dates( - target_columns = c( - "date_of_admission", - "date_of_birth", - "date_first_pcr_positive_test" - ) - ) %>% - cleanepi::check_date_sequence( - target_columns = c( - "date_of_birth", - "date_first_pcr_positive_test", - "date_of_admission" - ) - ) %>% - # using data_dictionary requires valid missing entries - cleanepi::replace_missing_values( - target_columns = "sex_fem_2", - na_strings = "-99" - ) %>% - cleanepi::clean_using_dictionary(dictionary = dat_dictionary) %>% - cleanepi::remove_constants() %>% - cleanepi::remove_duplicates( - target_columns = c("study_id", "date_of_birth") + cleanepi::standardize_dates( + target_columns = c( + "date_of_admission", + "date_of_birth", + "date_first_pcr_positive_test" ) - -dat_clean - - -# Create time span variable ---------------------------------------------- - -# What time span unit best describes the 'delay' from 'onset' to 'death'? -dat_timespan <- dat_clean %>% + ) %>% + cleanepi::check_date_sequence( + target_columns = c( + "date_of_birth", + "date_first_pcr_positive_test", + "date_of_admission" + ) + ) %>% + # using data_dictionary requires valid missing entries + cleanepi::replace_missing_values( + target_columns = "sex_fem_2", + na_strings = "-99" + ) %>% + cleanepi::clean_using_dictionary( + dictionary = dat_dictionary + ) %>% + cleanepi::remove_constants() %>% + cleanepi::remove_duplicates( + target_columns = c( + "study_id", + "date_of_birth" + ) + ) %>% cleanepi::timespan( target_column = "date_of_birth", end_date = Sys.Date(), span_unit = "years", span_column_name = "timespan_variable", span_remainder_unit = "months" - ) %>% - # skimr::skim(timespan_variable) - # Categorize the delay numerical variable - dplyr::mutate( - timespan_category = base::cut( - x = timespan_variable, - breaks = c(0, 20, 35, 60, 80), - include.lowest = TRUE, - right = FALSE - ) ) -dat_timespan - +dat_clean # nolint end \ No newline at end of file diff --git a/instructors/fig/01-practical-instructor-1-G2.R b/instructors/fig/01-practical-instructor-1-G2.R index e6ffa670..875a0af5 100644 --- a/instructors/fig/01-practical-instructor-1-G2.R +++ b/instructors/fig/01-practical-instructor-1-G2.R @@ -7,7 +7,6 @@ room_number <- 2 # Load packages ---------------------------------------------------------- library(cleanepi) -library(linelist) library(incidence2) library(tidyverse) @@ -59,45 +58,27 @@ dat_clean <- dat_raw %>% "date_outcome" ) ) %>% - cleanepi::convert_to_numeric(target_columns = "age") %>% + cleanepi::convert_to_numeric( + target_columns = "age" + ) %>% # dplyr::count(sex) # using data_dictionary requires valid missing entries cleanepi::replace_missing_values( target_columns = "sex", na_strings = "-99" ) %>% - cleanepi::clean_using_dictionary(dictionary = dat_dictionary) %>% + cleanepi::clean_using_dictionary( + dictionary = dat_dictionary + ) %>% cleanepi::remove_constants() %>% cleanepi::remove_duplicates( - target_columns = c("case_id", "case_name") + target_columns = c( + "case_id", + "case_name" + ) ) dat_clean -# Create time span variable ---------------------------------------------- - -# What time span unit best describes the 'delay' from 'onset' to 'death'? -dat_timespan <- dat_clean %>% - cleanepi::timespan( - target_column = "date_onset", - end_date = "date_outcome", - span_unit = "days", - span_column_name = "timespan_variable", - span_remainder_unit = NULL - ) %>% - # skimr::skim(timespan_variable) - # Categorize the delay numerical variable - dplyr::mutate( - timespan_category = base::cut( - x = timespan_variable, - breaks = c(0, 10, 15, 40), - include.lowest = TRUE, - right = FALSE - ) - ) - -dat_timespan - - # nolint end \ No newline at end of file diff --git a/instructors/fig/01-practical-instructor-1-G3.R b/instructors/fig/01-practical-instructor-1-G3.R index 0b08d15a..62b7ce55 100644 --- a/instructors/fig/01-practical-instructor-1-G3.R +++ b/instructors/fig/01-practical-instructor-1-G3.R @@ -7,7 +7,6 @@ room_number <- 3 # Load packages ---------------------------------------------------------- library(cleanepi) -library(linelist) library(incidence2) library(tidyverse) @@ -41,38 +40,20 @@ dat_raw # How many cleanepi functions did you use to get clean data? dat_clean <- dat_raw %>% cleanepi::standardize_column_names() %>% - cleanepi::standardize_dates(target_columns = "date") %>% # - cleanepi::convert_to_numeric(target_columns = "exp_num") %>% + cleanepi::standardize_dates( + target_columns = "date" + ) %>% # + cleanepi::convert_to_numeric( + target_columns = "exp_num" + ) %>% cleanepi::check_date_sequence( - target_columns = c("last_exp_date", "date") + target_columns = c( + "last_exp_date", + "date" + ) ) dat_clean -# Create time span variable ---------------------------------------------- - -# What time span unit best describes the 'delay' from 'onset' to 'death'? -dat_timespan <- dat_clean %>% - cleanepi::timespan( - target_column = "last_exp_date", - end_date = "date", - span_unit = "days", - span_column_name = "timespan_variable", - span_remainder_unit = NULL - ) %>% - # skimr::skim(timespan_variable) - # Categorize the delay numerical variable - dplyr::mutate( - timespan_category = base::cut( - x = timespan_variable, - breaks = c(0, 30, 100, 600), - include.lowest = TRUE, - right = FALSE - ) - ) - -dat_timespan - - # nolint end \ No newline at end of file diff --git a/instructors/fig/01-practical-instructor-2-G1.R b/instructors/fig/01-practical-instructor-2-G1.R index 3411c0db..3f905d65 100644 --- a/instructors/fig/01-practical-instructor-2-G1.R +++ b/instructors/fig/01-practical-instructor-2-G1.R @@ -5,51 +5,45 @@ room_number <- 1 -# Validate linelist ------------------------------------------------------ - -# Activate error message -linelist::lost_tags_action(action = "error") -# linelist::lost_tags_action(action = "warning") - -# Print tag types, names, and data to guide make_linelist -linelist::tags_types() -linelist::tags_names() -dat_timespan - -# Does the categorical variable of interest pass the validation step? -dat_validate <- dat_timespan %>% - # Tag variables - linelist::make_linelist( - id = "study_id", - date_reporting = "date_first_pcr_positive_test", - gender = "sex_fem_2", - age = "timespan_variable", - allow_extra = TRUE, - age_category = "timespan_category" - ) %>% - # Validate linelist - linelist::validate_linelist( - allow_extra = TRUE, - ref_types = linelist::tags_types( - age_category = c("factor"), - allow_extra = TRUE - ) - ) %>% - # Test safeguard - # dplyr::select(case_id, date_onset, sex) - # INSTEAD - linelist::tags_df() +# Load packages ---------------------------------------------------------- +library(cleanepi) +library(incidence2) +library(tidyverse) +# Read raw data ---------------------------------------------------------- +dat_linelist <- readr::read_rds( + "https://epiverse-trace.github.io/tutorials-early/data/covid_simulist.rds" + ) + +dat_linelist %>% dplyr::glimpse() + +# Describe delays -------------------------------------------------------- + +dat_delays <- dat_linelist %>% + cleanepi::timespan( + target_column = "date_onset", + end_date = "date_reporting", + span_unit = "days", + span_column_name = "delay_reporting" + ) + +dat_delays %>% + skimr::skim(delay_reporting) + +dat_delays %>% + ggplot(aes(delay_reporting)) + + geom_histogram(binwidth = 1) + + xlim(0,30) # Create incidence ------------------------------------------------------- # What is the most appropriate time-aggregate (days, months) to plot? -dat_incidence <- dat_validate %>% +dat_incidence <- dat_linelist %>% # Transform from individual-level to time-aggregate - incidence2::incidence( - date_index = "date_reporting", - groups = "age_category", # the categorical variable - interval = "month", + incidence2::incidence_( + date_index = c(date_onset, date_outcome), + groups = age_category, # the categorical variable + interval = "week", complete_dates = TRUE ) @@ -60,7 +54,8 @@ dat_incidence <- dat_validate %>% dat_incidence %>% plot( fill = "age_category", # the categorical variable - show_cases = TRUE, # + #nrow = 1, # 1 or 2 + show_cases = FALSE, # angle = 45, # n_breaks = 5 # ) diff --git a/instructors/fig/01-practical-instructor-2-G2.R b/instructors/fig/01-practical-instructor-2-G2.R index 1a0ae496..69e8abfc 100644 --- a/instructors/fig/01-practical-instructor-2-G2.R +++ b/instructors/fig/01-practical-instructor-2-G2.R @@ -5,44 +5,45 @@ room_number <- 2 -# Validate linelist ------------------------------------------------------ - -# Activate error message -linelist::lost_tags_action(action = "error") -# linelist::lost_tags_action(action = "warning") - -# Print tag types, names, and data to guide make_linelist -linelist::tags_types() -linelist::tags_names() -dat_timespan - -# Does the categorical variable of interest pass the validation step? -dat_validate <- dat_timespan %>% - # Tag variables - linelist::make_linelist( - id = "case_id", - date_onset = "date_onset", - gender = "sex", - age = "age", - outcome = "outcome" - ) %>% - # Validate linelist - linelist::validate_linelist() %>% - # Test safeguard - # dplyr::select(case_id, date_onset, sex) - # INSTEAD - linelist::tags_df() +# Load packages ---------------------------------------------------------- +library(cleanepi) +library(incidence2) +library(tidyverse) +# Read raw data ---------------------------------------------------------- +dat_linelist <- readr::read_rds( + "https://epiverse-trace.github.io/tutorials-early/data/ebola_simulist.rds" + ) + +dat_linelist %>% dplyr::glimpse() + +# Describe delays -------------------------------------------------------- + +dat_delays <- dat_linelist %>% + cleanepi::timespan( + target_column = "date_onset", + end_date = "date_reporting", + span_unit = "days", + span_column_name = "delay_reporting" + ) + +dat_delays %>% + skimr::skim(delay_reporting) + +dat_delays %>% + ggplot(aes(delay_reporting)) + + geom_histogram(binwidth = 1) + + xlim(0,30) # Create incidence ------------------------------------------------------- # What is the most appropriate time-aggregate (days, months) to plot? -dat_incidence <- dat_validate %>% +dat_incidence <- dat_linelist %>% # Transform from individual-level to time-aggregate - incidence2::incidence( - date_index = "date_onset", - groups = "outcome", # the categorical variable - interval = "day", + incidence2::incidence_( + date_index = c(date_onset, date_outcome), + groups = age_category, # the categorical variable + interval = "week", complete_dates = TRUE ) @@ -52,8 +53,11 @@ dat_incidence <- dat_validate %>% # Do arguments like 'fill', 'show_cases', 'angle', 'n_breaks' improve the plot? dat_incidence %>% plot( - angle = 45, # - n_breaks = 5 # + # fill = "age_category", # the categorical variable + #nrow = 1, # 1 or 2 + show_cases = FALSE, # + angle = 90, # + n_breaks = 3 # ) # Find plot() arguments at ?incidence2:::plot.incidence2() diff --git a/instructors/fig/01-practical-instructor-2-G3.R b/instructors/fig/01-practical-instructor-2-G3.R index 0fe293b8..d837409a 100644 --- a/instructors/fig/01-practical-instructor-2-G3.R +++ b/instructors/fig/01-practical-instructor-2-G3.R @@ -5,49 +5,44 @@ room_number <- 3 -# Validate linelist ------------------------------------------------------ - -# Activate error message -linelist::lost_tags_action(action = "error") -# linelist::lost_tags_action(action = "warning") - -# Print tag types, names, and data to guide make_linelist -linelist::tags_types() -linelist::tags_names() -dat_timespan - -# Does the categorical variable of interest pass the validation step? -dat_validate <- dat_timespan %>% - # Tag variables - linelist::make_linelist( - id = "pid", - allow_extra = TRUE, - last_exp_date = "last_exp_date", - last_vax_type = "last_vax_type" - ) %>% - # Validate linelist - linelist::validate_linelist( - allow_extra = TRUE, - ref_types = linelist::tags_types( - last_exp_date = c("Date"), - last_vax_type = c("character"), - allow_extra = TRUE - ) - ) %>% - # Test safeguard - # dplyr::select(case_id, date_onset, sex) - # INSTEAD - linelist::tags_df() +# Load packages ---------------------------------------------------------- +library(cleanepi) +library(incidence2) +library(tidyverse) +# Read raw data ---------------------------------------------------------- +dat_linelist <- readr::read_rds( + "https://epiverse-trace.github.io/tutorials-early/data/unknown_simulist.rds" + ) + +dat_linelist %>% dplyr::glimpse() + +# Describe delays -------------------------------------------------------- + +dat_delays <- dat_linelist %>% + cleanepi::timespan( + target_column = "date_onset", + end_date = "date_reporting", + span_unit = "days", + span_column_name = "delay_reporting" + ) + +dat_delays %>% + skimr::skim(delay_reporting) + +dat_delays %>% + ggplot(aes(delay_reporting)) + + geom_histogram(binwidth = 1) + + xlim(0,30) # Create incidence ------------------------------------------------------- # What is the most appropriate time-aggregate (days, months) to plot? -dat_incidence <- dat_validate %>% +dat_incidence <- dat_linelist %>% # Transform from individual-level to time-aggregate - incidence2::incidence( - date_index = "last_exp_date", - groups = "last_vax_type", # the categorical variable + incidence2::incidence_( + date_index = date_onset, + groups = c(sex, age_category), # the categorical variable interval = "month", complete_dates = TRUE ) @@ -58,7 +53,11 @@ dat_incidence <- dat_validate %>% # Do arguments like 'fill', 'show_cases', 'angle', 'n_breaks' improve the plot? dat_incidence %>% plot( - fill = "last_vax_type" # the categorical variable # + fill = "age_category", # the categorical variable + nrow = 1, # 1 or 2 + show_cases = FALSE, # + angle = 45, # + n_breaks = 5 # ) # Find plot() arguments at ?incidence2:::plot.incidence2() diff --git a/instructors/files/01-practical-tutors.md b/instructors/files/01-practical-tutors.md index ffa71842..f63a08dc 100644 --- a/instructors/files/01-practical-tutors.md +++ b/instructors/files/01-practical-tutors.md @@ -14,7 +14,6 @@ This practical is based in the following tutorial episodes: - -- - # Practical @@ -25,102 +24,84 @@ This practical has two activities. **Goal:** -Get a clean and standardized data frame using the following available -inputs: +Produce a clean and standardized data frame from the following input: -- Raw messy data frame in CSV format +- Raw, messy CSV file **Steps:** -- Open file `01-practical-activity-1.R` and complete all the lines - marked with `#`, following the detailed steps provided - within the R file. -- First, complete the argument to read the data. Paste the link as a - “string” in `read_csv()`. -- Second, complete the cleaning process. Add functions based on the data - needs. Connect them using the pipe `%>%`: - - Standardize column names - - Standardize dates - - Check date sequence - - Convert to numeric - - Replace missing values - - Clean using dictionary - - Remove constants - - Remove duplicates -- Third, complete the cleanepi::timespan() arguments. Access the help - manual running `?cleanepi::timespan()` in the console. -- Paste the outputs. Reply to questions. +1. Open `01-practical-activity-1.R` and complete every line marked with + `#`, following the instructions in the file. +2. Complete the argument in `read_csv()` by pasting the data URL link + as a string. +3. Complete the arguments of cleaning function. Remove functions based + on the data needs. +4. Paste screenshots of the data frame before and after cleaning. + Answer the questions below. **Questions:** -Within your room, Write your answers to these questions: +1. **Diagnose the raw data.** What cleaning operations are needed? + Which ones diagnose issues or perform cleaning actions? +2. **Time unit.** What time unit best describes the time span you need + to calculate? +3. **Report.** Print the report. Which features are most useful for you + as an analyst to understand or communicate the issues in the data + frame? -- Diagnose the raw data. What data cleaning operations need to be - performed on the dataset? Write all of them before writing the code. -- What time unit best describes the time span to calculate? -- Print the report: What features do you find useful to communicate with - a decision-maker? -- Compare: What differences do you identify from other room outputs? (if - available) +Discuss your answers with your group before sharing with the wider room. ### Inputs -| Room | Data | Link | Calculate time span | Categorize time span | -|----|----|----|----|----| -| 1 | Small linelist | | Age as of today | breaks = c(0, 20, 35, 60, 80) | -| 2 | Large linelist | | Delay from onset of symptoms to the time of death | breaks = c(0, 10, 15, 40) | -| 3 | Serology data [^1] | | Time from last exposure to vaccine | breaks = c(0, 30, 100, 600) | +| Room | Data | Link | Calculate time span | +|----|----|----|----| +| 1 | Small linelist | | Age (as of today) | +| 2 | Large linelist | | Time from symptom onset to death | +| 3 | Serology data [^1] | | Time from last vaccine dose to sample collection [^2] | -## Activity 2: Validate linelist and plot epicurve +## Activity 2: Plot delays and epicurves **Goal:** -Get a validated linelist and incidence plot using the following -available inputs: +Using a clean linelist data frame, produce: -- Clean data frame object +- A delay distribution plot showing time between two epidemiological + events. +- An incidence plot (epicurve) showing case counts over time. **Steps:** -- Open the file `01-practical-activity-2.R` and complete all the lines - marked with `#`, following the detailed steps provided - within the R file. -- First, complete linelist::make_linelist() arguments. -- Second, complete the {linelist} function that can validate a linelist. -- Third, complete the arguments of the incidence2::incidence() -- Fourth, keep, drop, or change argument values in function plot() -- Paste the outputs. Reply to questions. +1. Open `01-practical-activity-2.R` and complete every line marked with + `#`, following the instructions in the file. +2. Describe the pre-configured epidemiological delay. +3. Complete the arguments of `incidence2::incidence()` to generate an + incidence object. +4. Adjust the arguments of `plot()` to get the most informative + epicurve. Read the [`plot()` reference + manual](https://www.reconverse.org/incidence2/manual.html#sec:man-plot.incidence2) + to find available arguments. +5. Paste your plots and reply to the questions below. **Questions:** -Within your room, Write your answers to these questions: - -- In the validation step, Do you need to allow for extra variable names - and types for the `Date` and `Categorical` variable? - - *[Read this GitHub issue as a - hint](https://github.com/epiverse-trace/linelist/issues/176) to - allow for extra variables.* -- What is the most apprioriate time unit to aggregate the incidence - plot, based on visual inspection? -- Does keeping or dropping arguments like `fill`, `show_cases`, `angle`, - `n_breaks` improve the incidence plot? - - *[Read `plot()` reference - manual](https://www.reconverse.org/incidence2/manual.html#sec:man-plot.incidence2) - to find its arguments.* -- Interpret: How would you communicate these results to a - decision-maker? -- Compare: What differences do you identify from other room outputs? (if - available) +- Which combination of time unit and case categories best captures the + outbreak pattern and why? +- What does the shape of your epicurve suggest about how this outbreak + spread? +- Which is larger in your delay distribution: the mean or the median, + and what does that tell you about its shape? +- How might delays in the data collection process affect your + interpretation of the most recent cases? -### Inputs +Discuss your answers with your group before sharing with the wider room. -Use outputs from activity 1. +### Inputs -| Room | Date | Categorical variable | -|------|--------------------|----------------------| -| 1 | Date reporting | Age category | -| 2 | Date onset | Outcome | -| 3 | Last exposure date | Last vaccine type | +| Room | Disease | Link | +|----|----|----| +| 1 | COVID | | +| 2 | Ebola | | +| 3 | Unknown | | ### Solution @@ -142,7 +123,6 @@ room_number <- 1 # Load packages ---------------------------------------------------------- library(cleanepi) -library(linelist) library(incidence2) library(tidyverse) @@ -172,58 +152,44 @@ dat_raw # How many cleanepi functions did you use to get clean data? dat_clean <- dat_raw %>% cleanepi::standardize_column_names() %>% - cleanepi::standardize_dates( - target_columns = c( - "date_of_admission", - "date_of_birth", - "date_first_pcr_positive_test" - ) - ) %>% - cleanepi::check_date_sequence( - target_columns = c( - "date_of_birth", - "date_first_pcr_positive_test", - "date_of_admission" - ) - ) %>% - # using data_dictionary requires valid missing entries - cleanepi::replace_missing_values( - target_columns = "sex_fem_2", - na_strings = "-99" - ) %>% - cleanepi::clean_using_dictionary(dictionary = dat_dictionary) %>% - cleanepi::remove_constants() %>% - cleanepi::remove_duplicates( - target_columns = c("study_id", "date_of_birth") + cleanepi::standardize_dates( + target_columns = c( + "date_of_admission", + "date_of_birth", + "date_first_pcr_positive_test" ) - -dat_clean - - -# Create time span variable ---------------------------------------------- - -# What time span unit best describes the 'delay' from 'onset' to 'death'? -dat_timespan <- dat_clean %>% + ) %>% + cleanepi::check_date_sequence( + target_columns = c( + "date_of_birth", + "date_first_pcr_positive_test", + "date_of_admission" + ) + ) %>% + # using data_dictionary requires valid missing entries + cleanepi::replace_missing_values( + target_columns = "sex_fem_2", + na_strings = "-99" + ) %>% + cleanepi::clean_using_dictionary( + dictionary = dat_dictionary + ) %>% + cleanepi::remove_constants() %>% + cleanepi::remove_duplicates( + target_columns = c( + "study_id", + "date_of_birth" + ) + ) %>% cleanepi::timespan( target_column = "date_of_birth", end_date = Sys.Date(), span_unit = "years", span_column_name = "timespan_variable", span_remainder_unit = "months" - ) %>% - # skimr::skim(timespan_variable) - # Categorize the delay numerical variable - dplyr::mutate( - timespan_category = base::cut( - x = timespan_variable, - breaks = c(0, 20, 35, 60, 80), - include.lowest = TRUE, - right = FALSE - ) ) -dat_timespan - +dat_clean # nolint end ``` @@ -238,51 +204,45 @@ activity 2 room_number <- 1 -# Validate linelist ------------------------------------------------------ - -# Activate error message -linelist::lost_tags_action(action = "error") -# linelist::lost_tags_action(action = "warning") - -# Print tag types, names, and data to guide make_linelist -linelist::tags_types() -linelist::tags_names() -dat_timespan - -# Does the categorical variable of interest pass the validation step? -dat_validate <- dat_timespan %>% - # Tag variables - linelist::make_linelist( - id = "study_id", - date_reporting = "date_first_pcr_positive_test", - gender = "sex_fem_2", - age = "timespan_variable", - allow_extra = TRUE, - age_category = "timespan_category" - ) %>% - # Validate linelist - linelist::validate_linelist( - allow_extra = TRUE, - ref_types = linelist::tags_types( - age_category = c("factor"), - allow_extra = TRUE - ) - ) %>% - # Test safeguard - # dplyr::select(case_id, date_onset, sex) - # INSTEAD - linelist::tags_df() +# Load packages ---------------------------------------------------------- +library(cleanepi) +library(incidence2) +library(tidyverse) + +# Read raw data ---------------------------------------------------------- +dat_linelist <- readr::read_rds( + "https://epiverse-trace.github.io/tutorials-early/data/covid_simulist.rds" + ) + +dat_linelist %>% dplyr::glimpse() + +# Describe delays -------------------------------------------------------- +dat_delays <- dat_linelist %>% + cleanepi::timespan( + target_column = "date_onset", + end_date = "date_reporting", + span_unit = "days", + span_column_name = "delay_reporting" + ) + +dat_delays %>% + skimr::skim(delay_reporting) + +dat_delays %>% + ggplot(aes(delay_reporting)) + + geom_histogram(binwidth = 1) + + xlim(0,30) # Create incidence ------------------------------------------------------- # What is the most appropriate time-aggregate (days, months) to plot? -dat_incidence <- dat_validate %>% +dat_incidence <- dat_linelist %>% # Transform from individual-level to time-aggregate - incidence2::incidence( - date_index = "date_reporting", - groups = "age_category", # the categorical variable - interval = "month", + incidence2::incidence_( + date_index = c(date_onset, date_outcome), + groups = age_category, # the categorical variable + interval = "week", complete_dates = TRUE ) @@ -293,7 +253,8 @@ dat_incidence <- dat_validate %>% dat_incidence %>% plot( fill = "age_category", # the categorical variable - show_cases = TRUE, # + #nrow = 1, # 1 or 2 + show_cases = FALSE, # angle = 45, # n_breaks = 5 # ) @@ -317,7 +278,6 @@ room_number <- 2 # Load packages ---------------------------------------------------------- library(cleanepi) -library(linelist) library(incidence2) library(tidyverse) @@ -369,47 +329,29 @@ dat_clean <- dat_raw %>% "date_outcome" ) ) %>% - cleanepi::convert_to_numeric(target_columns = "age") %>% + cleanepi::convert_to_numeric( + target_columns = "age" + ) %>% # dplyr::count(sex) # using data_dictionary requires valid missing entries cleanepi::replace_missing_values( target_columns = "sex", na_strings = "-99" ) %>% - cleanepi::clean_using_dictionary(dictionary = dat_dictionary) %>% + cleanepi::clean_using_dictionary( + dictionary = dat_dictionary + ) %>% cleanepi::remove_constants() %>% cleanepi::remove_duplicates( - target_columns = c("case_id", "case_name") + target_columns = c( + "case_id", + "case_name" + ) ) dat_clean -# Create time span variable ---------------------------------------------- - -# What time span unit best describes the 'delay' from 'onset' to 'death'? -dat_timespan <- dat_clean %>% - cleanepi::timespan( - target_column = "date_onset", - end_date = "date_outcome", - span_unit = "days", - span_column_name = "timespan_variable", - span_remainder_unit = NULL - ) %>% - # skimr::skim(timespan_variable) - # Categorize the delay numerical variable - dplyr::mutate( - timespan_category = base::cut( - x = timespan_variable, - breaks = c(0, 10, 15, 40), - include.lowest = TRUE, - right = FALSE - ) - ) - -dat_timespan - - # nolint end ``` @@ -423,44 +365,45 @@ activity 2 room_number <- 2 -# Validate linelist ------------------------------------------------------ - -# Activate error message -linelist::lost_tags_action(action = "error") -# linelist::lost_tags_action(action = "warning") - -# Print tag types, names, and data to guide make_linelist -linelist::tags_types() -linelist::tags_names() -dat_timespan - -# Does the categorical variable of interest pass the validation step? -dat_validate <- dat_timespan %>% - # Tag variables - linelist::make_linelist( - id = "case_id", - date_onset = "date_onset", - gender = "sex", - age = "age", - outcome = "outcome" - ) %>% - # Validate linelist - linelist::validate_linelist() %>% - # Test safeguard - # dplyr::select(case_id, date_onset, sex) - # INSTEAD - linelist::tags_df() +# Load packages ---------------------------------------------------------- +library(cleanepi) +library(incidence2) +library(tidyverse) + +# Read raw data ---------------------------------------------------------- +dat_linelist <- readr::read_rds( + "https://epiverse-trace.github.io/tutorials-early/data/ebola_simulist.rds" + ) + +dat_linelist %>% dplyr::glimpse() + +# Describe delays -------------------------------------------------------- +dat_delays <- dat_linelist %>% + cleanepi::timespan( + target_column = "date_onset", + end_date = "date_reporting", + span_unit = "days", + span_column_name = "delay_reporting" + ) + +dat_delays %>% + skimr::skim(delay_reporting) + +dat_delays %>% + ggplot(aes(delay_reporting)) + + geom_histogram(binwidth = 1) + + xlim(0,30) # Create incidence ------------------------------------------------------- # What is the most appropriate time-aggregate (days, months) to plot? -dat_incidence <- dat_validate %>% +dat_incidence <- dat_linelist %>% # Transform from individual-level to time-aggregate - incidence2::incidence( - date_index = "date_onset", - groups = "outcome", # the categorical variable - interval = "day", + incidence2::incidence_( + date_index = c(date_onset, date_outcome), + groups = age_category, # the categorical variable + interval = "week", complete_dates = TRUE ) @@ -470,8 +413,11 @@ dat_incidence <- dat_validate %>% # Do arguments like 'fill', 'show_cases', 'angle', 'n_breaks' improve the plot? dat_incidence %>% plot( - angle = 45, # - n_breaks = 5 # + # fill = "age_category", # the categorical variable + #nrow = 1, # 1 or 2 + show_cases = FALSE, # + angle = 90, # + n_breaks = 3 # ) # Find plot() arguments at ?incidence2:::plot.incidence2() @@ -493,7 +439,6 @@ room_number <- 3 # Load packages ---------------------------------------------------------- library(cleanepi) -library(linelist) library(incidence2) library(tidyverse) @@ -527,40 +472,22 @@ dat_raw # How many cleanepi functions did you use to get clean data? dat_clean <- dat_raw %>% cleanepi::standardize_column_names() %>% - cleanepi::standardize_dates(target_columns = "date") %>% # - cleanepi::convert_to_numeric(target_columns = "exp_num") %>% + cleanepi::standardize_dates( + target_columns = "date" + ) %>% # + cleanepi::convert_to_numeric( + target_columns = "exp_num" + ) %>% cleanepi::check_date_sequence( - target_columns = c("last_exp_date", "date") + target_columns = c( + "last_exp_date", + "date" + ) ) dat_clean -# Create time span variable ---------------------------------------------- - -# What time span unit best describes the 'delay' from 'onset' to 'death'? -dat_timespan <- dat_clean %>% - cleanepi::timespan( - target_column = "last_exp_date", - end_date = "date", - span_unit = "days", - span_column_name = "timespan_variable", - span_remainder_unit = NULL - ) %>% - # skimr::skim(timespan_variable) - # Categorize the delay numerical variable - dplyr::mutate( - timespan_category = base::cut( - x = timespan_variable, - breaks = c(0, 30, 100, 600), - include.lowest = TRUE, - right = FALSE - ) - ) - -dat_timespan - - # nolint end ``` @@ -574,49 +501,44 @@ activity 2 room_number <- 3 -# Validate linelist ------------------------------------------------------ - -# Activate error message -linelist::lost_tags_action(action = "error") -# linelist::lost_tags_action(action = "warning") - -# Print tag types, names, and data to guide make_linelist -linelist::tags_types() -linelist::tags_names() -dat_timespan - -# Does the categorical variable of interest pass the validation step? -dat_validate <- dat_timespan %>% - # Tag variables - linelist::make_linelist( - id = "pid", - allow_extra = TRUE, - last_exp_date = "last_exp_date", - last_vax_type = "last_vax_type" - ) %>% - # Validate linelist - linelist::validate_linelist( - allow_extra = TRUE, - ref_types = linelist::tags_types( - last_exp_date = c("Date"), - last_vax_type = c("character"), - allow_extra = TRUE - ) - ) %>% - # Test safeguard - # dplyr::select(case_id, date_onset, sex) - # INSTEAD - linelist::tags_df() +# Load packages ---------------------------------------------------------- +library(cleanepi) +library(incidence2) +library(tidyverse) + +# Read raw data ---------------------------------------------------------- +dat_linelist <- readr::read_rds( + "https://epiverse-trace.github.io/tutorials-early/data/unknown_simulist.rds" + ) +dat_linelist %>% dplyr::glimpse() + +# Describe delays -------------------------------------------------------- + +dat_delays <- dat_linelist %>% + cleanepi::timespan( + target_column = "date_onset", + end_date = "date_reporting", + span_unit = "days", + span_column_name = "delay_reporting" + ) + +dat_delays %>% + skimr::skim(delay_reporting) + +dat_delays %>% + ggplot(aes(delay_reporting)) + + geom_histogram(binwidth = 1) + + xlim(0,30) # Create incidence ------------------------------------------------------- # What is the most appropriate time-aggregate (days, months) to plot? -dat_incidence <- dat_validate %>% +dat_incidence <- dat_linelist %>% # Transform from individual-level to time-aggregate - incidence2::incidence( - date_index = "last_exp_date", - groups = "last_vax_type", # the categorical variable + incidence2::incidence_( + date_index = date_onset, + groups = c(sex, age_category), # the categorical variable interval = "month", complete_dates = TRUE ) @@ -627,7 +549,11 @@ dat_incidence <- dat_validate %>% # Do arguments like 'fill', 'show_cases', 'angle', 'n_breaks' improve the plot? dat_incidence %>% plot( - fill = "last_vax_type" # the categorical variable # + fill = "age_category", # the categorical variable + nrow = 1, # 1 or 2 + show_cases = FALSE, # + angle = 45, # + n_breaks = 5 # ) # Find plot() arguments at ?incidence2:::plot.incidence2() @@ -647,17 +573,17 @@ dat_incidence %>% ##### activity 2 -| Room | Output | -|------|-----------------------------------------------------| -| 1 | ![image](https://hackmd.io/_uploads/ry5d6xnA1e.png) | -| 2 | ![image](https://hackmd.io/_uploads/SJ2f0e2Cyx.png) | -| 3 | ![image](https://hackmd.io/_uploads/H1-PRlhA1g.png) | +| Group | Delay | Incidence | +|----|----|----| +| 1 | ![image](https://hackmd.io/_uploads/SkIzvCrMMe.png) | ![image](https://hackmd.io/_uploads/rkIuVRBGMe.png) | +| 2 | ![image](https://hackmd.io/_uploads/rys4wCHGfl.png) | ![image](https://hackmd.io/_uploads/H1F-FAHGfe.png) | +| 3 | ![image](https://hackmd.io/_uploads/BkoSPCBMfg.png) | ![image](https://hackmd.io/_uploads/SJpIICSGzl.png) | -| Room | validation | time unit | incidence plot | -|----|----|----|----| -| 1 | The timespan_category (age categories) must be added as an extra variable. Declare age as “factor”. | An appropriate time unit to aggregated is using interval by month. | The argument `show_cases` can improve the visibility of `fill` categorical variables | -| 2 | It is not required to add an extra variable. | An appropriate time unit to aggregate is using interval by day. | Keeping angle and `n_breaks` works. Dropping `fill` keeps the facets making “died” are move visible. | -| 3 | last_exp_date and last_vax_type must be added as extra variables. Declare last_vax_type as “character”, and last_vax_date as “Date”. | An appropriate time unit to aggregated is using interval by month. | Keeping `fill` by last_vax_type works. | +| Room | Incidence arguments | Plot arguments | +|----|----|----| +| 1 | Date by onset and outcome help identify the low proportion of deaths among cases in time. | Keeping `fill` helps visualizing the delay and peak from cases to deaths in the whole population. | +| 2 | Date by onset and outcome help identify the high proportion of deaths among cases in time. | Dropping `fill` helps identify a higher proportion of deaths in older age groups. | +| 3 | Interval by month help identify a continuous growth and decay pattern. | Using `fill` with age category helps compare the higher proportion of cases in male compared to female. | #### Interpretation @@ -669,18 +595,31 @@ Cleaning with a data dictionary, we need to remove unconsistent missing values. We can use `dplyr::count()` to find this issue. -Validation - -- Using the `linelist::tags_df()` output can keep stable downstream - analysis. Jointly with `linelist::lost_tags_action(action = "error")` - we can improve the capacity to diagnose changes in the input data. - This can prevent getting misleading outputs from automatic daily code - runs or dashboards updates. - Epicurve -- The argument `show_cases` can improve the visibility of `fill` - categorical variables when the amount of observed cases is small. +- COVID: Onset cases peak sharply around 2023-W03, dominated by the + under-20 age group. Outcome dates are sparse and spread later, + reflecting the delay between onset and death. +- Ebola: Across all age groups, symptom onset dates peaks almost as high + as outcome dates (2023-W03 to W05). Possibly with a shorter delay from + onset to death than the COVID. +- Unknown: Males had substantially higher case counts than females + across all age groups, with both sexes peaking around mid-2023. Older + age groups (35-60 and 60-100) make up a larger share in males, while + females show a more even age distribution throughout. + +Delays + +- If the mean exceeds the median, the delay distribution is + right-skewed, meaning most samples are collected relatively soon after + exposure, but a small number of cases have unusually long delays that + pull the average upward. +- Recent cases are likely undercounted. If there is a lag between an + event occurring and it being recorded, the most recent time periods + will appear artificially low simply because many cases have not been + logged yet. This can make it look like there is a recent decline when + it is really just a reporting delay (also known as a right-censoring + effect). # Continue your learning path @@ -695,11 +634,20 @@ Explore the downstream analysis you can do with {incidence2} outputs # end -[^1]: Context of Serological data: Participants of a study are exposed - to COVID-19 vaccines, then their serum samples are collected and - challenged to emerging SARS-CoV-2 variants. They measure the titer - of this immunological response. The higher the titre, the higher the - antigenic response. Let’s focus on describing the change in the - frequency of vaccine categories (last vaccine exposure) through time - (last date of exposure). Ref: - https://www.thelancet.com/journals/laninf/article/PIIS1473-3099(24)00484-5/fulltext +[^1]: **Context of Serological data**: Participants of a study are + vaccinated against COVID-19 in multiple doses. Researchers collect + serum samples some days after each vaccination dose and expose the + serum to emerging SARS-CoV-2 variants. They measure the titre of + this immunological response. The higher the titre, the higher the + antigenic response. Ref: + + +[^2]: **How to read the repeated-measurements cohort dataset**: Subject + `pid = 1` was `Infection naive` (never exposed to infection). Was + exposed 2 times (`exp_num`) to the vaccine type BNT162b2 + (`last_vax_type`). The last vaccine exposure was on 2021-03-08 + (`last_exp_date`). One serum sample was obtained on 2021-03-10 + (`date`, two days after last vaccine exposure). The antibody titre + against `Alpha` and `Delta` SARS-CoV-2 variants (`titre_type`) was 5 + units (`value`). The antibody titre against `Ancestral` SARS-CoV-2 + variants was 176 units.