Processing DynamoDB CSVs

(Page WIP)

You can download your data in CSV format using DynamoDB. However, for some types of data, your data may not appear to be in the correct format. For example, for Movement data (collected using the Tracker system), row contains all data for that trial. We want to expand the data so that 1 row represents one timestep. Additionally, some datapoints are prefixed by the data type by DynamoDB. E.g., an observation of "hello!" may look like {"S": "hello!"} in your CSV. This is encoded as a json format, and we can write code to parse this json string and get out the data we want.

Here is an example for the Tracker data:

library(jsonlite) # need these packages
library(tidyverse) # need these packages

my_data <- read_csv("UXFData.basic_example.Trackers.csv")

my_data_processed <- my_data %>% 
  rowwise() %>% # row by row
  mutate(across(
    !(`ppid_session_dataname (S)`:`trial_num (N)`), # all pos, rot, time columns, NOT first two
    ~ list(as.numeric(jsonlite::fromJSON(.)[[1]])) # extract 1st element, which is the data vector, and convert to numeric
  )) %>% 
  ungroup() %>%
  unnest(`pos_x (L)`:`time (L)`)
  
write_csv(my_data_processed, "UXFData.basic_example.Trackers_processed.csv")