# Symposium: Challenges of data-driven approaches to multimodal data integration for daily human activity

### SAA Conf. 2021 (30/6-3/7): Capturing human development accross daily life contexts.
**Conf. website:** https://saa.dynage.uzh.ch

**Attendees:**
- Iris Yocarini (LIACS)
- Daniela Gawehns (LIACS)
- Stylianos Paraschiakos (LUMC)
- Thomas Papastergiou, V. Megalooikonomou (University of Patra)
- Lisa-Maria van Klaveren and Meadeh Nasri (Dept. Psychology, Univ. Leiden)

**Google sheets link for details:** https://docs.google.com/spreadsheets/d/1-cRJonrj4CsebyozRyrToPQgCK9rG2wtEIYG7izSDIw/edit?usp=sharing

## Details:
* Symposium without Discussant Submission
* Topics: Physical Health, Methods: Analysis
* Keywords: multimodal data integration, human activity recognition, open dataset, pattern mining, sensor fusion

### General symposium abstract (243 words):
The use of smartphone applications and wearable sensors provide us with tools for unobtrusive, continuous measurements on a high temporal resolution of (among others) a person’s physical activity, stress levels, sleep, mobility, and social interaction patterns. This gives us additional ways to gather information compared to traditional self-reports. To accomodate the complex system of everyday life behaviour, experiences, and functioning with its biological, psychological, and social aspects, information on these different modalities is ideally combined. This symposium focuses on the integration of information from different modalities to measure and quantify human activities in daily life, thereby taking a data-driven approach. In such a multimodal data integration, several challenges exist that have only gotten little attention in the field of daily human activity; such as integrating different sampling rates across datasets, heterogeneity between people, and determining the point at which data integration is optimal (i.e., feature-level vs. decision-level fusion). Throughout the different talks in this symposium, examples are given of the challenges of multimodal data integration to model human activity throughout different settings. In each of these settings data from different sources, ranging from data from a smartphone application, accelerometers, gyroscopes, magnetometers, indirect calorimetry, GPS, photoplethysmography, and RDIF sensors, need to be combined to describe the activity for varying groups of people, such as children, elderly, and wheelchair users. In addition to specific studies with multimodal data, we present an open-labelled dataset with multimodal data.

-------

## Symposium set-up

Presentations of approx. 10 minutes and 5 minutes for 1-2 questions per abstract, and at the end of all presentations, 10-15 minutes session of a round table discussion/questions. The following order of presentations and timing (total duration should not exceed 90 minutes):

* **Introduction (~5min)**
* **1st part, data collection (30 min):**
1. "A Public Domain Dataset to Simulate Children’s Free Play", Meadeh Nasri (and/or Lisa-Maria van Klaveren), and
2. "Integrating Qualitative and Quantitative Data: Mixed Methods revisited from a Pattern Mining Perspective", Daniela Gawehns
* **2nd part, model development (30 min):**
1. "Integrating multimodal data in human activity recognition of wheelchair activities", Iris Yocarini
2. "RNNs on monitoring Physical Activity Energy Expenditure in older individuals", Stylianos Paraschiakos
* **3rd part, complete user case (15 min):**
1. "Heterogeneous human daily life monitoring data integration, fusion and analysis for assessment and adverse event prediction", Thomas Papastergiou (and/or Vasileios Megalooikonomou)
* **4th part, round table discussion/questions (10 min)** with maybe Iris chairing.
Discussion points:

* Noise Reduction is one of the most impactful preprocessing steps. How can we make sure that the effects of the preprocessing are recorded?
* What do you prefer: an engineer who uses their knowledge to adjust, calibrate and correct signals OR a black box algorithm who does the same?
* Privacy concerns: how do we share (benchmark) datasets? GPS data, interdependent datasets, is anonymization possible?
* Integrating different hardware and platforms: How do we model difference in hardware (and between sensors of the same make?)

-------
## Individual abstracts

### A Public Domain Dataset to Simulate Children's Free Play
*Authors: M. Nasri, D. Gawehns, L. van Klaveren, M. Baratchi, A. Koutamanis, S. Giest, C. Rieffe*

Abstract:
In free play children engage in physically and socially reinforcing activities offering unique opportunities for learning and socialization that are often not available in structured activities. However, studies using sensor data so far have mainly focused on the detection and analyses of movements in structured environments leaving unstructured activities such as free play understudied.
Due to the complex nature of free play, one sensor type is not able to capture its quality. Therefore, a multi-modal approach fusing different sensor types is adopted. This approach has two advantages: Firstly, noise reduction methods can be improved. Secondly, methods that are able to map the complexity of free play in space, time and activity can be modified. To further develop these methods, a reliable dataset including free and structured play needs to be available.
For this dataset, 15 participants wore three types of sensors including GPS loggers, motion sensors (accelerometer, magnetometer and gyroscope) and proximity sensors (RFID) and performed play activities in two protocols: structured and unstructured. Both scenarios are video recorded, and videos are used as ground truth in order to evaluate the captured data.
Results of such data-driven approaches can be validated with labeled ground truth datasets. To the best of our knowledge, no such datasets exists for for the sensor types that are employed here and more specifically, for children’s unstructured interactions. A dataset simulating free play as well as structured activities will fill this gap and will enable researchers from different fields to validate and improve on their methods.

### Integrating Qualitative and Quantitative Data : Mixed Methods revisited from a Pattern Mining Perspective
*Authors: D. Gawehns, M. van Leeuwen, S.Portegijs, S. van Beek*

Abstract (244 words):
Mixed methods have been employed for a long time in the social, health, and behavioral sciences as a way to combine the best of two worlds, namely the reliability of quantitative methods and the expressiveness of qualitative data. In several research designs, the power of both types of methods are leveraged to inform each other or to be integrated.
With more so called big data being collected within mixed-methods research, new data science methods, such as pattern mining, are now also employed to correlate, cluster and segment data. Often, the extracted patterns are not easily interpretable and most methods ask the researcher to take a multitude of decisions (from pre-processing to parameter settings such as cluster size or thresholds). These modeling choices influence the results of the pattern mining pipeline and easily propagate into the interpretation and conclusion of research papers. Data-driven methods that are employed to mine patterns in multi-modal data aggravate this issue by adding more dimensions and additional parameters.
In this study, we show how qualitative information is not only integral to understanding and interpreting quantitative results, but is also needed to guide modeling choices made by the analyst. We will do this based on a case study in a Dutch nursing home, where wearables were used to collect activity data and data from fixed sensors and alarms were gathered in addition to observational, qualitative data that summarize the daily activities of residents at a dementia care ward.

### Integrating multimodal data in human activity recognition of wheelchair activities
*Authors: I. Yocarini, D. Hoevenaars, S. Paraschiakos, W. Kraaij*

Abstract (237 words):
Human activity recognition (HAR) provides context information that has been shown to be valuable in energy expenditure estimation (EEE). This is also true for wheelchair activities performed by wheelchair users. Improved EEE (and HAR) is especially important for this population for which physical activity is limited and in which increased risk on obesity and associated risk of cardiovascular disease and mortality exist. Whereas monitoring physical activity may aid in increasing fitness levels, the sensor software used to track activity is mostly trained on non-wheelchair activities and therefore often inaccurate for wheelchair users. To date, most studies on HAR have mainly used sensor data from accelerometers, gyroscopes or magnetometers. However, recent developments in wearable photoplethysmography (PPG) also allow us to measure heart rate, an indication of the intensity level of the physical activity of the wearer. In this study, we assess the use of heart rate data in classifying wheelchair activities in addition to accelerometer data to improve HAR in wheelchair activities. To do so, we build a classifier using data from different modalities (i.e., from wearable accelerometers and a PPG) and evaluate different fusion models for the integration of these different data streams. To handle the different sampling rates in the different type of sensors, a decision-level (i.e., late) fusion technique is applied in which separate classification models are built for each modality. Subsequently, these different classifiers are merged through a meta-learner producing the final activity class.

### RNNs on monitoring Physical Activity Energy Expenditure in older individuals
*Authors: S. Paraschiakos, Cláudio Rebelo de Sá, Jeremiah Okai, Eline P. Slagboom, Marian Beekman, Arno Knobbe*

Abstract (249 words):
Quantifying physical activity energy expenditure (PAEE) of older people, has the potential to stimulate vital and healthy ageing by inducing behavioural changes and linking them to personal health gains. To be able to measure PAEE in everyday life, methods from wearable accelerometers have been developed, however, mainly targeted towards younger people. Since older subjects differ in energy requirements and range of physical activities, the current models may not be suitable for PAEE estimation. Because past activities influence present PAEE, we propose a modeling approach known for its ability to model sequential data, the Recurrent Neural Network (RNN). To train the RNN, we used the GOTOV dataset with 34 healthy participants of 60 years and older, performing 16 activities. We used accelerometers placed on wrist and ankle, and measurements of indirect calorimetry. After optimization, we propose an architecture consisting of an RNN with 3 GRU layers and a feedforward network combining both accelerometer and participant-level data. In this paper, we describe our efforts to go beyond the standard facilities of a GRU-based RNN, with the aim of achieving accuracy surpassing the state of the art. These efforts include switching aggregation function from mean to dispersion measures, combining temporal and static data (person-specific details) and adding predicted symbolic activity data by a previously trained ML model. The resulting architecture increases its performance by approximatelly 10% while decreasing training input by a factor of 10. It can thus be employed to investigate associations of PAEE with vitality parameters related to healthy ageing.

### Heterogeneous human daily life monitoring data integration, fusion and analysis for assessment and adverse event prediction
*Authors: T. Papastergiou, E. I. Zacharaki, K. Deltouzos, S. Kalogiannis, J. Ellul and V. Megalooikonomou*

Abstract:
Technological advances have made available enormous amounts of continuous data related to human activity and behavior. We present a real life sensing and intervention platform that was developed to better understand frailty of older people, provide quantitative and qualitative measures of frailty through advanced data analytics approaches on multiparametric data and predict short and long-term outcome, risk of frailty and adverse events through a safe, unobtrusive and acceptable system for the ageing population. Data from physical, cognitive, psychological, and social domains are integrated and fused through various preprocessing techniques while a virtual patient model is used to provide a structured machine-readable representation (low- and high-level information) of a person’s data. Physical activity recognition from wearable sensors is also performed as a step for physiological monitoring and assessment. In this talk we focus on data integration and fusion that happens at the data level, the feature level and the decision level as well as on big data analytics techniques that include multiple instance learning, deep convolutional neural networks, and tensor decomposition techniques that are used for assessment of frailty level and prediction of adverse events.