{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tremor analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial shows how to run the tremor pipeline to obtain aggregated tremor measures from gyroscope sensor data. Before following along, make sure all data preparation steps have been followed in the data preparation tutorial. \n",
    "\n",
    "In this tutorial, we use two days of data from a participant of the Personalized Parkinson Project to demonstrate the functionalities. Since `ParaDigMa` expects contiguous time series, the collected data was stored in two segments each with contiguous timestamps. Per segment, we load the data and perform the following steps:\n",
    "1. Preprocess the time series data\n",
    "2. Extract tremor features\n",
    "3. Detect tremor\n",
    "4. Quantify tremor\n",
    "\n",
    "We then combine the output of the different segments for the final step:\n",
    "\n",
    "5. Compute aggregated tremor measures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load example data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we start by loading a single contiguous time series (segment), for which we continue running steps 1-3. [Below](#multiple_segments_cell) we show how to run these steps for multiple segments.\n",
    "\n",
    "We use the interally developed `TSDF` ([documentation](https://biomarkersparkinson.github.io/tsdf/)) to load and store data [[1](https://arxiv.org/abs/2211.11294)]. Depending on the file extension of your time series data, examples of other Python functions for loading the data into memory include:\n",
    "- _.csv_: `pandas.read_csv()` ([documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html))\n",
    "- _.json_: `json.load()` ([documentation](https://docs.python.org/3/library/json.html#json.load))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from paradigma.util import load_tsdf_dataframe\n",
    "\n",
    "# Set the path to where the prepared data is saved and load the data.\n",
    "# Note: the test data is stored in TSDF, but you can load your data in your own way\n",
    "path_to_data =  Path('../../example_data')\n",
    "path_to_prepared_data = path_to_data / 'imu'\n",
    "\n",
    "segment_nr  = '0001' \n",
    "\n",
    "df_data, metadata_time, metadata_values = load_tsdf_dataframe(path_to_prepared_data, prefix=f'IMU_segment{segment_nr}')\n",
    "\n",
    "df_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Preprocess data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMU sensors collect data at a fixed sampling frequency, but the sampling rate is not uniform, causing variation in time differences between timestamps. The [preprocess_imu_data](https://github.com/biomarkersParkinson/paradigma/blob/main/src/paradigma/preprocessing.py#:~:text=preprocess_imu_data) function therefore resamples the timestamps to be uniformly distributed, and then interpolates IMU values at these new timestamps using the original timestamps and corresponding IMU values. If the difference between timestamps is larger than a specified tolerance (`config.tolerance`, in seconds), it will return an error that the timestamps are not contiguous.  If you still want to process the data in this case, you can create segments from discontiguous samples using the function [`create_segments`](https://github.com/biomarkersParkinson/paradigma/blob/main/src/paradigma/segmenting.py) and analyze these segments consecutively as shown in [here](#multiple_segments_cell). By setting `sensor` to 'gyroscope', only gyroscope data is preprocessed and the accelerometer data is removed from the dataframe. Also a `watch_side` should be provided, although for the tremor analysis it does not matter whether this is the correct side since the tremor features are not influenced by the gyroscope axes orientation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from paradigma.config import IMUConfig\n",
    "from paradigma.constants import DataColumns\n",
    "from paradigma.preprocessing import preprocess_imu_data\n",
    "\n",
    "# Set column names: replace DataColumn.* with your actual column names. \n",
    "# It is only necessary to set the columns that are present in your data, and\n",
    "# only if they differ from the default names defined in DataColumns.\n",
    "column_mapping = {\n",
    "    'TIME': DataColumns.TIME,\n",
    "    'ACCELEROMETER_X': DataColumns.ACCELEROMETER_X,\n",
    "    'ACCELEROMETER_Y': DataColumns.ACCELEROMETER_Y,\n",
    "    'ACCELEROMETER_Z': DataColumns.ACCELEROMETER_Z,\n",
    "    'GYROSCOPE_X': DataColumns.GYROSCOPE_X,\n",
    "    'GYROSCOPE_Y': DataColumns.GYROSCOPE_Y,\n",
    "    'GYROSCOPE_Z': DataColumns.GYROSCOPE_Z,\n",
    "}\n",
    "\n",
    "config = IMUConfig(column_mapping)\n",
    "print(f'The data is resampled to {config.resampling_frequency} Hz.')\n",
    "print(f'The tolerance for checking contiguous timestamps is set to {config.tolerance:.3f} seconds.')\n",
    "\n",
    "df_preprocessed_data = preprocess_imu_data(df_data, config, sensor='gyroscope', watch_side='left')\n",
    "\n",
    "df_preprocessed_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Extract tremor features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function [`extract_tremor_features`](https://github.com/biomarkersParkinson/paradigma/blob/main/src/paradigma/pipelines/tremor_pipeline.py#:~:text=extract_tremor_features) extracts windows from the preprocessed gyroscope data using non-overlapping windows of length `config.window_length_s`. Next, from these windows the tremor features are extracted: 12 mel-frequency cepstral coefficients (MFCCs), frequency of the peak in the power spectral density, power below tremor (0.5 - 3 Hz), and power around the tremor peak. The latter is not used for tremor detection, but stored for tremor quantification in Step 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from paradigma.config import TremorConfig\n",
    "from paradigma.pipelines.tremor_pipeline import extract_tremor_features\n",
    "\n",
    "config = TremorConfig(step='features')\n",
    "print(f'The window length is {config.window_length_s} seconds')\n",
    "\n",
    "df_features = extract_tremor_features(df_preprocessed_data, config)\n",
    "\n",
    "df_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Detect tremor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function [`detect_tremor`](https://github.com/biomarkersParkinson/paradigma/blob/main/src/paradigma/pipelines/tremor_pipeline.py#:~:text=detect_tremor) uses a pretrained logistic regression classifier to predict the tremor probability (`pred_tremor_proba`) for each window, based on the MFCCs. Using the prespecified threshold, a tremor label of 0 (no tremor) or 1 (tremor) is assigned (`pred_tremor_logreg`). Furthermore, the detected tremor windows are checked for rest tremor in two ways. First, the frequency of the peak should be between 3-7 Hz. Second, we want to exclude windows with significant arm movements. We consider a window to have significant arm movement if `below_tremor_power` exceeds `config.movement_threshold`. The final tremor label is saved in `pred_tremor_checked`. A label for predicted arm at rest (`pred_arm_at_rest`, which is 1 when at rest and 0 when not at rest) was also saved, to control for the amount of arm movement during the observed time period when aggregating the amount of tremor time in Step 4 (if a person is moving their arm, they cannot have rest tremor)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from importlib.resources import files\n",
    "from paradigma.pipelines.tremor_pipeline import detect_tremor\n",
    "\n",
    "print(f'A threshold of {config.movement_threshold} deg\\u00b2/s\\u00b2 \\\n",
    "is used to determine whether the arm is at rest or in stable posture.')\n",
    "\n",
    "# Load the pre-trained logistic regression classifier\n",
    "tremor_detection_classifier_package_filename = 'tremor_detection_clf_package.pkl'\n",
    "full_path_to_classifier_package = files('paradigma') / 'assets' / tremor_detection_classifier_package_filename\n",
    "\n",
    "# Use the logistic regression classifier to detect tremor and check for rest tremor\n",
    "df_predictions = detect_tremor(df_features, config, full_path_to_classifier_package)\n",
    "\n",
    "df_predictions[[config.time_colname, 'pred_tremor_proba', 'pred_tremor_logreg', 'pred_arm_at_rest', 'pred_tremor_checked']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Store as TSDF\n",
    "The predicted probabilities (and optionally other features) can be stored and loaded in TSDF as demonstrated below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tsdf\n",
    "from paradigma.util import write_df_data\n",
    "\n",
    "# Set 'path_to_data' to the directory where you want to save the data\n",
    "metadata_time_store = tsdf.TSDFMetadata(metadata_time.get_plain_tsdf_dict_copy(), path_to_data)\n",
    "metadata_values_store = tsdf.TSDFMetadata(metadata_values.get_plain_tsdf_dict_copy(), path_to_data)\n",
    "\n",
    "# Select the columns to be saved \n",
    "metadata_time_store.channels = [config.time_colname]\n",
    "metadata_values_store.channels = ['tremor_power', 'pred_tremor_proba', 'pred_tremor_logreg', 'pred_arm_at_rest', 'pred_tremor_checked']\n",
    "\n",
    "# Set the units\n",
    "metadata_time_store.units = ['Relative seconds']\n",
    "metadata_values_store.units = ['Unitless', 'Unitless', 'Unitless', 'Unitless', 'Unitless']  \n",
    "metadata_time_store.data_type = float\n",
    "metadata_values_store.data_type = float\n",
    "\n",
    "# Set the filenames\n",
    "meta_store_filename = f'segment{segment_nr}_meta.json'\n",
    "values_store_filename = meta_store_filename.replace('_meta.json', '_values.bin')\n",
    "time_store_filename = meta_store_filename.replace('_meta.json', '_time.bin')\n",
    "\n",
    "metadata_values_store.file_name = values_store_filename\n",
    "metadata_time_store.file_name = time_store_filename\n",
    "\n",
    "write_df_data(metadata_time_store, metadata_values_store, path_to_data, meta_store_filename, df_predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_predictions, _, _ = load_tsdf_dataframe(path_to_data, prefix=f'segment{segment_nr}')\n",
    "df_predictions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Quantify tremor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tremor power of all predicted tremor windows (where `pred_tremor_checked` is 1) is used for tremor quantification. A datetime column is also added, providing necessary information before aggregating over specified hours in Step 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import datetime\n",
    "import pytz\n",
    "\n",
    "df_quantification = df_predictions[[config.time_colname, 'pred_arm_at_rest', 'pred_tremor_checked','tremor_power']].copy()\n",
    "df_quantification.loc[df_predictions['pred_tremor_checked'] == 0, 'tremor_power'] = None # tremor power of non-tremor windows is set to None\n",
    "\n",
    "# Create datetime column based on the start time of the segment\n",
    "start_time = datetime.datetime.strptime(metadata_time.start_iso8601, '%Y-%m-%dT%H:%M:%SZ')\n",
    "start_time = start_time.replace(tzinfo=pytz.timezone('UTC')).astimezone(pytz.timezone('CET')) # convert to correct timezone if necessary\n",
    "df_quantification[f'{config.time_colname}_dt'] = start_time + pd.to_timedelta(df_quantification[config.time_colname], unit=\"s\") \n",
    "df_quantification = df_quantification[[config.time_colname, f'{config.time_colname}_dt', 'pred_arm_at_rest', 'pred_tremor_checked', 'tremor_power']]\n",
    "\n",
    "df_quantification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run steps 1 - 4 for all segments <a id='multiple_segments_cell'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your data is also stored in multiple segments, you can modify `segments` in the cell below to a list of the filenames of your respective segmented data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from importlib.resources import files\n",
    "import datetime\n",
    "import pytz\n",
    "import pandas as pd\n",
    "\n",
    "from paradigma.util import load_tsdf_dataframe\n",
    "from paradigma.config import IMUConfig, TremorConfig\n",
    "from paradigma.preprocessing import preprocess_imu_data\n",
    "from paradigma.pipelines.tremor_pipeline import extract_tremor_features, detect_tremor\n",
    "\n",
    "# Set the path to where the prepared data is saved\n",
    "path_to_data =  Path('../../example_data')\n",
    "path_to_prepared_data = path_to_data / 'imu'\n",
    "\n",
    "# Load the pre-trained logistic regression classifier\n",
    "tremor_detection_classifier_package_filename = 'tremor_detection_clf_package.pkl'\n",
    "full_path_to_classifier_package = files('paradigma') / 'assets' / tremor_detection_classifier_package_filename\n",
    "\n",
    "# Create a list of dataframes to store the quantifications of all segments\n",
    "list_df_quantifications = []\n",
    "\n",
    "segments  = ['0001','0002'] # list with all  available segments\n",
    "\n",
    "for segment_nr in segments:\n",
    "    \n",
    "    # Load the data\n",
    "    df_data, metadata_time, _ = load_tsdf_dataframe(path_to_prepared_data, prefix='IMU_segment'+segment_nr)\n",
    "\n",
    "    # 1: Preprocess the data\n",
    "    # Change column names if necessary by creating parameter column_mapping (see previous cells for an example)\n",
    "    config = IMUConfig()\n",
    "    df_preprocessed_data = preprocess_imu_data(df_data, config, sensor='gyroscope', watch_side='left')\n",
    "\n",
    "    # 2: Extract features\n",
    "    config = TremorConfig(step='features')\n",
    "    df_features = extract_tremor_features(df_preprocessed_data, config)\n",
    "\n",
    "    # 3: Detect tremor\n",
    "    df_predictions = detect_tremor(df_features, config, full_path_to_classifier_package)\n",
    "\n",
    "    # 4: Quantify tremor\n",
    "    df_quantification = df_predictions[[config.time_colname, 'pred_arm_at_rest', 'pred_tremor_checked','tremor_power']].copy()\n",
    "    df_quantification.loc[df_predictions['pred_tremor_checked'] == 0, 'tremor_power'] = None\n",
    "\n",
    "    # Create datetime column based on the start time of the segment\n",
    "    start_time = datetime.datetime.strptime(metadata_time.start_iso8601, '%Y-%m-%dT%H:%M:%SZ')\n",
    "    start_time = start_time.replace(tzinfo=pytz.timezone('UTC')).astimezone(pytz.timezone('CET')) # convert to correct timezone if necessary\n",
    "    df_quantification[f'{config.time_colname}_dt'] = start_time + pd.to_timedelta(df_quantification[config.time_colname], unit=\"s\") \n",
    "    df_quantification = df_quantification[[config.time_colname, f'{config.time_colname}_dt', 'pred_arm_at_rest', 'pred_tremor_checked', 'tremor_power']]\n",
    "\n",
    "    # Add the quantifications of the current segment to the list\n",
    "    df_quantification['segment_nr'] = segment_nr\n",
    "    list_df_quantifications.append(df_quantification)\n",
    "\n",
    "df_quantification = pd.concat(list_df_quantifications, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Compute aggregated tremor measures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final step is to compute the amount of tremor time and tremor power with the function [`aggregate_tremor`](https://github.com/biomarkersParkinson/paradigma/blob/main/src/paradigma/pipelines/tremor_pipeline.py#:~:text=aggregate_tremor), which aggregates over all windows in the input dataframe. Depending on the size of the input dateframe, you could select the hours and days (both optional) that you want to include in this analysis. In this case we use data collected between 8 am and 10 pm (specified as `select_hours_start` and `select_hours_end`), and days with at least 10 hours of data (`min_hours_per_day`) based on. Based on the selected data, we compute aggregated measures for tremor time and tremor power:\n",
    "- Tremor time is calculated as the number of detected tremor windows, as percentage of the number of windows while the arm is at rest or in stable posture (when `below_tremor_power` does not exceed `config.movement_threshold`). This way the tremor time is controlled for the amount of time the arm is at rest or in stable posture, when rest tremor and re-emergent tremor could occur.\n",
    "- For tremor power the following aggregates are derived: the mode, median and 90th percentile of tremor power (specified in `config.aggregates_tremor_power`). The median and modal tremor power reflect the typical tremor severity, whereas the 90th percentile reflects the maximal tremor severity within the observed timeframe. The modal tremor power is computed as the peak in the probability density function of tremor power, which is evaluated at the points specified in `config.evaluation_points_tremor_power` (300 points between 0 and 6 log tremor power). The aggregated tremor measures and metadata are stored in a json file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pprint\n",
    "from paradigma.util import select_hours, select_days\n",
    "from paradigma.pipelines.tremor_pipeline import aggregate_tremor\n",
    "\n",
    "select_hours_start = '08:00' # you can specifiy the hours and minutes here\n",
    "select_hours_end = '22:00'\n",
    "min_hours_per_day = 10\n",
    "\n",
    "print(f'Before aggregation we select data collected between {select_hours_start} \\\n",
    "and {select_hours_end}. We also select days with at least {min_hours_per_day} hours of data.')\n",
    "print(f'The following tremor power aggregates are derived: {config.aggregates_tremor_power}.')\n",
    "\n",
    "# Select the hours that should be included in the analysis\n",
    "df_quantification = select_hours(df_quantification, select_hours_start, select_hours_end)\n",
    "\n",
    "# Remove days with less than the specified minimum amount of hours\n",
    "df_quantification = select_days(df_quantification, min_hours_per_day)\n",
    "\n",
    "# Compute the aggregated measures\n",
    "config = TremorConfig()\n",
    "d_tremor_aggregates = aggregate_tremor(df = df_quantification, config = config)\n",
    "\n",
    "pprint.pprint(d_tremor_aggregates)"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 2
}