{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Heart rate analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial shows how to extract heart rate estimates using photoplethysmography (PPG) data and accelerometer data. The pipeline consists of a stepwise approach to determine signal quality, assessing both PPG morphology and accounting for periodic artifacts using the accelerometer. Based on the signal quality, we extract high-quality segments and estimate the heart rate for every 2 s using the smoothed pseudo Wigner-Ville Distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This pipeline requires accelerometer and PPG data to run. In this example we loaded data from a participant of the Personalized Parkinson Project. We load the corresponding dataframes using the [load_tsdf_dataframe](https://github.com/biomarkersParkinson/paradigma/blob/main/src/paradigma/util.py#:~:text=load_tsdf_dataframe) function. The channel `green` represents the values obtained with PPG using green light.\n",
    "\n",
    "In this example we use the interally developed `TSDF` ([documentation](https://biomarkersparkinson.github.io/tsdf/)) to load and store data [[1](https://arxiv.org/abs/2211.11294)]. \n",
    "\n",
    "However, we are aware that there are other common data formats. For example, the following functions can be used depending on the file extension of the data:\n",
    "- _.csv_: `pandas.read_csv()` ([documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html))\n",
    "- _.json_: `json.load()` ([documentation](https://docs.python.org/3/library/json.html#json.load))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>green</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00000</td>\n",
       "      <td>649511</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.00996</td>\n",
       "      <td>648214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.01992</td>\n",
       "      <td>646786</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.02988</td>\n",
       "      <td>645334</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.03984</td>\n",
       "      <td>644317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64770</th>\n",
       "      <td>644.54720</td>\n",
       "      <td>553884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64771</th>\n",
       "      <td>644.55716</td>\n",
       "      <td>554088</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64772</th>\n",
       "      <td>644.56712</td>\n",
       "      <td>554240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64773</th>\n",
       "      <td>644.57708</td>\n",
       "      <td>555134</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64774</th>\n",
       "      <td>644.58704</td>\n",
       "      <td>557205</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>64775 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            time   green\n",
       "0        0.00000  649511\n",
       "1        0.00996  648214\n",
       "2        0.01992  646786\n",
       "3        0.02988  645334\n",
       "4        0.03984  644317\n",
       "...          ...     ...\n",
       "64770  644.54720  553884\n",
       "64771  644.55716  554088\n",
       "64772  644.56712  554240\n",
       "64773  644.57708  555134\n",
       "64774  644.58704  557205\n",
       "\n",
       "[64775 rows x 2 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>accelerometer_x</th>\n",
       "      <th>accelerometer_y</th>\n",
       "      <th>accelerometer_z</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.550718</td>\n",
       "      <td>0.574163</td>\n",
       "      <td>-0.273684</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.01004</td>\n",
       "      <td>0.535885</td>\n",
       "      <td>0.623445</td>\n",
       "      <td>-0.254545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02008</td>\n",
       "      <td>0.504306</td>\n",
       "      <td>0.651675</td>\n",
       "      <td>-0.251675</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03012</td>\n",
       "      <td>0.488517</td>\n",
       "      <td>0.686603</td>\n",
       "      <td>-0.265550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.04016</td>\n",
       "      <td>0.494258</td>\n",
       "      <td>0.725359</td>\n",
       "      <td>-0.278469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72942</th>\n",
       "      <td>730.74468</td>\n",
       "      <td>0.234928</td>\n",
       "      <td>-0.516268</td>\n",
       "      <td>-0.802871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72943</th>\n",
       "      <td>730.75472</td>\n",
       "      <td>0.245455</td>\n",
       "      <td>-0.514354</td>\n",
       "      <td>-0.806699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72944</th>\n",
       "      <td>730.76476</td>\n",
       "      <td>0.243541</td>\n",
       "      <td>-0.511005</td>\n",
       "      <td>-0.807177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72945</th>\n",
       "      <td>730.77480</td>\n",
       "      <td>0.240191</td>\n",
       "      <td>-0.514354</td>\n",
       "      <td>-0.808134</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72946</th>\n",
       "      <td>730.78484</td>\n",
       "      <td>0.243541</td>\n",
       "      <td>-0.511005</td>\n",
       "      <td>-0.808134</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>72947 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            time  accelerometer_x  accelerometer_y  accelerometer_z\n",
       "0        0.00000         0.550718         0.574163        -0.273684\n",
       "1        0.01004         0.535885         0.623445        -0.254545\n",
       "2        0.02008         0.504306         0.651675        -0.251675\n",
       "3        0.03012         0.488517         0.686603        -0.265550\n",
       "4        0.04016         0.494258         0.725359        -0.278469\n",
       "...          ...              ...              ...              ...\n",
       "72942  730.74468         0.234928        -0.516268        -0.802871\n",
       "72943  730.75472         0.245455        -0.514354        -0.806699\n",
       "72944  730.76476         0.243541        -0.511005        -0.807177\n",
       "72945  730.77480         0.240191        -0.514354        -0.808134\n",
       "72946  730.78484         0.243541        -0.511005        -0.808134\n",
       "\n",
       "[72947 rows x 4 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "from paradigma.util import load_tsdf_dataframe\n",
    "\n",
    "path_to_data = Path('../../tests/data')\n",
    "path_to_prepared_data = path_to_data / '1.prepared_data'\n",
    "\n",
    "ppg_prefix = 'PPG'\n",
    "imu_prefix = 'IMU'\n",
    "\n",
    "df_ppg, metadata_time_ppg, _ = load_tsdf_dataframe(\n",
    "    path_to_data=path_to_prepared_data / ppg_prefix, \n",
    "    prefix=ppg_prefix\n",
    ")\n",
    "df_imu, metadata_time_imu, _ = load_tsdf_dataframe(\n",
    "    path_to_data=path_to_prepared_data / imu_prefix, \n",
    "    prefix=imu_prefix\n",
    ")\n",
    "\n",
    "# Drop the gyroscope columns from the IMU data\n",
    "cols_to_drop = df_imu.filter(regex='^gyroscope_').columns\n",
    "df_acc = df_imu.drop(cols_to_drop, axis=1)\n",
    "\n",
    "display(df_ppg, df_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Preprocess data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step after loading the data is preprocessing using the [preprocess_ppg_data](https://github.com/biomarkersParkinson/paradigma/blob/main/src/paradigma/preprocessing.py#:~:text=preprocess_ppg_data). This begins by isolating segments containing both PPG and IMU data, discarding portions where one modality (e.g., PPG) extends beyond the other, such as when the PPG recording is longer than the accelerometer data. This functionality requires the starting times (`metadata_time_ppg.start_iso8601` and `metadata_time_imu.start_iso8601`) in iso8601 format as inputs. After this step, the preprocess_ppg_data function resamples the PPG and accelerometer data to uniformly distributed timestamps, addressing the fixed but non-uniform sampling rates of the sensors. After this, a bandpass Butterworth filter (4th-order, bandpass frequencies: 0.4--3.5 Hz) is applied to the PPG signal, while a high-pass Butterworth filter (4th-order, cut-off frequency: 0.2 Hz) is applied to the accelerometer data.\n",
    "\n",
    "Note: the printed shapes are (rows, columns) with each row corresponding to a single data point and each column representing a data column (e.g.time). The number of rows of the overlapping segments of PPG and accelerometer are not the same due to sampling differences (other sensors and possibly other sampling frequencies)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original data shapes:\n",
      "- PPG data: (64775, 2)\n",
      "- Accelerometer data: (72947, 7)\n",
      "Overlapping preprocessed data shapes:\n",
      "- PPG data: (19338, 2)\n",
      "- Accelerometer data: (64459, 4)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>green</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>-1434.856838</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.033333</td>\n",
       "      <td>-3461.717789</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.066667</td>\n",
       "      <td>-5172.913796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.100000</td>\n",
       "      <td>-6343.203160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.133333</td>\n",
       "      <td>-6875.904054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19333</th>\n",
       "      <td>644.433333</td>\n",
       "      <td>-2403.547097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19334</th>\n",
       "      <td>644.466667</td>\n",
       "      <td>-7434.311944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19335</th>\n",
       "      <td>644.500000</td>\n",
       "      <td>-8214.847078</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19336</th>\n",
       "      <td>644.533333</td>\n",
       "      <td>-5194.393329</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19337</th>\n",
       "      <td>644.566667</td>\n",
       "      <td>70.147000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>19338 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             time        green\n",
       "0        0.000000 -1434.856838\n",
       "1        0.033333 -3461.717789\n",
       "2        0.066667 -5172.913796\n",
       "3        0.100000 -6343.203160\n",
       "4        0.133333 -6875.904054\n",
       "...           ...          ...\n",
       "19333  644.433333 -2403.547097\n",
       "19334  644.466667 -7434.311944\n",
       "19335  644.500000 -8214.847078\n",
       "19336  644.533333 -5194.393329\n",
       "19337  644.566667    70.147000\n",
       "\n",
       "[19338 rows x 2 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>accelerometer_x</th>\n",
       "      <th>accelerometer_y</th>\n",
       "      <th>accelerometer_z</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.053078</td>\n",
       "      <td>0.010040</td>\n",
       "      <td>-0.273154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.01</td>\n",
       "      <td>0.038337</td>\n",
       "      <td>0.058802</td>\n",
       "      <td>-0.256899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02</td>\n",
       "      <td>0.006824</td>\n",
       "      <td>0.086559</td>\n",
       "      <td>-0.256739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03</td>\n",
       "      <td>-0.009156</td>\n",
       "      <td>0.120855</td>\n",
       "      <td>-0.273280</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.04</td>\n",
       "      <td>-0.003770</td>\n",
       "      <td>0.159316</td>\n",
       "      <td>-0.289007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64454</th>\n",
       "      <td>644.54</td>\n",
       "      <td>0.104185</td>\n",
       "      <td>0.101627</td>\n",
       "      <td>-0.029342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64455</th>\n",
       "      <td>644.55</td>\n",
       "      <td>0.120403</td>\n",
       "      <td>0.090413</td>\n",
       "      <td>0.051113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64456</th>\n",
       "      <td>644.56</td>\n",
       "      <td>0.101289</td>\n",
       "      <td>0.128684</td>\n",
       "      <td>0.037951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64457</th>\n",
       "      <td>644.57</td>\n",
       "      <td>0.073139</td>\n",
       "      <td>0.137399</td>\n",
       "      <td>0.030834</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64458</th>\n",
       "      <td>644.58</td>\n",
       "      <td>0.004578</td>\n",
       "      <td>0.102862</td>\n",
       "      <td>0.025627</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>64459 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         time  accelerometer_x  accelerometer_y  accelerometer_z\n",
       "0        0.00         0.053078         0.010040        -0.273154\n",
       "1        0.01         0.038337         0.058802        -0.256899\n",
       "2        0.02         0.006824         0.086559        -0.256739\n",
       "3        0.03        -0.009156         0.120855        -0.273280\n",
       "4        0.04        -0.003770         0.159316        -0.289007\n",
       "...       ...              ...              ...              ...\n",
       "64454  644.54         0.104185         0.101627        -0.029342\n",
       "64455  644.55         0.120403         0.090413         0.051113\n",
       "64456  644.56         0.101289         0.128684         0.037951\n",
       "64457  644.57         0.073139         0.137399         0.030834\n",
       "64458  644.58         0.004578         0.102862         0.025627\n",
       "\n",
       "[64459 rows x 4 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from paradigma.config import PPGConfig, IMUConfig\n",
    "from paradigma.preprocessing import preprocess_ppg_data\n",
    "\n",
    "ppg_config = PPGConfig()\n",
    "imu_config = IMUConfig()\n",
    "\n",
    "print(f\"Original data shapes:\\n- PPG data: {df_ppg.shape}\\n- Accelerometer data: {df_imu.shape}\")\n",
    "df_ppg_proc, df_acc_proc = preprocess_ppg_data(\n",
    "    df_ppg=df_ppg, \n",
    "    df_acc=df_acc, \n",
    "    ppg_config=ppg_config, \n",
    "    imu_config=imu_config, \n",
    "    start_time_ppg=metadata_time_ppg.start_iso8601,\n",
    "    start_time_imu=metadata_time_imu.start_iso8601\n",
    ")\n",
    "\n",
    "print(f\"Overlapping preprocessed data shapes:\\n- PPG data: {df_ppg_proc.shape}\\n- Accelerometer data: {df_acc_proc.shape}\")\n",
    "display(df_ppg_proc, df_acc_proc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Extract signal quality features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The preprocessed data (PPG & accelerometer) is windowed into overlapping windows of length `ppg_config.window_length_s` with a window step of `ppg_config.window_step_length_s`. From the PPG windows 10 time- and frequency domain features are extracted to assess PPG morphology and from the accelerometer windows one relative power feature is calculated to assess periodic motion artifacts.\n",
    "\n",
    "The detailed steps are encapsulated in `extract_signal_quality_features` (documentation can be found [here](https://github.com/biomarkersParkinson/paradigma/blob/main/src/paradigma/pipelines/heart_rate_pipeline.py#:~:text=extract_signal_quality_features))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The default window length for the signal quality feature extraction is set to 6 seconds.\n",
      "The default step size for the signal quality feature extraction is set to 1 seconds.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>var</th>\n",
       "      <th>mean</th>\n",
       "      <th>median</th>\n",
       "      <th>kurtosis</th>\n",
       "      <th>skewness</th>\n",
       "      <th>signal_to_noise</th>\n",
       "      <th>auto_corr</th>\n",
       "      <th>f_dom</th>\n",
       "      <th>rel_power</th>\n",
       "      <th>spectral_entropy</th>\n",
       "      <th>acc_power_ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.697510e+07</td>\n",
       "      <td>3418.696392</td>\n",
       "      <td>2892.815105</td>\n",
       "      <td>2.540447</td>\n",
       "      <td>0.277237</td>\n",
       "      <td>3.241634</td>\n",
       "      <td>0.282452</td>\n",
       "      <td>1.171875</td>\n",
       "      <td>0.179922</td>\n",
       "      <td>0.561186</td>\n",
       "      <td>0.014196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.635033e+07</td>\n",
       "      <td>3350.769158</td>\n",
       "      <td>2892.815105</td>\n",
       "      <td>2.647951</td>\n",
       "      <td>0.420163</td>\n",
       "      <td>3.202313</td>\n",
       "      <td>0.276526</td>\n",
       "      <td>1.171875</td>\n",
       "      <td>0.201972</td>\n",
       "      <td>0.547812</td>\n",
       "      <td>0.015343</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1.921248e+07</td>\n",
       "      <td>3698.963099</td>\n",
       "      <td>3310.773556</td>\n",
       "      <td>2.333687</td>\n",
       "      <td>0.481974</td>\n",
       "      <td>3.521882</td>\n",
       "      <td>0.210984</td>\n",
       "      <td>0.820312</td>\n",
       "      <td>0.264922</td>\n",
       "      <td>0.509015</td>\n",
       "      <td>0.042583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>1.646562e+07</td>\n",
       "      <td>3389.325775</td>\n",
       "      <td>2946.588526</td>\n",
       "      <td>2.608187</td>\n",
       "      <td>0.733913</td>\n",
       "      <td>3.346236</td>\n",
       "      <td>0.225823</td>\n",
       "      <td>0.703125</td>\n",
       "      <td>0.270707</td>\n",
       "      <td>0.496051</td>\n",
       "      <td>0.034215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>1.603135e+07</td>\n",
       "      <td>3364.366313</td>\n",
       "      <td>2912.280798</td>\n",
       "      <td>2.268768</td>\n",
       "      <td>0.440357</td>\n",
       "      <td>3.314565</td>\n",
       "      <td>0.305998</td>\n",
       "      <td>0.820312</td>\n",
       "      <td>0.264097</td>\n",
       "      <td>0.496697</td>\n",
       "      <td>0.056204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>634</th>\n",
       "      <td>634.0</td>\n",
       "      <td>2.281660e+06</td>\n",
       "      <td>1106.006037</td>\n",
       "      <td>838.307729</td>\n",
       "      <td>4.058037</td>\n",
       "      <td>0.674184</td>\n",
       "      <td>2.072660</td>\n",
       "      <td>0.172052</td>\n",
       "      <td>0.703125</td>\n",
       "      <td>0.204028</td>\n",
       "      <td>0.577570</td>\n",
       "      <td>0.060486</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>635</th>\n",
       "      <td>635.0</td>\n",
       "      <td>2.224441e+06</td>\n",
       "      <td>1113.194933</td>\n",
       "      <td>838.307729</td>\n",
       "      <td>3.684159</td>\n",
       "      <td>0.067979</td>\n",
       "      <td>2.118019</td>\n",
       "      <td>0.236408</td>\n",
       "      <td>0.703125</td>\n",
       "      <td>0.221044</td>\n",
       "      <td>0.560362</td>\n",
       "      <td>0.053120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>636</th>\n",
       "      <td>636.0</td>\n",
       "      <td>5.103141e+06</td>\n",
       "      <td>1627.618918</td>\n",
       "      <td>1030.113303</td>\n",
       "      <td>3.913068</td>\n",
       "      <td>0.852331</td>\n",
       "      <td>2.040094</td>\n",
       "      <td>0.235822</td>\n",
       "      <td>0.585938</td>\n",
       "      <td>0.157667</td>\n",
       "      <td>0.514717</td>\n",
       "      <td>0.058853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>637</th>\n",
       "      <td>637.0</td>\n",
       "      <td>7.428728e+06</td>\n",
       "      <td>2093.736007</td>\n",
       "      <td>1520.640067</td>\n",
       "      <td>3.188753</td>\n",
       "      <td>0.305780</td>\n",
       "      <td>2.457446</td>\n",
       "      <td>0.322034</td>\n",
       "      <td>0.468750</td>\n",
       "      <td>0.150582</td>\n",
       "      <td>0.458477</td>\n",
       "      <td>0.061269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>638</th>\n",
       "      <td>638.0</td>\n",
       "      <td>1.549534e+07</td>\n",
       "      <td>2974.462555</td>\n",
       "      <td>2482.703944</td>\n",
       "      <td>4.042171</td>\n",
       "      <td>-0.628008</td>\n",
       "      <td>2.228730</td>\n",
       "      <td>0.095918</td>\n",
       "      <td>0.468750</td>\n",
       "      <td>0.102976</td>\n",
       "      <td>0.493830</td>\n",
       "      <td>0.105554</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>639 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      time           var         mean       median  kurtosis  skewness  \\\n",
       "0      0.0  1.697510e+07  3418.696392  2892.815105  2.540447  0.277237   \n",
       "1      1.0  1.635033e+07  3350.769158  2892.815105  2.647951  0.420163   \n",
       "2      2.0  1.921248e+07  3698.963099  3310.773556  2.333687  0.481974   \n",
       "3      3.0  1.646562e+07  3389.325775  2946.588526  2.608187  0.733913   \n",
       "4      4.0  1.603135e+07  3364.366313  2912.280798  2.268768  0.440357   \n",
       "..     ...           ...          ...          ...       ...       ...   \n",
       "634  634.0  2.281660e+06  1106.006037   838.307729  4.058037  0.674184   \n",
       "635  635.0  2.224441e+06  1113.194933   838.307729  3.684159  0.067979   \n",
       "636  636.0  5.103141e+06  1627.618918  1030.113303  3.913068  0.852331   \n",
       "637  637.0  7.428728e+06  2093.736007  1520.640067  3.188753  0.305780   \n",
       "638  638.0  1.549534e+07  2974.462555  2482.703944  4.042171 -0.628008   \n",
       "\n",
       "     signal_to_noise  auto_corr     f_dom  rel_power  spectral_entropy  \\\n",
       "0           3.241634   0.282452  1.171875   0.179922          0.561186   \n",
       "1           3.202313   0.276526  1.171875   0.201972          0.547812   \n",
       "2           3.521882   0.210984  0.820312   0.264922          0.509015   \n",
       "3           3.346236   0.225823  0.703125   0.270707          0.496051   \n",
       "4           3.314565   0.305998  0.820312   0.264097          0.496697   \n",
       "..               ...        ...       ...        ...               ...   \n",
       "634         2.072660   0.172052  0.703125   0.204028          0.577570   \n",
       "635         2.118019   0.236408  0.703125   0.221044          0.560362   \n",
       "636         2.040094   0.235822  0.585938   0.157667          0.514717   \n",
       "637         2.457446   0.322034  0.468750   0.150582          0.458477   \n",
       "638         2.228730   0.095918  0.468750   0.102976          0.493830   \n",
       "\n",
       "     acc_power_ratio  \n",
       "0           0.014196  \n",
       "1           0.015343  \n",
       "2           0.042583  \n",
       "3           0.034215  \n",
       "4           0.056204  \n",
       "..               ...  \n",
       "634         0.060486  \n",
       "635         0.053120  \n",
       "636         0.058853  \n",
       "637         0.061269  \n",
       "638         0.105554  \n",
       "\n",
       "[639 rows x 12 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from paradigma.config import HeartRateConfig\n",
    "from paradigma.pipelines.heart_rate_pipeline import extract_signal_quality_features\n",
    "\n",
    "ppg_config = HeartRateConfig('ppg')\n",
    "acc_config = HeartRateConfig('imu')\n",
    "\n",
    "print(\"The default window length for the signal quality feature extraction is set to\", ppg_config.window_length_s, \"seconds.\")\n",
    "print(\"The default step size for the signal quality feature extraction is set to\", ppg_config.window_step_length_s, \"seconds.\")\n",
    "\n",
    "df_features = extract_signal_quality_features(\n",
    "    df_ppg=df_ppg_proc,\n",
    "    df_acc=df_acc_proc,\n",
    "    ppg_config=ppg_config, \n",
    "    acc_config=acc_config, \n",
    ")\n",
    "\n",
    "df_features\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Signal quality classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A trained logistic classifier is used to predict PPG signal quality and returns the `pred_sqa_proba`, which is the posterior probability of a PPG window to look like the typical PPG morphology (higher probability indicates toward the typical PPG morphology). The relative power feature from the accelerometer is compared to a threshold for periodic artifacts and therefore `pred_sqa_acc_label` is used to return a label indicating predicted periodic motion artifacts (label 0) or no periodic motion artifacts (label 1).\n",
    "\n",
    "The classification step is implemented in `signal_quality_classification` (documentation can be found [here](https://github.com/biomarkersParkinson/paradigma/blob/main/src/paradigma/pipelines/heart_rate_pipeline.py#:~:text=signal_quality_classification))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>pred_sqa_proba</th>\n",
       "      <th>pred_sqa_acc_label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.006031</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.011725</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>0.068063</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.080808</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.074591</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>634</th>\n",
       "      <td>634.0</td>\n",
       "      <td>0.001387</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>635</th>\n",
       "      <td>635.0</td>\n",
       "      <td>0.001570</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>636</th>\n",
       "      <td>636.0</td>\n",
       "      <td>0.000380</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>637</th>\n",
       "      <td>637.0</td>\n",
       "      <td>0.000406</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>638</th>\n",
       "      <td>638.0</td>\n",
       "      <td>0.000004</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>639 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      time  pred_sqa_proba  pred_sqa_acc_label\n",
       "0      0.0        0.006031                   1\n",
       "1      1.0        0.011725                   1\n",
       "2      2.0        0.068063                   1\n",
       "3      3.0        0.080808                   1\n",
       "4      4.0        0.074591                   1\n",
       "..     ...             ...                 ...\n",
       "634  634.0        0.001387                   1\n",
       "635  635.0        0.001570                   1\n",
       "636  636.0        0.000380                   1\n",
       "637  637.0        0.000406                   1\n",
       "638  638.0        0.000004                   1\n",
       "\n",
       "[639 rows x 3 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from importlib.resources import files\n",
    "from paradigma.pipelines.heart_rate_pipeline import signal_quality_classification\n",
    "\n",
    "ppg_quality_classifier_package_filename = 'ppg_quality_clf_package.pkl'\n",
    "full_path_to_classifier_package = files('paradigma') / 'assets' / ppg_quality_classifier_package_filename\n",
    "\n",
    "config = HeartRateConfig()\n",
    "\n",
    "df_sqa = signal_quality_classification(\n",
    "    df=df_features, \n",
    "    config=config, \n",
    "    full_path_to_classifier_package=full_path_to_classifier_package\n",
    ")\n",
    "\n",
    "df_sqa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Heart rate estimation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For heart rate estimation, we extract segments of `config.tfd_length` using [estimate_heart_rate](https://github.com/biomarkersParkinson/paradigma/blob/main/src/paradigma/pipelines/heart_rate_pipeline.py#:~:text=estimate_heart_rate). We calculate the smoothed-pseudo Wigner-Ville Distribution (SPWVD) to obtain the frequency content of the PPG signal over time. We extract for every timestamp in the SPWVD the frequency with the highest power. For every non-overlapping 2 s window we average the corresponding frequencies to obtain a heart rate per window.\n",
    "\n",
    "Note: for the test data we set the tfd_length to 10 s instead of the default of 30 s, because the small PPP test data doesn't have 30 s of consecutive high-quality PPG data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The standard default minimal window length for the heart rate extraction is set to 30 seconds.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>heart_rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>56.0</td>\n",
       "      <td>86.404715</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>58.0</td>\n",
       "      <td>86.640472</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60.0</td>\n",
       "      <td>86.345776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>62.0</td>\n",
       "      <td>84.872299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>64.0</td>\n",
       "      <td>84.872299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>66.0</td>\n",
       "      <td>84.194499</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   time  heart_rate\n",
       "0  56.0   86.404715\n",
       "1  58.0   86.640472\n",
       "2  60.0   86.345776\n",
       "3  62.0   84.872299\n",
       "4  64.0   84.872299\n",
       "5  66.0   84.194499"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from paradigma.pipelines.heart_rate_pipeline import estimate_heart_rate\n",
    "\n",
    "print(\"The standard default minimal window length for the heart rate extraction is set to\", config.tfd_length, \"seconds.\")\n",
    "# set the minimal window length for the heart rate extraction to 10 seconds instead of default of 30 seconds.\n",
    "config.set_tfd_length(10)       \n",
    "\n",
    "df_hr = estimate_heart_rate(\n",
    "    df_sqa=df_sqa, \n",
    "    df_ppg_preprocessed=df_ppg_proc, \n",
    "    config=config\n",
    ")\n",
    "\n",
    "df_hr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Heart rate aggregation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final step is to aggregate all 2 s heart rate estimates using [aggregate_heart_rate](https://github.com/biomarkersParkinson/paradigma/blob/main/src/paradigma/pipelines/heart_rate_pipeline.py#:~:text=aggregate_heart_rate). In the current example, the mode and 99th percentile are calculated. We hypothesize that the mode gives representation of the resting heart rate while the 99th percentile indicates the maximum heart rate. In Parkinson's disease, we expect that these two measures could reflect autonomic (dys)functioning. The `nr_hr_est` in the metadata indicates based on how many 2 s windows these aggregates are determined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'hr_aggregates': {'99p_heart_rate': 86.62868369351669,\n",
      "                   'mode_heart_rate': 84.8722986247544},\n",
      " 'metadata': {'nr_hr_est': 6}}\n"
     ]
    }
   ],
   "source": [
    "import pprint\n",
    "from paradigma.pipelines.heart_rate_pipeline import aggregate_heart_rate\n",
    "\n",
    "hr_values = df_hr['heart_rate'].values\n",
    "df_hr_agg = aggregate_heart_rate(\n",
    "    hr_values=hr_values, \n",
    "    aggregates = ['mode', '99p']\n",
    ")\n",
    "\n",
    "pprint.pprint(df_hr_agg)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "paradigma-Fn6RLG4_-py3.11",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}