Basic reading and writing (pandas)
These are some examples on how to read and write TSDF data into and from a numpy array, using the tsdf library.
import tsdf
Load some data
# Load the metadata
metadata_path = "data/ppp_format_meta.json"
metadata_dict = tsdf.load_metadata_from_path(metadata_path)
metadata_time = metadata_dict["ppp_format_time.bin"]
metadata_samples = metadata_dict["ppp_format_samples.bin"]
# Load the data
df = tsdf.load_dataframe_from_binaries([metadata_time, metadata_samples], tsdf.constants.ConcatenationType.columns)
# Print some info
print(f"Data shape:\t {df.shape}")
df.head()
Data shape: (17, 7)
| time | acceleration_x | acceleration_y | acceleration_z | rotation_x | rotation_y | rotation_z | |
|---|---|---|---|---|---|---|---|
| 0 | 0.374540 | 6 | 1 | 3 | 2 | 5 | 3 |
| 1 | 0.950714 | 7 | 9 | 1 | 4 | 6 | 0 |
| 2 | 0.731994 | 9 | 2 | 2 | 6 | 7 | 1 |
| 3 | 0.598659 | 7 | 4 | 3 | 2 | 7 | 2 |
| 4 | 0.156019 | 7 | 4 | 2 | 0 | 5 | 5 |
Perform basic data processing
scale_factors = getattr(metadata_samples, "scale_factors")
print(f"Scale factors: {scale_factors}")
# Perform scaling
df_scaled = df.copy()
df_scaled.iloc[:, 1:] = df_scaled.iloc[:, 1:].multiply(scale_factors, axis=1)
# Print some info
print(f"Data shape:\t\t {df.shape}")
df_scaled.head()
Scale factors: [0.00469378, 0.00469378, 0.00469378, 0.06097561, 0.06097561, 0.06097561]
Data shape: (17, 7)
/tmp/ipykernel_2035/4007094660.py:6: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '0 0.028163
1 0.032856
2 0.042244
3 0.032856
4 0.032856
5 0.018775
6 0.023469
7 0.000000
8 0.028163
9 0.009388
10 0.042244
11 0.042244
12 0.037550
13 0.037550
14 0.014081
15 0.037550
16 0.042244
Name: acceleration_x, dtype: float64' has dtype incompatible with int16, please explicitly cast to a compatible dtype first.
df_scaled.iloc[:, 1:] = df_scaled.iloc[:, 1:].multiply(scale_factors, axis=1)
/tmp/ipykernel_2035/4007094660.py:6: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '0 0.004694
1 0.042244
2 0.009388
3 0.018775
4 0.018775
5 0.000000
6 0.004694
7 0.014081
8 0.014081
9 0.042244
10 0.028163
11 0.028163
12 0.009388
13 0.028163
14 0.028163
15 0.018775
16 0.037550
Name: acceleration_y, dtype: float64' has dtype incompatible with int16, please explicitly cast to a compatible dtype first.
df_scaled.iloc[:, 1:] = df_scaled.iloc[:, 1:].multiply(scale_factors, axis=1)
/tmp/ipykernel_2035/4007094660.py:6: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '0 0.014081
1 0.004694
2 0.009388
3 0.014081
4 0.009388
5 0.004694
6 0.014081
7 0.009388
8 0.037550
9 0.023469
10 0.004694
11 0.018775
12 0.018775
13 0.037550
14 0.032856
15 0.014081
16 0.042244
Name: acceleration_z, dtype: float64' has dtype incompatible with int16, please explicitly cast to a compatible dtype first.
df_scaled.iloc[:, 1:] = df_scaled.iloc[:, 1:].multiply(scale_factors, axis=1)
/tmp/ipykernel_2035/4007094660.py:6: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '0 0.121951
1 0.243902
2 0.365854
3 0.121951
4 0.000000
5 0.426829
6 0.060976
7 0.548780
8 0.426829
9 0.121951
10 0.060976
11 0.182927
12 0.243902
13 0.243902
14 0.060976
15 0.182927
16 0.365854
Name: rotation_x, dtype: float64' has dtype incompatible with int16, please explicitly cast to a compatible dtype first.
df_scaled.iloc[:, 1:] = df_scaled.iloc[:, 1:].multiply(scale_factors, axis=1)
/tmp/ipykernel_2035/4007094660.py:6: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '0 0.304878
1 0.365854
2 0.426829
3 0.426829
4 0.304878
5 0.243902
6 0.243902
7 0.304878
8 0.000000
9 0.548780
10 0.060976
11 0.304878
12 0.304878
13 0.426829
14 0.304878
15 0.060976
16 0.243902
Name: rotation_y, dtype: float64' has dtype incompatible with int16, please explicitly cast to a compatible dtype first.
df_scaled.iloc[:, 1:] = df_scaled.iloc[:, 1:].multiply(scale_factors, axis=1)
/tmp/ipykernel_2035/4007094660.py:6: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '0 0.182927
1 0.000000
2 0.060976
3 0.121951
4 0.304878
5 0.243902
6 0.182927
7 0.304878
8 0.426829
9 0.487805
10 0.121951
11 0.487805
12 0.121951
13 0.487805
14 0.182927
15 0.182927
16 0.060976
Name: rotation_z, dtype: float64' has dtype incompatible with int16, please explicitly cast to a compatible dtype first.
df_scaled.iloc[:, 1:] = df_scaled.iloc[:, 1:].multiply(scale_factors, axis=1)
| time | acceleration_x | acceleration_y | acceleration_z | rotation_x | rotation_y | rotation_z | |
|---|---|---|---|---|---|---|---|
| 0 | 0.374540 | 0.028163 | 0.004694 | 0.014081 | 0.121951 | 0.304878 | 0.182927 |
| 1 | 0.950714 | 0.032856 | 0.042244 | 0.004694 | 0.243902 | 0.365854 | 0.000000 |
| 2 | 0.731994 | 0.042244 | 0.009388 | 0.009388 | 0.365854 | 0.426829 | 0.060976 |
| 3 | 0.598659 | 0.032856 | 0.018775 | 0.014081 | 0.121951 | 0.426829 | 0.121951 |
| 4 | 0.156019 | 0.032856 | 0.018775 | 0.009388 | 0.000000 | 0.304878 | 0.304878 |
Write the processed data
Write the processed data in binary format. The call updates the metadata object with the data attributes of the dataframe.
# The new name of the file
output_bin_filename = "tmp_pandas_example_processed.bin"
metadata_samples.__setattr__("file_name", output_bin_filename)
# Write the data to a new binary file
tsdf.write_dataframe_to_binaries("data", df_scaled, [metadata_time, metadata_samples])
print(f"File written to data/{output_bin_filename}")
File written to data/tmp_pandas_example_processed.bin
Write the TSDF metadata file
# Write new metadata file
output_meta_filename = "tmp_pandas_example_processed_meta.json"
tsdf.write_metadata([metadata_time, metadata_samples], output_meta_filename)
print(f"File written to data/{output_meta_filename}")
File written to data/tmp_pandas_example_processed_meta.json