Basic reading and writing (pandas)¶
These are some examples on how to read and write TSDF data into and from a numpy array, using the tsdf
library.
In [1]:
Copied!
import tsdf
import tsdf
Load some data¶
In [2]:
Copied!
# Load the metadata
metadata_path = "data/ppp_format_meta.json"
metadata_dict = tsdf.load_metadata_from_path(metadata_path)
metadata_time = metadata_dict["ppp_format_time.bin"]
metadata_samples = metadata_dict["ppp_format_samples.bin"]
# Load the data
df = tsdf.load_binaries_to_dataframe([metadata_time, metadata_samples], tsdf.constants.ConcatenationType.columns)
# Print some info
print(f"Data shape:\t {df.shape}")
df.head()
# Load the metadata
metadata_path = "data/ppp_format_meta.json"
metadata_dict = tsdf.load_metadata_from_path(metadata_path)
metadata_time = metadata_dict["ppp_format_time.bin"]
metadata_samples = metadata_dict["ppp_format_samples.bin"]
# Load the data
df = tsdf.load_binaries_to_dataframe([metadata_time, metadata_samples], tsdf.constants.ConcatenationType.columns)
# Print some info
print(f"Data shape:\t {df.shape}")
df.head()
Data shape: (17, 7)
Out[2]:
time | acceleration_x | acceleration_y | acceleration_z | rotation_x | rotation_y | rotation_z | |
---|---|---|---|---|---|---|---|
0 | 0.374540 | 6 | 1 | 3 | 2 | 5 | 3 |
1 | 0.950714 | 7 | 9 | 1 | 4 | 6 | 0 |
2 | 0.731994 | 9 | 2 | 2 | 6 | 7 | 1 |
3 | 0.598659 | 7 | 4 | 3 | 2 | 7 | 2 |
4 | 0.156019 | 7 | 4 | 2 | 0 | 5 | 5 |
Perform basic data processing¶
In [3]:
Copied!
scale_factors = getattr(metadata_samples, "scale_factors")
print(f"Scale factors: {scale_factors}")
# Perform scaling
df_scaled = df.copy()
df_scaled.iloc[:, 1:] = df_scaled.iloc[:, 1:].multiply(scale_factors, axis=1)
# Print some info
print(f"Data shape:\t\t {df.shape}")
df_scaled.head()
scale_factors = getattr(metadata_samples, "scale_factors")
print(f"Scale factors: {scale_factors}")
# Perform scaling
df_scaled = df.copy()
df_scaled.iloc[:, 1:] = df_scaled.iloc[:, 1:].multiply(scale_factors, axis=1)
# Print some info
print(f"Data shape:\t\t {df.shape}")
df_scaled.head()
Scale factors: [0.00469378, 0.00469378, 0.00469378, 0.06097561, 0.06097561, 0.06097561] Data shape: (17, 7)
Out[3]:
time | acceleration_x | acceleration_y | acceleration_z | rotation_x | rotation_y | rotation_z | |
---|---|---|---|---|---|---|---|
0 | 0.374540 | 0.028163 | 0.004694 | 0.014081 | 0.121951 | 0.304878 | 0.182927 |
1 | 0.950714 | 0.032856 | 0.042244 | 0.004694 | 0.243902 | 0.365854 | 0.000000 |
2 | 0.731994 | 0.042244 | 0.009388 | 0.009388 | 0.365854 | 0.426829 | 0.060976 |
3 | 0.598659 | 0.032856 | 0.018775 | 0.014081 | 0.121951 | 0.426829 | 0.121951 |
4 | 0.156019 | 0.032856 | 0.018775 | 0.009388 | 0.000000 | 0.304878 | 0.304878 |
Write the processed data¶
Write the processed data in binary format. The call updates the metadata object with the data attributes of the dataframe.
In [4]:
Copied!
# The new name of the file
output_bin_filename = "tmp_pandas_example_processed.bin"
metadata_samples.__setattr__("file_name", output_bin_filename)
# Write the data to a new binary file
tsdf.write_dataframe_to_binaries("data", df_scaled, [metadata_time, metadata_samples])
print(f"File written to data/{output_bin_filename}")
# The new name of the file
output_bin_filename = "tmp_pandas_example_processed.bin"
metadata_samples.__setattr__("file_name", output_bin_filename)
# Write the data to a new binary file
tsdf.write_dataframe_to_binaries("data", df_scaled, [metadata_time, metadata_samples])
print(f"File written to data/{output_bin_filename}")
File written to data/tmp_pandas_example_processed.bin
Write the TSDF metadata file¶
In [5]:
Copied!
# Write new metadata file
output_meta_filename = "tmp_pandas_example_processed_meta.json"
tsdf.write_metadata([metadata_time, metadata_samples], output_meta_filename)
print(f"File written to data/{output_meta_filename}")
# Write new metadata file
output_meta_filename = "tmp_pandas_example_processed_meta.json"
tsdf.write_metadata([metadata_time, metadata_samples], output_meta_filename)
print(f"File written to data/{output_meta_filename}")
File written to data/tmp_pandas_example_processed_meta.json