- Modify `build_features` function to support iterative processing across nested directories, enhancing the system's ability to handle larger datasets and varied input structures.
- Replace direct usage of `FeatureExtractor` class with `ExtractTimeFeatures` function, which now acts as a wrapper to include this class, facilitating streamlined integration and maintenance of feature extraction processes.
- Implement `extract_numbers` function using regex to parse filenames and extract numeric identifiers, used for labels when training with SVM
- Switch output from `.npz` to `.csv` format in `build_features`, offering better compatibility with data analysis tools and readability.
- Update documentation and comments within the code to reflect changes in functionality and usage of the new feature extraction setup.
Closes#4
Update the README.md file in the data/processed directory to provide clearer instructions on how to load the data from the desired Dx_TESTy.csv file. This change enhances the usability of the data files for analysis.
- Create a Python script to generate CSV files in a structured folder hierarchy under `data/processed` with specific damage levels and tests.
- Add a `.gitignore` file to exclude CSV files from Git tracking, enhancing data privacy and reducing repository size.
- Include a `README.md` in the `data` directory to clearly document the directory structure, file content, and their intended use for clarity and better usability.
Closes#7
This commit adds a new script `start.sh` that automates the process of processing raw data, building features, and training a model. The script uses Python scripts from the `src` directory to perform these tasks. The processed data is saved in the `data/processed` directory, the feature matrix is saved in the `data/features` directory, and the trained model is saved in the `models` directory.
The purpose of these changes is to streamline the data processing and model training workflow, making it easier to reproduce and iterate on the results.
This commit adds code to the `03_feature_extraction.ipynb` notebook to print time-domain features. The features include mean, max, peak, peak-to-peak, RMS, variance, standard deviation, power, crest factor, form factor, pulse indicator, margin, kurtosis, and skewness. The features are calculated using the `FeatureExtractor` class and displayed in a pandas DataFrame.
This commit adds the "python.analysis.extraPaths" setting to the VSCode settings.json file. The setting includes the "./code/src/features" directory as an additional path for Python analysis. This change improves the analysis capabilities within the VSCode environment.
Closes#3
Introduce a new testing script that generates mockup data and applies the FeatureExtractor class to calculate and display features. This test script assists in verifying the functionality of the feature extraction methods with controlled input data.
- Implement FeatureExtractor class in time_domain_features.py for calculating statistical features from dataset columns.
- Create build_features.py script to automate feature extraction from processed data and save results in a structured format.
- Adjust build_features.py to read processed data, utilize FeatureExtractor, and save feature matrix.
This update supports enhanced analysis capabilities within the thesis-project structure, allowing for more sophisticated data processing and model training stages.
Closes#1
The code changes add a new file `time_domain_features.py` that contains a `FeatureExtractor` class. This class calculates various time domain features for a given dataset. The features include mean, max, peak, peak-to-peak, RMS, variance, standard deviation, power, crest factor, form factor, pulse indicator, margin, kurtosis, and skewness.
The class takes a file path as input and reads the data from a CSV file. It assumes the data to analyze is in the first column. The calculated features are stored in a dictionary.
The commit message suggests that the purpose of the changes is to add a new class for time domain feature extraction.
