nuluh/thesis
1
0
Fork 0
Code Issues 47 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Add 'labels' Column to Time-domain Feature Extraction DataFrame #10

New Issue
Closed
opened 2024-08-20 06:11:35 +00:00 by nuluh · 0 comments
nuluh commented 2024-08-20 06:11:35 +00:00 (Migrated from github.com)
Copy Link

Description

We need to include a 'labels' column in our feature extraction DataFrame to facilitate downstream tasks such as training machine learning models. Currently, the DataFrame generated by the build_features function only contains extracted features, and lacks any form of labeling for these features.

Expected Behavior

The DataFrame should include a 'labels' column where each row corresponds to the label of the dataset from which the features were extracted.

Current Behavior

The current implementation generates a DataFrame without a 'labels' column. This absence prevents us from using the DataFrame directly in supervised learning scenarios. Here's the DataFrame features head looks like:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 50 entries, 0 to 49
Data columns (total 14 columns):
 #   Column              Non-Null Count  Dtype  
---  ------              --------------  -----  
 0   Mean                50 non-null     float64
 1   Max                 50 non-null     float64
 2   Peak (Pm)           50 non-null     float64
 3   Peak-to-Peak (Pk)   50 non-null     float64
 4   RMS                 50 non-null     float64
 5   Variance            50 non-null     float64
 6   Standard Deviation  50 non-null     float64
 7   Power               50 non-null     float64
 8   Crest Factor        50 non-null     float64
 9   Form Factor         50 non-null     float64
 10  Pulse Indicator     50 non-null     float64
 11  Margin              50 non-null     float64
 12  Kurtosis            50 non-null     float64
 13  Skewness            50 non-null     float64
dtypes: float64(14)
memory usage: 5.6 KB

Possible Solution

Modify the build_features function to append a 'labels' column to the DataFrame. This column could be derived from the directory names or a specific pattern in the filenames, depending on how our data is structured.

Steps to Reproduce

  1. Run the build_features script with the current setup.
  2. Observe that the resulting DataFrame saved as combined_features.csv does not include a 'labels' column.

Context (Environment)

The feature extraction is crucial for our model training, and having labeled data is necessary for any supervised learning approach. The absence of labels impacts our ability to directly train models using the extracted features.

Possible Implementation

Here's a potential snippet for how we might modify the build_features function:

def build_features(input_dir, output_dir):
    all_features = []  # List to store all feature dicts
    for nth_damage in os.listdir(input_dir):
        nth_damage_path = os.path.join(input_dir, nth_damage)
        if os.path.isdir(nth_damage_path):
            for nth_test in os.listdir(nth_damage_path):
                nth_test_path = os.path.join(nth_damage_path, nth_test)
                if nth_test_path.endswith('.csv'):  # Ensure it's a CSV file
                    features = ExtractTimeFeatures(nth_test_path)
                    features['label'] = nth_damage  # assuming label is the directory name
                    all_features.append(features)
    df = pd.DataFrame(all_features)
    output_file_path = os.path.join(output_dir, 'combined_features.csv')
    df.to_csv(output_file_path, index=False)
## Description We need to include a 'labels' column in our feature extraction DataFrame to facilitate downstream tasks such as training machine learning models. Currently, the DataFrame generated by the `build_features` function only contains extracted features, and lacks any form of labeling for these features. ## Expected Behavior The DataFrame should include a 'labels' column where each row corresponds to the label of the dataset from which the features were extracted. ## Current Behavior The current implementation generates a DataFrame without a 'labels' column. This absence prevents us from using the DataFrame directly in supervised learning scenarios. Here's the DataFrame features head looks like: ``` <class 'pandas.core.frame.DataFrame'> RangeIndex: 50 entries, 0 to 49 Data columns (total 14 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Mean 50 non-null float64 1 Max 50 non-null float64 2 Peak (Pm) 50 non-null float64 3 Peak-to-Peak (Pk) 50 non-null float64 4 RMS 50 non-null float64 5 Variance 50 non-null float64 6 Standard Deviation 50 non-null float64 7 Power 50 non-null float64 8 Crest Factor 50 non-null float64 9 Form Factor 50 non-null float64 10 Pulse Indicator 50 non-null float64 11 Margin 50 non-null float64 12 Kurtosis 50 non-null float64 13 Skewness 50 non-null float64 dtypes: float64(14) memory usage: 5.6 KB ``` ## Possible Solution Modify the `build_features` function to append a 'labels' column to the DataFrame. This column could be derived from the directory names or a specific pattern in the filenames, depending on how our data is structured. ## Steps to Reproduce 1. Run the `build_features` script with the current setup. 2. Observe that the resulting DataFrame saved as `combined_features.csv` does not include a 'labels' column. ## Context (Environment) The feature extraction is crucial for our model training, and having labeled data is necessary for any supervised learning approach. The absence of labels impacts our ability to directly train models using the extracted features. ## Possible Implementation Here's a potential snippet for how we might modify the `build_features` function: ```python def build_features(input_dir, output_dir): all_features = [] # List to store all feature dicts for nth_damage in os.listdir(input_dir): nth_damage_path = os.path.join(input_dir, nth_damage) if os.path.isdir(nth_damage_path): for nth_test in os.listdir(nth_damage_path): nth_test_path = os.path.join(nth_damage_path, nth_test) if nth_test_path.endswith('.csv'): # Ensure it's a CSV file features = ExtractTimeFeatures(nth_test_path) features['label'] = nth_damage # assuming label is the directory name all_features.append(features) df = pd.DataFrame(all_features) output_file_path = os.path.join(output_dir, 'combined_features.csv') df.to_csv(output_file_path, index=False)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something isn't working
 critical
Block Progress
 documentation
Improvements or additions to documentation
 documentclass
Improvements or additions to LaTeX documentclass
 duplicate
This issue or pull request already exists
 enhancement
New feature or request
 good first issue
Good for newcomers
 help wanted
Extra attention is needed
 invalid
This doesn't seem right
 latex
Related to the writings
 question
Further information is requested
 wontfix
This will not be worked on
No Label bug enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
nuluh
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: nuluh/thesis#10
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?