Merge pull request #36 from nuluh/35-bug-oserror-for-non-existent-folder-when-running-convertpy

fix(data): ensure output directories are created before saving files

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,115 @@
+name: Bug Report
+description: Report a bug or unexpected behavior
+title: "[BUG] "
+labels: ["bug"]
+assignees:
+  - ${{github.actor}}
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to fill out this bug report!
+        
+  - type: textarea
+    id: description
+    attributes:
+      label: Bug Description
+      description: A clear and concise description of what the bug is
+      placeholder: When I run the script, it crashes when processing large datasets...
+    validations:
+      required: true
+      
+  - type: textarea
+    id: reproduction
+    attributes:
+      label: Steps to Reproduce
+      description: Steps to reproduce the behavior
+      placeholder: |
+        1. Go to notebook '...'
+        2. Run cell #...
+        3. See error
+    validations:
+      required: true
+      
+  - type: textarea
+    id: expected
+    attributes:
+      label: Expected Behavior
+      description: What did you expect to happen?
+      placeholder: The analysis should complete successfully and generate the visualization
+    validations:
+      required: true
+      
+  - type: textarea
+    id: actual
+    attributes:
+      label: Actual Behavior
+      description: What actually happened?
+      placeholder: The script crashes with a memory error after processing 1000 samples
+    validations:
+      required: true
+      
+  - type: textarea
+    id: logs
+    attributes:
+      label: Error Logs
+      description: Paste any relevant logs or error messages
+      render: shell
+      placeholder: |
+        Traceback (most recent call last):
+          File "script.py", line 42, in <module>
+            main()
+          File "script.py", line 28, in main
+            process_data(data)
+        MemoryError: ...
+    validations:
+      required: false
+      
+  - type: dropdown
+    id: component
+    attributes:
+      label: Component
+      description: Which part of the thesis project is affected?
+      options:
+        - LaTeX Document
+        - Python Source Code
+        - Jupyter Notebook
+        - Data Processing
+        - ML Model
+        - Visualization
+        - Build/Environment
+    validations:
+      required: true
+      
+  - type: input
+    id: version
+    attributes:
+      label: Version/Commit
+      description: Which version or commit hash are you using?
+      placeholder: v0.2.3 or 8d5b9a7
+    validations:
+      required: true
+      
+  - type: textarea
+    id: environment
+    attributes:
+      label: Environment
+      description: Information about your environment
+      placeholder: |
+        - OS: [e.g. Ubuntu 22.04]
+        - Python: [e.g. 3.9.5]
+        - Relevant packages and versions:
+          - numpy: 1.22.3
+          - scikit-learn: 1.0.2
+          - tensorflow: 2.9.1
+    validations:
+      required: false
+      
+  - type: textarea
+    id: additional
+    attributes:
+      label: Additional Context
+      description: Any other context or screenshots about the problem
+      placeholder: Add any other context about the problem here...
+    validations:
+      required: false

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,12 @@
+blank_issues_enabled: false
+contact_links:
+  - name: Documentation
+    url: ../docs/README.md
+    about: Check the documentation before creating an issue
+
+# Template configurations
+templates:
+  - name: bug_report.yml
+  - name: feature_request.yml
+  - name: experiment.yml
+  - name: documentation.yml

.github/ISSUE_TEMPLATE/documentation.yml

@@ -0,0 +1,116 @@
+name: Documentation
+description: Improvements or additions to documentation
+title: "[DOC] "
+labels: ["documentation"]
+assignees:
+  - ${{github.actor}}
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Use this template for documentation-related tasks for your thesis project.
+        
+  - type: dropdown
+    id: doc_type
+    attributes:
+      label: Documentation Type
+      description: What type of documentation is this issue about?
+      options:
+        - Thesis Chapter/Section
+        - Code Documentation
+        - Experiment Documentation
+        - README/Project Documentation
+        - Literature Review
+        - Methodology Description
+        - Results Analysis
+        - API Reference
+    validations:
+      required: true
+      
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: Describe what needs to be documented
+      placeholder: Need to document the data preprocessing pipeline including all transformation steps and rationale
+    validations:
+      required: true
+      
+  - type: textarea
+    id: current_state
+    attributes:
+      label: Current State
+      description: What's the current state of the documentation (if any)?
+      placeholder: Currently there are some comments in the code but no comprehensive documentation of the preprocessing steps
+    validations:
+      required: false
+      
+  - type: textarea
+    id: proposed_changes
+    attributes:
+      label: Proposed Changes
+      description: What specific documentation changes do you want to make?
+      placeholder: |
+        1. Create a dedicated markdown file describing each preprocessing step
+        2. Add docstrings to all preprocessing functions
+        3. Create a diagram showing the data flow
+        4. Document parameter choices and their justification
+    validations:
+      required: true
+      
+  - type: input
+    id: location
+    attributes:
+      label: Documentation Location
+      description: Where will this documentation be stored?
+      placeholder: docs/data_preprocessing.md or src/preprocessing/README.md
+    validations:
+      required: true
+      
+  - type: dropdown
+    id: priority
+    attributes:
+      label: Priority
+      description: How important is this documentation?
+      options:
+        - Critical (required for thesis)
+        - High (important for understanding)
+        - Medium (helpful but not urgent)
+        - Low (nice to have)
+    validations:
+      required: true
+      
+  - type: dropdown
+    id: audience
+    attributes:
+      label: Target Audience
+      description: Who is the primary audience for this documentation?
+      options:
+        - Thesis Committee/Reviewers
+        - Future Self
+        - Other Researchers
+        - Technical Readers
+        - Non-technical Readers
+        - Multiple Audiences
+    validations:
+      required: true
+      
+  - type: textarea
+    id: references
+    attributes:
+      label: References
+      description: Any papers, documentation or other materials related to this documentation task
+      placeholder: |
+        - Smith et al. (2022). "Best practices in machine learning documentation"
+        - Code in src/preprocessing/normalize.py
+    validations:
+      required: false
+      
+  - type: textarea
+    id: notes
+    attributes:
+      label: Additional Notes
+      description: Any other relevant information
+      placeholder: This documentation will be referenced in Chapter 3 of the thesis
+    validations:
+      required: false

.github/ISSUE_TEMPLATE/experiment.yml

@@ -0,0 +1,124 @@
+# .github/ISSUE_TEMPLATE/experiment.yml
+name: Experiment
+description: Document a new ML experiment
+title: "[EXP] "
+labels: ["experiment"]
+assignees:
+  - ${{github.actor}}
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Use this template to document a new experiment for your thesis.
+        
+  - type: textarea
+    id: hypothesis
+    attributes:
+      label: Hypothesis
+      description: What is the hypothesis you're testing with this experiment?
+      placeholder: Using a deeper network with residual connections will improve accuracy on the imbalanced dataset without increasing overfitting
+    validations:
+      required: true
+      
+  - type: textarea
+    id: background
+    attributes:
+      label: Background & Motivation
+      description: Background context and why this experiment is important
+      placeholder: Previous experiments showed promising results but suffered from overfitting. Recent literature suggests that...
+    validations:
+      required: true
+      
+  - type: textarea
+    id: dataset
+    attributes:
+      label: Dataset
+      description: What data will you use for this experiment?
+      placeholder: |
+        - Dataset: MNIST with augmentation
+        - Preprocessing: Standardization + random rotation
+        - Train/Test Split: 80/20
+        - Validation strategy: 5-fold cross-validation
+    validations:
+      required: true
+      
+  - type: textarea
+    id: methodology
+    attributes:
+      label: Methodology
+      description: How will you conduct the experiment?
+      placeholder: |
+        1. Implement ResNet architecture with varying depths (18, 34, 50)
+        2. Train with early stopping (patience=10)
+        3. Compare against baseline CNN from experiment #23
+        4. Analyze learning curves and performance metrics
+    validations:
+      required: true
+      
+  - type: textarea
+    id: parameters
+    attributes:
+      label: Parameters & Hyperparameters
+      description: List the key parameters for this experiment
+      placeholder: |
+        - Learning rate: 0.001 with Adam optimizer
+        - Batch size: 64
+        - Epochs: Max 100 with early stopping
+        - Dropout rate: 0.3
+        - L2 regularization: 1e-4
+    validations:
+      required: true
+      
+  - type: textarea
+    id: metrics
+    attributes:
+      label: Evaluation Metrics
+      description: How will you evaluate the results?
+      placeholder: |
+        - Accuracy
+        - F1-score (macro-averaged)
+        - ROC-AUC
+        - Training vs. validation loss curves
+        - Inference time
+    validations:
+      required: true
+      
+  - type: input
+    id: notebook
+    attributes:
+      label: Notebook Location
+      description: Where will the experiment notebook be stored?
+      placeholder: notebooks/experiment_resnet_comparison.ipynb
+    validations:
+      required: false
+      
+  - type: textarea
+    id: dependencies
+    attributes:
+      label: Dependencies
+      description: What other issues or tasks does this experiment depend on?
+      placeholder: |
+        - Depends on issue #42 (Data preprocessing pipeline)
+        - Requires completion of issue #51 (Baseline model)
+    validations:
+      required: false
+      
+  - type: textarea
+    id: references
+    attributes:
+      label: References
+      description: Any papers, documentation or other materials relevant to this experiment
+      placeholder: |
+        - He et al. (2016). "Deep Residual Learning for Image Recognition"
+        - My previous experiment #23 (baseline CNN)
+    validations:
+      required: false
+      
+  - type: textarea
+    id: notes
+    attributes:
+      label: Additional Notes
+      description: Any other relevant information
+      placeholder: This experiment may require significant GPU resources. Expected runtime is ~3 hours on Tesla V100.
+    validations:
+      required: false

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,99 @@
+# .github/ISSUE_TEMPLATE/feature_request.yml
+name: Feature Request
+description: Suggest a new feature or enhancement
+title: "[FEAT] "
+labels: ["enhancement"]
+assignees:
+  - ${{github.actor}}
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to propose a new feature!
+        
+  - type: textarea
+    id: problem
+    attributes:
+      label: Problem Statement
+      description: What problem are you trying to solve with this feature?
+      placeholder: I'm frustrated when trying to analyze different model results because I need to manually compare them...
+    validations:
+      required: true
+      
+  - type: textarea
+    id: solution
+    attributes:
+      label: Proposed Solution
+      description: Describe the solution you'd like to implement
+      placeholder: Create a visualization utility that automatically compares results across multiple models and experiments
+    validations:
+      required: true
+      
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: Alternatives Considered
+      description: Describe alternatives you've considered
+      placeholder: I considered using an external tool, but integrating directly would provide better workflow
+    validations:
+      required: false
+      
+  - type: dropdown
+    id: component
+    attributes:
+      label: Component
+      description: Which part of the thesis project would this feature affect?
+      options:
+        - LaTeX Document
+        - Python Source Code
+        - Jupyter Notebook
+        - Data Processing
+        - ML Model
+        - Visualization
+        - Build/Environment
+        - Multiple Components
+    validations:
+      required: true
+      
+  - type: dropdown
+    id: priority
+    attributes:
+      label: Priority
+      description: How important is this feature for your thesis progression?
+      options:
+        - Critical (blocks progress)
+        - High (significantly improves workflow)
+        - Medium (nice to have)
+        - Low (minor improvement)
+    validations:
+      required: true
+      
+  - type: textarea
+    id: implementation
+    attributes:
+      label: Implementation Ideas
+      description: Any initial thoughts on how to implement this feature?
+      placeholder: |
+        - Could use matplotlib's subplot feature
+        - Would need to standardize the model output format
+        - Should include statistical significance tests
+    validations:
+      required: false
+      
+  - type: textarea
+    id: benefits
+    attributes:
+      label: Expected Benefits
+      description: How will this feature benefit your thesis work?
+      placeholder: This will save time in analysis and provide more consistent comparisons across experiments
+    validations:
+      required: true
+      
+  - type: textarea
+    id: additional
+    attributes:
+      label: Additional Context
+      description: Any other context, screenshots, or reference material
+      placeholder: Here's a paper that uses a similar approach...
+    validations:
+      required: false

LICENSE

@@ -0,0 +1,7 @@
+Copyright 2024 Rifqi D. Panuluh
+
+All Rights Reserved.
+
+This repository is for viewing purposes only. No part of this repository, including but not limited to the code, files, and documentation, may be copied, reproduced, modified, or distributed in any form or by any means without the prior written permission of the copyright holder.
+
+Unauthorized use, distribution, or modification of this repository may result in legal action.

README.md

@@ -0,0 +1,18 @@
+## Summary
+
+This repository contains the work related to my thesis, which focuses on damage localization prediction. The research explores the application of machine learning techniques to structural health monitoring.
+
+**Note:** This repository does not contain the secondary data used in the analysis. The code is designed to work with data from the [QUGS (Qatar University Grandstand Simulator)](https://www.structuralvibration.com/benchmark/qugs/) dataset, which is not included here.
+
+The repository is private and access is restricted only to those who have been given explicit permission by the owner. Access is provided solely for the purpose of brief review or seeking technical guidance.
+
+## Restrictions
+
+- **No Derivative Works or Cloning:** Any form of copying, cloning, or creating derivative works based on this repository is strictly prohibited.
+- **Limited Access:** Use beyond brief review or collaboration is not allowed without prior permission from the owner.
+
+---
+
+All contents of this repository, including the thesis idea, code, and associated data, are copyrighted © 2024 by Rifqi Panuluh. Unauthorized use or duplication is prohibited.
+
+[LICENSE](https://github.com/nuluh/thesis?tab=License-1-ov-file#readme)

data/QUGS/convert.py

@@ -36,6 +36,7 @@ def create_damage_files(base_path, output_base, prefix):
             print(f"Creating {output_file_1} from taking zzz{prefix}D{file_index}.TXT")
             print("Taking datetime column on index 0...")
             print(f"Taking `{top_sensor}`...")
+            os.makedirs(os.path.dirname(output_file_1), exist_ok=True)
             df[['Time', top_sensor]].to_csv(output_file_1, index=False)
             print(Fore.GREEN + "Done")
 
@@ -44,6 +45,7 @@ def create_damage_files(base_path, output_base, prefix):
             print(f"Creating {output_file_2} from taking zzz{prefix}D{file_index}.TXT")
             print("Taking datetime column on index 0...")
             print(f"Taking `{bottom_sensor}`...")
+            os.makedirs(os.path.dirname(output_file_2), exist_ok=True)
             df[['Time', bottom_sensor]].to_csv(output_file_2, index=False)
             print(Fore.GREEN + "Done")
             print("---")
@@ -58,8 +60,8 @@ def main():
     prefix = sys.argv[3]  # Define output directory
 
     # Create output folders if they don't exist
-    for i in range(1, 5):
+    # for i in range(1, 7):
-        os.makedirs(os.path.join(output_base, f'DAMAGE_{i}'), exist_ok=True)
+    #     os.makedirs(os.path.join(output_base, f'DAMAGE_{i}'), exist_ok=True)
 
     create_damage_files(base_path, output_base, prefix)
     print(Fore.YELLOW + Style.BRIGHT + "All files have been created successfully.")

docs/CONTRIBUTING.md

@@ -0,0 +1,66 @@
+This document outlines the process for developing and contributing to my own thesis project. By following these guidelines, this will ensure consistent quality and maintain a clear development history.
+
+## Development Workflow
+
+### 1. Issue Creation
+Before working on any new feature, experiment, or bug fix:
+- Create a GitHub issue using the appropriate template
+- Assign it to myself
+- Add relevant labels
+- Link it to the project board if applicable
+
+### 2. Branching Strategy
+Use the following branch naming convention:
+- `feature/<issue-number>-short-description`
+- `bugfix/<issue-number>-short-description`
+- `experiment/<issue-number>-short-description`
+- `doc/<issue-number>-short-description`
+
+Always branch from `main` for new features/experiments.
+
+### 3. Development Process
+- Make regular, atomic commits following the commit message template
+- Include the issue number in commit messages (e.g., "#42")
+- Push changes at the end of each work session
+
+### 4. Code Quality
+- Follow PEP 8 guidelines for Python code
+- Document functions with docstrings
+- Maintain test coverage for custom functions
+- Keep notebooks clean and well-documented
+
+### 5. Pull Requests
+Even working alone, use PRs for significant changes:
+- Create a PR from your feature branch to `main`
+- Reference the issue(s) it resolves
+- Include a summary of changes
+- Self-review the PR before merging
+
+### 6. Versioning
+Follow semantic versioning:
+- Major version: Significant thesis milestones or structural changes
+- Minor version: New experiments, features, or chapters
+- Patch version: Bug fixes and minor improvements
+
+### 7. Documentation
+Update documentation with each significant change:
+- Keep README current
+- Update function documentation
+- Maintain clear experiment descriptions in notebooks
+- Record significant decisions and findings
+
+## LaTeX Guidelines
+- Use consistent citation style
+- Break long sections into multiple files
+- Use meaningful label names for cross-references
+- Consider using version-control friendly LaTeX practices (one sentence per line)
+
+## Experiment Tracking
+For each experiment:
+- Create an issue documenting the experiment design
+- Reference related papers and previous experiments
+- Document parameters and results in the notebook
+- Summarize findings in the issue before closing
+
+## Commit Categories
+Use the categories defined in the commit template to clearly classify changes.