.. _tabular-data-preparation: ================ Data Preparation ================ This page explains how to prepare your data for NeurEco Tabular models. Proper data preparation is essential for building accurate models. .. contents:: On this page :local: :depth: 2 Supported File Formats ====================== NeurEco accepts data in the following formats: .. list-table:: :header-rows: 1 :widths: 20 30 50 * - Format - Extension - Notes * - CSV - ``.csv`` - Delimiter: ``;`` or ``,`` (auto-detected). Optional header row. * - NumPy - ``.npy`` - Standard NumPy binary format * - MATLAB - ``.mat`` - MATLAB data file format **Supported data types:** ``int``, ``float``, ``double`` Data Structure ============== Input Data ---------- Input data must be a 2D array with shape ``(n_samples, n_input_features)``: - **Rows:** Individual samples/observations - **Columns:** Input features/variables **Example:** For 1000 samples with 5 input features, shape is ``(1000, 5)`` Output Data ----------- Output data requirements vary by template: **Regression & Classification:** - Shape: ``(n_samples, n_output_features)`` - Must have the same number of samples as input data **Compression:** - No separate output files required - The model learns to reconstruct the input (autoencoder) Template-Specific Requirements ============================== Regression ---------- - **Inputs:** Continuous or categorical numerical values - **Outputs:** Continuous numerical values - Both inputs and outputs should be numerical arrays **Example structure:** .. code-block:: text x_train.csv (1000 samples, 5 features): temp_diff;wind_dir;wind_speed;sunshine;humidity 12.5;180;5.2;0.8;65 8.3;90;3.1;0.2;78 ... y_train.csv (1000 samples, 1 output): energy_consumption 245.6 189.2 ... Classification -------------- - **Inputs:** Same as Regression - **Outputs:** Must be **one-hot encoded** **One-hot encoding:** Each output row contains exactly one ``1`` (indicating the class) and ``0`` elsewhere. **Example for 3-class classification:** .. code-block:: text y_train.csv (one-hot encoded): class_A;class_B;class_C 1;0;0 0;1;0 0;0;1 1;0;0 ... .. warning:: NeurEco expects one-hot encoded outputs for classification. If your data has integer labels (0, 1, 2, ...), you must convert them to one-hot format before training. Compression ----------- - **Inputs:** The data you want to compress - **Outputs:** Not required (the target is to reconstruct the input) You can provide multiple input files if your data is organized in chunks. Loading Data ============ .. tabs:: .. tab:: GUI **Loading Training Data** 1. In the **Settings** tab, click on **Training data** 2. Browse and select your input file(s) 3. If using Regression or Classification, also select your output file(s) **Loading Validation Data (Optional)** - Either let NeurEco automatically split your training data (using **Validation percentage**) - Or provide separate validation files by clicking **Validation data** **Loading Test Data (Optional)** - Click **Test data** to add files for final evaluation .. tab:: Python API **Loading from NumPy arrays:** .. code-block:: python import numpy as np from NeurEco import NeurEcoTabular as Tabular # Load your data x_train = np.genfromtxt("x_train.csv", delimiter=";", skip_header=True) y_train = np.genfromtxt("y_train.csv", delimiter=";", skip_header=True) # Ensure correct shape for outputs y_train = np.reshape(y_train, (-1, 1)) # If single output column # For Regression config = Tabular.RegressorBuildConfig( input_data=x_train, output_data=y_train, # ... other parameters ) # For Classification config = Tabular.ClassifierBuildConfig( input_data=x_train, output_data=y_train, # Must be one-hot encoded # ... other parameters ) # For Compression (no output_data needed) config = Tabular.CompressorBuildConfig( input_data=x_train, # ... other parameters ) **Loading from files directly:** .. code-block:: python # You can also pass file paths instead of arrays config = Tabular.RegressorBuildConfig( input_data="path/to/x_train.csv", output_data="path/to/y_train.csv", # ... other parameters ) .. tab:: CLI In your configuration file, specify the data paths: .. code-block:: json { "neurecoRegression_build": { "exc_filenames": [ "x_train.csv" ], "output_filenames": [ "y_train.csv" ], "test_exc_filenames": [ "x_test.csv" ], "test_output_filenames": [ "y_test.csv" ] } } For Compression, use the root key ``neurecoCompression_build`` and omit the output file fields. Data Normalization ================== NeurEco automatically normalizes your data before training. Understanding normalization options helps you get better results. How Normalization Works ----------------------- NeurEco applies the formula: .. math:: x_{normalized} = \frac{x - shift}{scale} Where ``shift`` and ``scale`` are computed based on your chosen methods. Shift Methods ------------- .. list-table:: :header-rows: 1 :widths: 20 80 * - Method - Description * - ``mean`` - Shift by the mean value (centers data around 0) * - ``min_centered`` - Shift so the minimum value becomes -1 * - ``auto`` - Automatically chooses: ``mean`` if data mean is far from 0, otherwise ``none`` * - ``none`` - No shifting applied Scale Methods ------------- .. list-table:: :header-rows: 1 :widths: 20 80 * - Method - Description * - ``max`` - Scale by the maximum absolute value * - ``max_centered`` - Scale so the range becomes [-1, 1] * - ``std`` - Scale by the standard deviation * - ``auto`` - Automatically chooses based on data characteristics * - ``none`` - No scaling applied Per-Feature Normalization ------------------------- The **normalize per feature** option controls whether normalization is applied independently to each feature or globally across all features. **Set to True when:** - Features represent different physical quantities (e.g., temperature and pressure) - You want each feature to have equal importance **Set to False when:** - Features represent the same quantity (e.g., temperatures at different locations) - You're working with field data .. tabs:: .. tab:: GUI In the **Settings** tab, under **Normalization**: - Select shift and scale methods from dropdowns - Check/uncheck **Normalize per feature** for inputs and outputs .. tab:: Python API .. code-block:: python config = Tabular.RegressorBuildConfig( input_data=x_train, output_data=y_train, # Input normalization inputs_shifting="min_centered", inputs_scaling="max_centered", inputs_normalize_per_feature=True, # Output normalization outputs_shifting="auto", outputs_scaling="auto", outputs_normalize_per_feature=False, ) .. tab:: CLI .. code-block:: json { "neurecoRegression_build": { "input_normalization": { "shift_type": "min_centered", "scale_type": "max_centered", "normalize_per_feature": true }, "output_normalization": { "shift_type": "auto", "scale_type": "auto", "normalize_per_feature": false } } } .. tip:: If the default normalization doesn't suit your needs, you can normalize your data yourself before passing it to NeurEco and set normalization to ``none``. Validation and Test Data ======================== Training, Validation, and Test Sets ----------------------------------- - **Training data:** Used to train the model - **Validation data:** Used during training to detect overfitting and stop enrichment - **Test data:** Used after training to evaluate final model performance Automatic Splitting ------------------- If you don't provide separate validation data, NeurEco automatically splits your training data: - Default: 33.33% for validation, 66.67% for training - Adjust via the **Validation percentage** parameter .. tabs:: .. tab:: GUI In **Settings**, adjust the **Validation percentage** slider or enter a value. .. tab:: Python API .. code-block:: python config = Tabular.RegressorBuildConfig( input_data=x_train, output_data=y_train, valid_percentage=25.0, # 25% for validation ) .. tab:: CLI .. code-block:: json { "neurecoRegression_build": { "DevSettings": { "valid_percentage": 25.0 } } } Providing Separate Validation Data ---------------------------------- For more control, provide your own validation set: .. tabs:: .. tab:: GUI Click **Validation data** and select your validation files. .. tab:: Python API .. code-block:: python config = Tabular.RegressorBuildConfig( input_data=x_train, output_data=y_train, validation_input_data=x_valid, validation_output_data=y_valid, ) .. tab:: CLI .. code-block:: json { "neurecoRegression_build": { "exc_filenames": ["x_train.csv"], "output_filenames": ["y_train.csv"], "validation_exc_filenames": ["x_valid.csv"], "validation_output_filenames": ["y_valid.csv"] } } Best Practices ============== 1. **Check data quality:** Remove duplicates, handle missing values, and identify outliers before training 2. **Ensure consistent shapes:** Input and output files must have the same number of samples 3. **Use appropriate normalization:** Match normalization settings to your data characteristics (see :ref:`best-practices`) 4. **Choose validation data carefully:** The validation set should represent real-world usage scenarios 5. **Reserve test data:** Keep a portion of data completely separate for final evaluation Next Steps ========== Once your data is prepared, proceed to :ref:`building models `.