.. _dynamic-data-preparation:

================
Data Preparation
================

This page explains how to format your time-series data for use with NeurEco Dynamic models, whether for training or evaluation.

.. contents:: On this page
   :local:
   :depth: 2

Data Structure Overview
=======================

Dynamic models work with two arrays per time series, and each array begins with a time column (column 0):

1. **Excitation (input) array**: column 0 is time, the remaining columns are the input signals driving the system
2. **Output array**: column 0 is time, the remaining columns are the target system responses

Both arrays of a sequence must carry the **same time column**. Multiple sequences (experiments, simulations, or recordings) can be combined for training.

.. note::

   **Deprecated separate-time form.** Passing the time as a separate array
   (``train_time_list`` and its ``valid_``/``test_`` counterparts, with the
   excitation/output arrays then holding pure excitations/outputs) is still
   supported for backward compatibility but emits a ``DeprecatedParameterWarning``.
   Prefer placing the time in the first column of each excitation and output array.

Critical Requirements
=====================

.. warning::

   Violating these requirements will cause errors or incorrect results.

Requirements for All Dynamic Models
-----------------------------------

1. **Matching row counts**: For each sequence, the excitation and output arrays must have the **same number of rows** and share the **same time column** (column 0)

2. **Consistent feature counts**: All excitation arrays must have the same number of columns; all output arrays must have the same number of columns (the leading time column included)

3. **Corresponding indices**: Lists must maintain correspondence — the i-th excitation and output arrays must belong to the same sequence

Template-Specific Requirements
------------------------------

.. list-table::
   :header-rows: 1
   :widths: 30 35 35

   * - Requirement
     - ODE
     - Convolutional Dynamic
   * - Time stamp ordering
     - **Required** — the time column (column 0) must be sorted in strictly increasing order (no repeated time stamps)
     - **Required** — the time column (column 0) must be sorted in strictly increasing order (no repeated time stamps)
   * - Time step uniformity
     - **Not required** — can handle non-uniform time steps (as long as they are strictly increasing)
     - **Required** — must be uniform within and across sequences
   * - Time step consistency
     - Can vary between sequences
     - Must be identical across all sequences

What Can Vary Between Sequences
-------------------------------

- Duration (number of time steps)
- Start time
- Time step spacing (ODE only)

What Must Be Constant
---------------------

- Number of input features
- Number of output features
- Time step (Convolutional Dynamic only)

Data Formats by Interface
=========================

.. tabs::

   .. tab:: GUI

      **File Formats**

      The GUI accepts data in the following formats:

      - **CSV** with ``;`` or ``,`` separator
      - **NumPy** ``.npy`` files
      - **MATLAB** ``.mat`` files

      Allowed data types: ``int``, ``float``, ``double``

      **Input (Excitation) Files**

      Each input file contains a table where:

      - **First column**: Time variable (uniformly spaced for Convolutional Dynamic; for ODE any spacing is allowed, provided the values are sorted in strictly increasing order)
      - **Remaining columns**: Input features (excitations)
      - **Each row**: Time point and corresponding input values
      - CSV files may include a header row

      *Example CSV structure:*

      .. code-block:: text

         time;force;temperature;pressure
         0.00;1.2;25.3;101.5
         0.01;1.4;25.4;101.6
         0.02;1.3;25.5;101.7
         ...

      **Output Files**

      Each output file contains a table where:

      - **First column**: Time variable (must match the corresponding input file)
      - **Remaining columns**: Output features
      - **Each row**: Time point and corresponding output values
      - CSV files may include a header row

      *Example CSV structure:*

      .. code-block:: text

         time;position;velocity
         0.00;0.0;0.0
         0.01;0.012;0.24
         0.02;0.048;0.47
         ...

      **Multiple Sequences**

      When your data comes from multiple experiments:

      - Provide multiple input and output file pairs
      - Maintain correspondence between input and output files
      - For Convolutional Dynamic: all files must use the same time step
      - For ODE: time steps can vary between sequences
      - Different sequences can have different durations and start times

   .. tab:: Python API

      **Data Format**

      The API expects data as **lists of NumPy arrays**. Each element in a list represents a separate time series.

      You need two parallel lists, and every array carries the time in its first column:

      - **Excitation (input) list**: arrays whose column 0 is time, remaining columns are input signals
      - **Output list**: arrays whose column 0 is time, remaining columns are target outputs

      Supported NumPy data types: ``int``, ``float``, ``double``

      **The Time Column**

      The first column (column 0) of every excitation and output array holds the
      time stamps. It must be:

      - **Sorted in strictly increasing order** — no repeated time stamps (e.g. ``0.1, 0.2, 0.5`` is valid; ``0.1, 0.5, 0.2`` is not)
      - **Uniformly spaced** for Convolutional Dynamic; ODE accepts non-uniform spacing as long as the time stamps are strictly increasing
      - **Identical** in the excitation array and the output array of the same sequence

      The ordering must be *strict* because the time step between two consecutive
      samples is used as a denominator during integration: equal (or decreasing)
      time stamps would give a zero or negative :math:`dt`, which is not allowed.

      .. code-block:: python

         import numpy as np

         # Uniform spacing (required for Convolutional Dynamic, works for ODE)
         time = np.linspace(0, 10, 1001)  # 0.00, 0.01, 0.02, ..., 10.00
         time = np.arange(0, 10.01, 0.01)

         # Non-uniform but strictly increasing spacing (ODE only)
         time = np.array([0.0, 0.1, 0.15, 0.3, 0.5, 1.0, 2.0, 5.0])

      **Excitation (Input) Arrays**

      Each excitation array is 2-dimensional:

      - **Rows** = time steps
      - **Column 0** = time
      - **Remaining columns** = input features

      .. code-block:: python

         # 1001 time steps, time + 3 input features
         time = np.linspace(0, 10, 1001)
         features = np.random.randn(1001, 3)
         excitation = np.column_stack([time, features])  # shape (1001, 4)

         # Row 0: time[0] and the input values at time[0]
         # Row 1: time[1] and the input values at time[1]
         # etc.

      **Output Arrays**

      Each output array is 2-dimensional:

      - **Rows** = time steps (must match the excitation array)
      - **Column 0** = time (the same time column as the excitation array)
      - **Remaining columns** = output features

      .. code-block:: python

         # 1001 time steps, time + 2 output features
         targets = np.random.randn(1001, 2)
         output = np.column_stack([time, targets])  # shape (1001, 3)

      **Multiple Sequences Example**

      .. code-block:: python

         import numpy as np

         # Common time step for all sequences
         dt = 0.01

         # Experiment 1: 10 seconds
         time_1 = np.arange(0, 10, dt)
         exc_1 = np.column_stack([time_1, np.random.randn(len(time_1), 3)])
         out_1 = np.column_stack([time_1, np.random.randn(len(time_1), 2)])

         # Experiment 2: 5 seconds
         time_2 = np.arange(0, 5, dt)
         exc_2 = np.column_stack([time_2, np.random.randn(len(time_2), 3)])
         out_2 = np.column_stack([time_2, np.random.randn(len(time_2), 2)])

         # Experiment 3: 8 seconds, starting at t=2
         time_3 = np.arange(2, 10, dt)
         exc_3 = np.column_stack([time_3, np.random.randn(len(time_3), 3)])
         out_3 = np.column_stack([time_3, np.random.randn(len(time_3), 2)])

         # Each array: column 0 = time, remaining columns = excitations / outputs.
         train_exc_list = [exc_1, exc_2, exc_3]
         train_out_list = [out_1, out_2, out_3]

   .. tab:: CLI

      **File Formats**

      The CLI accepts the same formats as the GUI:

      - **CSV** with ``;`` or ``,`` separator
      - **NumPy** ``.npy`` files
      - **MATLAB** ``.mat`` files

      **Configuration File**

      Specify data files in the configuration JSON:

      .. code-block:: json

         {
             "neurecoODE_build": {
                 "exc_filenames": [
                     "data/experiment1_input.csv",
                     "data/experiment2_input.csv"
                 ],
                 "output_filenames": [
                     "data/experiment1_output.csv",
                     "data/experiment2_output.csv"
                 ]
             }
         }

      File indices must correspond (first input with first output, etc.). A
      Convolutional Dynamic configuration (``neurecoConvolutiveDynamic_build``) lists
      the same files under ``data``, as ``training.inputs`` and ``training.targets``
      (see :ref:`cli-reference`).

Data Normalization
==================

NeurEco automatically normalizes your data before training. Understanding normalization options helps you get better results.

How Normalization Works
-----------------------

NeurEco applies the formula:

.. math::

   x_{normalized} = \frac{x - shift}{scale}

Where ``shift`` and ``scale`` are computed based on your chosen methods.

.. note::

   ODE and Convolutional Dynamic templates have different default normalization options optimized for their respective use cases.

Shift Methods
-------------

.. list-table::
   :header-rows: 1
   :widths: 20 40 40

   * - Method
     - Description
     - Default for
   * - ``mean``
     - Shift by the mean value (centers data around 0)
     - ODE
   * - ``min``
     - Shift by the minimum value
     - Convolutional Dynamic
   * - ``none``
     - No shifting applied
     - —

Scale Methods
-------------

.. list-table::
   :header-rows: 1
   :widths: 20 40 40

   * - Method
     - Description
     - Default for
   * - ``l2``
     - Scale by the L2 norm
     - ODE
   * - ``minmax``
     - Scale to [0, 1] range based on min/max values
     - Convolutional Dynamic
   * - ``max``
     - Scale by the maximum absolute value
     - —
   * - ``std``
     - Scale by the standard deviation
     - —
   * - ``none``
     - No scaling applied
     - —

Per-Feature Normalization
-------------------------

The **normalize per feature** option controls whether normalization is applied independently to each feature or globally across all features.

**Set to True when:**

- Features represent different physical quantities (e.g., temperature and pressure)
- Features have vastly different scales or units
- You want each feature to contribute equally

**Set to False when:**

- Features represent the same quantity (e.g., temperatures at different locations)
- You're working with field data where relative magnitudes matter

.. tabs::

   .. tab:: GUI

      In the **Build** panel, under **Normalization**:

      - Select shift and scale methods from dropdowns
      - Check/uncheck **Normalize per feature** for inputs and outputs

   .. tab:: Python API (ODE)

      .. code-block:: python

         # Each array: column 0 = time, remaining columns = excitations / outputs.
         config = Dynamic.ODEBuildConfig(
             train_exc_list=train_exc_list,
             train_out_list=train_out_list,
             # Input normalization
             inputs_shifting="mean",
             inputs_scaling="l2",
             inputs_normalize_per_feature=True,
             # Output normalization
             outputs_shifting="mean",
             outputs_scaling="l2",
             outputs_normalize_per_feature=True,
         )

   .. tab:: Python API (Convolutional Dynamic)

      .. code-block:: python

         # Each array: column 0 = time, remaining columns = excitations / outputs.
         config = Dynamic.ConvolutiveDynamicBuildConfig(
             train_exc_list=train_exc_list,
             train_out_list=train_out_list,
             # Input normalization
             inputs_shifting="min",
             inputs_scaling="minmax",
             inputs_normalize_per_feature=True,
             # Output normalization
             outputs_shifting="min",
             outputs_scaling="minmax",
             outputs_normalize_per_feature=True,
         )

   .. tab:: CLI (ODE)

      .. code-block:: json

         {
             "neurecoODE_build": {
                 "settings": {
                     "input_normalization": {
                         "shift_type": "mean",
                         "scale_type": "l2",
                         "normalize_per_feature": true
                     },
                     "output_normalization": {
                         "shift_type": "mean",
                         "scale_type": "l2",
                         "normalize_per_feature": true
                     }
                 }
             }
         }

   .. tab:: CLI (Convolutional Dynamic)

      .. code-block:: json

         {
             "neurecoConvolutiveDynamic_build": {
                 "normalizers": {
                     "inputs": {
                         "shift_type": "min",
                         "scale_type": "minmax",
                         "normalize_per_feature": true
                     },
                     "targets": {
                         "shift_type": "min",
                         "scale_type": "minmax",
                         "normalize_per_feature": true
                     }
                 }
             }
         }

.. tip::

   If the default normalization doesn't suit your needs, you can normalize your data yourself before passing it to NeurEco and set normalization to ``none``.

Complete Python Example
=======================

.. code-block:: python

   import numpy as np
   from NeurEco import NeurEcoDynamic as Dynamic

   # Define common time step
   dt = 0.001  # 1 ms

   # Experiment 1: 5 seconds, 3 inputs, 2 outputs (time in column 0)
   t1 = np.arange(0, 5, dt)
   x1 = np.column_stack([
       t1,                             # Column 0: time
       np.sin(2 * np.pi * 1.0 * t1),   # 1 Hz sine wave
       np.sin(2 * np.pi * 2.5 * t1),   # 2.5 Hz sine wave
       np.random.randn(len(t1)) * 0.1  # Noise
   ])
   y1 = np.column_stack([
       t1,                                  # Column 0: time
       np.sin(2 * np.pi * 1.0 * t1 + 0.5),  # Phase-shifted response
       np.cos(2 * np.pi * 2.5 * t1)          # Another response
   ])

   # Experiment 2: 3 seconds
   t2 = np.arange(0, 3, dt)
   x2 = np.column_stack([
       t2,                             # Column 0: time
       np.sin(2 * np.pi * 1.5 * t2),
       np.sin(2 * np.pi * 3.0 * t2),
       np.random.randn(len(t2)) * 0.1
   ])
   y2 = np.column_stack([
       t2,                                  # Column 0: time
       np.sin(2 * np.pi * 1.5 * t2 + 0.5),
       np.cos(2 * np.pi * 3.0 * t2)
   ])

   # Each array: column 0 = time, remaining columns = excitations / outputs.
   train_exc_list = [x1, x2]
   train_out_list = [y1, y2]

   # Verify shapes
   for i, (x, y) in enumerate(zip(train_exc_list, train_out_list)):
       print(f"Sequence {i}: excitation={x.shape}, output={y.shape}")

   # Output:
   # Sequence 0: excitation=(5000, 4), output=(5000, 3)
   # Sequence 1: excitation=(3000, 4), output=(3000, 3)

   # Now ready to use with NeurEco ODE or Convolutional Dynamic
   model = Dynamic.ODE()
   config = Dynamic.ODEBuildConfig(
       train_exc_list=train_exc_list,
       train_out_list=train_out_list,
       # ... other configuration options
   )

Next Steps
==========

- :ref:`Build ODE models <ode-building>`
- :ref:`Build Convolutional Dynamic models <convolutive-building>`