.. _tabular-evaluating-models:

=================
Evaluating Models
=================

This page covers how to evaluate your NeurEco Tabular models to assess their performance on new data.

.. contents:: On this page
   :local:
   :depth: 2

Why Evaluate?
=============

Evaluation helps you:

- **Verify accuracy:** Check how well the model predicts on unseen data
- **Compare checkpoints:** Find the best model from training history
- **Detect overfitting:** Identify if the model memorized training data rather than learning patterns
- **Validate deployment readiness:** Ensure the model meets your accuracy requirements

Evaluation Workflow
===================

.. tabs::

   .. tab:: GUI

      **1. Access the Evaluation panel**

      Switch to the **Evaluation** tab in NeurEco GUI. This panel becomes available after the first checkpoint is saved during training.

      **2. Load evaluation data**

      - Select evaluation data from the dropdown list (if already loaded)
      - Or click **+** to add new evaluation files

      **3. Provide reference outputs (optional)**

      To compare predictions against known values:

      - Load the corresponding output file
      - NeurEco will compute and display error metrics

      **4. Select checkpoint (optional)**

      Use the checkpoint slider to evaluate different models from the training history.

      **5. View results**

      - See predictions in the results panel
      - View error metrics if reference outputs were provided
      - Export results to CSV, NumPy, or MATLAB format

   .. tab:: Python API

      **Basic evaluation:**

      .. code-block:: python

         from NeurEco import NeurEcoTabular as Tabular
         import numpy as np

         # Load the model
         model = Tabular.Regressor()  # or Classifier(), Compressor()
         model.load("./model/regression.ednn")

         # Load test data
         x_test = np.genfromtxt("x_test.csv", delimiter=";", skip_header=True)

         # Evaluate
         predictions = model.evaluate(x_test)

      **With error computation:**

      .. code-block:: python

         # Load reference outputs
         y_test = np.genfromtxt("y_test.csv", delimiter=";", skip_header=True)
         y_test = np.reshape(y_test, (-1, 1))

         # Compute error
         error = model.compute_error(predictions, y_test)
         print(f"Evaluation error: {error}")

      **Evaluating a specific checkpoint:**

      .. code-block:: python

         # Load a specific checkpoint model (index 5 = "Checkpoint 6" in the log and the GUI)
         model = Tabular.Regressor()
         model.load_model_from_checkpoint("./model/regression.checkpoint", model_index=5)

         predictions = model.evaluate(x_test)

      ``load()`` only loads a saved model file. Indices start at 0: "Checkpoint N" in
      the build log and the GUI is ``model_index`` N - 1.

   .. tab:: CLI

      **1. Create an evaluation configuration file**

      Create ``eval.conf``:

      .. code-block:: json

         {
             "neurecoRegression_evaluate": {
                 "neureco_filename": "./model/regression.ednn",
                 "exc_filenames": [
                     "x_test.csv"
                 ],
                 "optional_output_reference": [
                     "y_test.csv"
                 ],
                 "write_model_output_to_directory": "./evaluation_results"
             }
         }

      **2. Run the evaluation**

      .. code-block:: shell

         neureco Regression evaluate eval.conf

      Results will be saved to the specified output directory. For a Classification
      or a Compression model, use ``neurecoClassification_evaluate`` or
      ``neurecoCompression_evaluate`` and the matching template in the command.

Template-Specific Evaluation
============================

Regression
----------

For regression models, evaluation produces continuous predicted values.

**Primary metric — L2 Relative Error:**

NeurEco uses the L2 relative error during training, making it the primary metric for evaluating model quality. The ``Regressor`` model provides a built-in method to compute it:

.. code-block:: python

   predictions = model.evaluate(x_test)

   # L2 relative error (primary metric used by NeurEco)
   l2_relative_error = model.compute_error(predictions, y_test)
   print(f"L2 Relative Error: {l2_relative_error:.6f}")

**Additional metrics:**

- **L2 Relative Error (mean-shifted)** — Useful when data has a significant offset from zero
- **Mean Absolute Error (MAE)** — Average absolute difference between predictions and actual values
- **R² score** — Coefficient of determination

.. code-block:: python

   # L2 relative error with mean shift (useful for data with offset)
   pred_shifted = predictions - np.mean(predictions)
   ref_shifted = y_test - np.mean(y_test)
   l2_relative_shifted = model.compute_error(pred_shifted, ref_shifted)
   print(f"L2 Relative Error (mean-shifted): {l2_relative_shifted:.6f}")

   # Other common metrics
   mae = np.mean(np.abs(predictions - y_test))
   ss_res = np.sum((y_test - predictions) ** 2)
   ss_tot = np.sum((y_test - np.mean(y_test)) ** 2)
   r2 = 1 - (ss_res / ss_tot)

   print(f"MAE: {mae}")
   print(f"R² Score: {r2:.4f}")

Classification
--------------

For classification models, evaluation produces probability-like values for each class.

**Computing accuracy:**

The ``Classifier`` model provides a built-in method to compute accuracy:

.. code-block:: python

   predictions = model.evaluate(x_test)

   # Compute accuracy
   accuracy = model.compute_error(predictions, y_test)
   print(f"Accuracy: {accuracy * 100:.2f}%")

**Confusion matrix:**

The ``Classifier`` model provides a built-in method to plot the confusion matrix:

.. code-block:: python

   # Plot confusion matrix (also returns the matrix as np.ndarray)
   cm = model.plot_confusion_matrix(
       prediction=predictions,
       reference=y_test,
       class_names=["Class A", "Class B", "Class C"],
       show_percentages=True,
       title="Model Performance",
       save_path="confusion_matrix.png"  # Optional: save to file
   )

Compression
-----------

For compression models, evaluation reconstructs the input data.

**Reconstruction quality:**

The ``Compressor`` model provides a built-in method to compute the L2 relative reconstruction error:

.. code-block:: python

   # Complete model: input -> compressed -> reconstructed
   reconstructed = model.evaluate(x_test)

   # Compute reconstruction error (L2 relative error)
   reconstruction_error = model.compute_error(reconstructed, x_test)
   print(f"Reconstruction L2 Relative Error: {reconstruction_error:.6f}")

   # Per-sample error
   per_sample_error = np.mean((x_test - reconstructed) ** 2, axis=1)
   print(f"Max sample error: {np.max(per_sample_error)}")

**Evaluating compression ratio:**

.. code-block:: python

   # Load encoder separately
   encoder = Tabular.Compressor()
   encoder.load("./model/encoder.ednn")

   compressed = encoder.evaluate(x_test)

   original_dim = x_test.shape[1]
   compressed_dim = compressed.shape[1]
   ratio = original_dim / compressed_dim

   print(f"Original dimensions: {original_dim}")
   print(f"Compressed dimensions: {compressed_dim}")
   print(f"Compression ratio: {ratio:.1f}x")

Comparing Checkpoints
=====================

To find the best model from training history:

.. seealso::

   Using the test data to choose a checkpoint is safe with NeurEco — the test set
   is never used during the build. See :ref:`checkpoint-test-data`.

.. tabs::

   .. tab:: GUI

      Use the checkpoint slider in the Evaluation panel to browse models and compare their performance on your test data.

   .. tab:: Python API

      .. code-block:: python

         checkpoint_path = "./model/regression.checkpoint"
         best_error = float("inf")
         best_index = 0

         model = Tabular.Regressor()
         n_models = model.get_number_of_networks_from_checkpoint(checkpoint_path)

         for index in range(n_models):
             model.load_model_from_checkpoint(checkpoint_path, model_index=index)

             predictions = model.evaluate(x_test)
             error = model.compute_error(predictions, y_test)

             if error < best_error:
                 best_error = error
                 best_index = index

         model.delete()

         # The build log and the GUI number checkpoints from 1
         print(f"Best model: Checkpoint {best_index + 1} (index {best_index}) with error: {best_error}")

Exporting Evaluation Results
============================

.. tabs::

   .. tab:: GUI

      In the Evaluation panel, click the export button to save predictions in:

      - CSV format
      - NumPy format (.npy)
      - MATLAB format (.mat)

      Check **Also export inputs/outputs files** to save the original input and reference files next to the predictions.

   .. tab:: Python API

      .. code-block:: python

         predictions = model.evaluate(x_test)

         # Save as CSV
         np.savetxt("predictions.csv", predictions, delimiter=";")

         # Save as NumPy
         np.save("predictions.npy", predictions)

         # Save as MATLAB
         from scipy.io import savemat
         savemat("predictions.mat", {"predictions": predictions})

   .. tab:: CLI

      Results are automatically saved to the directory specified in ``write_model_output_to_directory``.

Best Practices
==============

1. **Always use separate test data:** Never evaluate on training or validation data

2. **Compare multiple checkpoints:** The final model isn't always the best

3. **Use appropriate metrics:** L2 relative error for regression, accuracy for classification, reconstruction error for compression

4. **Check for edge cases:** Evaluate on boundary conditions and unusual inputs

5. **Validate generalization:** Test on data that represents real-world usage

Next Steps
==========

- :ref:`Export your model <tabular-exporting-models>` for deployment
- :ref:`Use advanced features <tabular-advanced-features>` like sensitivity analysis