Evaluating Models#

This page covers how to evaluate your NeurEco Tabular models to assess their performance on new data.

On this page

Why Evaluate?#

Evaluation helps you:

  • Verify accuracy: Check how well the model predicts on unseen data

  • Compare checkpoints: Find the best model from training history

  • Detect overfitting: Identify if the model memorized training data rather than learning patterns

  • Validate deployment readiness: Ensure the model meets your accuracy requirements

Evaluation Workflow#

1. Access the Evaluation panel

Switch to the Evaluation tab in NeurEco GUI. This panel becomes available after the first checkpoint is saved during training.

2. Load evaluation data

  • Select evaluation data from the dropdown list (if already loaded)

  • Or click + to add new evaluation files

3. Provide reference outputs (optional)

To compare predictions against known values:

  • Load the corresponding output file

  • NeurEco will compute and display error metrics

4. Select checkpoint (optional)

Use the checkpoint slider to evaluate different models from the training history.

5. View results

  • See predictions in the results panel

  • View error metrics if reference outputs were provided

  • Export results to CSV, NumPy, or MATLAB format

Template-Specific Evaluation#

Regression#

For regression models, evaluation produces continuous predicted values.

Primary metric — L2 Relative Error:

NeurEco uses the L2 relative error during training, making it the primary metric for evaluating model quality. The Regressor model provides a built-in method to compute it:

predictions = model.evaluate(x_test)

# L2 relative error (primary metric used by NeurEco)
l2_relative_error = model.compute_error(predictions, y_test)
print(f"L2 Relative Error: {l2_relative_error:.6f}")

Additional metrics:

  • L2 Relative Error (mean-shifted) — Useful when data has a significant offset from zero

  • Mean Absolute Error (MAE) — Average absolute difference between predictions and actual values

  • R² score — Coefficient of determination

# L2 relative error with mean shift (useful for data with offset)
pred_shifted = predictions - np.mean(predictions)
ref_shifted = y_test - np.mean(y_test)
l2_relative_shifted = model.compute_error(pred_shifted, ref_shifted)
print(f"L2 Relative Error (mean-shifted): {l2_relative_shifted:.6f}")

# Other common metrics
mae = np.mean(np.abs(predictions - y_test))
ss_res = np.sum((y_test - predictions) ** 2)
ss_tot = np.sum((y_test - np.mean(y_test)) ** 2)
r2 = 1 - (ss_res / ss_tot)

print(f"MAE: {mae}")
print(f"R² Score: {r2:.4f}")

Classification#

For classification models, evaluation produces probability-like values for each class.

Computing accuracy:

The Classifier model provides a built-in method to compute accuracy:

predictions = model.evaluate(x_test)

# Compute accuracy
accuracy = model.compute_error(predictions, y_test)
print(f"Accuracy: {accuracy * 100:.2f}%")

Confusion matrix:

The Classifier model provides a built-in method to plot the confusion matrix:

# Plot confusion matrix (also returns the matrix as np.ndarray)
cm = model.plot_confusion_matrix(
    prediction=predictions,
    reference=y_test,
    class_names=["Class A", "Class B", "Class C"],
    show_percentages=True,
    title="Model Performance",
    save_path="confusion_matrix.png"  # Optional: save to file
)

Compression#

For compression models, evaluation reconstructs the input data.

Reconstruction quality:

The Compressor model provides a built-in method to compute the L2 relative reconstruction error:

# Complete model: input -> compressed -> reconstructed
reconstructed = model.evaluate(x_test)

# Compute reconstruction error (L2 relative error)
reconstruction_error = model.compute_error(reconstructed, x_test)
print(f"Reconstruction L2 Relative Error: {reconstruction_error:.6f}")

# Per-sample error
per_sample_error = np.mean((x_test - reconstructed) ** 2, axis=1)
print(f"Max sample error: {np.max(per_sample_error)}")

Evaluating compression ratio:

# Load encoder separately
encoder = Tabular.Compressor()
encoder.load("./model/encoder.ednn")

compressed = encoder.evaluate(x_test)

original_dim = x_test.shape[1]
compressed_dim = compressed.shape[1]
ratio = original_dim / compressed_dim

print(f"Original dimensions: {original_dim}")
print(f"Compressed dimensions: {compressed_dim}")
print(f"Compression ratio: {ratio:.1f}x")

Comparing Checkpoints#

To find the best model from training history:

See also

Using the test data to choose a checkpoint is safe with NeurEco — the test set is never used during the build. See checkpoint-test-data.

Use the checkpoint slider in the Evaluation panel to browse models and compare their performance on your test data.

Exporting Evaluation Results#

In the Evaluation panel, click the export button to save predictions in:

  • CSV format

  • NumPy format (.npy)

  • MATLAB format (.mat)

Check Also export inputs/outputs files to save the original input and reference files next to the predictions.

Best Practices#

  1. Always use separate test data: Never evaluate on training or validation data

  2. Compare multiple checkpoints: The final model isn’t always the best

  3. Use appropriate metrics: L2 relative error for regression, accuracy for classification, reconstruction error for compression

  4. Check for edge cases: Evaluate on boundary conditions and unusual inputs

  5. Validate generalization: Test on data that represents real-world usage

Next Steps#