Building Compression

This page covers building Tabular Compression models for dimensionality reduction and autoencoder applications.

When to Use Compression

Use the Compression template when:

• You want to reduce data dimensionality (compress high-dimensional data)

• You need to compress and reconstruct signals (autoencoder)

• You’re building IoT applications that require efficient data transmission

• You want to perform feature extraction or anomaly detection

How Compression Works

Unlike Regression and Classification, Compression doesn’t require separate output files. The model learns to:

1. Compress: Transform input data into a lower-dimensional representation

2. Decompress: Reconstruct the original input from the compressed representation

NeurEco produces three models:

• Complete model: Full encoder + decoder pipeline

• Compression model (encoder): Input -> compressed representation

• Decompression model (decoder): Compressed representation -> reconstructed output

This separation is particularly useful for IoT applications where:

• A small encoder runs on the device (sensor, microcontroller)

• A larger decoder runs on the server

Building a Compression Model

GUIPython APICLI

1. Create a new project

Launch NeurEco GUI and select Tabular Compression.

2. Configure settings

In the Settings tab:

• Training data: Select your input file(s) only (no output file needed)

• Compression tolerance: Maximum acceptable reconstruction error

• Minimum compression coefficients: Minimum latent space dimension

• Compressor/Decompressor size ratio: Balance between encoder and decoder size

3. Start the build

Click the BUILD button. NeurEco will:

• Build both encoder and decoder networks

• Optimize for reconstruction accuracy within tolerance

• Save all three models (complete, encoder, decoder)

Compression-Specific Parameters

These parameters are unique to Compression:

Parameter	Default	Description
compress_tolerance	0.01	Maximum acceptable reconstruction error (relative)
minimum_compression_coefficients	1	Minimum number of latent space dimensions
compress_decompress_size_ratio	1.0	Ratio of encoder size to decoder size (0 < ratio <= 1)
write_compression_model_to	None	Output path for encoder model (.ednn)
write_decompression_model_to	None	Output path for decoder model (.ednn)

Understanding Compression Parameters

Compression Tolerance

The compress_tolerance parameter defines the maximum acceptable reconstruction error:

• Lower values (e.g., 0.001): Higher accuracy, potentially larger latent space

• Higher values (e.g., 0.1): More aggressive compression, some accuracy loss

NeurEco will find the smallest latent dimension that achieves the specified tolerance.

Minimum Compression Coefficients

The minimum_compression_coefficients parameter sets a floor for the latent space dimension:

• Set higher if you know your data has a certain intrinsic dimensionality

• Useful when you need at least N compression coefficients for downstream tasks

Compressor/Decompressor Size Ratio

The compress_decompress_size_ratio parameter (range: 0 < ratio <= 1) controls the balance between encoder and decoder network sizes:

• ratio = 1.0: Balanced encoder and decoder

• ratio < 1.0: Smaller encoder, larger decoder

IoT Use Case:

For IoT applications, set ratio < 1.0 to create:

• A small encoder that runs on resource-constrained devices

• A larger decoder that runs on powerful servers

This reduces:

• Memory footprint on the device

• Computation time on the device

• Radio transmission power (smaller model, less transmitted data)

Using Compression Models

After building, you have three models:

Complete Model (end-to-end)

# Load complete model
model = Tabular.Compressor()
model.load("./model/compression_complete.ednn")

# Compress and decompress in one step
reconstructed = model.evaluate(x_test)

Encoder Only

# Load encoder
encoder = Tabular.Compressor()
encoder.load("./model/encoder.ednn")

# Get compressed representation
compressed = encoder.evaluate(x_test)
print(f"Compressed from {x_test.shape[1]} to {compressed.shape[1]} dimensions")

Decoder Only

# Load decoder
decoder = Tabular.Compressor()
decoder.load("./model/decoder.ednn")

# Reconstruct from compressed representation
reconstructed = decoder.evaluate(compressed)

Complete Parameter Reference

Parameter	Default	Description
input_data	Required	Training inputs (array or file path)
write_model_to	Required	Output path for complete model (.ednn)
write_compression_model_to	None	Output path for encoder (.ednn)
write_decompression_model_to	None	Output path for decoder (.ednn)
checkpoint_address	None	Path for checkpoints
compress_tolerance	0.01	Maximum reconstruction error
minimum_compression_coefficients	1	Minimum latent dimensions
compress_decompress_size_ratio	1.0	Encoder/decoder size balance
valid_percentage	33.33	Validation split percentage
inputs_shifting	“auto”	Input shift method
inputs_scaling	“auto”	Input scale method
inputs_normalize_per_feature	True	Per-feature input normalization
initial_beta_reg	0.1	Initial regularization coefficient (see The Regularization Coefficient)
final_learning	True	Push weight optimization to its limit using the validation data (see Final Learning)
links_maximum_number	0	Max network connections (0=unlimited)
use_gpu	False	Enable GPU acceleration
gpu_id	0	GPU device ID

Example: Signal Compression for IoT

Here’s an example compressing sensor data for IoT transmission:

Python API

from NeurEco import NeurEcoTabular as Tabular
import numpy as np

# Load sensor data (e.g., 210 measurements per sample)
x_train = np.genfromtxt("sensor/measurements.csv", delimiter=";", skip_header=True)
print(f"Input dimension: {x_train.shape[1]}")

# Create and build the compression model
model = Tabular.Compressor()

config = Tabular.CompressorBuildConfig(
    input_data=x_train,
    write_model_to="./sensor_model/complete.ednn",
    write_compression_model_to="./sensor_model/encoder.ednn",
    write_decompression_model_to="./sensor_model/decoder.ednn",
    checkpoint_address="./sensor_model/compression.checkpoint",

    # Compression settings
    compress_tolerance=0.01,
    minimum_compression_coefficients=1,
    compress_decompress_size_ratio=0.3,  # Small encoder for IoT device
)

model.build(config)

# Test the compression
encoder = Tabular.Compressor()
encoder.load("./sensor_model/encoder.ednn")

compressed = encoder.evaluate(x_train[:10])
print(f"Compressed to {compressed.shape[1]} coefficients")
print(f"Compression ratio: {x_train.shape[1] / compressed.shape[1]:.1f}x")

# Verify reconstruction quality
decoder = Tabular.Compressor()
decoder.load("./sensor_model/decoder.ednn")

reconstructed = decoder.evaluate(compressed)
error = decoder.compute_error(reconstructed, x_train[:10])
print(f"Reconstruction L2 Relative Error: {error:.6f}")

encoder.delete()
decoder.delete()

Next Steps

• Evaluate your model to check reconstruction quality

• Export for deployment

• See advanced features for compression coefficient visualization