.. _tabular-building-compression: ==================== Building Compression ==================== This page covers building **Tabular Compression** models for dimensionality reduction and autoencoder applications. .. contents:: On this page :local: :depth: 2 When to Use Compression ======================= Use the Compression template when: - You want to **reduce data dimensionality** (compress high-dimensional data) - You need to **compress and reconstruct signals** (autoencoder) - You're building **IoT applications** that require efficient data transmission - You want to perform **feature extraction** or **anomaly detection** How Compression Works ===================== Unlike Regression and Classification, Compression doesn't require separate output files. The model learns to: 1. **Compress:** Transform input data into a lower-dimensional representation 2. **Decompress:** Reconstruct the original input from the compressed representation NeurEco produces **three models**: - **Complete model:** Full encoder + decoder pipeline - **Compression model (encoder):** Input -> compressed representation - **Decompression model (decoder):** Compressed representation -> reconstructed output This separation is particularly useful for IoT applications where: - A **small encoder** runs on the device (sensor, microcontroller) - A **larger decoder** runs on the server Building a Compression Model ============================ .. tabs:: .. tab:: GUI **1. Create a new project** Launch NeurEco GUI and select **Tabular Compression**. **2. Configure settings** In the **Settings** tab: - **Training data:** Select your input file(s) only (no output file needed) - **Compression tolerance:** Maximum acceptable reconstruction error - **Minimum compression coefficients:** Minimum latent space dimension - **Compressor/Decompressor size ratio:** Balance between encoder and decoder size **3. Start the build** Click the **BUILD** button. NeurEco will: - Build both encoder and decoder networks - Optimize for reconstruction accuracy within tolerance - Save all three models (complete, encoder, decoder) .. tab:: Python API **1. Import and prepare data** .. code-block:: python from NeurEco import NeurEcoTabular as Tabular import numpy as np # Load data (no separate output needed) x_train = np.genfromtxt("data/x_train.csv", delimiter=";", skip_header=True) **2. Create the model** .. code-block:: python model = Tabular.Compressor() **3. Configure and build** .. code-block:: python config = Tabular.CompressorBuildConfig( # Required: Data (no output_data needed) input_data=x_train, # Required: Output paths for all three models write_model_to="./model/compression_complete.ednn", write_compression_model_to="./model/encoder.ednn", write_decompression_model_to="./model/decoder.ednn", checkpoint_address="./model/compression.checkpoint", # Compression-specific parameters compress_tolerance=0.01, # Max reconstruction error minimum_compression_coefficients=1, # Min latent dimensions compress_decompress_size_ratio=1.0, # Encoder/decoder balance # Optional: Validation valid_percentage=33.33, # Optional: Normalization (inputs only) inputs_shifting="min_centered", inputs_scaling="max_centered", inputs_normalize_per_feature=True, ) model.build(config) .. tab:: CLI **1. Create a configuration file** Create ``build_compression.conf``: .. code-block:: json { "neurecoCompression_build": { "DevSettings": { "disconnect_inputs_if_possible": true, "final_learning": true, "initial_beta_reg": 0.1, "valid_percentage": 33.33, "compressor_decompressor_ratio": 1.0 }, "UserSettings": { "gpu_id": 0, "use_gpu": false }, "compress_tolerance": 0.01, "minimum_compression_coefficients": 1, "checkpoint_address": "./model/compression.checkpoint", "exc_filenames": ["x_train.csv"], "input_normalization": { "normalize_per_feature": true, "scale_type": "max_centered", "shift_type": "min_centered" }, "output_normalization": { "normalize_per_feature": true, "scale_type": "max_centered", "shift_type": "min_centered" }, "write_model_to": "./model/compression_complete.ednn", "write_compression_model_to": "./model/encoder.ednn", "write_decompression_model_to": "./model/decoder.ednn" } } In a configuration file, ``compress_tolerance`` and ``minimum_compression_coefficients`` sit at the top level, and the size ratio (``compress_decompress_size_ratio`` in Python) is ``DevSettings.compressor_decompressor_ratio``. The output normalization is the same as the input normalization, as in the Python API. **2. Run the build** .. code-block:: shell neureco Compression build build_compression.conf Compression-Specific Parameters =============================== These parameters are unique to Compression: .. list-table:: :header-rows: 1 :widths: 30 15 55 * - Parameter - Default - Description * - ``compress_tolerance`` - 0.01 - Maximum acceptable reconstruction error (relative) * - ``minimum_compression_coefficients`` - 1 - Minimum number of latent space dimensions * - ``compress_decompress_size_ratio`` - 1.0 - Ratio of encoder size to decoder size (0 < ratio <= 1) * - ``write_compression_model_to`` - None - Output path for encoder model (.ednn) * - ``write_decompression_model_to`` - None - Output path for decoder model (.ednn) Understanding Compression Parameters ==================================== Compression Tolerance --------------------- The ``compress_tolerance`` parameter defines the maximum acceptable reconstruction error: - **Lower values (e.g., 0.001):** Higher accuracy, potentially larger latent space - **Higher values (e.g., 0.1):** More aggressive compression, some accuracy loss NeurEco will find the smallest latent dimension that achieves the specified tolerance. Minimum Compression Coefficients -------------------------------- The ``minimum_compression_coefficients`` parameter sets a floor for the latent space dimension: - Set higher if you know your data has a certain intrinsic dimensionality - Useful when you need at least N compression coefficients for downstream tasks Compressor/Decompressor Size Ratio ---------------------------------- The ``compress_decompress_size_ratio`` parameter (range: 0 < ratio <= 1) controls the balance between encoder and decoder network sizes: - **ratio = 1.0:** Balanced encoder and decoder - **ratio < 1.0:** Smaller encoder, larger decoder **IoT Use Case:** For IoT applications, set ``ratio < 1.0`` to create: - A **small encoder** that runs on resource-constrained devices - A **larger decoder** that runs on powerful servers This reduces: - Memory footprint on the device - Computation time on the device - Radio transmission power (smaller model, less transmitted data) Using Compression Models ======================== After building, you have three models: **Complete Model (end-to-end)** .. code-block:: python # Load complete model model = Tabular.Compressor() model.load("./model/compression_complete.ednn") # Compress and decompress in one step reconstructed = model.evaluate(x_test) **Encoder Only** .. code-block:: python # Load encoder encoder = Tabular.Compressor() encoder.load("./model/encoder.ednn") # Get compressed representation compressed = encoder.evaluate(x_test) print(f"Compressed from {x_test.shape[1]} to {compressed.shape[1]} dimensions") **Decoder Only** .. code-block:: python # Load decoder decoder = Tabular.Compressor() decoder.load("./model/decoder.ednn") # Reconstruct from compressed representation reconstructed = decoder.evaluate(compressed) Complete Parameter Reference ============================ .. list-table:: :header-rows: 1 :widths: 30 15 55 * - Parameter - Default - Description * - ``input_data`` - Required - Training inputs (array or file path) * - ``write_model_to`` - Required - Output path for complete model (.ednn) * - ``write_compression_model_to`` - None - Output path for encoder (.ednn) * - ``write_decompression_model_to`` - None - Output path for decoder (.ednn) * - ``checkpoint_address`` - None - Path for checkpoints * - ``compress_tolerance`` - 0.01 - Maximum reconstruction error * - ``minimum_compression_coefficients`` - 1 - Minimum latent dimensions * - ``compress_decompress_size_ratio`` - 1.0 - Encoder/decoder size balance * - ``valid_percentage`` - 33.33 - Validation split percentage * - ``inputs_shifting`` - "auto" - Input shift method * - ``inputs_scaling`` - "auto" - Input scale method * - ``inputs_normalize_per_feature`` - True - Per-feature input normalization * - ``initial_beta_reg`` - 0.1 - Initial regularization coefficient (see :ref:`regularization-coefficient`) * - ``final_learning`` - True - Push weight optimization to its limit using the validation data (see :ref:`final-learning`) * - ``links_maximum_number`` - 0 - Max network connections (0=unlimited) * - ``use_gpu`` - False - Enable GPU acceleration * - ``gpu_id`` - 0 - GPU device ID Example: Signal Compression for IoT =================================== Here's an example compressing sensor data for IoT transmission: .. tabs:: .. tab:: Python API .. code-block:: python from NeurEco import NeurEcoTabular as Tabular import numpy as np # Load sensor data (e.g., 210 measurements per sample) x_train = np.genfromtxt("sensor/measurements.csv", delimiter=";", skip_header=True) print(f"Input dimension: {x_train.shape[1]}") # Create and build the compression model model = Tabular.Compressor() config = Tabular.CompressorBuildConfig( input_data=x_train, write_model_to="./sensor_model/complete.ednn", write_compression_model_to="./sensor_model/encoder.ednn", write_decompression_model_to="./sensor_model/decoder.ednn", checkpoint_address="./sensor_model/compression.checkpoint", # Compression settings compress_tolerance=0.01, minimum_compression_coefficients=1, compress_decompress_size_ratio=0.3, # Small encoder for IoT device ) model.build(config) # Test the compression encoder = Tabular.Compressor() encoder.load("./sensor_model/encoder.ednn") compressed = encoder.evaluate(x_train[:10]) print(f"Compressed to {compressed.shape[1]} coefficients") print(f"Compression ratio: {x_train.shape[1] / compressed.shape[1]:.1f}x") # Verify reconstruction quality decoder = Tabular.Compressor() decoder.load("./sensor_model/decoder.ednn") reconstructed = decoder.evaluate(compressed) error = decoder.compute_error(reconstructed, x_train[:10]) print(f"Reconstruction L2 Relative Error: {error:.6f}") encoder.delete() decoder.delete() Next Steps ========== - :ref:`Evaluate your model ` to check reconstruction quality - :ref:`Export for deployment ` - See :ref:`advanced features ` for compression coefficient visualization