Privacy-Preserving Collaborative Medical Image Segmentation Using Latent Transform Networks

Collaborative training across multiple institutions is becoming essential for building reliable medical image segmentation models. However, privacy regulations, data silos, and uneven data availability prevent hospitals from sharing raw scans or annotations, limiting the ability to train generalizable models. Latent-space collaboration frameworks such as privacy-segmentation framework (SF) offer a promising alternative, but such methods still face challenges in segmentation accuracy and vulnerability to latent inversion and membership-inference attacks. This work introduces a privacy-preserving collaborative medical image segmentation framework (PPCMI-SF) designed for heterogeneous medical datasets. The approach combines skip-connected autoencoders for images and masks with a keyed latent transform that applies client-specific orthogonal mixing and permutation to protect latent features before they are shared. A unified mapping network on the server-side performs multi-scale latent-to-latent translation, enabling segmentation inference without exposing raw data. Experiments on four datasets: PSFH ultrasound, ultrasound nerve segmentation, FUMPE CTA, and cardiac MRI show that the proposed PPCMI-SF consistently achieves high Dice scores and improved boundary accuracy, as reflected by lower 95th percentile Hausdorff distance (HD95) and average symmetric surface distance (ASD) compared to the current state-of-the-art and performs competitively with privacy-agnostic baselines. Privacy tests confirm strong resistance to inversion and membership attacks, and the overall system achieves real-time inference with low communication overhead. These results demonstrate that accurate and efficient medical image segmentation can be achieved without compromising data privacy in multi-institution settings.

Key Contributions

Keyed Latent Transform (KLT) using client-specific orthogonal mixing and permutation to protect shared latent features against inversion and membership inference attacks
Skip-connected autoencoders for image and mask encoding that improve segmentation fidelity (Dice, HD95, ASD) while keeping raw data on-client
Server-side unified mapping network for multi-scale latent-to-latent translation enabling inference without raw data exposure, with real-time throughput

🛡️ Threat Analysis

Model Inversion Attack

The KLT (keyed latent transform) is specifically designed to prevent adversaries from reconstructing original medical images from shared latent representations — a direct defense against latent inversion / model inversion attacks, evaluated empirically in privacy tests.

Membership Inference Attack

The paper explicitly evaluates and confirms resistance to membership inference attacks on the shared latent features, making MIA defense a primary security objective alongside inversion resistance.