Capsule Networks for 3D Data

We introduce 3D Capsule Networks as a generic framework that is suited to capture extrinsic or intrinsic object properties such as object orientation or part labels. Our quest is to learn robust, flexible and generalizable 3D object representations without requiring heavy annotation efforts or supervision. The use of capsule networks allows us to go beyond conventional 3D generative models by constructing a structured latent space where certain factors of shape variations, such as object parts, can be disentangled into independent sub-spaces. Such embeddings can either be explicit like quaternions representing part poses enabling equivariant network architectures or purely implicit. While the learned representations can already be used in typical tasks like classification or pose estimation, they also allow for generation. Our novel decoders then act on the latent capsules to reconstruct 3D points in a self-supervised manner. Currently, we present two approaches, one for (un)supervised, part-based 3D shape processing across rigid / non-rigid shapes and cross-/within-category [CVPR'19, IJCV'22] and the other for classification and pose estimation. Thanks to the use of capsule networks, our methods enjoy an added structure on the latent space which enables multiple applications such as part pose estimation, part interpolation or replacement. Please refer to the papers and videos for futher information.