Hyperlib

What is Hyperlib?

Hyperlib is a cutting-edge python library that makes it easy to create the next generation of neural networks in the hyperbolic space (as opposed to Euclidean).

Hyperbolic geometry is different from Euclidean geometry - It has more capacity which means it can fit a wider range of data. Hyperbolic geometry is particularly suited to data that has an underlying hierarchical structure. There’s also a growing amount of research documenting the benefits of modelling the brain using hyperbolic over Euclidean geometry.

Hyperlib makes hyperbolic neural networks accessible, abstracting away all of the complicated maths. We hope it will inspire further hyperbolic research.

Non-Euclidean deep learning

Since the famed 'Image-net' moment in 2012 deep learning has become the dominant field within Artificial Intelligence research. Deep learning can be defined as training an Artificial Neural Network that has many layers within it to learn representations of input data that makes it easier to perform a task. A task, in this case, can be anything from classifying whether an image is a photograph of a cat or a dog, to predicting if it will rain in New York tomorrow.

Deep learning models have been able to outperform humans in a number of tasks, as Deepmind demonstrated most recently with their AlphaFold 2 model, but these models are still limited in a number of ways. One limitation is the geometry that the majority of these models work within. Recent works have explored the ways in which deep learning models can be improved by generalising them to alternative geometries.

From Euclidean to Hyperbolic Geometry

Geometry is, with arithmetic, one of the oldest branches of mathematics. It is concerned with properties of space that are related with distance, shape, size, and relative position of figures. Since Euclid's famous works 2000 years ago the word ‘geometry’ has been synonymous with Euclidean geometry, the geometry of flat space. This is despite the discovery of alternative geometries by Lobachevesky, Bolyai, Gauss, and Riemann and the numerous limitations that occur as a result of using flat geometry. The video below provides a great overview of the different geometries.

Because of it's flat nature Euclidean space simply doesn't fit many types of data structures that we often need to work with. The core example is the hierarchy, or, its abstract network representation, the tree. Social networks, human skeletons, sentences in natural language all possess a tree-like structure or can be represented hierarchically. It's also widely accepted that human beings use a hierarchy to organise object categories. Recently there has been a surge of interest in applying machine learning to non-Euclidean geometries such as Hyperbolic and Spherical/Elliptic geometries as the limits of Euclidean geometry become clear.

Parallel lines as shown in each of the three geometries

The curved nature of the hyperbolic space means that it has the capacity to embed tree structures without distorting them out of shape. It is impossible to embed a tree structure without distortion in the Euclidean space, even with an unbounded number of dimensions. However, this task becomes surprisingly easy in the hyperbolic space with only 2 dimensions. Apart from a superior ability to embed trees, neural networks trained in the hyperbolic space have a number of other potential advantages including :

• Better generalization capabilities of the model, with less over-fitting, computational complexity, requirement of training data.

• Reduction in the number of model parameters and embedding dimensions.

• Providing low distortion embedding, preserving the local and global geometric information.

• Better model understanding and interpretation.

Lines of equal length as shown in the Hyperbolic space

Using Hyperlib

Hyperlib implements common Neural Network components in the hyperbolic space (using the Poincare model). Hyperlib uses Tensorflow as a backend and can easily be used with Keras and is meant to help Data Scientists, Machine Learning Engineers, Researchers and others to implement hyperbolic neural networks.

You can install Hyperlib using:

pip install hyperlib

Once installed you can use hyperbolic layers just like you would use any other. Here's a starter script to get you started:

import tensorflow as tf

from tensorflow import keras

from hyperlib.nn.layers.lin_hyp import LinearHyperbolic

from hyperlib.nn.optimizers.rsgd import RSGD

from hyperlib.manifold.poincare import Poincare

# Create layers

hyperbolic_layer_1 = LinearHyperbolic(32, Poincare(), 1)

hyperbolic_layer_2 = LinearHyperbolic(32, Poincare(), 1)

output_layer = LinearHyperbolic(10, Poincare(), 1)

# Create optimizer

optimizer = RSGD(learning_rate=0.1)

# Create model architecture

model = tf.keras.models.Sequential([

  hyperbolic_layer_1,

  hyperbolic_layer_2,

  output_layer

])

# Compile the model with the Riemannian optimizer            

model.compile(

    optimizer=optimizer,

    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),

    metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],

)

Interesting use cases

As mentioned earlier Hyperbolic Networks are ideal when you're working with data that has some sort of hierarchical structure. They have already been applied to Natural Language Processing where Hyperbolic word embeddings have been shown to outperform their Euclidean counterparts. Elsewhere, hyperbolic networks have been used in image classification due to their ability to learn higher-level semantic representations.

Get in touch

Hyperlib is brought to you by Nalex.ai. We will be continuing to explore the hyperbolic space and we invite you to join us on our journey by checking our blog posts and following us on Twitter at @frankly_francis and @lex_akeem.

References

[1] Chami, I., Ying, R., Ré, C. and Leskovec, J. Hyperbolic Graph Convolutional Neural Networks. NIPS 2019.

[2] Nickel, M. and Kiela, D. Poincaré embeddings for learning hierarchical representations. NIPS 2017.

[3] Khrulkov, Mirvakhabova, Ustinova, Oseledets, Lempitsky. Hyperbolic Image Embeddings.

[4] Wei Peng, Varanka, Mostafa, Shi, Zhao. Hyperbolic Deep Neural Networks: A Survey.

[5]Sala, Chami et al. Into the Wild: Machine Learning In Non-Euclidean Spaces