This notebook explains how to perform GP inference using the Markovflow CVIGaussianProcess model. Here, we perform binary classification with time as the input.
As with GPR, the observations do not have to be regularly spaced. However, they do need to be sequential. We denote the input/output tuples as \((x_i, y_i)_{1 \leq i \leq n}\), where \(x_i\) is a scalar value and \(y_i \in \{0, 1\}\).
Our probabilistic model for this data is:
where \(\Phi\) is a function that maps \(f(x_i)\) to \([0, 1]\), the probability that \(y_i=1\). In practice, we choose \(\Phi\) to be the standard normal cumulative distribution function (also known as the probit function) which maps to \([0, 1]\).
NOTE: If you have difficulty running this notebook, consider clearing the output and then restarting the kernel.
[1]:
# Setup import warnings # Turn off warnings warnings.simplefilter('ignore') import matplotlib.pyplot as plt import numpy as np import tensorflow as tf from gpflow import default_float from gpflow.ci_utils import ci_niter from gpflow.likelihoods import Bernoulli from markovflow.models.variational_cvi import CVIGaussianProcess from markovflow.kernels import Matern52 np.random.seed(0) FLOAT_TYPE = default_float() # uncomment in notebook # try: # from IPython import get_ipython # get_ipython().run_line_magic('matplotlib', 'inline') # except AttributeError: # print('Magic function can only be used in IPython environment') # matplotlib.use('Agg') plt.rcParams["figure.figsize"] = [15, 8]
2022-09-17 14:46:07.193013: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.7.13/x64/lib 2022-09-17 14:46:07.193050: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
First, let’s generate some binary data \(X = (x_1, \dots, x_n)\) and \(Y = (y_1, \dots, y_n)^T\).
[2]:
# Generate some observations num_data = 300 time_points = np.linspace(0 , 1, num_data) F = np.cos(time_points * 20).reshape(-1, 1) observations = (F + np.random.randn(*F.shape) > 0).astype(float) data = (time_points, observations)
[3]:
kernel = Matern52(lengthscale=.2, variance=5.0) # We see Matern12 has only two dimensions (therefore there is less risk of overparameterising) print(kernel.state_dim)
3
2022-09-17 14:46:08.665765: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2022-09-17 14:46:08.665964: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.7.13/x64/lib 2022-09-17 14:46:08.665974: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303) 2022-09-17 14:46:08.665998: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (fv-az178-774): /proc/driver/nvidia/version does not exist 2022-09-17 14:46:08.666248: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2022-09-17 14:46:08.666360: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
This is a classification problem with outputs between [0,1], so we create a variational GP model using a Bernoulli likelihood.
[0,1]
[4]:
# Create a likelihood object likelihood = Bernoulli() input_data = (tf.constant(time_points), tf.constant(observations)) vgpc = CVIGaussianProcess(input_data=input_data, kernel=kernel, likelihood=likelihood, learning_rate=.1) def plot_model(model): f_mu, f_var = model.posterior.predict_f(time_points) f_mu = f_mu.numpy() f_std = np.sqrt(f_var) plt.figure(figsize=(10, 6)) plt.plot(time_points, observations, 'kx', ms=8, mew=2, label='data') plt.plot(time_points, F, 'b', ms=8, mew=2, label='underlying $f$') plt.plot(time_points, f_mu, 'C0', ms=8, mew=2, label='posterior prediction') plt.fill_between( time_points, y1 = (f_mu - 2 * f_std).reshape(-1,), y2 = (f_mu + 2 * f_std).reshape(-1,), alpha=.2, facecolor='C0' ) plt.xlabel("Time") plt.ylabel("Label") plt.legend() plt.show() plot_model(vgpc) max_iter = ci_niter(500) for i in range(max_iter): if i % 10 == 0: print(i, vgpc.elbo()) vgpc.update_sites() plot_model(vgpc)
0 tf.Tensor(-3625.587221755262, shape=(), dtype=float64) 10 tf.Tensor(-556.6376706768804, shape=(), dtype=float64) 20 tf.Tensor(-531.5701690613641, shape=(), dtype=float64) 30 tf.Tensor(-525.4783223926928, shape=(), dtype=float64) 40 tf.Tensor(-523.4811612909392, shape=(), dtype=float64) 50 tf.Tensor(-522.7472761951153, shape=(), dtype=float64) 60 tf.Tensor(-522.4636221954515, shape=(), dtype=float64) 70 tf.Tensor(-522.3510786289505, shape=(), dtype=float64) 80 tf.Tensor(-522.305729478675, shape=(), dtype=float64) 90 tf.Tensor(-522.2872750728598, shape=(), dtype=float64) 100 tf.Tensor(-522.2797170000238, shape=(), dtype=float64) 110 tf.Tensor(-522.2766088950966, shape=(), dtype=float64) 120 tf.Tensor(-522.2753283679884, shape=(), dtype=float64) 130 tf.Tensor(-522.2747984304037, shape=(), dtype=float64) 140 tf.Tensor(-522.274580032272, shape=(), dtype=float64) 150 tf.Tensor(-522.2744887538497, shape=(), dtype=float64) 160 tf.Tensor(-522.274451839274, shape=(), dtype=float64) 170 tf.Tensor(-522.2744363440205, shape=(), dtype=float64) 180 tf.Tensor(-522.2744295248319, shape=(), dtype=float64) 190 tf.Tensor(-522.2744275149942, shape=(), dtype=float64) 200 tf.Tensor(-522.2744266514927, shape=(), dtype=float64) 210 tf.Tensor(-522.2744262640493, shape=(), dtype=float64) 220 tf.Tensor(-522.274426143615, shape=(), dtype=float64) 230 tf.Tensor(-522.2744261053072, shape=(), dtype=float64) 240 tf.Tensor(-522.2744260925317, shape=(), dtype=float64) 250 tf.Tensor(-522.2744260881852, shape=(), dtype=float64) 260 tf.Tensor(-522.2744260866937, shape=(), dtype=float64) 270 tf.Tensor(-522.2744260861798, shape=(), dtype=float64) 280 tf.Tensor(-522.2744260860023, shape=(), dtype=float64) 290 tf.Tensor(-522.274426085941, shape=(), dtype=float64) 300 tf.Tensor(-522.2744260859197, shape=(), dtype=float64) 310 tf.Tensor(-522.2744260859123, shape=(), dtype=float64) 320 tf.Tensor(-522.2744260859097, shape=(), dtype=float64) 330 tf.Tensor(-522.2744260859089, shape=(), dtype=float64) 340 tf.Tensor(-522.2744260859085, shape=(), dtype=float64) 350 tf.Tensor(-522.2744260859084, shape=(), dtype=float64) 360 tf.Tensor(-522.2744260859084, shape=(), dtype=float64) 370 tf.Tensor(-522.2744260859084, shape=(), dtype=float64) 380 tf.Tensor(-522.2744260859084, shape=(), dtype=float64) 390 tf.Tensor(-522.2744260859084, shape=(), dtype=float64) 400 tf.Tensor(-522.2744260859084, shape=(), dtype=float64) 410 tf.Tensor(-522.2744260859084, shape=(), dtype=float64) 420 tf.Tensor(-522.2744260859084, shape=(), dtype=float64) 430 tf.Tensor(-522.2744260859084, shape=(), dtype=float64) 440 tf.Tensor(-522.2744260859084, shape=(), dtype=float64) 450 tf.Tensor(-522.2744260859084, shape=(), dtype=float64) 460 tf.Tensor(-522.2744260859084, shape=(), dtype=float64) 470 tf.Tensor(-522.2744260859084, shape=(), dtype=float64) 480 tf.Tensor(-522.2744260859084, shape=(), dtype=float64) 490 tf.Tensor(-522.2744260859084, shape=(), dtype=float64)