gpp_math_test

Contents:

gpp_math_test.hpp

Functions for testing gpp_math’s GP and EI functionality.

Tests are broken into two main groups:

  • ping (unit) tests for GP outputs (mean, cholesky/variance) and EI (for the general and one sample cases)
  • unit + integration tests for optimization methods

The ping tests are set up the same way as the ping tests in gpp_covariance_test; using the function evaluator and ping framework defined in gpp_test_utils.

There is also a consistency check between general MC-based EI calculation and the analytic one sample case.

Finally, we have tests for EI optimization. These include multithreading tests (verifying that each core does what is expected) as well as integration tests for EI optimization. Unit tests for optimizers live in gpp_optimization_test.hpp/cpp. These integration tests use constructed data but exercise all the same code paths used for hyperparameter optimization in production.

namespace optimal_learning

Macro to allow restrict as a keyword for C++ compilation and CUDA/nvcc compilation. See related entry in gpp_common.hpp for more details.

Enums

ExpectedImprovementEvaluationMode enum

Enum for specifying which EI evaluation mode to test.

Values:

  • kAnalytic = = 0 -

    test analytic evaluation

  • kMonteCarlo = = 1 -

    test monte-carlo evaluation

gpp_math_test.cpp

Routines to test the functions in gpp_math.cpp.

The tests verify GaussianProcess, ExpectedImprovementEvaluator (+OnePotentialSample), and EI optimization from gpp_math.cpp.

  1. Ping testing (verifying analytic gradient computation against finite difference approximations)

    1. Following gpp_covariance_test.cpp, we define classes (PingGPMean + other GP ping, PingExpectedImprovement) for evaluating those functions + their spatial gradients.

      Some Pingable classes for GP functions are less general than their gpp_covariance_test or gpp_model_selection_test counterparts, since GP derivative functions sometimes return sparse or incomplete data (e.g., gradient of mean returned as a vector instead of a diagonal matrix; gradient of variance only differentiates wrt a single point at a time); hence we need specialized handlers for testing.

    2. Ping for derivative accuracy (PingGPComponentTest, PingEITest); these unit test the analytic derivatives.

  2. Monte-Carlo EI vs analytic EI validation: the monte-carlo versions are run to “high” accuracy and checked against analytic formulae when applicable

  3. Gradient Descent: using polynomials and other simple fucntions with analytically known optima to verify that the algorithm(s) underlying EI optimization are performing correctly.

  4. Single-threaded vs multi-threaded EI optimization validation: single and multi-threaded runs are checked to have the same output.

  5. End-to-end test of the EI optimization process for the analytic and monte-carlo cases. These tests use constructed data for inputs but otherwise exercise the same code paths used for EI optimization in production.

Variables

constexpr char const *const kName

int dim_

spatial dimension (e.g., entries per point of points_sampled)

int num_to_sample_

number of points currently being sampled

number of potential future samples; gradients are evaluated wrt these points (i.e., the “q” in q,p-EI)

int num_sampled_

number of points in points_sampled

bool gradients_already_computed_

whether gradients been computed and storedwhether this class is ready for use

std::vector< double > noise_variance_

\sigma_n^2, the noise variance

std::vector< double > points_sampled_

coordinates of already-sampled points, X

std::vector< double > points_sampled_value_

function values at points_sampled, y

std::vector< double > grad_mu_

the gradient of the GP mean evaluated at union_of_points, wrt union_of_points[0:num_to_sample]

SquareExponential sqexp_covariance_

covariance class (for computing covariance and its gradients)

GaussianProcess gaussian_process_

gaussian process used for computations

std::vector< double > grad_variance_

the gradient of the GP variance evaluated at union_of_points, wrt union_of_points[0:num_to_sample]

the gradient of the cholesky factorization of the GP variance evaluated at union_of_points, wrt union_of_points[0:num_to_sample]

int num_being_sampled_

number of points being sampled concurrently (i.e., the “p” in q,p-EI)

number of points being sampled concurrently (i.e., the “p” in q,p-EI). Must be 0 for the analytic case.

std::vector< double > points_being_sampled_

points that are being sampled in concurrently experiments

std::vector< double > grad_EI_

the gradient of EI at union_of_points, wrt union_of_points[0:num_to_sample]

ExpectedImprovementEvaluator ei_evaluator_

expected improvement evaluator object that specifies the parameters & GP for EI evaluation

namespace optimal_learning

Macro to allow restrict as a keyword for C++ compilation and CUDA/nvcc compilation. See related entry in gpp_common.hpp for more details.

Functions

int PingGPMeanTest()

Pings the gradients (spatial) of the GP mean 50 times with randomly generated test cases

Returns:
number of ping/test failures

Checks that the gradients (spatial) of the GP mean are computed correctly.

Returns:
number of test failures: 0 if all is working well.

int PingGPVarianceTest()

Pings the gradients (spatial) of the GP variance 50 times with randomly generated test cases

Returns:
number of ping/test failures

Checks that the gradients (spatial) of the GP variance are computed correctly.

Returns:
number of test failures: 0 if all is working well.

int PingGPCholeskyVarianceTest()

Wrapper to ping the gradients (spatial) of the cholesky factorization.

Returns:
number of ping/test failures

Checks that the gradients (spatial) of the cholesky factorization of GP variance are computed correctly.

Returns:
number of test failures: 0 if all is working well.

template < typename EIEvaluator >
OL_WARN_UNUSED_RESULT int PingEITest(int num_to_sample, int num_being_sampled, double epsilon, double tolerance_fine, double tolerance_coarse, double input_output_ratio)

Pings the gradients (spatial) of the EI 50 times with randomly generated test cases Works with various EI evaluators (e.g., MC, analytic formulae)

Parameters:
num_to_sample:number of potential future samples; gradients are evaluated wrt these points (i.e., the “q” in q,p-EI)
num_being_sampled:
 number of points being sampled in concurrent experiments (i.e., the “p” in q,p-EI)
epsilon:coarse, fine h sizes to use in finite difference computation
tolerance_fine:desired amount of deviation from the exact rate
tolerance_coarse:
 maximum allowable abmount of deviation from the exact rate
input_output_ratio:
 for ||analytic_gradient||/||input|| < input_output_ratio, ping testing is not performed, see PingDerivative()
Returns:
number of ping/test failures

int PingEIGeneralTest()

Pings the gradients (spatial) of the EI 50 times with randomly generated test cases

Returns:
number of ping/test failures

Checks that the gradients (spatial) of Expected Improvement are computed correctly.

Returns:
number of test failures: 0 if all is working well.

int PingEIOnePotentialSampleTest()

Pings the gradients (spatial) of the EI (one potential sample special case) 50 times with randomly generated test cases

Returns:
number of ping/test failures

Checks the gradients (spatial) of Expected Improvement (in the special case of only 1 potential sample) are computed correctly.

Returns:
number of test failures: 0 if all is working well.

int EIOnePotentialSampleEdgeCasesTest()

Test cases where analytic EI would attempt to compute 0/0 without variance lower bounds.

The bounds are OnePotentialSampleExpectedImprovementEvaluator::kMinimumVarianceEI and kMinimumVarianceGradEI. See those class docs for more details.

These particular test cases arose from plotting EI (easy since dim = 1) and checking that EI and grad_EI were being computed appropriately at the specified locations. The test cases are purposely simple; the requirement was that they trigger behavior that would result in 0/0 without minimum variance thresholds.

Without the aforementioned thresholds, 1D analytic EI could attempt 0/0 = (best_so_far - gp_mean) / sqrt(gp_variance) The easiest way to do cause these conditions is to compute EI at (or near) one of points_sampled such that gp_mean == best_so_far and gp_variance == 0. (Although these conditions can arise elsewhere; try plotting the test case in the code.)

Returns:
number of test failures

int RunEIConsistencyTests()

Generates a set of 50 random test cases for expected improvement with only one potential sample. The general EI (which uses MC integration) is evaluated to reasonably high accuracy (while not taking too long to run) and compared against the analytic formula version for consistency. The gradients (spatial) of EI are also checked.

Returns:
number of cases where analytic and monte-carlo EI do not match

Tests that the general EI + grad EI computation (using MC integration) is consistent with the special analytic case of EI when there is only ONE potential point to sample.

Returns:
number of test failures: 0 if all is working well.

int RunGPTests()

Runs a battery of tests for the GP and EI functions, including ping tests for:

  • GP mean
  • GP variance
  • cholesky decomposition of the GP variance
  • Expected Improvement
  • Expected Improvement special case: only ONE potential point to sample

and edge case testing for:

  • 1D Analytic Expected Improvement
Returns:
number of test failures: 0 if all is working well.

int MultithreadedEIOptimizationTest(ExpectedImprovementEvaluationMode ei_mode)

Tests that single & multithreaded EI optimization produce the exact same results.

We do this by first setting up EI optimization in a single threaded scenario with 2 starting points and 2 random number generators. Optimization is run one from starting point 0 with RNG 0, and then again from starting point 1 with RNG 1.

Then we run the optimization multithreaded (with 2 threads) over both starting points simultaneously. One of the threads will see the winning (point, RNG) pair from the single-threaded won. Hence one result point will match with the single threaded results exactly.

Then we re-run the multithreaded optimization, swapping the position of the RNGs and starting points. If thread 0 won in the previous test, thread 1 will win here (and vice versa).

Note that it’s tricky to run single-threaded optimization over both starting points simultaneously because we won’t know which (point, RNG) pair won (which is required to ascertain the ‘winner’ since we are not computing EI accurately enough to avoid error).

Checks that multithreaded EI optimization behaves the same way that single threaded does.

Parameters:
ei_mode:ei evaluation mode to test (analytic or monte carlo)
Returns:
number of test failures: 0 if EI multi/single threaded optimization are consistent

int ExpectedImprovementOptimizationMultipleSamplesTest()

At the moment, this test is very bare-bones. It checks:

  1. method succeeds
  2. points returned are all inside the specified domain
  3. points returned are not within epsilon of each other (i.e., distinct)
  4. result of gradient-descent optimization is no worse than result of a random search
  5. final grad EI is sufficiently small

The test sets up a toy problem by repeatedly drawing from a GP with made-up hyperparameters. Then it runs EI optimization, attempting to sample 3 points simultaneously.

Checks that ComputeOptimalPointsToSample works on a tensor product domain. This test exercises the the code tested in: ExpectedImprovementOptimizationTest(kTensorProduct, ei_mode) for ei_mode = {kAnalytic, kMonteCarlo}.

This test checks the generation of multiple, simultaneous experimental points to sample.

Returns:
number of test failures: 0 if EI optimization is working properly

int EvaluateEIAtPointListTest()

Tests EvaluateEIAtPointList (computes EI at a specified list of points, multithreaded). Checks that the returned best point is in fact the best. Verifies multithreaded consistency.

Returns:
number of test failures: 0 if function evaluation is working properly

int ExpectedImprovementOptimizationTest(DomainTypes domain_type, ExpectedImprovementEvaluationMode ei_mode)

Checks that EI optimization is working on tensor product or simplex domain using analytic or monte-carlo EI evaluation.

Parameters:
domain_type:type of the domain to test on (e.g., tensor product, simplex)
ei_mode:ei evaluation mode to test (analytic or monte carlo)
Returns:
number of test failures: 0 if EI optimization is working properly