mxnet.test_utils

Tools for testing.

Classes

DummyIter(real_iter)

A dummy iterator that always returns the same batch of data (the first data batch of the real data iter).

EnvManager(key, val)

Environment variable setter and unsetter via with idiom

Functions

almost_equal(a, b[, rtol, atol, equal_nan, …])

Test if two numpy arrays are almost equal.

almost_equal_ignore_nan(a, b[, rtol, atol])

Test that two NumPy arrays are almost equal (ignoring NaN in either array).

assert_almost_equal(a, b[, rtol, atol, …])

Test that two numpy arrays are almost equal.

assert_almost_equal_ignore_nan(a, b[, rtol, …])

Test that two NumPy arrays are almost equal (ignoring NaN in either array).

assert_almost_equal_with_err(a, b[, rtol, …])

Test that two numpy arrays are almost equal within given error rate.

assert_exception(f, exception_type, *args, …)

Test that function f will throw an exception of type given by exception_type

assign_each(the_input, function)

Return ndarray composed of passing each array value through some function

assign_each2(input1, input2, function)

Return ndarray composed of passing two array values through some function

check_consistency(sym, ctx_list[, scale, …])

Check symbol gives the same output for different running context

check_numeric_gradient(sym, location[, …])

Verify an operation by checking backward pass via finite difference method.

check_speed(sym[, location, ctx, N, …])

Check the running speed of a symbol.

check_symbolic_backward(sym, location, …)

Compares a symbol’s backward results with the expected ones.

check_symbolic_forward(sym, location, expected)

Compares a symbol’s forward results with the expected ones.

chi_square_check(generator, buckets, probs)

Run the chi-square test for the generator.

collapse_sum_like(a, shape)

Given a as a numpy ndarray, perform reduce_sum on a over the axes that do not exist in shape.

compare_ndarray_tuple(t1, t2[, rtol, atol])

Compare ndarray tuple.

compare_optimizer(opt1, opt2, shape, dtype)

Compare opt1 and opt2.

create_sparse_array(shape, stype[, …])

Create a sparse array, For Rsp, assure indices are in a canonical format

create_sparse_array_zd(shape, stype, density)

Create sparse array, using only rsp_indices to determine density

default_context()

Get default context for regression test.

default_dtype()

Get default data type for regression test.

discard_stderr()

Discards error output of a routine if invoked as:

download(url[, fname, dirname, overwrite, …])

Download an given URL

download_model(model_name[, dst_dir, meta_info])

Download a model from data.mxnet.io

find_max_violation(a, b[, rtol, atol])

Finds and returns the location of maximum violation.

gen_buckets_probs_with_ppf(ppf, nbuckets)

Generate the buckets and probabilities for chi_square test when the ppf (Quantile function) is specified.

get_atol([atol])

Get default numerical threshold for regression test.

get_bz2_data(data_dir, data_name, url, …)

Download and extract bz2 data.

get_cifar10()

Downloads CIFAR10 dataset into a directory in the current directory with the name data, and then extracts all files into the directory data/cifar.

get_etol([etol])

Get default numerical threshold for regression test.

get_im2rec_path([home_env])

Get path to the im2rec.py tool

get_mnist()

Download and load the MNIST dataset

get_mnist_iterator(batch_size, input_shape)

Returns training and validation iterators for MNIST dataset

get_mnist_pkl()

Downloads MNIST dataset as a pkl.gz into a directory in the current directory

get_mnist_ubyte()

Downloads ubyte version of the MNIST dataset into a directory in the current directory with the name data and extracts all files in the zip archive to this directory.

get_rtol([rtol])

Get default numerical threshold for regression test.

get_zip_data(data_dir, url, data_origin_name)

Download and extract zip data.

has_tvm_ops()

Returns True if MXNet is compiled with TVM generated operators.

is_cd_run()

Checks if the test is running as part of a Continuous Delivery run

is_op_runnable()

Returns True for all CPU tests.

list_gpus()

Return a list of GPUs

mean_check(generator, mu, sigma[, nsamples])

Test the generator by matching the mean.

np_reduce(dat, axis, keepdims, numpy_reduce_func)

Compatible reduce for old version of NumPy.

numeric_grad(executor, location[, …])

Calculates a numeric gradient via finite difference method.

rand_ndarray(shape[, stype, density, dtype, …])

Generate a random sparse ndarray.

rand_sparse_ndarray(shape, stype[, density, …])

Generate a random sparse ndarray.

random_arrays(*shapes)

Generate some random numpy arrays.

random_sample(population, k)

Return a k length list of the elements chosen from the population sequence.

retry(n)

Retry n times before failing for stochastic test cases.

same(a, b)

Test if two NumPy arrays are the same.

same_array(array1, array2)

Check whether two NDArrays sharing the same memory block

same_symbol_structure(sym1, sym2)

Compare two symbols to check if they have the same computation graph structure.

set_default_context(ctx)

Set default context.

set_env_var(key, val[, default_val])

Set environment variable

shuffle_csr_column_indices(csr)

Shuffle CSR column indices per row

simple_forward(sym[, ctx, is_train])

A simple forward function for a symbol.

var_check(generator, sigma[, nsamples])

Test the generator by matching the variance.

verify_generator(generator, buckets, probs)

Verify whether the generator is correct using chi-square testing.
class mxnet.test_utils.DummyIter(real_iter)[source]

Bases: mxnet.io.io.DataIter

A dummy iterator that always returns the same batch of data (the first data batch of the real data iter). This is usually used for speed testing.

Parameters

real_iter (mx.io.DataIter) – The real data iterator where the first batch of data comes from

Methods

next()

Get a data batch from iterator.
next()[source]

Get a data batch from iterator. The first data batch of real iter is always returned. StopIteration will never be raised.

Returns

The data of next batch.

Return type

DataBatch

class mxnet.test_utils.EnvManager(key, val)[source]

Bases: object

Environment variable setter and unsetter via with idiom

mxnet.test_utils.almost_equal(a, b, rtol=None, atol=None, equal_nan=False, use_broadcast=True)[source]

Test if two numpy arrays are almost equal.

mxnet.test_utils.almost_equal_ignore_nan(a, b, rtol=None, atol=None)[source]

Test that two NumPy arrays are almost equal (ignoring NaN in either array). Combines a relative and absolute measure of approximate eqality. If either the relative or absolute check passes, the arrays are considered equal. Including an absolute check resolves issues with the relative check where all array values are close to zero.

Parameters

  • a (np.ndarray) –

  • b (np.ndarray) –

  • rtol (None or float) – The relative threshold. Default threshold will be used if set to None.

  • atol (None or float) – The absolute threshold. Default threshold will be used if set to None.

mxnet.test_utils.assert_almost_equal(a, b, rtol=None, atol=None, names=('a', 'b'), equal_nan=False, use_broadcast=True, mismatches=(10, 10))[source]

Test that two numpy arrays are almost equal. Raise exception message if not.

Parameters

  • a (np.ndarray or mx.nd.array) –

  • b (np.ndarray or mx.nd.array) –

  • rtol (None or float) – The relative threshold. Default threshold will be used if set to None.

  • atol (None or float) – The absolute threshold. Default threshold will be used if set to None.

  • names (tuple of names, optional) – The names used in error message when an exception occurs

  • equal_nan (boolean, optional) – The flag determining how to treat NAN values in comparison

  • mismatches (tuple of mismatches) – Maximum number of mismatches to be printed (mismatches[0]) and determine (mismatches[1])

mxnet.test_utils.assert_almost_equal_ignore_nan(a, b, rtol=None, atol=None, names=('a', 'b'))[source]

Test that two NumPy arrays are almost equal (ignoring NaN in either array). Combines a relative and absolute measure of approximate eqality. If either the relative or absolute check passes, the arrays are considered equal. Including an absolute check resolves issues with the relative check where all array values are close to zero.

Parameters

  • a (np.ndarray) –

  • b (np.ndarray) –

  • rtol (None or float) – The relative threshold. Default threshold will be used if set to None.

  • atol (None or float) – The absolute threshold. Default threshold will be used if set to None.

mxnet.test_utils.assert_almost_equal_with_err(a, b, rtol=None, atol=None, etol=None, names=('a', 'b'), equal_nan=False)[source]

Test that two numpy arrays are almost equal within given error rate. Raise exception message if not.

Parameters

  • a (np.ndarray) –

  • b (np.ndarray) –

  • threshold (None or float) – The checking threshold. Default threshold will be used if set to None.

  • etol (None or float) – The error rate threshold. If etol is float, return true if error_rate < etol even if any error is found.

mxnet.test_utils.assert_exception(f, exception_type, *args, **kwargs)[source]

Test that function f will throw an exception of type given by exception_type

mxnet.test_utils.assign_each(the_input, function)[source]

Return ndarray composed of passing each array value through some function

mxnet.test_utils.assign_each2(input1, input2, function)[source]

Return ndarray composed of passing two array values through some function

mxnet.test_utils.check_consistency(sym, ctx_list, scale=1.0, grad_req='write', arg_params=None, aux_params=None, tol=None, raise_on_err=True, ground_truth=None, equal_nan=False, use_uniform=False, rand_type=<class 'numpy.float64'>)[source]

Check symbol gives the same output for different running context

Parameters

  • sym (Symbol or list of Symbols) – Symbol(s) to run the consistency test.

  • ctx_list (list) – Running context. See example for more detail.

  • scale (float, optional) – Standard deviation of the inner normal distribution. Used in initialization.

  • grad_req (str or list of str or dict of str to str) – Gradient requirement.

  • use_unifrom (bool) – Optional, When flag set to true, random input data generated follows uniform distribution, not normal distribution

  • rand_type (np.dtype) – casts the randomly generated data to this type Optional, when input data is passed via arg_params, defaults to np.float64 (numpy float default)

Examples

>>> # create the symbol
>>> sym = mx.sym.Convolution(num_filter=3, kernel=(3,3), name='conv')
>>> # initialize the running context
>>> ctx_list =[{'ctx': mx.gpu(0), 'conv_data': (2, 2, 10, 10), 'type_dict': {'conv_data': np.float64}}, {'ctx': mx.gpu(0), 'conv_data': (2, 2, 10, 10), 'type_dict': {'conv_data': np.float32}}, {'ctx': mx.gpu(0), 'conv_data': (2, 2, 10, 10), 'type_dict': {'conv_data': np.float16}}, {'ctx': mx.cpu(0), 'conv_data': (2, 2, 10, 10), 'type_dict': {'conv_data': np.float64}}, {'ctx': mx.cpu(0), 'conv_data': (2, 2, 10, 10), 'type_dict': {'conv_data': np.float32}}]
>>> check_consistency(sym, ctx_list)
>>> sym = mx.sym.Concat(name='concat', num_args=2)
>>> ctx_list = [{'ctx': mx.gpu(0), 'concat_arg1': (2, 10), 'concat_arg0': (2, 10),  'type_dict': {'concat_arg0': np.float64, 'concat_arg1': np.float64}}, {'ctx': mx.gpu(0), 'concat_arg1': (2, 10), 'concat_arg0': (2, 10),  'type_dict': {'concat_arg0': np.float32, 'concat_arg1': np.float32}}, {'ctx': mx.gpu(0), 'concat_arg1': (2, 10), 'concat_arg0': (2, 10),  'type_dict': {'concat_arg0': np.float16, 'concat_arg1': np.float16}}, {'ctx': mx.cpu(0), 'concat_arg1': (2, 10), 'concat_arg0': (2, 10),  'type_dict': {'concat_arg0': np.float64, 'concat_arg1': np.float64}}, {'ctx': mx.cpu(0), 'concat_arg1': (2, 10), 'concat_arg0': (2, 10),  'type_dict': {'concat_arg0': np.float32, 'concat_arg1': np.float32}}]
>>> check_consistency(sym, ctx_list)
mxnet.test_utils.check_numeric_gradient(sym, location, aux_states=None, numeric_eps=0.001, rtol=0.01, atol=None, grad_nodes=None, use_forward_train=True, ctx=None, grad_stype_dict=None, dtype=<class 'numpy.float32'>)[source]

Verify an operation by checking backward pass via finite difference method.

Based on Theano’s theano.gradient.verify_grad [1]

Parameters

  • sym (Symbol) – Symbol containing op to test

  • location (list or tuple or dict) –

    Argument values used as location to compute gradient

    • if type is list of numpy.ndarray, inner elements should have the same order as mxnet.sym.list_arguments().

    • if type is dict of str -> numpy.ndarray, maps the name of arguments to the corresponding numpy.ndarray.

    In either case, value of all the arguments must be provided.

  • aux_states (list or tuple or dict, optional) – The auxiliary states required when generating the executor for the symbol.

  • numeric_eps (float, optional) – Delta for the finite difference method that approximates the gradient.

  • check_eps (float, optional) – relative error eps used when comparing numeric grad to symbolic grad.

  • grad_nodes (None or list or tuple or dict, optional) – Names of the nodes to check gradient on

  • use_forward_train (bool) – Whether to use is_train=True when computing the finite-difference.

  • ctx (Context, optional) – Check the gradient computation on the specified device.

  • grad_stype_dict (dict of str->str, optional) – Storage type dictionary for gradient ndarrays.

  • dtype (np.float16 or np.float32 or np.float64) – Datatype for mx.nd.array.

References

[1] https://github.com/Theano/Theano/blob/master/theano/gradient.py

mxnet.test_utils.check_speed(sym, location=None, ctx=None, N=20, grad_req=None, typ='whole', **kwargs)[source]

Check the running speed of a symbol.

Parameters

  • sym (Symbol) – Symbol to run the speed test.

  • location (none or dict of str to np.ndarray) – Location to evaluate the inner executor.

  • ctx (Context) – Running context.

  • N (int, optional) – Repeat times.

  • grad_req (None or str or list of str or dict of str to str, optional) – Gradient requirements.

  • typ (str, optional) –

    “whole” or “forward”

    • ”whole”

      Test the forward_backward speed.

    • ”forward”

      Only test the forward speed.

mxnet.test_utils.check_symbolic_backward(sym, location, out_grads, expected, rtol=1e-05, atol=None, aux_states=None, grad_req='write', ctx=None, grad_stypes=None, equal_nan=False, dtype=<class 'numpy.float32'>)[source]

Compares a symbol’s backward results with the expected ones. Prints error messages if the backward results are not the same as the expected results.

Parameters

  • sym (Symbol) – output symbol

  • location (list of np.ndarray or dict of str to np.ndarray) –

    The evaluation point

    • if type is list of np.ndarray

      Contains all the NumPy arrays corresponding to mx.sym.list_arguments.

    • if type is dict of str to np.ndarray

      Contains the mapping between argument names and their values.

  • out_grads (None or list of np.ndarray or dict of str to np.ndarray) –

    NumPys arrays corresponding to sym.outputs for incomming gradient.

    • if type is list of np.ndarray

      Contains arrays corresponding to exe.outputs.

    • if type is dict of str to np.ndarray

      contains mapping between mxnet.sym.list_output() and Executor.outputs

  • expected (list of np.ndarray or dict of str to np.ndarray) –

    expected gradient values

    • if type is list of np.ndarray

      Contains arrays corresponding to exe.grad_arrays

    • if type is dict of str to np.ndarray

      Contains mapping between sym.list_arguments() and exe.outputs.

  • check_eps (float, optional) – Relative error to check to.

  • aux_states (list of np.ndarray or dict of str to np.ndarray) –

  • grad_req (str or list of str or dict of str to str, optional) – Gradient requirements. ‘write’, ‘add’ or ‘null’.

  • ctx (Context, optional) – Running context.

  • grad_stypes (dict of str->str) – dictionary of mapping argument name to stype for the gradient

  • equal_nan (Boolean) – if True, nan is a valid value for checking equivalency (ie nan == nan)

  • dtype (np.float16 or np.float32 or np.float64) – Datatype for mx.nd.array.

Example

>>> lhs = mx.symbol.Variable('lhs')
>>> rhs = mx.symbol.Variable('rhs')
>>> sym_add = mx.symbol.elemwise_add(lhs, rhs)
>>> mat1 = np.array([[1, 2], [3, 4]])
>>> mat2 = np.array([[5, 6], [7, 8]])
>>> grad1 = mx.nd.zeros(shape)
>>> grad2 = mx.nd.zeros(shape)
>>> exec_add = sym_add.bind(default_context(), args={'lhs': mat1, 'rhs': mat2},
... args_grad={'lhs': grad1, 'rhs': grad2}, grad_req={'lhs': 'write', 'rhs': 'write'})
>>> exec_add.forward(is_train=True)
>>> ograd = mx.nd.ones(shape)
>>> grad_expected = ograd.copy().asnumpy()
>>> check_symbolic_backward(sym_add, [mat1, mat2], [ograd], [grad_expected, grad_expected])
mxnet.test_utils.check_symbolic_forward(sym, location, expected, rtol=0.0001, atol=None, aux_states=None, ctx=None, equal_nan=False, dtype=<class 'numpy.float32'>)[source]

Compares a symbol’s forward results with the expected ones. Prints error messages if the forward results are not the same as the expected ones.

Parameters

  • sym (Symbol) – output symbol

  • location (list of np.ndarray or dict of str to np.ndarray) –

    The evaluation point

    • if type is list of np.ndarray

      Contains all the numpy arrays corresponding to sym.list_arguments().

    • if type is dict of str to np.ndarray

      Contains the mapping between argument names and their values.

  • expected (list of np.ndarray or dict of str to np.ndarray) –

    The expected output value

    • if type is list of np.ndarray

      Contains arrays corresponding to exe.outputs.

    • if type is dict of str to np.ndarray

      Contains mapping between sym.list_output() and exe.outputs.

  • check_eps (float, optional) – Relative error to check to.

  • aux_states (list of np.ndarray of dict, optional) –

    • if type is list of np.ndarray

      Contains all the NumPy arrays corresponding to sym.list_auxiliary_states

    • if type is dict of str to np.ndarray

      Contains the mapping between names of auxiliary states and their values.

  • ctx (Context, optional) – running context

  • dtype ("asnumpy" or np.float16 or np.float32 or np.float64) – If dtype is “asnumpy” then the mx.nd.array created will have the same type as th numpy array from which it is copied. Otherwise, dtype is the explicit datatype for all mx.nd.array objects created in this function.

  • equal_nan (Boolean) – if True, nan is a valid value for checking equivalency (ie nan == nan)

Example

>>> shape = (2, 2)
>>> lhs = mx.symbol.Variable('lhs')
>>> rhs = mx.symbol.Variable('rhs')
>>> sym_dot = mx.symbol.dot(lhs, rhs)
>>> mat1 = np.array([[1, 2], [3, 4]])
>>> mat2 = np.array([[5, 6], [7, 8]])
>>> ret_expected = np.array([[19, 22], [43, 50]])
>>> check_symbolic_forward(sym_dot, [mat1, mat2], [ret_expected])
mxnet.test_utils.chi_square_check(generator, buckets, probs, nsamples=1000000)[source]

Run the chi-square test for the generator. The generator can be both continuous and discrete.

If the generator is continuous, the buckets should contain tuples of (range_min, range_max) and the probs should be the corresponding ideal probability within the specific ranges. Otherwise, the buckets should contain all the possible values generated over the discrete distribution and the probs should be groud-truth probability.

Usually the user is required to specify the probs parameter.

After obtaining the p value, we could further use the standard p > 0.05 (alpha) threshold to get the final result.

Examples:

buckets, probs = gen_buckets_probs_with_ppf(lambda x: ss.norm.ppf(x, 0, 1), 5)
generator = lambda x: np.random.normal(0, 1.0, size=x)
p = chi_square_check(generator=generator, buckets=buckets, probs=probs)
assert(p > 0.05)
Parameters

  • generator (function) – A function that is assumed to generate i.i.d samples from a specific distribution. generator(N) should generate N random samples.

  • buckets (list of tuple or list of number) – The buckets to run the chi-square the test. Make sure that the buckets cover the whole range of the distribution. Also, the buckets must be in ascending order and have no intersection

  • probs (list or tuple) – The ground-truth probability of the random value fall in a specific bucket.

  • nsamples (int) – The number of samples to generate for the testing

Returns

  • p (float) – p value that the generator has the expected distribution. A higher value indicates a larger confidence

  • obs_freq (list) – Observed frequency of buckets

  • expected_freq (list) – The expected (ground-truth) frequency of the buckets

mxnet.test_utils.collapse_sum_like(a, shape)[source]

Given a as a numpy ndarray, perform reduce_sum on a over the axes that do not exist in shape. Note that an ndarray with shape must be broadcastable to a.

mxnet.test_utils.compare_ndarray_tuple(t1, t2, rtol=None, atol=None)[source]

Compare ndarray tuple.

mxnet.test_utils.compare_optimizer(opt1, opt2, shape, dtype, w_stype='default', g_stype='default', rtol=0.0001, atol=1e-05, compare_states=True)[source]

Compare opt1 and opt2.

mxnet.test_utils.create_sparse_array(shape, stype, data_init=None, rsp_indices=None, dtype=None, modifier_func=None, density=0.5, shuffle_csr_indices=False)[source]

Create a sparse array, For Rsp, assure indices are in a canonical format

mxnet.test_utils.create_sparse_array_zd(shape, stype, density, data_init=None, rsp_indices=None, dtype=None, modifier_func=None, shuffle_csr_indices=False)[source]

Create sparse array, using only rsp_indices to determine density

mxnet.test_utils.default_context()[source]

Get default context for regression test.

mxnet.test_utils.default_dtype()[source]

Get default data type for regression test.

mxnet.test_utils.discard_stderr()[source]

Discards error output of a routine if invoked as:

with discard_stderr():

…

mxnet.test_utils.download(url, fname=None, dirname=None, overwrite=False, retries=5)[source]

Download an given URL

Parameters

  • url (str) – URL to download

  • fname (str, optional) – filename of the downloaded file. If None, then will guess a filename from url.

  • dirname (str, optional) – output directory name. If None, then guess from fname or use the current directory

  • overwrite (bool, optional) – Default is false, which means skipping download if the local file exists. If true, then download the url to overwrite the local file if exists.

  • retries (integer, default 5) – The number of times to attempt the download in case of failure or non 200 return codes

Returns

The filename of the downloaded file

Return type

str

mxnet.test_utils.download_model(model_name, dst_dir='./', meta_info=None)[source]

Download a model from data.mxnet.io

Parameters

  • model_name (str) – Model name to download

  • dst_dir (str) – Destination Directory to download the model

  • meta_info (dict of dict) – Mapping from model_name to dict of the following structure: {‘symbol’: url, ‘params’: url}

Returns

Return type

Two element tuple containing model_name and epoch for the params saved

mxnet.test_utils.find_max_violation(a, b, rtol=None, atol=None)[source]

Finds and returns the location of maximum violation.

mxnet.test_utils.gen_buckets_probs_with_ppf(ppf, nbuckets)[source]
Generate the buckets and probabilities for chi_square test when the ppf (Quantile function)

is specified.

Parameters

  • ppf (function) – The Quantile function that takes a probability and maps it back to a value. It’s the inverse of the cdf function

  • nbuckets (int) – size of the buckets

Returns

  • buckets (list of tuple) – The generated buckets

  • probs (list) – The generate probabilities

mxnet.test_utils.get_atol(atol=None)[source]

Get default numerical threshold for regression test.

mxnet.test_utils.get_bz2_data(data_dir, data_name, url, data_origin_name)[source]

Download and extract bz2 data.

Parameters

  • data_dir (str) – Absolute or relative path of the directory name to store bz2 files

  • data_name (str) – Name of the output file in which bz2 contents will be extracted

  • url (str) – URL to download data from

  • data_origin_name (str) – Name of the downloaded b2 file

Examples

>>> get_bz2_data("data_dir", "kdda.t",
                 "https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/kdda.t.bz2",
                 "kdda.t.bz2")
mxnet.test_utils.get_cifar10()[source]

Downloads CIFAR10 dataset into a directory in the current directory with the name data, and then extracts all files into the directory data/cifar.

mxnet.test_utils.get_etol(etol=None)[source]

Get default numerical threshold for regression test.

mxnet.test_utils.get_im2rec_path(home_env='MXNET_HOME')[source]

Get path to the im2rec.py tool

Parameters

home_env (str) – Env variable that holds the path to the MXNET folder

Returns

The path to im2rec.py

Return type

str

mxnet.test_utils.get_mnist()[source]

Download and load the MNIST dataset

Returns

A dict containing the data

Return type

dict

mxnet.test_utils.get_mnist_iterator(batch_size, input_shape, num_parts=1, part_index=0)[source]

Returns training and validation iterators for MNIST dataset

mxnet.test_utils.get_mnist_pkl()[source]

Downloads MNIST dataset as a pkl.gz into a directory in the current directory with the name data

mxnet.test_utils.get_mnist_ubyte()[source]

Downloads ubyte version of the MNIST dataset into a directory in the current directory with the name data and extracts all files in the zip archive to this directory.

mxnet.test_utils.get_rtol(rtol=None)[source]

Get default numerical threshold for regression test.

mxnet.test_utils.get_zip_data(data_dir, url, data_origin_name)[source]

Download and extract zip data.

Parameters

  • data_dir (str) – Absolute or relative path of the directory name to store zip files

  • url (str) – URL to download data from

  • data_origin_name (str) – Name of the downloaded zip file

Examples

>>> get_zip_data("data_dir",
                 "http://files.grouplens.org/datasets/movielens/ml-10m.zip",
                 "ml-10m.zip")
mxnet.test_utils.has_tvm_ops()[source]

Returns True if MXNet is compiled with TVM generated operators. If current ctx is GPU, it only returns True for CUDA compute capability > 52 where FP16 is supported.

mxnet.test_utils.is_cd_run()[source]

Checks if the test is running as part of a Continuous Delivery run

mxnet.test_utils.is_op_runnable()[source]

Returns True for all CPU tests. Returns True for GPU tests that are either of the following. 1. Built with USE_TVM_OP=0. 2. Built with USE_TVM_OP=1, but with compute capability >= 53.

mxnet.test_utils.list_gpus()[source]

Return a list of GPUs

Returns

If there are n GPUs, then return a list [0,1,…,n-1]. Otherwise returns [].

Return type

list of int

mxnet.test_utils.mean_check(generator, mu, sigma, nsamples=1000000)[source]

Test the generator by matching the mean.

We test the sample mean by checking if it falls inside the range

(mu - 3 * sigma / sqrt(n), mu + 3 * sigma / sqrt(n))

References:

@incollection{goucher2009beautiful,
      title={Beautiful Testing: Leading Professionals Reveal How They Improve Software},
      author={Goucher, Adam and Riley, Tim},
      year={2009},
      chapter=10
}

Examples:

generator = lambda x: np.random.normal(0, 1.0, size=x)
mean_check_ret = mean_check(generator, 0, 1.0)
Parameters

  • generator (function) – The generator function. It’s expected to generate N i.i.d samples by calling generator(N).

  • mu (float) –

  • sigma (float) –

  • nsamples (int) –

Returns

ret – Whether the mean test succeeds

Return type

bool

mxnet.test_utils.np_reduce(dat, axis, keepdims, numpy_reduce_func)[source]

Compatible reduce for old version of NumPy.

Parameters

  • dat (np.ndarray) – Same as NumPy.

  • axis (None or int or list-like) – Same as NumPy.

  • keepdims (bool) – Same as NumPy.

  • numpy_reduce_func (function) – A NumPy reducing function like np.sum or np.max.

mxnet.test_utils.numeric_grad(executor, location, aux_states=None, eps=0.0001, use_forward_train=True, dtype=<class 'numpy.float32'>)[source]

Calculates a numeric gradient via finite difference method.

Class based on Theano’s theano.gradient.numeric_grad [1]

Parameters

  • executor (Executor) – Executor that computes the forward pass.

  • location (list of numpy.ndarray or dict of str to numpy.ndarray) – Argument values used as location to compute gradient Maps the name of arguments to the corresponding numpy.ndarray. Value of all the arguments must be provided.

  • aux_states (None or list of numpy.ndarray or dict of str to numpy.ndarray, optional) – Auxiliary states values used as location to compute gradient Maps the name of aux_states to the corresponding numpy.ndarray. Value of all the auxiliary arguments must be provided.

  • eps (float, optional) – Epsilon for the finite-difference method.

  • use_forward_train (bool, optional) – Whether to use is_train=True in testing.

  • dtype (np.float16 or np.float32 or np.float64) – Datatype for mx.nd.array.

References

..[1] https://github.com/Theano/Theano/blob/master/theano/gradient.py

mxnet.test_utils.rand_ndarray(shape, stype='default', density=None, dtype=None, modifier_func=None, shuffle_csr_indices=False, distribution=None, ctx=None)[source]

Generate a random sparse ndarray. Returns the generated ndarray.

mxnet.test_utils.rand_sparse_ndarray(shape, stype, density=None, dtype=None, distribution=None, data_init=None, rsp_indices=None, modifier_func=None, shuffle_csr_indices=False, ctx=None)[source]

Generate a random sparse ndarray. Returns the ndarray, value(np) and indices(np)

Parameters

  • shape (list or tuple) –

  • stype (str) – valid values: “csr” or “row_sparse”

  • density (float, optional) – should be between 0 and 1

  • distribution (str, optional) – valid values: “uniform” or “powerlaw”

  • dtype (numpy.dtype, optional) – default value is None

Returns

Return type

Result of type CSRNDArray or RowSparseNDArray

Examples

Below is an example of the powerlaw distribution with csr as the stype. It calculates the nnz using the shape and density. It fills up the ndarray with exponentially increasing number of elements. If there are enough unused_nnzs, n+1th row will have twice more nnzs compared to nth row. else, remaining unused_nnzs will be used in n+1th row If number of cols is too small and we have already reached column size it will fill up all following columns in all followings rows until we reach the required density.

>>> csr_arr, _ = rand_sparse_ndarray(shape=(5, 16), stype="csr",
                                     density=0.50, distribution="powerlaw")
>>> indptr = csr_arr.indptr.asnumpy()
>>> indices = csr_arr.indices.asnumpy()
>>> data = csr_arr.data.asnumpy()
>>> row2nnz = len(data[indptr[1]:indptr[2]])
>>> row3nnz = len(data[indptr[2]:indptr[3]])
>>> assert(row3nnz == 2*row2nnz)
>>> row4nnz = len(data[indptr[3]:indptr[4]])
>>> assert(row4nnz == 2*row3nnz)
mxnet.test_utils.random_arrays(*shapes)[source]

Generate some random numpy arrays.

mxnet.test_utils.random_sample(population, k)[source]

Return a k length list of the elements chosen from the population sequence.

mxnet.test_utils.retry(n)[source]

Retry n times before failing for stochastic test cases.

mxnet.test_utils.same(a, b)[source]

Test if two NumPy arrays are the same.

Parameters

  • a (np.ndarray) –

  • b (np.ndarray) –

mxnet.test_utils.same_array(array1, array2)[source]

Check whether two NDArrays sharing the same memory block

Parameters

  • array1 (NDArray) – First NDArray to be checked

  • array2 (NDArray) – Second NDArray to be checked

Returns

Whether two NDArrays share the same memory

Return type

bool

mxnet.test_utils.same_symbol_structure(sym1, sym2)[source]

Compare two symbols to check if they have the same computation graph structure. Returns true if operator corresponding to a particular node id is same in both symbols for all nodes

mxnet.test_utils.set_default_context(ctx)[source]

Set default context.

mxnet.test_utils.set_env_var(key, val, default_val='')[source]

Set environment variable

Parameters

  • key (str) – Env var to set

  • val (str) – New value assigned to the env var

  • default_val (str, optional) – Default value returned if the env var doesn’t exist

Returns

The value of env var before it is set to the new value

Return type

str

mxnet.test_utils.shuffle_csr_column_indices(csr)[source]

Shuffle CSR column indices per row This allows validation of unordered column indices, which is not a requirement for a valid CSR matrix

mxnet.test_utils.simple_forward(sym, ctx=None, is_train=False, **inputs)[source]

A simple forward function for a symbol.

Primarily used in doctest to test the functionality of a symbol. Takes NumPy arrays as inputs and outputs are also converted to NumPy arrays.

Parameters

  • ctx (Context) – If None, will take the default context.

  • inputs (keyword arguments) – Mapping each input name to a NumPy array.

Returns

  • The result as a numpy array. Multiple results will

  • be returned as a list of NumPy arrays.

mxnet.test_utils.var_check(generator, sigma, nsamples=1000000)[source]

Test the generator by matching the variance. It will need a large number of samples and is not recommended to use

We test the sample variance by checking if it falls inside the range

(sigma^2 - 3 * sqrt(2 * sigma^4 / (n-1)), sigma^2 + 3 * sqrt(2 * sigma^4 / (n-1)))

References:

@incollection{goucher2009beautiful,
      title={Beautiful Testing: Leading Professionals Reveal How They Improve Software},
      author={Goucher, Adam and Riley, Tim},
      year={2009},
      chapter=10
}

Examples:

generator = lambda x: np.random.normal(0, 1.0, size=x)
var_check_ret = var_check(generator, 0, 1.0)
Parameters

  • generator (function) – The generator function. It’s expected to generate N i.i.d samples by calling generator(N).

  • sigma (float) –

  • nsamples (int) –

Returns

ret – Whether the variance test succeeds

Return type

bool

mxnet.test_utils.verify_generator(generator, buckets, probs, nsamples=1000000, nrepeat=5, success_rate=0.2, alpha=0.05)[source]

Verify whether the generator is correct using chi-square testing.

The test is repeated for “nrepeat” times and we check if the success rate is

above the threshold (25% by default).

Parameters

  • generator (function) –

    A function that is assumed to generate i.i.d samples from a specific distribution.

    generator(N) should generate N random samples.

  • buckets (list of tuple or list of number) –

    The buckets to run the chi-square the test. Make sure that the buckets cover

    the whole range of the distribution. Also, the buckets must be in ascending order and have no intersection

  • probs (list or tuple) – The ground-truth probability of the random value fall in a specific bucket.

  • nsamples (int) – The number of samples to generate for the testing

  • nrepeat (int) – The times to repeat the test

  • success_rate (float) – The desired success rate

  • alpha (float) – The desired threshold for type-I error i.e. when a true null hypothesis is rejected

Returns

cs_ret_l – The p values of the chi-square test.

Return type

list