Batch Normalization

Collaboration diagram for Batch Normalization:

Functions
mkldnn_status_t MKLDNN_API	mkldnn_batch_normalization_forward_desc_init (mkldnn_batch_normalization_desc_t *bnrm_desc, mkldnn_prop_kind_t prop_kind, const mkldnn_memory_desc_t *data_desc, float epsilon, unsigned flags)
mkldnn_status_t MKLDNN_API	mkldnn_batch_normalization_backward_desc_init (mkldnn_batch_normalization_desc_t *bnrm_desc, mkldnn_prop_kind_t prop_kind, const mkldnn_memory_desc_t *diff_data_desc, const mkldnn_memory_desc_t *data_desc, float epsilon, unsigned flags)

Detailed Description

A primitive to perform batch normalization.

Both forward and backward passes support in-place operation; that is, src and dst point to the same memory for forward pass, and diff_dst and diff_src point to the same memory for backward pass.

Batch normalization supports different flavors controlled by mkldnn_batch_normalization_desc_t. For example, batch normalization can compute the mean and variance on its own or take them as inputs. It can either perform scaling and shifting using gamma and beta parameters or not. Optionally, it can also perform a fused ReLU, which in case of training would also require a workspace.

See also

mkldnn_batch_normalization_desc_t

dev_guide_batch_normalization in developer guide

cpp_api_batch_normalization in cpp_api

Function Documentation

mkldnn_status_t MKLDNN_API mkldnn_batch_normalization_backward_desc_init	(	mkldnn_batch_normalization_desc_t *	bnrm_desc,
		mkldnn_prop_kind_t	prop_kind,
		const mkldnn_memory_desc_t *	diff_data_desc,
		const mkldnn_memory_desc_t *	data_desc,
		float	epsilon,
		unsigned	flags
	)

Initializes a batch normalization descriptor bnrm_desc for backward propagation with respect to data and scale-shift parameters using memory descriptors data_desc and diff_data_desc, normalization parameter epsilon, and flags set using bit flags of type mkldnn_batch_normalization_desc_t.

Inputs:

• src (mkldnn_query_src_md, 0)
• mean (mkldnn_query_src_md, 1)
• variance (mkldnn_query_src_md, 2)
• diff_dst (mkldnn_query_diff_dst_md, 0)
• scale_and_shift (mkldnn_query_weights_md, 0), if mkldnn_use_scaleshift bit-flags is set in flags
• workspace (mkldnn_query_workspace_md, 0), if mkldnn_fuse_norm_relu bit-flags is set in flags

Outputs:

• diff_src (mkldnn_query_diff_src_md, 0)
• diff_scale_and_shift (mkldnn_query_diff_weights_md, 0), if mkldnn_use_scaleshift bit-flags is set in flags and prop_kind = mkldnn_backward

Note

in-place operation is supported, i.e. diff_src points to the same memory as diff_dst.

See also

mkldnn_batch_normalization_desc_t

mkldnn_status_t MKLDNN_API mkldnn_batch_normalization_forward_desc_init	(	mkldnn_batch_normalization_desc_t *	bnrm_desc,
		mkldnn_prop_kind_t	prop_kind,
		const mkldnn_memory_desc_t *	data_desc,
		float	epsilon,
		unsigned	flags
	)

Initializes a batch normalization descriptor bnrm_desc for forward propagation using prop_kind (possible values are mkldnn_forward_training and mkldnn_forward_inference), memory descriptor data_desc, normalization parameter epsilon, and flags set using bit flags of type mkldnn_batch_normalization_desc_t.

Inputs:

• src (mkldnn_query_src_md, 0)
• mean (mkldnn_query_src_md, 1), if mkldnn_use_global_stats bit-flags is set in flags
• variance (mkldnn_query_src_md, 2), if mkldnn_use_global_stats bit-flags is set in flags
• scale_and_shift (mkldnn_query_weights_md, 0), if mkldnn_use_scaleshift bit-flags is set in flags

Outputs:

• dst (mkldnn_query_dst_md, 0)
• mean (mkldnn_query_dst_md, 1), if mkldnn_use_global_stats bit-flags is not set in flags prop_kind = mkldnn_forward_training
• variance (mkldnn_query_dst_md, 2), if mkldnn_use_global_stats bit-flags is not set in flags and prop_kind = mkldnn_forward_training
• workspace (mkldnn_query_workspace_md, 0), if mkldnn_fuse_norm_relu bit-flags is set in flags and prop_kind = mkldnn_forward_training

Note

In-place operation is supported; that is, dst points to the same memory as src.

See also

mkldnn_batch_normalization_desc_t