a iterator that was backed by a thread to pull data eagerly from a single producer into a bounded buffer the consumer can pull the data at its own rate More...

#include <threadediter.h>

Inheritance diagram for dmlc::ThreadedIter< DType >:

Collaboration diagram for dmlc::ThreadedIter< DType >:

Classes
class	Producer
	producer class interface that threaditer used as source to preduce the content More...

Public Member Functions
	ThreadedIter (size_t max_capacity=8)
	constructor More...

virtual	~ThreadedIter (void)
	destructor More...

void	Destroy (void)
	destroy all the related resources this is equivalent to destructor, can be used to destroy the threaditer when user think it is appropriate, it is safe to call this multiple times More...

void	set_max_capacity (size_t max_capacity)
	set maximum capacity of the queue More...

void	Init (std::shared_ptr< Producer > producer)
	initialize the producer and start the thread can only be called once More...

void	Init (std::function< bool(DType **)> next, std::function< void()> beforefirst=NotImplemented)
	initialize the producer and start the thread pass in two function(closure) of producer to represent the producer the beforefirst function is optional, and defaults to not implemented NOTE: the closure must remain valid until the ThreadedIter destructs More...

bool	Next (DType **out_dptr)
	get the next data, this function is threadsafe More...

void	Recycle (DType **inout_dptr)
	recycle the data cell, this function is threadsafe the threaditer can reuse the data cell for future data loading More...

void	ThrowExceptionIfSet (void)
	Rethrows exception which is set by the producer. More...

void	ClearException (void)
	clears exception_ptr, called from Init More...

virtual bool	Next (void)
	adapt the iterator interface's Next NOTE: the call to this function is not threadsafe use the other Next instead More...

virtual const DType &	Value (void) const
	adapt the iterator interface's Value NOTE: the call to this function is not threadsafe use the other Next instead More...

virtual void	BeforeFirst (void)
	set the iterator before first location More...

Public Member Functions inherited from dmlc::DataIter< DType >
virtual	~DataIter (void) DMLC_THROW_EXCEPTION
	destructor More...

Detailed Description

template<typename DType>
class dmlc::ThreadedIter< DType >

a iterator that was backed by a thread to pull data eagerly from a single producer into a bounded buffer the consumer can pull the data at its own rate

NOTE: thread concurrency cost time, make sure to store big blob of data in DType

Usage example:

ThreadedIter<DType> iter;
iter.Init(&producer);
// the following code can be in parallel
DType *dptr;
while (iter.Next(&dptr)) {
  // do something on dptr
  // recycle the space
  iter.Recycle(&dptr);
}

Template Parameters

DType the type of data blob we support

Constructor & Destructor Documentation

template<typename DType >

dmlc::ThreadedIter< DType >::ThreadedIter ( size_t max_capacity = 8 )

inlineexplicit

constructor

Parameters

max_capacity maximum capacity of the queue

template<typename DType >

virtual dmlc::ThreadedIter< DType >::~ThreadedIter ( void )

inlinevirtual

destructor

Member Function Documentation

template<typename DType >

virtual void dmlc::ThreadedIter< DType >::BeforeFirst ( void )

inlinevirtual

set the iterator before first location

Implements dmlc::DataIter< DType >.

template<typename DType >

void dmlc::ThreadedIter< DType >::ClearException ( void )

inline

clears exception_ptr, called from Init

template<typename DType >

void dmlc::ThreadedIter< DType >::Destroy ( void )

inline

destroy all the related resources this is equivalent to destructor, can be used to destroy the threaditer when user think it is appropriate, it is safe to call this multiple times

template<typename DType >

void dmlc::ThreadedIter< DType >::Init ( std::shared_ptr< Producer > producer )

inline

initialize the producer and start the thread can only be called once

Parameters

producer pointer to the producer

template<typename DType >

void dmlc::ThreadedIter< DType >::Init	(	std::function< bool(DType **)>	next,
		std::function< void()>	beforefirst = `NotImplemented`
	)

inline

initialize the producer and start the thread pass in two function(closure) of producer to represent the producer the beforefirst function is optional, and defaults to not implemented NOTE: the closure must remain valid until the ThreadedIter destructs

Parameters

next	the function called to get next element, see Producer.Next
beforefirst	the function to call to reset the producer, see Producer.BeforeFirst

template<typename DType >

bool dmlc::ThreadedIter< DType >::Next ( DType ** out_dptr )

inline

get the next data, this function is threadsafe

Parameters

out_dptr used to hold the pointer to the record after the function call, the caller takes ownership of the pointer the caller can call recycle to return ownership back to the threaditer so that the pointer can be re-used

Returns: true if there is next record, false if we reach the end

See also: Recycle

template<typename DType >

virtual bool dmlc::ThreadedIter< DType >::Next ( void )

inlinevirtual

adapt the iterator interface's Next NOTE: the call to this function is not threadsafe use the other Next instead

Returns: true if there is next record, false if we reach the end

Implements dmlc::DataIter< DType >.

template<typename DType >

void dmlc::ThreadedIter< DType >::Recycle ( DType ** inout_dptr )

inline

recycle the data cell, this function is threadsafe the threaditer can reuse the data cell for future data loading

Parameters

inout_dptr pointer to the dptr to recycle, after the function call the content of inout_dptr will be set to NULL

template<typename DType >

void dmlc::ThreadedIter< DType >::set_max_capacity ( size_t max_capacity )

inline

set maximum capacity of the queue

Parameters

max_capacity maximum capacity of the queue

template<typename DType >

void dmlc::ThreadedIter< DType >::ThrowExceptionIfSet ( void )

inline

Rethrows exception which is set by the producer.

template<typename DType >

virtual const DType& dmlc::ThreadedIter< DType >::Value ( void ) const

inlinevirtual

adapt the iterator interface's Value NOTE: the call to this function is not threadsafe use the other Next instead

Implements dmlc::DataIter< DType >.

The documentation for this class was generated from the following file:

/work/mxnet/3rdparty/dmlc-core/include/dmlc/threadediter.h

Classes

Public Member Functions

Detailed Description

template<typename DType> class dmlc::ThreadedIter< DType >

Constructor & Destructor Documentation

Member Function Documentation

template<typename DType>
class dmlc::ThreadedIter< DType >