SHOGUN  v1.1.0
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
List of all members | Public Member Functions | Protected Member Functions
CCrossValidation Class Reference

Detailed Description

base class for cross-validation evaluation. Given a learning machine, a splitting strategy, an evaluation criterium, features and correspnding labels, this provides an interface for cross-validation. Results may be retrieved using the evaluate method. A number of repetitions may be specified for obtaining more accurate results. The arithmetic mean of different runs is returned along with confidence intervals, if a p-value is specified. Default number of runs is one, confidence interval combutation is disabled.

This class calculates an evaluation criterium of every fold and then calculates the arithmetic mean of all folds. This is for example suitable for the AUC or for Accuracy. However, for example F1-measure may not be merged this way (result will be biased). To solve this, different sub-classes may average results of each cross validation fold differently by overwriting the evaluate_one_run method.

See [Forman, G. and Scholz, M. (2009). Apples-to-apples in cross-validation studies: Pitfalls in classifier performance measurement. Technical report, HP Laboratories.] for details on this subject.

Definition at line 81 of file CrossValidation.h.

Inheritance diagram for CCrossValidation:
Inheritance graph
[legend]

Public Member Functions

 CCrossValidation ()
 CCrossValidation (CMachine *machine, CFeatures *features, CLabels *labels, CSplittingStrategy *splitting_strategy, CEvaluation *evaluation_criterium)
virtual ~CCrossValidation ()
EEvaluationDirection get_evaluation_direction ()
CrossValidationResult evaluate ()
CMachineget_machine () const
void set_num_runs (int32_t num_runs)
void set_conf_int_alpha (float64_t m_conf_int_alpha)
virtual const char * get_name () const
- Public Member Functions inherited from CSGObject
 CSGObject ()
 CSGObject (const CSGObject &orig)
virtual ~CSGObject ()
virtual bool is_generic (EPrimitiveType *generic) const
template<class T >
void set_generic ()
void unset_generic ()
virtual void print_serializable (const char *prefix="")
virtual bool save_serializable (CSerializableFile *file, const char *prefix="")
virtual bool load_serializable (CSerializableFile *file, const char *prefix="")
void set_global_io (SGIO *io)
SGIOget_global_io ()
void set_global_parallel (Parallel *parallel)
Parallelget_global_parallel ()
void set_global_version (Version *version)
Versionget_global_version ()
SGVector< char * > get_modelsel_names ()
char * get_modsel_param_descr (const char *param_name)
index_t get_modsel_param_index (const char *param_name)

Protected Member Functions

virtual float64_t evaluate_one_run ()
- Protected Member Functions inherited from CSGObject
virtual void load_serializable_pre () throw (ShogunException)
virtual void load_serializable_post () throw (ShogunException)
virtual void save_serializable_pre () throw (ShogunException)
virtual void save_serializable_post () throw (ShogunException)

Additional Inherited Members

- Public Attributes inherited from CSGObject
SGIOio
Parallelparallel
Versionversion
Parameterm_parameters
Parameterm_model_selection_parameters

Constructor & Destructor Documentation

constructor

Definition at line 20 of file CrossValidation.cpp.

CCrossValidation ( CMachine machine,
CFeatures features,
CLabels labels,
CSplittingStrategy splitting_strategy,
CEvaluation evaluation_criterium 
)

constructor

Parameters
machinelearning machine to use
featuresfeatures to use for cross-validation
labelslabels that correspond to the features
splitting_strategysplitting strategy to use
evaluation_criteriumevaluation criterium to use

Definition at line 25 of file CrossValidation.cpp.

~CCrossValidation ( )
virtual

destructor

Definition at line 44 of file CrossValidation.cpp.

Member Function Documentation

CrossValidationResult evaluate ( )

method for evaluation. Performs cross-validation. Is repeated m_num_runs. If this number is larger than one, a confidence interval is calculated if m_conf_int_alpha is (0<p<1). By default m_num_runs=1 and m_conf_int_alpha=0

Returns
result of evaluation

Definition at line 87 of file CrossValidation.cpp.

float64_t evaluate_one_run ( )
protectedvirtual

Evaluates one single cross-validation run. Current implementation evaluates each fold separately and then calculates arithmetic mean. Suitable for accuracy and AUC for example. NOT for F1-measure. Has to be overridden by sub-classes if results have to be merged differently

Returns
evaluation result of one cross-validation run

Definition at line 136 of file CrossValidation.cpp.

EEvaluationDirection get_evaluation_direction ( )
Returns
in which direction is the best evaluation value?

Definition at line 53 of file CrossValidation.cpp.

CMachine * get_machine ( ) const
Returns
underlying learning machine

Definition at line 81 of file CrossValidation.cpp.

virtual const char* get_name ( ) const
virtual
Returns
name of the SGSerializable

Implements CSGObject.

Definition at line 123 of file CrossValidation.h.

void set_conf_int_alpha ( float64_t  m_conf_int_alpha)

setter for the number of runs to use for evaluation

Definition at line 117 of file CrossValidation.cpp.

void set_num_runs ( int32_t  num_runs)

setter for the number of runs to use for evaluation

Definition at line 128 of file CrossValidation.cpp.


The documentation for this class was generated from the following files:

SHOGUN Machine Learning Toolbox - Documentation