Classifiers used in image processing and computer vision are frequent subject of research and exploitation in applications. This contribution does not directly involve research in the classification itself but rather introduces a systematic approach of evaluation of image classifiers, comparison between the classifiers, and "tuning" the classifiers for particular applications. The proposed approach is included in an open software system for evaluation of the image classifiers. The contribution also demonstrates application of the system on several selected classifiers and discusses the possibilities and results.