Usage Scenarios

Script-Only

This scenario basically does not require programming and reveals only the basic capabilities of the library.

To use it, the appropriate environment must be initialized (see Installation). Further, being in the terminal in the project directory in a prepared virtual environment, you can interact with RevelioNN through the following scripts.

• Converting a pre-trained model to the RevelioNN format

$ python convert_to_rvl_format.py <model_filepath> <main_net_modules_directory> <main_net_module_name> <main_net_class> <transformation_name> <img_size_name> <num_channels_name> <rvl_filename> <class_label>

• Printing a dictionary of layers of the main neural network

$ python print_layers_dict.py <main_net_modules_directory> <main_net_module_name> <main_net_class> -l <layer_types>

• Training of the main networks

$ python train_main_nets.py <main_net_modules_directory> <module_name> <main_net_class> <transformation_name> <img_size_name> <num_channels_name> <path_to_images> <path_to_train_csv> <path_to_valid_csv> <image_names_column> -l <label_columns> -d <device>

• Evaluation of the main networks

$ python evaluate_main_nets.py <path_to_images> <path_to_test_csv> <image_names_column> <main_net_modules_directory> -m <main_model_filenames> -d <device>

• Training a single mapping network

$ python train_single_mapping_net.py <main_model_filepath> <main_net_modules_directory> <path_to_images> <path_to_train_csv> <path_to_valid_csv> <image_names_column> <label_column> --layers_types <layer_types> --layers <layers> --num_neurons <num_neurons> -d <device>

• Training a simultaneous mapping network

$ python train_simultaneous_mapping_net.py <main_model_filename> <main_net_modules_directory> <path_to_images> <path_to_train_csv> <path_to_valid_csv> <image_names_column> --label_columns <label_columns> --layers_types <layer_types> --decoder_channels <decoder_channels> --num_shared_neurons <num_shared_neurons> --num_output_neurons <num_output_neurons> -d <device>

• Evaluation of the mapping network

$ python evaluate_mapping_net.py <path_to_images> <path_to_test_csv> <image_names_column> main_net_modules_directory -m <mapping_model_filename> -d <device>

• Extracting concepts from an image

$ python extract_concepts_from_image.py <path_to_img> <main_models_directory> <main_net_modules_directory> -m <mapping_model_filepaths> -d <device>

• Formation of logical explanations based on ontology

$ python form_logical_explanations.py <path_to_img> <path_to_ontology> <concepts_map_directory> <concepts_map_module_name> <concepts_map_name> <target_concept> <main_models_directory> <main_net_modules_directory> -m <mapping_model_filenames> -d <device>

• Formation of visual explanations

$ python form_visual_explanations.py <path_to_img> <mapping_model_filepath> <main_models_directory> <main_net_modules_directory> --window_size <window_size> --stride <stride>

To get detailed information on each of the scripts, you need to run:

$ python <script_name>.py --help

Program-Level

To use the API, follow these steps:

1. Import convert_to_rvl_format() function:

   from revelionn.utils.model import convert_to_rvl_format
   

   Call this function by passing the data of the previously declared network model as parameters:

   convert_to_rvl_format(main_model, filename, class_label, module_name, main_net_class, transformation_name, img_size, num_channels)
   

2. Import MappingTrainer class:

   from revelionn.mapping_trainer import MappingTrainer
   

3. Initialize the MappingTrainer object and define a list of layer types to be identified in the convolutional network. It provides a training/evaluation interface:

   • MappingTrainer.train_single_model() trains a single mapping network for a given concept based on the activations of given layers;

   • MappingTrainer.train_simultaneous_model() trains a simultaneous mapping network for a given set of concepts based on the activations of layers of previously defined types;

   • MappingTrainer.train_simultaneous_model_semisupervised() trains a simultaneous mapping network for a given set of concepts using semi-supervised learning, in which a semantic loss is calculated for unlabeled samples, taking into account the relationships between the concepts;

   • MappingTrainer.evaluate_model() evaluates the mapping network model on the test set using the ROC AUC.

4. Once the mapping network is trained, you can form logical and visual explanations. To do this, you must first load the trained network model via load_mapping_model().

   from revelionn.utils.model import load_mapping_model
   
   main_module, mapping_module, activation_extractor, transformation, img_size =
   load_mapping_model(mapping_model_filepath, main_models_directory, main_net_modules_directory, device)
   

5. To form logical explanations using an ontology, one must first extract the concepts relevant to the target concept from the image, and then transfer the extracted concepts and their probabilities to the reasoning module along with the ontology. This can be done as follows:

   from revelionn.utils.explanation import extract_concepts_from_img, explain_target_concept
   
   image = Image.open(image_path)
   main_concepts, extracted_concepts, mapping_probabilities = extract_concepts_from_img(main_module,
                                                                                     mapping_module,
                                                                                     image,
                                                                                     transformation)
   justifications = explain_target_concept(extracted_concepts, mapping_probabilities, concepts_map, target_concept,
                                        jar_filepath, owl_ontology, temp_files_path)
   print(justifications)
   

6. Visual explanations can be formed as follows:

   import matplotlib.pyplot as plt
   from revelionn.occlusion import perform_occlusion
   
   perform_occlusion(main_module, mapping_module, activation_extractor, transformation, img_size,
                  image_path, window_size, stride, threads)
   plt.show()