edflow.data.believers.meta module

Summary

Classes:

MetaDataset

The MetaDataset allows for easy data reading using a simple interface.

Functions:

clean_keys	Removes all loader information from the keys.
load_labels	param root Where to look for the labels.
loader_from_key	Returns the name, loader pair given a key.
setup_loaders	Creates a map of key -> function pairs, which can be used to postprocess label values at each __getitem__ call.

Reference

class edflow.data.believers.meta.MetaDataset(root)

Bases: edflow.data.dataset_mixin.DatasetMixin

The MetaDataset allows for easy data reading using a simple interface.

All you need to do is hand the constructor a path and it will look for all data in a special format and load it as numpy arrays. If further specified in a meta data file or the name of the label array, when calling the getitem method of the dataset, a special loader function will be called.

Let’s take a look at an example data folder of the following structure:

root/
├ meta.yaml
├ images/
│  ├ image_1.png
│  ├ image_2.png
│  ├ image_3.png
...
│  └ image_10000.png
├ image:image-*-10000-*-str.npy
├ attr1-*-10000-*-int.npy
├ attr2-*-10000x2-*-int.npy
└ kps-*-10000x17x3-*-int.npy

The meta.yaml file looks like this:

description: |
    This is a dataset which loads images.
    All paths to the images are in the label `image`.

loader_kwargs:
    image:
        support: "-1->1"

The resulting dataset has the following labels:

• image_: the paths to the images. Note the extra _ at the end.

• attr1

• attr2

• kps

When using the __getitem__ method of the dataset, the image loader will be applied to the image label at the given index and the image will be loaded from the given path.

As we have specifed loader kweyword arguments, we will get the images with a support of [-1, 1].

__init__(root)

Parameters

root (str) – Where to look for all the data.

get_example(idx)

Loads all loadable data from the labels.

Parameters

idx (int) – The index of the example to load

show()

edflow.data.believers.meta.setup_loaders(labels, meta_dict)

Creates a map of key -> function pairs, which can be used to postprocess label values at each __getitem__ call.

Loaders defined in meta_dict supersede those definde in the label keys.

Parameters

• labels (dict(str, numpy.memmap)) – Labels contain all load-easy dataset relevant data. If the key follows the pattern name:loader, this function will try to finde the corresponding loader in DEFAULT_LOADERS.

• meta_dict (dict) – A dictionary containing all dataset relevent information, which is the same for all examples. This function will try to find the entry loaders in the dictionary, which must contain another dict with name:loader pairs. Here loader must be either an entry in DEFAULT_LOADERS or a loadable import path. You can additionally define an entry loader_kwargs, which must contain name:dict pairs. The dictionary is passed as keyword arguments to the loader corresponding to name.

Returns

• loaders (dict) – Name, function pairs, to apply loading logic based on the labels with the specified names.

• loader_kwargs (dict) – Name, dict pairs. The dicts are passed to the loader functions as keyword arguments.

edflow.data.believers.meta.load_labels(root)

Parameters

root (str) – Where to look for the labels.

Returns

labels – All labels as np.memmap s.

Return type

dict

edflow.data.believers.meta.clean_keys(labels, loaders)

Removes all loader information from the keys.

Parameters

labels (dict(str, numpy.memmap)) – Labels contain all load-easy dataset relevant data.

Returns

labels – The original labels, with keys without the :loader part.

Return type

dict(str, numpy.memmap)

edflow.data.believers.meta.loader_from_key(key)

Returns the name, loader pair given a key.