edflow.data.agnostics.late_loading module

Summary

Classes:

LateLoadingDataset

The LateLoadingDataset allows to work with examples containing Callables, which are evaluated by this Dataset.

Functions:

expand

Reference

class edflow.data.agnostics.late_loading.LateLoadingDataset(base_dset)

Bases: edflow.data.dataset_mixin.DatasetMixin

The LateLoadingDataset allows to work with examples containing Callables, which are evaluated by this Dataset. This way you can define data loading routines for images or other time consuming things in a base dataset, then add lots of data rearranging logic on top of this base dataset and in the end only load the subset of examples, you really want to use by calling the routines.

class BaseDset:
    def get_example(self, idx):
        def _loading_routine():
            load_image(idx)

        return {'image': _loading_routine}

class AnchorDset:
    def __init__(self):
        B = BaseDset()

        self.S = SequenceDataset(B, 5)

    def get_example(self, idx):
        ex = self.S[idx]

        out = {}
        out['anchor1'] = ex['image'][0]
        out['anchor2'] = ex['image'][-1]

        return out


final_dset = LateLoadingDataset(AnchorDset())

__init__(base_dset)

Initialize self. See help(type(self)) for accurate signature.

get_example(idx)

Note

Please the documentation of DatasetMixin to not be confused.

Add default behaviour for datasets defining an attribute data, which in turn is a dataset. This happens often when stacking several datasets on top of each other.

The default behaviour now is to return self.data.get_example(idx) if possible, and otherwise revert to the original behaviour.

edflow.data.agnostics.late_loading.expand(value)