edflow.data.agnostics.concatenated module

Summary

Classes:

DisjunctExampleConcatenatedDataset	Concatenates a list of disjunct datasets.
ExampleConcatenatedDataset	Concatenates a list of datasets along the example axis.

Reference

class edflow.data.agnostics.concatenated.ExampleConcatenatedDataset(*datasets)

Bases: edflow.data.dataset_mixin.DatasetMixin

Concatenates a list of datasets along the example axis.

Note

All datasets must be of same length and must return examples with the same keys and behind those keys with the same type and shape.

If dataset A returns examples of form {'a': x, 'b': x} and dataset B of form {'a': y, 'b': y} the ExampleConcatenatedDataset(A, B) return examples of form {'a': [x, y], 'b': [x, y]}.

__init__(*datasets)

Parameters

*datasets (DatasetMixin) – All the datasets to concatenate.

set_example_pars(start=None, stop=None, step=None)

Allows to manipulate the length and step of the returned example lists.

property labels

Now each index corresponds to a sequence of labels.

get_example(i)

Note

Please the documentation of DatasetMixin to not be confused.

Add default behaviour for datasets defining an attribute data, which in turn is a dataset. This happens often when stacking several datasets on top of each other.

The default behaviour now is to return self.data.get_example(idx) if possible, and otherwise revert to the original behaviour.

class edflow.data.agnostics.concatenated.DisjunctExampleConcatenatedDataset(*datasets, disjunct=True, same_length=True)

Bases: edflow.data.dataset_mixin.DatasetMixin

Concatenates a list of disjunct datasets.

Note

All datasets must be of same length and labels and returned keys must be disjunct. If labels or keys are not disjunct, set the optional parameter disjunct to False, to use the value of the last dataset containing the key. Datasets can have different length if same_length is set to False.

If dataset A returns examples of form {'a': w, 'b': x} and dataset B of form {'c': y, 'd': z} the DisjunctExampleConcatenatedDataset(A, B) return examples of form {'a': w, 'b': x, 'c': y, 'd': z}.

__init__(*datasets, disjunct=True, same_length=True)

Parameters

• *datasets (DatasetMixin) – All the datasets to concatenate.

• disjunct (bool) – labels and returned keys do not have to be disjunct. Last datasetet overwrites values

• same_length (bool) – Datasets do not have to be of same length. Concatenated dataset has length of smallest dataset.

get_example(i)

Note

Please the documentation of DatasetMixin to not be confused.

Add default behaviour for datasets defining an attribute data, which in turn is a dataset. This happens often when stacking several datasets on top of each other.

The default behaviour now is to return self.data.get_example(idx) if possible, and otherwise revert to the original behaviour.