edflow.nn.tf_nn module

Summary

Functions:

add_coordinates	Given an input_tensor, adds 2 channelw ith x and y coordinates to the feature maps.
apply_partwise	Applies function func on all parts separately.
downsample	Downsampling by stride 2 convolution
flatten	returns a flat version of x –> [N, -1] :param x: :type x: tensor
get_name	utlity for keeping track of layer names
hourglass_model	A U-net or hourglass style image-to-image model with skip-connections
int_shape	short for x.shape.as_list()
make_ema	apply exponential moving average to variable
make_mask_colors	make a color array using the specified colormap for n_parts classes :param n_parts: how many classes there are in the mask :type n_parts: int :param cmap: matplotlib colormap handle
make_model	Create model with fixed kwargs.
mask2rgb	Convert tensor with masks [N, H, W, C] to an RGB tensor [N, H, W, 3] using argmax over channels.
model_arg_scope	Create new counter and apply arg scope to all arg scoped nn operations.
nin	a network in network layer (1x1 CONV)
np_mask2rgb	numpy equivalent of @mask2rgb convert tensor with masks [N, H, W, C] to an RGB tensor [N, H, W, 3] using argmax over channels.
np_one_hot	numpy equivalent of tf.one_hot returns targets as one hot matrix
np_to_float	cast x to float32
probs_to_mu_L	Calculate mean and covariance (cholesky decomposition of covariance) for each channel of probs tensor of keypoint probabilites [bn, h, w, n_kp] mean calculated on a grid of scale [-1, 1]
probs_to_mu_sigma	Calculate mean and covariance matrix for each channel of spatial probability maps Mean and covariance are caluclated on a grid of scale [-1, 1]
tf_hm	Returns Gaussian densitiy function based on μ and L for each batch index and part L is the cholesky decomposition of the covariance matrix : Σ = L L^T
upsample	2D upsampling layer.

Reference

edflow.nn.tf_nn.model_arg_scope(**kwargs)

Create new counter and apply arg scope to all arg scoped nn operations.

edflow.nn.tf_nn.make_model(name, template, **kwargs)

Create model with fixed kwargs.

edflow.nn.tf_nn.int_shape(x)

short for x.shape.as_list()

edflow.nn.tf_nn.get_name(layer_name, counters)

utlity for keeping track of layer names

edflow.nn.tf_nn.apply_partwise(input_, func)

Applies function func on all parts separately. Parts are in channel 3. The input is reshaped to map the parts to the batch axis and then the function is applied :param input_: [b, h, w, parts, features] :type input_: tensor :param func: a NN function to apply to each part individually :type func: callable

Returns

Return type

[b, out_h, out_w, parts, out_features]

edflow.nn.tf_nn.nin(x, num_units)

a network in network layer (1x1 CONV)

edflow.nn.tf_nn.downsample(x, num_units)

Downsampling by stride 2 convolution

equivalent to x = conv2d(x, num_units, stride = [2, 2])

Parameters

• x (tensor) – input

• num_units – number of feature map in the output

edflow.nn.tf_nn.upsample(x, num_units, method='subpixel')

2D upsampling layer.

Parameters

• x (tensor) – input

• num_units – number of feature maps in the output

• method – upsampling method. A string from: “conv_transposed”, “nearest_neighbor”, “linear”, “subpixel” Subpixel means that every upsampled pixel gets its own filter.

Returns

Return type

upsampled input

edflow.nn.tf_nn.flatten(x)

returns a flat version of x –> [N, -1] :param x: :type x: tensor

edflow.nn.tf_nn.mask2rgb(mask)

Convert tensor with masks [N, H, W, C] to an RGB tensor [N, H, W, 3] using argmax over channels. :param mask: an array of shape [N, H, W, C] :type mask: ndarray :param Returns: RGB visualization in shape [N, H, W, 3] :param ——-:

edflow.nn.tf_nn.np_one_hot(targets, n_classes)

numpy equivalent of tf.one_hot returns targets as one hot matrix

Parameters

• targets (ndarray) – array of target classes

• n_classes (int) – how many classes there are overall

• Returns (ndarray) – one-hot array with shape [n, n_classes]

• ------- –

edflow.nn.tf_nn.np_to_float(x)

cast x to float32

edflow.nn.tf_nn.np_mask2rgb(mask)

numpy equivalent of @mask2rgb convert tensor with masks [N, H, W, C] to an RGB tensor [N, H, W, 3] using argmax over channels. :param mask: an array of shape [N, H, W, C] :type mask: ndarray :param Returns: RGB visualization in shape [N, H, W, 3] :param ——-:

edflow.nn.tf_nn.make_mask_colors(n_parts, cmap=<Mock name='mock.pyplot.cm.inferno' id='140096290281312'>)

make a color array using the specified colormap for n_parts classes :param n_parts: how many classes there are in the mask :type n_parts: int :param cmap: matplotlib colormap handle

Returns

colors – an array with shape [n_parts, 3] representing colors in the range [0, 1].

Return type

ndarray

edflow.nn.tf_nn.hourglass_model(x, config, extra_resnets, n_out=3, activation='relu', upsample_method='subpixel', coords=False)

A U-net or hourglass style image-to-image model with skip-connections

Parameters

• x (tensor) – input tensor to unet

• config (list) – a list of ints specifying the number of feature maps on each scale of the unet in the downsampling path for the upsampling path, the list will be reversed For example [32, 64] will use 32 channels on scale 0 (without downsampling) and 64 channels on scale 1 once downsampled).

• extra_resnets (int) – how many extra res blocks to use at the bottleneck

• n_out (int) – number of final output feature maps of the unet. 3 for RGB

• activation (str) – a string specifying the activation function to use. See @activate for options.

• upsample_method (list of str or str) – a str specifying the upsampling method or a list of str specifying the upsampling method for each scale individually. See @upsample for possible options.

• coords (True) – if coord conv should be used.

Examples

tf.enable_eager_execution() x = tf.ones((1, 128, 128, 3)) config = [32, 64] extra_resnets = 0 upsample_method = “subpixel” activation = “leaky_relu” coords = False

unet = make_model(“unet”, hourglass_model, config=config, extra_resnets= extra_resnets, upsample_method=upsample_method, activation=activation) y = unet(x)

# plotting the output should look random because we did not train anything im = np.concatenate([x, y], axis=1) plt.imshow(np.squeeze(im))

edflow.nn.tf_nn.make_ema(init_value, value, decay=0.99)

apply exponential moving average to variable

Parameters

• init_value (float) – initial value for moving average variable

• value (variable) – tf variable to apply update ops on

• decay (float) – decay parameter

Returns

• avg_value (variable with exponential moving average)

• update_ema (tensorflow update operation for exponential moving average)

Examples

# usage within edflow Trainer.make_loss_ops. Apply EMA to discriminator accuracy avg_acc, update_ema = make_ema(0.5, dis_accuracy, decay) self.update_ops.append(update_ema) self.log_ops[“dis_acc”] = avg_acc

edflow.nn.tf_nn.add_coordinates(input_tensor, with_r=False)

Given an input_tensor, adds 2 channelw ith x and y coordinates to the feature maps. This was introduced in coordConv (2018ACS_liuIntriguingFailingConvolutionalNeuralNetworks). :param input_tensor: Tensor of shape [N, H, W, C] :type input_tensor: tensor :param with_r: if True, euclidian radius will also be added as channel :type with_r: bool :param Returns: :param ret: :type ret: input_tensor concatenated with x and y coordinates and maybe euclidian distance. :param ——-:

edflow.nn.tf_nn.probs_to_mu_L(probs, scaling_factor, inv=True)

Calculate mean and covariance (cholesky decomposition of covariance) for each channel of probs tensor of keypoint probabilites [bn, h, w, n_kp] mean calculated on a grid of scale [-1, 1]

Parameters

• probs (tensor) – tensor of shape [b, h, w, k] where each channel along axis 3 is interpreted as an unnormalized probability density.

• scaling_factor (tensor) – tensor of shape [b, 1, 1, k] representing normalizing the normalizing constant of the density

• inv (bool) – if True, returns covariance matrix of density. Else returns inverse of covariance matrix aka precision matrix

Returns

• mu (tensor) – tensor of shape [b, k, 2] representing partwise mean coordinates of x and y for each item in the batch

• L (tensor) –

  tensor of shape [b, k, 2, 2] representing partwise cholesky decomposition of covariance

  matrix for each item in the batch.

Example

from matplotlib import pyplot as plt
tf.enable_eager_execution()
import numpy as np
import tensorflow.contrib.distributions as tfd

_means = [-0.5, 0, 0.5]
means = tf.ones((3, 1, 2), dtype=tf.float32) * np.array(_means).reshape((3, 1, 1))
means = tf.concat([means, means, means[::-1, ...]], axis=1)
means = tf.reshape(means, (-1, 2))

var_ = 0.1
rho = 0.5
cov = [[var_, rho * var_], [rho * var_, var_]]
scale = tf.cholesky(cov)
scale = tf.stack([scale] * 3, axis=0)
scale = tf.stack([scale] * 3, axis=0)
scale = tf.reshape(scale, (-1, 2, 2))

mvn = tfd.MultivariateNormalTriL(
    loc=means,
    scale_tril=scale)

h = 100
w = 100
y_t = tf.tile(tf.reshape(tf.linspace(-1., 1., h), [h, 1]), [1, w])
x_t = tf.tile(tf.reshape(tf.linspace(-1., 1., w), [1, w]), [h, 1])
y_t = tf.expand_dims(y_t, axis=-1)
x_t = tf.expand_dims(x_t, axis=-1)
meshgrid = tf.concat([y_t, x_t], axis=-1)
meshgrid = tf.expand_dims(meshgrid, 0)
meshgrid = tf.expand_dims(meshgrid, 3)  # 1, h, w, 1, 2

blob = mvn.prob(meshgrid)
blob = tf.reshape(blob, (100, 100, 3, 3))
blob = tf.transpose(blob, perm=[2, 0, 1, 3])

norm_const = np.sum(blob, axis=(1, 2), keepdims=True)
mu, L = nn.probs_to_mu_L(blob / norm_const, 1, inv=False)

bn, h, w, nk = blob.get_shape().as_list()
estimated_blob = nn.tf_hm(h, w, mu, L)

fig, ax = plt.subplots(2, 3, figsize=(9, 6))
for b in range(len(_means)):
    ax[0, b].imshow(np.squeeze(blob[b, ...]))
    ax[0, b].set_title("target_blobs")
    ax[0, b].set_axis_off()

for b in range(len(_means)):
    ax[1, b].imshow(np.squeeze(estimated_blob[b, ...]))
    ax[1, b].set_title("estimated_blobs")
    ax[1, b].set_axis_off()

edflow.nn.tf_nn.probs_to_mu_sigma(probs)

Calculate mean and covariance matrix for each channel of spatial probability maps Mean and covariance are caluclated on a grid of scale [-1, 1]

Parameters

probs (tensor) – tensor of shape [N, H, W, C] where each channel along axis 3 is interpreted as a probability density.

Returns

• mu (tensor) – tensor of shape [N, C, 2] representing partwise mean coordinates of x and y for each item in the batch

• sigma (tensor) – tensor of shape [N, C, 2, 2] representing covariance matrix matrix for each item in the batch

Example

mu, sigma = nn.probs_to_mu_sigma(spatial_probability_maps)

edflow.nn.tf_nn.tf_hm(h, w, mu, L)

Returns Gaussian densitiy function based on μ and L for each batch index and part L is the cholesky decomposition of the covariance matrix : Σ = L L^T

Parameters

• h (int) – heigh ot output map

• w (int) – width of output map

• mu (tensor) – mean of gaussian part and batch item. Shape [b, p, 2]. Mean in range [-1, 1] with respect to height and width

• L (tensor) – cholesky decomposition of covariance matrix for each batch item and part. Shape [b, p, 2, 2]

• order –

Returns

density – gaussian blob for each part and batch idx. Shape [b, h, w, p]

Return type

tensor

Example

from matplotlib import pyplot as plt
tf.enable_eager_execution()
import numpy as np
import tensorflow as tf
import tensorflow.contrib.distributions as tfd

# create Target Blobs
_means = [-0.5, 0, 0.5]
means = tf.ones((3, 1, 2), dtype=tf.float32) * np.array(_means).reshape((3, 1, 1))
means = tf.concat([means, means, means[::-1, ...]], axis=1)
means = tf.reshape(means, (-1, 2))

var_ = 0.1
rho = 0.5
cov = [[var_, rho * var_],
       [rho * var_, var_]]
scale = tf.cholesky(cov)
scale = tf.stack([scale] * 3, axis=0)
scale = tf.stack([scale] * 3, axis=0)
scale = tf.reshape(scale, (-1, 2, 2))

mvn = tfd.MultivariateNormalTriL(
    loc=means,
    scale_tril=scale)

h = 100
w = 100
y_t = tf.tile(tf.reshape(tf.linspace(-1., 1., h), [h, 1]), [1, w])
x_t = tf.tile(tf.reshape(tf.linspace(-1., 1., w), [1, w]), [h, 1])
y_t = tf.expand_dims(y_t, axis=-1)
x_t = tf.expand_dims(x_t, axis=-1)
meshgrid = tf.concat([y_t, x_t], axis=-1)
meshgrid = tf.expand_dims(meshgrid, 0)
meshgrid = tf.expand_dims(meshgrid, 3)  # 1, h, w, 1, 2

blob = mvn.prob(meshgrid)
blob = tf.reshape(blob, (100, 100, 3, 3))
blob = tf.transpose(blob, perm=[2, 0, 1, 3])

# Estimate mean and L
norm_const = np.sum(blob, axis=(1, 2), keepdims=True)
mu, L = nn.probs_to_mu_L(blob / norm_const, 1, inv=False)

bn, h, w, nk = blob.get_shape().as_list()

# Estimate blob based on mu and L
estimated_blob = nn.tf_hm(h, w, mu, L)

# plot
fig, ax = plt.subplots(2, 3, figsize=(9, 6))
for b in range(len(_means)):
    ax[0, b].imshow(np.squeeze(blob[b, ...]))
    ax[0, b].set_title("target_blobs")
    ax[0, b].set_axis_off()

for b in range(len(_means)):
    ax[1, b].imshow(np.squeeze(estimated_blob[b, ...]))
    ax[1, b].set_title("estimated_blobs")
    ax[1, b].set_axis_off()