[WIP] TensorLayer 2.0

[DEKHTIARJonathan]

NETWORK API REFACTORING - TO DO LIST

Finished work:

• activation.py:
  • PReluLayer:
    • refactored
    • tested
  • PRelu6Layer:
    • refactored
    • tested
  • PTRelu6Layer:
    • refactored
    • tested
• convolution/
  • AtrousConv1dLayer:
    • refactored => Need to be implemented
    • tested
  • AtrousConv2dLayer:
    • refactored
    • tested
  • AtrousDeConv2dLayer:
    • refactored
    • tested
  • BinaryConv2d:
    • refactored
    • tested
  • Conv1d:
    • refactored
    • tested
  • Conv2d:
    • refactored
    • tested
  • Conv1dLayer:
    • refactored
    • tested
  • Conv2dLayer:
    • refactored
    • tested
  • Conv3dLayer:
    • refactored
    • tested
  • DeConv2d:
    • refactored
    • tested
  • DeConv3d:
    • refactored
    • tested
  • DeConv2dLayer:
    • refactored
    • tested
  • DeConv3dLayer:
    • refactored
    • tested
  • DeformableConv2d:
    • refactored
    • tested
  • DepthwiseConv2d:
    • refactored
    • tested
  • DorefaConv2d:
    • refactored
    • tested
  • GroupConv2d:
    • refactored
    • tested
  • QuantizedConv2d:
    • refactored
    • tested
  • QuantizedConv2dWithBN:
    • refactored
    • tested
  • SeparableConv1d:
    • refactored
    • tested
  • SeparableConv2d:
    • refactored
    • tested
  • SubpixelConv1d:
    • refactored
    • tested
  • SubpixelConv2d:
    • refactored
    • tested
  • TernaryConv2d:
    • refactored
    • tested
• dense/
  • BinaryDenseLayer:
    • refactored
    • tested
  • DenseLayer:
    • refactored
    • tested
  • DorefaDenseLayer:
    • refactored
    • tested
  • DropconnectDenseLayer:
    • refactored
    • tested
  • QuantizedDense:
    • refactored
    • tested
  • QuantizedDenseWithBN:
    • refactored
    • tested
  • TernaryDenseLayer:
    • refactored
    • tested
• dropout.py
  • DropoutLayer:
    • refactored
    • tested
• extend.py
  • ExpandDimsLayer:
    • refactored
    • tested
  • TileLayer:
    • refactored
    • tested
• image_resampling.py
  • UpSampling2dLayer:
    • refactored
    • tested
  • DownSampling2dLayer:
    • refactored
    • tested
• importer.py
  • SlimNetsLayer:
    • refactored
    • tested
  • KerasLayer:
    • refactored
    • tested
• inputs.py
  • InputLayer:
    • refactored
    • tested
  • OneHotInputLayer:
    • refactored
    • tested
  • Word2vecEmbeddingInputlayer:
    • refactored
    • tested
  • EmbeddingInputlayer:
    • refactored
    • tested
  • AverageEmbeddingInputlayer:
    • refactored
    • tested
• lambda_layers.py
  • LambdaLayer:
    • refactored
    • tested
  • ElementwiseLambdaLayer:
    • refactored
    • tested
• merge.py
  • ConcatLayer:
    • refactored
    • tested
  • ElementwiseLayer:
    • refactored
    • tested
• noise.py
  • GaussianNoiseLayer:
    • refactored
    • tested
• normalization.py
  • BatchNormLayer:
    • refactored
    • tested
  • InstanceNormLayer:
    • refactored
    • tested
  • GroupNormLayer:
    • refactored
    • tested
  • LayerNormLayer:
    • refactored
    • tested
  • LocalResponseNormLayer:
    • refactored
    • tested
  • SwitchNormLayer:
    • refactored
    • tested
• padding.py
  • PadLayer:
    • refactored
    • tested
  • ZeroPad1d:
    • refactored
    • tested
  • ZeroPad2d:
    • refactored
    • tested
  • ZeroPad3d:
    • refactored
    • tested
• pooling/
  • MaxPool1d:
    • refactored
    • tested
  • MaxPool2d:
    • refactored
    • tested
  • MaxPool3d:
    • refactored
    • tested
  • MeanPool1d:
    • refactored
    • tested
  • MeanPool2d:
    • refactored
    • tested
  • MeanPool3d:
    • refactored
    • tested
  • GlobalMaxPool1d:
    • refactored
    • tested
  • GlobalMaxPool2d:
    • refactored
    • tested
  • GlobalMaxPool3d:
    • refactored
    • tested
  • GlobalMeanPool1d:
    • refactored
    • tested
  • GlobalMeanPool2d:
    • refactored
    • tested
  • GlobalMeanPool3d:
    • refactored
    • tested
  • PoolLayer:
    • refactored
    • tested
• quantize.py
  • SignLayer:
    • refactored
    • tested
• recurrent/
  • RNNLayer:
    • refactored
    • tested
  • BiRNNLayer:
    • refactored
    • tested
  • ConvLSTMLayer:
    • refactored
    • tested
  • DynamicRNNLayer:
    • refactored
    • tested
  • BiDynamicRNNLayer:
    • refactored
    • tested
  • Seq2Seq:
    • refactored
    • tested # => Not Tested, need to be in TL Repo [WIP] for tensorlayer 2.0 seq2seq-chatbot#17
• reshape.py
  • FlattenLayer:
    • refactored
    • tested
  • ReshapeLayer:
    • refactored
    • tested
  • TransposeLayer:
    • refactored
    • tested
• scale.py
  • ScaleLayer:
    • refactored
    • tested
• spatial_transformer.py
  • SpatialTransformer2dAffineLayer:
    • refactored
    • tested
• stack.py
  • StackLayer:
    • refactored
    • tested
  • UnStackLayer: => Need to be checked! Not working
    • refactored
    • tested
• time_distribution.py
  • TimeDistributedLayer:
    • refactored
    • tested

Unittests Status:

• test_activations.py
• test_array_ops.py
• test_decorators.py
• test_documentation.py
• test_layers_activation.py
• test_layers_basic.py
• test_layers_convolution_1d.py
• test_layers_convolution_2d.py
• test_layers_convolution_3d.py
• test_layers_convolution_deformable.py
• test_layers_core.py
• test_layers_extend.py
• test_layers_flow_control.py
• test_layers_importer.py
• test_layers_merge.py
• test_layers_normalization.py
• test_layers_padding.py
• test_layers_pooling.py
• test_layers_recurrent.py
• test_layers_reshape.py
• test_layers_spatial_transformer.py
• test_layers_stack.py
• test_layers_super_resolution.py
• test_layers_time_distributed.py
• test_logging.py
• test_logging_hyperdash.py
• test_mnist_simple.py
• test_models.py
• test_network_custom_2d.py
• test_network_custom_input_layers.py
• test_network_custom_multiple_inputs.py
• test_network_custom_multiple_outputs.py
• test_network_sequential_1d.py
• test_network_sequential_2d.py
• test_network_sequential_3d.py
• test_network_sequential_rnn.py
• test_optimizer_amsgrad.py
• test_pydocstyle.py
• test_reuse_mlp.py
• test_tf_layers.py
• test_timeout.py
• test_utils_predict.py
• test_yapf_format.py

Work to be done

Layers

• contrib
  • ROIPoolingLayer:
    • refactored
    • tested
• convolution/
  • AtrousConv1dLayer:
    • refactored => Need to be implemented
    • tested
  • DeformableConv2d:
    • refactored
    • tested
• lambda_layers.py
  • ElementwiseLambdaLayer:
    • refactored
    • tested
• merge.py
  • ElementwiseLayer:
    • refactored
    • tested
• normalization.py
  • GroupNormLayer:
    • refactored
    • tested
• pooling/
  • PoolLayer:
    • refactored
    • tested
• quantize.py
  • SignLayer:
    • refactored
    • tested
• spatial_transformer.py
  • SpatialTransformer2dAffineLayer:
    • refactored
    • tested
• recurrent/
  • Seq2Seq:
    • refactored
    • tested # => Not Tested, need to be in TL Repo [WIP] for tensorlayer 2.0 seq2seq-chatbot#17
• stack.py
  • StackLayer:
    • refactored
    • tested
  • UnStackLayer: => Need to be checked! Not working
    • refactored
    • tested
• time_distribution.py
  • TimeDistributedLayer:
    • refactored
    • tested

TensorLayer Hub

• tl.models => became tl.hub
  • VGG16
  • VGG19
  • MobileNet
  • SqueezeNet

Examples

• basic_tutorials
  • refactored
  • tested
• data_process
  • refactored
  • tested
• database
  • refactored
  • tested
• deprecated_tutorials
  • refactored
  • tested
• distributed_training
  • refactored
  • tested
• keras_tfslim
  • refactored
  • tested
• pretrained_cnn
  • refactored
  • tested
• quantized_net
  • refactored
  • tested
• reinforcement_learning
  • refactored
  • tested
• text_classification
  • refactored
  • tested
• text_generation
  • refactored
  • tested
• text_ptb
  • refactored
  • tested
• text_word_embedding
  • refactored
  • tested

More TODO

• add scope_name into the layer name
• don't check duplicated layer name in core layer

Test code:

plh = tf.placeholder(tf.float16, (100, 32))
net = tl.layers.InputLayer(name='in')(plh)
with tf.variable_scope('test'):
    net = tl.layers.DenseLayer(n_units=50, act=tf.nn.relu, name="dense")(net)
with tf.variable_scope('test', reuse=True):
    net = tl.layers.DenseLayer(n_units=50, act=tf.nn.relu, name="dense")(net)
print(net['test/dense'])
print(net['test/dense_2'])
assert len(net.all_weights) == 2
print(net.all_params)  # give a warning, but still works

How to do with manual-compile?:

with tf.variable_scope('test'):
    model.add(tl.layers.DenseLayer(n_units=50, act=tf.nn.relu, name="seq_layer_9"))
with tf.variable_scope('test', reuse=True):
    model.add(tl.layers.DenseLayer(n_units=50, act=tf.nn.relu, name="seq_layer_9"))
plh = tf.placeholder(tf.float16, (100, 32))
net = model.compile(plc)
print(net['test/seq_layer_9'])

• _params and _variables are everywhere ....
  • assign_params, get_variables_with_name, print_params

[2wins]

Related Issue: NCHW/NHWC, channels_first/last (Issue #561, PR #640)
We need to refer to comments on the posts.

[DEKHTIARJonathan]

Closing to delete the branch