Affine transformation APIs for high-performance image augmentation

CHANGELOG.md

@@ -72,6 +72,17 @@ To release a new version, please update the changelog as followed:
 ### Added
 - Layer:
   - Release `GroupNormLayer` (PR #850)
+- Image affine transformation APIs
+  - `affine_rotation_matrix` (PR #857)
+  - `affine_horizontal_flip_matrix` (PR #857)
+  - `affine_vertical_flip_matrix` (PR #857)
+  - `affine_shift_matrix` (PR #857)
+  - `affine_shear_matrix` (PR #857)
+  - `affine_zoom_matrix` (PR #857)
+  - `affine_transform_cv2` (PR #857)
+  - `affine_transform_keypoints` (PR #857)
+- Affine transformation tutorial
+  - `examples/data_process/tutorial_fast_affine_transform.py` (PR #857)
 
 ### Changed
 
@@ -95,6 +106,8 @@ To release a new version, please update the changelog as followed:
 ### Contributors
 - @2wins: #850 #855
 - @DEKHTIARJonathan: #853
+- @zsdonghao: #857
+- @luomai: #857
 
 ## [1.10.1] - 2018-09-07
 
@@ -121,6 +134,7 @@ To release a new version, please update the changelog as followed:
 - @ndiy: #819 
 - @zsdonghao: #818
 
+
 ## [1.10.0] - 2018-09-02
 
 ### Added

README.md

@@ -4,17 +4,15 @@
     </div>
 </a>
 
-[![English Documentation](https://img.shields.io/badge/documentation-english-blue.svg)](https://tensorlayer.readthedocs.io/)
-[![Chinese Documentation](https://img.shields.io/badge/documentation-%E4%B8%AD%E6%96%87-blue.svg)](https://tensorlayercn.readthedocs.io/)
-[![Chinese Book](https://img.shields.io/badge/book-%E4%B8%AD%E6%96%87-blue.svg)](http://www.broadview.com.cn/book/5059/)
-[![Downloads](http://pepy.tech/badge/tensorlayer)](http://pepy.tech/project/tensorlayer)
+
 
 
 ![PyPI Stable Version](http://ec2-35-178-47-120.eu-west-2.compute.amazonaws.com/github/release/tensorlayer/tensorlayer.svg?label=PyPI%20-%20Release)
 ![PyPI RC Version](http://ec2-35-178-47-120.eu-west-2.compute.amazonaws.com/github/release/tensorlayer/tensorlayer/all.svg?label=PyPI%20-%20Pre-Release)
 [![Github commits (since latest release)](http://ec2-35-178-47-120.eu-west-2.compute.amazonaws.com/github/commits-since/tensorlayer/tensorlayer/latest.svg)](https://github.com/tensorlayer/tensorlayer/compare/1.10.1...master)
 [![PyPI - Python Version](http://ec2-35-178-47-120.eu-west-2.compute.amazonaws.com/pypi/pyversions/tensorlayer.svg)](https://pypi.org/project/tensorlayer/)
 [![Supported TF Version](http://ec2-35-178-47-120.eu-west-2.compute.amazonaws.com/badge/tensorflow-1.6.0+-blue.svg)](https://github.com/tensorflow/tensorflow/releases)
+[![Downloads](http://pepy.tech/badge/tensorlayer)](http://pepy.tech/project/tensorlayer)
 
 [![Build Status](http://ec2-35-178-47-120.eu-west-2.compute.amazonaws.com/travis/tensorlayer/tensorlayer/master.svg?label=Travis)](https://travis-ci.org/tensorlayer/tensorlayer)
 [![CircleCI](http://ec2-35-178-47-120.eu-west-2.compute.amazonaws.com/circleci/project/github/tensorlayer/tensorlayer/master.svg?label=Docker%20Build)](https://circleci.com/gh/tensorlayer/tensorlayer/tree/master)
@@ -62,16 +60,26 @@ TensorLayer has a fast growing usage among top researchers and engineers, from u
 Imperial College London, UC Berkeley, Carnegie Mellon University, Stanford University, and
 University of Technology of Compiegne (UTC), and companies like Google, Microsoft, Alibaba, Tencent, Xiaomi, and Bloomberg.
 
-# Tutorials, Examples and Applications
+# Tutorials and Real-World Applications
+
+You can find a large collection of tutorials, examples and real-world applications using TensorLayer through the following space:
 
 <a href="https://github.com/tensorlayer/awesome-tensorlayer/blob/master/readme.md" target="\_blank">
 	<div align="center">
 		<img src="img/awesome-mentioned.png" width="40%"/>
 	</div>
 </a>
 
+# Documentation
+
+TensorLayer has extensive documentation for both beginners and professionals. The documentation is available in
+both English and Chinese. Please click the following icons to find the documents you need:
+
+[![English Documentation](https://img.shields.io/badge/documentation-english-blue.svg)](https://tensorlayer.readthedocs.io/)
+[![Chinese Documentation](https://img.shields.io/badge/documentation-%E4%B8%AD%E6%96%87-blue.svg)](https://tensorlayercn.readthedocs.io/)
+[![Chinese Book](https://img.shields.io/badge/book-%E4%B8%AD%E6%96%87-blue.svg)](http://www.broadview.com.cn/book/5059/)
 
-# Installation Guide
+# Install
 
 TensorLayer has pre-requisites including TensorFlow, numpy, and others. For GPU support, CUDA and cuDNN are required.
 The simplest way to install TensorLayer is to use the **Py**thon **P**ackage **I**ndex (PyPI):
@@ -97,7 +105,7 @@ pip install https://github.com/tensorlayer/tensorlayer/archive/master.zip
 # pip install https://github.com/tensorlayer/tensorlayer/archive/<branch-name>.zip
 ```
 
-## Using Docker - a ready-to-use environment
+## Using Docker
 
 The [TensorLayer containers](https://hub.docker.com/r/tensorlayer/tensorlayer/) are built on top of the official [TensorFlow containers](https://hub.docker.com/r/tensorflow/tensorflow/):
 

docs/index.rst

@@ -3,7 +3,7 @@ Welcome to TensorLayer
 
 
 .. image:: user/my_figs/tl_transparent_logo.png
-  :scale: 30 %
+  :width: 30 %
   :align: center
   :target: https://github.com/tensorlayer/tensorlayer
 
@@ -50,14 +50,14 @@ method, this part of the documentation is for you.
   modules/activation
   modules/array_ops
   modules/cost
+  modules/prepro
   modules/distributed
   modules/files
   modules/iterate
   modules/layers
   modules/models
   modules/nlp
   modules/optimizers
-  modules/prepro
   modules/rein
   modules/utils
   modules/visualize

docs/modules/prepro.rst

@@ -1,18 +1,23 @@
-API - Preprocessing
+API - Data Pre-processing
 =========================
 
-
-We provide abundant data augmentation and processing functions by using Numpy, Scipy, Threading and Queue.
-However, we recommend you to use TensorFlow operation function like ``tf.image.central_crop``,
-more TensorFlow data augmentation method can be found
-`here <https://www.tensorflow.org/api_guides/python/image.html>`_ and ``tutorial_cifar10_tfrecord.py``.
-Some of the code in this package are borrowed from Keras.
-
 .. automodule:: tensorlayer.prepro
 
 .. autosummary::
 
-   threading_data
+   affine_rotation_matrix
+   affine_horizontal_flip_matrix
+   affine_vertical_flip_matrix
+   affine_shift_matrix
+   affine_shear_matrix
+   affine_zoom_matrix
+   affine_respective_zoom_matrix
+
+   transform_matrix_offset_center
+   affine_transform
+   affine_transform_cv2
+   affine_transform_keypoints
+   projective_transform_by_points
 
    rotation
    rotation_multi
@@ -33,6 +38,7 @@ Some of the code in this package are borrowed from Keras.
    elastic_transform_multi
 
    zoom
+   respective_zoom
    zoom_multi
 
    brightness
@@ -56,10 +62,6 @@ Some of the code in this package are borrowed from Keras.
 
    drop
 
-   transform_matrix_offset_center
-   apply_transform
-   projective_transform_by_points
-
    array_to_img
 
    find_contours
@@ -88,7 +90,7 @@ Some of the code in this package are borrowed from Keras.
    obj_box_zoom
 
    keypoint_random_crop
-   keypoint_random_crop2
+   keypoint_resize_random_crop
    keypoint_random_rotate
    keypoint_random_flip
    keypoint_random_resize
@@ -103,10 +105,166 @@ Some of the code in this package are borrowed from Keras.
    sequences_get_mask
 
 
+..
+  Threading
+  ------------
+  .. autofunction:: threading_data
+
+
+Affine Transform
+----------------
+
+
+Python can be FAST
+^^^^^^^^^^^^^^^^^^
+
+Image augmentation is a critical step in deep learning.
+Though TensorFlow has provided ``tf.image``,
+image augmentation often remains as a key bottleneck.
+``tf.image`` has three limitations:
+
+- Real-world visual tasks such as object detection, segmentation, and pose estimation
+  must cope with image meta-data (e.g., coordinates).
+  These data are beyond ``tf.image``
+  which processes images as tensors.
+
+- ``tf.image`` operators
+  breaks the pure Python programing experience (i.e., users have to
+  use ``tf.py_func`` in order to call image functions written in Python); however,
+  frequent uses of ``tf.py_func`` slow down TensorFlow,
+  making users hard to balance flexibility and performance.
+
+- ``tf.image`` API is inflexible. Image operations are
+  performed in an order. They are hard to jointly optimize. More importantly,
+  sequential image operations can significantly
+  reduces the quality of images, thus affecting training accuracy.
+
+
+TensorLayer addresses these limitations by providing a
+high-performance image augmentation API in Python.
+This API bases on affine transformation and ``cv2.wrapAffine``.
+It allows you to combine multiple image processing functions into
+a single matrix operation. This combined operation
+is executed by the fast ``cv2`` library, offering 78x performance improvement (observed in
+`openpose-plus <https://github.com/tensorlayer/openpose-plus>`_ for example).
+The following example illustrates the rationale
+behind this tremendous speed up.
+
+
+Example
+^^^^^^^
+
+The source code of complete examples can be found \
+`here <https://github.com/tensorlayer/tensorlayer/tree/master/examples/data_process/tutorial_fast_affine_transform.py>`__.
+The following is a typical Python program that applies rotation, shifting, flipping, zooming and shearing to an image,
+
+.. code-block:: python
+
+    image = tl.vis.read_image('tiger.jpeg')
+
+    xx = tl.prepro.rotation(image, rg=-20, is_random=False)
+    xx = tl.prepro.flip_axis(xx, axis=1, is_random=False)
+    xx = tl.prepro.shear2(xx, shear=(0., -0.2), is_random=False)
+    xx = tl.prepro.zoom(xx, zoom_range=1/0.8)
+    xx = tl.prepro.shift(xx, wrg=-0.1, hrg=0, is_random=False)
+
+    tl.vis.save_image(xx, '_result_slow.png')
+
+
+However, by leveraging affine transformation, image operations can be combined into one:
+
+.. code-block:: python
+
+    # 1. Create required affine transformation matrices
+    M_rotate = tl.prepro.affine_rotation_matrix(angle=20)
+    M_flip = tl.prepro.affine_horizontal_flip_matrix(prob=1)
+    M_shift = tl.prepro.affine_shift_matrix(wrg=0.1, hrg=0, h=h, w=w)
+    M_shear = tl.prepro.affine_shear_matrix(x_shear=0.2, y_shear=0)
+    M_zoom = tl.prepro.affine_zoom_matrix(zoom_range=0.8)
+
+    # 2. Combine matrices
+    # NOTE: operations are applied in a reversed order (i.e., rotation is performed first)
+    M_combined = M_shift.dot(M_zoom).dot(M_shear).dot(M_flip).dot(M_rotate)
+
+    # 3. Convert the matrix from Cartesian coordinates (the origin in the middle of image)
+    # to image coordinates (the origin on the top-left of image)
+    transform_matrix = tl.prepro.transform_matrix_offset_center(M_combined, x=w, y=h)
+
+    # 4. Transform the image using a single operation
+    result = tl.prepro.affine_transform_cv2(image, transform_matrix)  # 76 times faster
+
+    tl.vis.save_image(result, '_result_fast.png')
+
+
+The following figure illustrates the rational behind combined affine transformation.
+
+.. image:: ../images/affine_transform_why.jpg
+  :width: 100 %
+  :align: center
+
+
+Using combined affine transformation has two key benefits. First, it allows \
+you to leverage a pure Python API to achieve orders of magnitudes of speed up in image augmentation,
+and thus prevent data pre-processing from becoming a bottleneck in training. \
+Second, performing sequential image transformation requires multiple image interpolations. \
+This produces low-quality input images. In contrast, a combined transformation performs the \
+interpolation only once, and thus
+preserve the content in an image. The following figure illustrates these two benefits:
+
+.. image:: ../images/affine_transform_comparison.jpg
+  :width: 100 %
+  :align: center
+
+
+Get rotation matrix
+^^^^^^^^^^^^^^^^^^^^^^^^^
+.. autofunction:: affine_rotation_matrix
+
+Get horizontal flipping matrix
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. autofunction:: affine_horizontal_flip_matrix
+
+Get vertical flipping matrix
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. autofunction:: affine_vertical_flip_matrix
+
+Get shifting matrix
+^^^^^^^^^^^^^^^^^^^^^^^^^
+.. autofunction:: affine_shift_matrix
+
+Get shearing matrix
+^^^^^^^^^^^^^^^^^^^^^^^^^
+.. autofunction:: affine_shear_matrix
+
+Get zooming matrix
+^^^^^^^^^^^^^^^^^^^^^^^^^
+.. autofunction:: affine_zoom_matrix
+
+Get respective zooming matrix
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. autofunction:: affine_respective_zoom_matrix
+
+Cartesian to image coordinates
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. autofunction:: transform_matrix_offset_center
+
+..
+    Apply image transform
+    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+    .. autofunction:: affine_transform
+
+Apply image transform
+^^^^^^^^^^^^^^^^^^^^^
+.. autofunction:: affine_transform_cv2
+
+Apply keypoint transform
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. autofunction:: affine_transform_keypoints
+
+Projective transform by points
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. autofunction:: projective_transform_by_points
 
-Threading
-------------
-.. autofunction:: threading_data
 
 Images
 -----------
@@ -160,6 +318,10 @@ Zoom
 .. autofunction:: zoom
 .. autofunction:: zoom_multi
 
+Respective Zoom
+^^^^^^^^^^^^^^^^^
+.. autofunction:: respective_zoom
+
 Brightness
 ^^^^^^^^^^^^
 .. autofunction:: brightness
@@ -203,18 +365,6 @@ Noise
 ^^^^^^^^^^^^^^
 .. autofunction:: drop
 
-Transform matrix offset
-^^^^^^^^^^^^^^^^^^^^^^^^^
-.. autofunction:: transform_matrix_offset_center
-
-Apply affine transform by matrix
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-.. autofunction:: apply_transform
-
-Projective transform by points
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-.. autofunction:: projective_transform_by_points
-
 Numpy and PIL
 ^^^^^^^^^^^^^^
 .. autofunction:: array_to_img
@@ -430,7 +580,9 @@ Image Aug - Crop
 ^^^^^^^^^^^^^^^^^^^^
 .. autofunction:: keypoint_random_crop
 
-.. autofunction:: keypoint_random_crop2
+Image Aug - Resize then Crop
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. autofunction:: keypoint_resize_random_crop
 
 Image Aug - Rotate
 ^^^^^^^^^^^^^^^^^^^^