1- API - Preprocessing
1+ API - Data Pre-processing
22=========================
33
4-
5- We provide abundant data augmentation and processing functions by using Numpy, Scipy, Threading and Queue.
6- However, we recommend you to use TensorFlow operation function like ``tf.image.central_crop ``,
7- more TensorFlow data augmentation method can be found
8- `here <https://www.tensorflow.org/api_guides/python/image.html >`_ and ``tutorial_cifar10_tfrecord.py ``.
9- Some of the code in this package are borrowed from Keras.
10-
114.. automodule :: tensorlayer.prepro
125
136.. autosummary ::
147
15- threading_data
8+ affine_rotation_matrix
9+ affine_horizontal_flip_matrix
10+ affine_vertical_flip_matrix
11+ affine_shift_matrix
12+ affine_shear_matrix
13+ affine_zoom_matrix
14+ affine_respective_zoom_matrix
15+
16+ transform_matrix_offset_center
17+ affine_transform
18+ affine_transform_cv2
19+ affine_transform_keypoints
20+ projective_transform_by_points
1621
1722 rotation
1823 rotation_multi
@@ -33,6 +38,7 @@ Some of the code in this package are borrowed from Keras.
3338 elastic_transform_multi
3439
3540 zoom
41+ respective_zoom
3642 zoom_multi
3743
3844 brightness
@@ -56,10 +62,6 @@ Some of the code in this package are borrowed from Keras.
5662
5763 drop
5864
59- transform_matrix_offset_center
60- apply_transform
61- projective_transform_by_points
62-
6365 array_to_img
6466
6567 find_contours
@@ -88,7 +90,7 @@ Some of the code in this package are borrowed from Keras.
8890 obj_box_zoom
8991
9092 keypoint_random_crop
91- keypoint_random_crop2
93+ keypoint_resize_random_crop
9294 keypoint_random_rotate
9395 keypoint_random_flip
9496 keypoint_random_resize
@@ -103,10 +105,166 @@ Some of the code in this package are borrowed from Keras.
103105 sequences_get_mask
104106
105107
108+ ..
109+ Threading
110+ ------------
111+ .. autofunction:: threading_data
112+
113+
114+ Affine Transform
115+ ----------------
116+
117+
118+ Python can be FAST
119+ ^^^^^^^^^^^^^^^^^^
120+
121+ Image augmentation is a critical step in deep learning.
122+ Though TensorFlow has provided ``tf.image ``,
123+ image augmentation often remains as a key bottleneck.
124+ ``tf.image `` has three limitations:
125+
126+ - Real-world visual tasks such as object detection, segmentation, and pose estimation
127+ must cope with image meta-data (e.g., coordinates).
128+ These data are beyond ``tf.image ``
129+ which processes images as tensors.
130+
131+ - ``tf.image `` operators
132+ breaks the pure Python programing experience (i.e., users have to
133+ use ``tf.py_func `` in order to call image functions written in Python); however,
134+ frequent uses of ``tf.py_func `` slow down TensorFlow,
135+ making users hard to balance flexibility and performance.
136+
137+ - ``tf.image `` API is inflexible. Image operations are
138+ performed in an order. They are hard to jointly optimize. More importantly,
139+ sequential image operations can significantly
140+ reduces the quality of images, thus affecting training accuracy.
141+
142+
143+ TensorLayer addresses these limitations by providing a
144+ high-performance image augmentation API in Python.
145+ This API bases on affine transformation and ``cv2.wrapAffine ``.
146+ It allows you to combine multiple image processing functions into
147+ a single matrix operation. This combined operation
148+ is executed by the fast ``cv2 `` library, offering 78x performance improvement (observed in
149+ `openpose-plus <https://github.com/tensorlayer/openpose-plus >`_ for example).
150+ The following example illustrates the rationale
151+ behind this tremendous speed up.
152+
153+
154+ Example
155+ ^^^^^^^
156+
157+ The source code of complete examples can be found \
158+ `here <https://github.com/tensorlayer/tensorlayer/tree/master/examples/data_process/tutorial_fast_affine_transform.py >`__.
159+ The following is a typical Python program that applies rotation, shifting, flipping, zooming and shearing to an image,
160+
161+ .. code-block :: python
162+
163+ image = tl.vis.read_image(' tiger.jpeg'
164+
165+ xx = tl.prepro.rotation(image, rg = - 20 , is_random = False )
166+ xx = tl.prepro.flip_axis(xx, axis = 1 , is_random = False )
167+ xx = tl.prepro.shear2(xx, shear = (0 ., - 0.2 ), is_random = False )
168+ xx = tl.prepro.zoom(xx, zoom_range = 1 / 0.8 )
169+ xx = tl.prepro.shift(xx, wrg = - 0.1 , hrg = 0 , is_random = False )
170+
171+ tl.vis.save_image(xx, ' _result_slow.png'
172+
173+
174+
175+
176+ .. code-block :: python
177+
178+ # 1. Create required affine transformation matrices
179+ M_rotate = tl.prepro.affine_rotation_matrix(angle = 20 )
180+ M_flip = tl.prepro.affine_horizontal_flip_matrix(prob = 1 )
181+ M_shift = tl.prepro.affine_shift_matrix(wrg = 0.1 , hrg = 0 , h = h, w = w)
182+ M_shear = tl.prepro.affine_shear_matrix(x_shear = 0.2 , y_shear = 0 )
183+ M_zoom = tl.prepro.affine_zoom_matrix(zoom_range = 0.8 )
184+
185+ # 2. Combine matrices
186+ # NOTE : operations are applied in a reversed order (i.e., rotation is performed first)
187+ M_combined = M_shift.dot(M_zoom).dot(M_shear).dot(M_flip).dot(M_rotate)
188+
189+ # 3. Convert the matrix from Cartesian coordinates (the origin in the middle of image)
190+ # to image coordinates (the origin on the top-left of image)
191+ transform_matrix = tl.prepro.transform_matrix_offset_center(M_combined, x = w, y = h)
192+
193+ # 4. Transform the image using a single operation
194+ result = tl.prepro.affine_transform_cv2(image, transform_matrix) # 76 times faster
195+
196+ tl.vis.save_image(result, ' _result_fast.png'
197+
198+
199+
200+
201+ .. image :: ../images/affine_transform_why.jpg
202+ :width: 100 %
203+ :align: center
204+
205+
206+ Using combined affine transformation has two key benefits. First, it allows \
207+ you to leverage a pure Python API to achieve orders of magnitudes of speed up in image augmentation,
208+ and thus prevent data pre-processing from becoming a bottleneck in training. \
209+ Second, performing sequential image transformation requires multiple image interpolations. \
210+ This produces low-quality input images. In contrast, a combined transformation performs the \
211+ interpolation only once, and thus
212+ preserve the content in an image. The following figure illustrates these two benefits:
213+
214+ .. image :: ../images/affine_transform_comparison.jpg
215+ :width: 100 %
216+ :align: center
217+
218+
219+ Get rotation matrix
220+ ^^^^^^^^^^^^^^^^^^^^^^^^^
221+ .. autofunction :: affine_rotation_matrix
222+
223+ Get horizontal flipping matrix
224+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
225+ .. autofunction :: affine_horizontal_flip_matrix
226+
227+ Get vertical flipping matrix
228+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
229+ .. autofunction :: affine_vertical_flip_matrix
230+
231+ Get shifting matrix
232+ ^^^^^^^^^^^^^^^^^^^^^^^^^
233+ .. autofunction :: affine_shift_matrix
234+
235+ Get shearing matrix
236+ ^^^^^^^^^^^^^^^^^^^^^^^^^
237+ .. autofunction :: affine_shear_matrix
238+
239+ Get zooming matrix
240+ ^^^^^^^^^^^^^^^^^^^^^^^^^
241+ .. autofunction :: affine_zoom_matrix
242+
243+ Get respective zooming matrix
244+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
245+ .. autofunction :: affine_respective_zoom_matrix
246+
247+ Cartesian to image coordinates
248+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
249+ .. autofunction :: transform_matrix_offset_center
250+
251+ ..
252+ Apply image transform
253+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
254+ .. autofunction:: affine_transform
255+
256+ Apply image transform
257+ ^^^^^^^^^^^^^^^^^^^^^
258+ .. autofunction :: affine_transform_cv2
259+
260+ Apply keypoint transform
261+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
262+ .. autofunction :: affine_transform_keypoints
263+
264+ Projective transform by points
265+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
266+ .. autofunction :: projective_transform_by_points
106267
107- Threading
108- ------------
109- .. autofunction :: threading_data
110268
111269Images
112270-----------
@@ -160,6 +318,10 @@ Zoom
160318.. autofunction :: zoom
161319.. autofunction :: zoom_multi
162320
321+ Respective Zoom
322+ ^^^^^^^^^^^^^^^^^
323+ .. autofunction :: respective_zoom
324+
163325Brightness
164326^^^^^^^^^^^^
165327.. autofunction :: brightness
@@ -203,18 +365,6 @@ Noise
203365^^^^^^^^^^^^^^
204366.. autofunction :: drop
205367
206- Transform matrix offset
207- ^^^^^^^^^^^^^^^^^^^^^^^^^
208- .. autofunction :: transform_matrix_offset_center
209-
210- Apply affine transform by matrix
211- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
212- .. autofunction :: apply_transform
213-
214- Projective transform by points
215- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
216- .. autofunction :: projective_transform_by_points
217-
218368Numpy and PIL
219369^^^^^^^^^^^^^^
220370.. autofunction :: array_to_img
@@ -430,7 +580,9 @@ Image Aug - Crop
430580^^^^^^^^^^^^^^^^^^^^
431581.. autofunction :: keypoint_random_crop
432582
433- .. autofunction :: keypoint_random_crop2
583+ Image Aug - Resize then Crop
584+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
585+ .. autofunction :: keypoint_resize_random_crop
434586
435587Image Aug - Rotate
436588^^^^^^^^^^^^^^^^^^^^
0 commit comments