Skip to content

Commit 3fa6c50

Browse files
Mikulas Patockatorvalds
authored andcommitted
mm: optimize copy_page_to/from_iter_iovec
copy_page_to_iter_iovec() and copy_page_from_iter_iovec() copy some data to userspace or from userspace. These functions have a fast path where they map a page using kmap_atomic and a slow path where they use kmap. kmap is slower than kmap_atomic, so the fast path is preferred. However, on kernels without highmem support, kmap just calls page_address, so there is no need to avoid kmap. On kernels without highmem support, the fast path just increases code size (and cache footprint) and it doesn't improve copy performance in any way. This patch enables the fast path only if CONFIG_HIGHMEM is defined. Code size reduced by this patch: x86 (without highmem) 928 x86-64 960 sparc64 848 alpha 1136 pa-risc 1200 [[email protected]: use IS_ENABLED(), per Andi] Link: http://lkml.kernel.org/r/alpine.LRH.2.02.1607221711410.4818@file01.intranet.prod.int.rdu2.redhat.com Signed-off-by: Mikulas Patocka <[email protected]> Cc: Hugh Dickins <[email protected]> Cc: Michal Hocko <[email protected]> Cc: Alexander Viro <[email protected]> Cc: Mel Gorman <[email protected]> Cc: Johannes Weiner <[email protected]> Cc: Andi Kleen <[email protected]> Signed-off-by: Andrew Morton <[email protected]> Signed-off-by: Linus Torvalds <[email protected]>
1 parent 7e4411b commit 3fa6c50

File tree

1 file changed

+6
-2
lines changed

1 file changed

+6
-2
lines changed

lib/iov_iter.c

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -144,7 +144,7 @@ static size_t copy_page_to_iter_iovec(struct page *page, size_t offset, size_t b
144144
buf = iov->iov_base + skip;
145145
copy = min(bytes, iov->iov_len - skip);
146146

147-
if (!fault_in_pages_writeable(buf, copy)) {
147+
if (IS_ENABLED(CONFIG_HIGHMEM) && !fault_in_pages_writeable(buf, copy)) {
148148
kaddr = kmap_atomic(page);
149149
from = kaddr + offset;
150150

@@ -175,6 +175,7 @@ static size_t copy_page_to_iter_iovec(struct page *page, size_t offset, size_t b
175175
copy = min(bytes, iov->iov_len - skip);
176176
}
177177
/* Too bad - revert to non-atomic kmap */
178+
178179
kaddr = kmap(page);
179180
from = kaddr + offset;
180181
left = __copy_to_user(buf, from, copy);
@@ -193,6 +194,7 @@ static size_t copy_page_to_iter_iovec(struct page *page, size_t offset, size_t b
193194
bytes -= copy;
194195
}
195196
kunmap(page);
197+
196198
done:
197199
if (skip == iov->iov_len) {
198200
iov++;
@@ -225,7 +227,7 @@ static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t
225227
buf = iov->iov_base + skip;
226228
copy = min(bytes, iov->iov_len - skip);
227229

228-
if (!fault_in_pages_readable(buf, copy)) {
230+
if (IS_ENABLED(CONFIG_HIGHMEM) && !fault_in_pages_readable(buf, copy)) {
229231
kaddr = kmap_atomic(page);
230232
to = kaddr + offset;
231233

@@ -256,6 +258,7 @@ static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t
256258
copy = min(bytes, iov->iov_len - skip);
257259
}
258260
/* Too bad - revert to non-atomic kmap */
261+
259262
kaddr = kmap(page);
260263
to = kaddr + offset;
261264
left = __copy_from_user(to, buf, copy);
@@ -274,6 +277,7 @@ static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t
274277
bytes -= copy;
275278
}
276279
kunmap(page);
280+
277281
done:
278282
if (skip == iov->iov_len) {
279283
iov++;

0 commit comments

Comments
 (0)