rds_rdma: allocate FMR according to max_item_soft

When max_item on the pool is very large(say 170000), large number of
alloc_fmr request would fail with -EGAIN. This can greatly hurt
performance.

Actually, whether it is going to allocate or wait for pool flush
should be according to "max_item_soft" rather than "max_item" because
we are resizing the pool by changing "max_item_soft" not max_item.

Also, when successfully allocated a lower layer fmr, we should increase
"max_item_soft" incrementally, not set it immediately to "max_item".

Orabug: 21551548

Signed-off-by: Wengang Wang <[email protected]>
Reviewed-by: Chien-Hua Yen <[email protected]>

Author: Wengang-oracle

net/rds/ib_rdma.c

@@ -341,7 +341,8 @@ static struct rds_ib_mr *rds_ib_alloc_fmr(struct rds_ib_device *rds_ibdev,
 		 * We're fussy with enforcing the FMR limit, though. If the driver
 		 * tells us we can't use more than N fmrs, we shouldn't start
 		 * arguing with it */
-		if (atomic_inc_return(&pool->item_count) <= pool->max_items)
+		if (atomic_inc_return(&pool->item_count) <=
+					atomic_read(&pool->max_items_soft))
 			break;
 
 		atomic_dec(&pool->item_count);
@@ -431,9 +432,8 @@ static struct rds_ib_mr *rds_ib_alloc_fmr(struct rds_ib_device *rds_ibdev,
 	else
 		rds_ib_stats_inc(s_ib_rdma_mr_1m_alloc);
 
-	if (atomic_read(&pool->item_count) >
-		atomic_read(&pool->max_items_soft))
-		atomic_set(&pool->max_items_soft, pool->max_items);
+	if (atomic_read(&pool->item_count) > atomic_read(&pool->max_items_soft))
+		atomic_inc(&pool->max_items_soft);
 
 	return ibmr;