Skip to content

Commit 412758c

Browse files
almostivanIngo Molnar
authored andcommitted
jump label, locking/static_keys: Update docs
Signed-off-by: Jason Baron <[email protected]> Signed-off-by: Peter Zijlstra (Intel) <[email protected]> Cc: Andrew Morton <[email protected]> Cc: Linus Torvalds <[email protected]> Cc: Paul E. McKenney <[email protected]> Cc: Peter Zijlstra <[email protected]> Cc: Thomas Gleixner <[email protected]> Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] Link: http://lkml.kernel.org/r/6b50f2f6423a2244f37f4b1d2d6c211b9dcdf4f8.1438227999.git.jbaron@akamai.com Signed-off-by: Ingo Molnar <[email protected]>
1 parent 2bf9e0a commit 412758c

File tree

2 files changed

+98
-68
lines changed

2 files changed

+98
-68
lines changed

Documentation/static-keys.txt

Lines changed: 52 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,30 +1,45 @@
11
Static Keys
22
-----------
33

4-
By: Jason Baron <[email protected]>
4+
DEPRECATED API:
5+
6+
The use of 'struct static_key' directly, is now DEPRECATED. In addition
7+
static_key_{true,false}() is also DEPRECATED. IE DO NOT use the following:
8+
9+
struct static_key false = STATIC_KEY_INIT_FALSE;
10+
struct static_key true = STATIC_KEY_INIT_TRUE;
11+
static_key_true()
12+
static_key_false()
13+
14+
The updated API replacements are:
15+
16+
DEFINE_STATIC_KEY_TRUE(key);
17+
DEFINE_STATIC_KEY_FALSE(key);
18+
static_key_likely()
19+
statick_key_unlikely()
520

621
0) Abstract
722

823
Static keys allows the inclusion of seldom used features in
924
performance-sensitive fast-path kernel code, via a GCC feature and a code
1025
patching technique. A quick example:
1126

12-
struct static_key key = STATIC_KEY_INIT_FALSE;
27+
DEFINE_STATIC_KEY_FALSE(key);
1328

1429
...
1530

16-
if (static_key_false(&key))
31+
if (static_branch_unlikely(&key))
1732
do unlikely code
1833
else
1934
do likely code
2035

2136
...
22-
static_key_slow_inc();
37+
static_branch_enable(&key);
2338
...
24-
static_key_slow_inc();
39+
static_branch_disable(&key);
2540
...
2641

27-
The static_key_false() branch will be generated into the code with as little
42+
The static_branch_unlikely() branch will be generated into the code with as little
2843
impact to the likely code path as possible.
2944

3045

@@ -56,7 +71,7 @@ the branch site to change the branch direction.
5671

5772
For example, if we have a simple branch that is disabled by default:
5873

59-
if (static_key_false(&key))
74+
if (static_branch_unlikely(&key))
6075
printk("I am the true branch\n");
6176

6277
Thus, by default the 'printk' will not be emitted. And the code generated will
@@ -75,68 +90,55 @@ the basis for the static keys facility.
7590

7691
In order to make use of this optimization you must first define a key:
7792

78-
struct static_key key;
79-
80-
Which is initialized as:
81-
82-
struct static_key key = STATIC_KEY_INIT_TRUE;
93+
DEFINE_STATIC_KEY_TRUE(key);
8394

8495
or:
8596

86-
struct static_key key = STATIC_KEY_INIT_FALSE;
97+
DEFINE_STATIC_KEY_FALSE(key);
98+
8799

88-
If the key is not initialized, it is default false. The 'struct static_key',
89-
must be a 'global'. That is, it can't be allocated on the stack or dynamically
100+
The key must be global, that is, it can't be allocated on the stack or dynamically
90101
allocated at run-time.
91102

92103
The key is then used in code as:
93104

94-
if (static_key_false(&key))
105+
if (static_branch_unlikely(&key))
95106
do unlikely code
96107
else
97108
do likely code
98109

99110
Or:
100111

101-
if (static_key_true(&key))
112+
if (static_branch_likely(&key))
102113
do likely code
103114
else
104115
do unlikely code
105116

106-
A key that is initialized via 'STATIC_KEY_INIT_FALSE', must be used in a
107-
'static_key_false()' construct. Likewise, a key initialized via
108-
'STATIC_KEY_INIT_TRUE' must be used in a 'static_key_true()' construct. A
109-
single key can be used in many branches, but all the branches must match the
110-
way that the key has been initialized.
117+
Keys defined via DEFINE_STATIC_KEY_TRUE(), or DEFINE_STATIC_KEY_FALSE, may
118+
be used in either static_branch_likely() or static_branch_unlikely()
119+
statemnts.
111120

112-
The branch(es) can then be switched via:
121+
Branch(es) can be set true via:
113122

114-
static_key_slow_inc(&key);
115-
...
116-
static_key_slow_dec(&key);
123+
static_branch_enable(&key);
117124

118-
Thus, 'static_key_slow_inc()' means 'make the branch true', and
119-
'static_key_slow_dec()' means 'make the branch false' with appropriate
120-
reference counting. For example, if the key is initialized true, a
121-
static_key_slow_dec(), will switch the branch to false. And a subsequent
122-
static_key_slow_inc(), will change the branch back to true. Likewise, if the
123-
key is initialized false, a 'static_key_slow_inc()', will change the branch to
124-
true. And then a 'static_key_slow_dec()', will again make the branch false.
125+
or false via:
126+
127+
static_branch_disable(&key);
125128

126-
An example usage in the kernel is the implementation of tracepoints:
129+
The branch(es) can then be switched via reference counts:
127130

128-
static inline void trace_##name(proto) \
129-
{ \
130-
if (static_key_false(&__tracepoint_##name.key)) \
131-
__DO_TRACE(&__tracepoint_##name, \
132-
TP_PROTO(data_proto), \
133-
TP_ARGS(data_args), \
134-
TP_CONDITION(cond)); \
135-
}
131+
static_branch_inc(&key);
132+
...
133+
static_branch_dec(&key);
136134

137-
Tracepoints are disabled by default, and can be placed in performance critical
138-
pieces of the kernel. Thus, by using a static key, the tracepoints can have
139-
absolutely minimal impact when not in use.
135+
Thus, 'static_branch_inc()' means 'make the branch true', and
136+
'static_branch_dec()' means 'make the branch false' with appropriate
137+
reference counting. For example, if the key is initialized true, a
138+
static_branch_dec(), will switch the branch to false. And a subsequent
139+
static_branch_inc(), will change the branch back to true. Likewise, if the
140+
key is initialized false, a 'static_branch_inc()', will change the branch to
141+
true. And then a 'static_branch_dec()', will again make the branch false.
140142

141143

142144
4) Architecture level code patching interface, 'jump labels'
@@ -150,9 +152,12 @@ simply fall back to a traditional, load, test, and jump sequence.
150152

151153
* #define JUMP_LABEL_NOP_SIZE, see: arch/x86/include/asm/jump_label.h
152154

153-
* __always_inline bool arch_static_branch(struct static_key *key), see:
155+
* __always_inline bool arch_static_branch(struct static_key *key, bool branch), see:
154156
arch/x86/include/asm/jump_label.h
155157

158+
* __always_inline bool arch_static_branch_jump(struct static_key *key, bool branch),
159+
see: arch/x86/include/asm/jump_label.h
160+
156161
* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type),
157162
see: arch/x86/kernel/jump_label.c
158163

@@ -173,7 +178,7 @@ SYSCALL_DEFINE0(getppid)
173178
{
174179
int pid;
175180

176-
+ if (static_key_false(&key))
181+
+ if (static_branch_unlikely(&key))
177182
+ printk("I am the true branch\n");
178183

179184
rcu_read_lock();

include/linux/jump_label.h

Lines changed: 46 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -7,17 +7,52 @@
77
* Copyright (C) 2009-2012 Jason Baron <[email protected]>
88
* Copyright (C) 2011-2012 Peter Zijlstra <[email protected]>
99
*
10+
* DEPRECATED API:
11+
*
12+
* The use of 'struct static_key' directly, is now DEPRECATED. In addition
13+
* static_key_{true,false}() is also DEPRECATED. IE DO NOT use the following:
14+
*
15+
* struct static_key false = STATIC_KEY_INIT_FALSE;
16+
* struct static_key true = STATIC_KEY_INIT_TRUE;
17+
* static_key_true()
18+
* static_key_false()
19+
*
20+
* The updated API replacements are:
21+
*
22+
* DEFINE_STATIC_KEY_TRUE(key);
23+
* DEFINE_STATIC_KEY_FALSE(key);
24+
* static_key_likely()
25+
* statick_key_unlikely()
26+
*
1027
* Jump labels provide an interface to generate dynamic branches using
11-
* self-modifying code. Assuming toolchain and architecture support, the result
12-
* of a "if (static_key_false(&key))" statement is an unconditional branch (which
13-
* defaults to false - and the true block is placed out of line).
28+
* self-modifying code. Assuming toolchain and architecture support, if we
29+
* define a "key" that is initially false via "DEFINE_STATIC_KEY_FALSE(key)",
30+
* an "if (static_branch_unlikely(&key))" statement is an unconditional branch
31+
* (which defaults to false - and the true block is placed out of line).
32+
* Similarly, we can define an initially true key via
33+
* "DEFINE_STATIC_KEY_TRUE(key)", and use it in the same
34+
* "if (static_branch_unlikely(&key))", in which case we will generate an
35+
* unconditional branch to the out-of-line true branch. Keys that are
36+
* initially true or false can be using in both static_branch_unlikely()
37+
* and static_branch_likely() statements.
1438
*
15-
* However at runtime we can change the branch target using
16-
* static_key_slow_{inc,dec}(). These function as a 'reference' count on the key
17-
* object, and for as long as there are references all branches referring to
18-
* that particular key will point to the (out of line) true block.
39+
* At runtime we can change the branch target by setting the key
40+
* to true via a call to static_branch_enable(), or false using
41+
* static_branch_disable(). If the direction of the branch is switched by
42+
* these calls then we run-time modify the branch target via a
43+
* no-op -> jump or jump -> no-op conversion. For example, for an
44+
* initially false key that is used in an "if (static_branch_unlikely(&key))"
45+
* statement, setting the key to true requires us to patch in a jump
46+
* to the out-of-line of true branch.
1947
*
20-
* Since this relies on modifying code, the static_key_slow_{inc,dec}() functions
48+
* In addtion to static_branch_{enable,disable}, we can also reference count
49+
* the key or branch direction via static_branch_{inc,dec}. Thus,
50+
* static_branch_inc() can be thought of as a 'make more true' and
51+
* static_branch_dec() as a 'make more false'. The inc()/dec()
52+
* interface is meant to be used exclusively from the inc()/dec() for a given
53+
* key.
54+
*
55+
* Since this relies on modifying code, the branch modifying functions
2156
* must be considered absolute slow paths (machine wide synchronization etc.).
2257
* OTOH, since the affected branches are unconditional, their runtime overhead
2358
* will be absolutely minimal, esp. in the default (off) case where the total
@@ -29,20 +64,10 @@
2964
* cause significant performance degradation. Struct static_key_deferred and
3065
* static_key_slow_dec_deferred() provide for this.
3166
*
32-
* Lacking toolchain and or architecture support, jump labels fall back to a simple
33-
* conditional branch.
34-
*
35-
* struct static_key my_key = STATIC_KEY_INIT_TRUE;
36-
*
37-
* if (static_key_true(&my_key)) {
38-
* }
39-
*
40-
* will result in the true case being in-line and starts the key with a single
41-
* reference. Mixing static_key_true() and static_key_false() on the same key is not
42-
* allowed.
67+
* Lacking toolchain and or architecture support, static keys fall back to a
68+
* simple conditional branch.
4369
*
44-
* Not initializing the key (static data is initialized to 0s anyway) is the
45-
* same as using STATIC_KEY_INIT_FALSE.
70+
* Additional babbling in: Documentation/static-keys.txt
4671
*/
4772

4873
#if defined(CC_HAVE_ASM_GOTO) && defined(CONFIG_JUMP_LABEL)

0 commit comments

Comments
 (0)