Skip to content

Commit 5463bfd

Browse files
tzanussirostedt
tzanussi
authored and
rostedt
committed
tracing: Add support for named hist triggers
Allow users to define 'named' hist triggers. All triggers created with the same 'name=xxx' option will update the same shared histogram data. This expands the hist trigger syntax from this: # echo hist:keys=xxx ... [ if filter] > event/trigger to this: # echo hist:name=xxx:keys=xxx ... [ if filter] > event/trigger Named histograms must use a 'compatible' set of keys and values, which means each event added to a set of named triggers must have the same names and types. Reading the 'hist' file of any of the participating events will produce the same output as any other participating event, which is to be expected since they share the same data. Link: http://lkml.kernel.org/r/1dbc84ee3322a75daaf5b3ef1d0cc0a2fb682fc7.1457029949.git.tom.zanussi@linux.intel.com Signed-off-by: Tom Zanussi <[email protected]> Tested-by: Masami Hiramatsu <[email protected]> Reviewed-by: Namhyung Kim <[email protected]> Signed-off-by: Steven Rostedt <[email protected]>
1 parent db1388b commit 5463bfd

File tree

3 files changed

+407
-29
lines changed

3 files changed

+407
-29
lines changed

‎Documentation/trace/events.txt

Lines changed: 266 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -524,7 +524,7 @@ The following commands are supported:
524524

525525
hist:keys=<field1[,field2,...]>[:values=<field1[,field2,...]>]
526526
[:sort=<field1[,field2,...]>][:size=#entries][:pause][:continue]
527-
[:clear] [if <filter>]
527+
[:clear][:name=histname1] [if <filter>]
528528

529529
When a matching event is hit, an entry is added to a hash table
530530
using the key(s) and value(s) named. Keys and values correspond to
@@ -546,18 +546,28 @@ The following commands are supported:
546546
specified by the 'sort' keyword. If more than one field is
547547
specified, the result will be a 'sort within a sort': the first key
548548
is taken to be the primary sort key and the second the secondary
549-
key.
549+
key. If a hist trigger is given a name using the 'name' parameter,
550+
its histogram data will be shared with other triggers of the same
551+
name, and trigger hits will update this common data. Only triggers
552+
with 'compatible' fields can be combined in this way; triggers are
553+
'compatible' if the fields named in the trigger share the same
554+
number and type of fields and those fields also have the same names.
555+
Note that any two events always share the compatible 'hitcount' and
556+
'stacktrace' fields and can therefore be combined using those
557+
fields, however pointless that may be.
550558

551559
'hist' triggers add a 'hist' file to each event's subdirectory.
552560
Reading the 'hist' file for the event will dump the hash table in
553561
its entirety to stdout. If there are multiple hist triggers
554562
attached to an event, there will be a table for each trigger in the
555-
output. Each printed hash table entry is a simple list of the keys
556-
and values comprising the entry; keys are printed first and are
557-
delineated by curly braces, and are followed by the set of value
558-
fields for the entry. By default, numeric fields are displayed as
559-
base-10 integers. This can be modified by appending any of the
560-
following modifiers to the field name:
563+
output. The table displayed for a named trigger will be the same as
564+
any other instance having the same name. Each printed hash table
565+
entry is a simple list of the keys and values comprising the entry;
566+
keys are printed first and are delineated by curly braces, and are
567+
followed by the set of value fields for the entry. By default,
568+
numeric fields are displayed as base-10 integers. This can be
569+
modified by appending any of the following modifiers to the field
570+
name:
561571

562572
.hex display a number as a hex value
563573
.sym display an address as a symbol
@@ -1809,3 +1819,251 @@ The following commands are supported:
18091819
Hits: 0
18101820
Entries: 0
18111821
Dropped: 0
1822+
1823+
Named triggers can be used to have triggers share a common set of
1824+
histogram data. This capability is mostly useful for combining the
1825+
output of events generated by tracepoints contained inside inline
1826+
functions, but names can be used in a hist trigger on any event.
1827+
For example, these two triggers when hit will update the same 'len'
1828+
field in the shared 'foo' histogram data:
1829+
1830+
# echo 'hist:name=foo:keys=skbaddr.hex:vals=len' > \
1831+
/sys/kernel/debug/tracing/events/net/netif_receive_skb/trigger
1832+
# echo 'hist:name=foo:keys=skbaddr.hex:vals=len' > \
1833+
/sys/kernel/debug/tracing/events/net/netif_rx/trigger
1834+
1835+
You can see that they're updating common histogram data by reading
1836+
each event's hist files at the same time:
1837+
1838+
# cat /sys/kernel/debug/tracing/events/net/netif_receive_skb/hist;
1839+
cat /sys/kernel/debug/tracing/events/net/netif_rx/hist
1840+
1841+
# event histogram
1842+
#
1843+
# trigger info: hist:name=foo:keys=skbaddr.hex:vals=hitcount,len:sort=hitcount:size=2048 [active]
1844+
#
1845+
1846+
{ skbaddr: ffff88000ad53500 } hitcount: 1 len: 46
1847+
{ skbaddr: ffff8800af5a1500 } hitcount: 1 len: 76
1848+
{ skbaddr: ffff8800d62a1900 } hitcount: 1 len: 46
1849+
{ skbaddr: ffff8800d2bccb00 } hitcount: 1 len: 468
1850+
{ skbaddr: ffff8800d3c69900 } hitcount: 1 len: 46
1851+
{ skbaddr: ffff88009ff09100 } hitcount: 1 len: 52
1852+
{ skbaddr: ffff88010f13ab00 } hitcount: 1 len: 168
1853+
{ skbaddr: ffff88006a54f400 } hitcount: 1 len: 46
1854+
{ skbaddr: ffff8800d2bcc500 } hitcount: 1 len: 260
1855+
{ skbaddr: ffff880064505000 } hitcount: 1 len: 46
1856+
{ skbaddr: ffff8800baf24e00 } hitcount: 1 len: 32
1857+
{ skbaddr: ffff88009fe0ad00 } hitcount: 1 len: 46
1858+
{ skbaddr: ffff8800d3edff00 } hitcount: 1 len: 44
1859+
{ skbaddr: ffff88009fe0b400 } hitcount: 1 len: 168
1860+
{ skbaddr: ffff8800a1c55a00 } hitcount: 1 len: 40
1861+
{ skbaddr: ffff8800d2bcd100 } hitcount: 1 len: 40
1862+
{ skbaddr: ffff880064505f00 } hitcount: 1 len: 174
1863+
{ skbaddr: ffff8800a8bff200 } hitcount: 1 len: 160
1864+
{ skbaddr: ffff880044e3cc00 } hitcount: 1 len: 76
1865+
{ skbaddr: ffff8800a8bfe700 } hitcount: 1 len: 46
1866+
{ skbaddr: ffff8800d2bcdc00 } hitcount: 1 len: 32
1867+
{ skbaddr: ffff8800a1f64800 } hitcount: 1 len: 46
1868+
{ skbaddr: ffff8800d2bcde00 } hitcount: 1 len: 988
1869+
{ skbaddr: ffff88006a5dea00 } hitcount: 1 len: 46
1870+
{ skbaddr: ffff88002e37a200 } hitcount: 1 len: 44
1871+
{ skbaddr: ffff8800a1f32c00 } hitcount: 2 len: 676
1872+
{ skbaddr: ffff88000ad52600 } hitcount: 2 len: 107
1873+
{ skbaddr: ffff8800a1f91e00 } hitcount: 2 len: 92
1874+
{ skbaddr: ffff8800af5a0200 } hitcount: 2 len: 142
1875+
{ skbaddr: ffff8800d2bcc600 } hitcount: 2 len: 220
1876+
{ skbaddr: ffff8800ba36f500 } hitcount: 2 len: 92
1877+
{ skbaddr: ffff8800d021f800 } hitcount: 2 len: 92
1878+
{ skbaddr: ffff8800a1f33600 } hitcount: 2 len: 675
1879+
{ skbaddr: ffff8800a8bfff00 } hitcount: 3 len: 138
1880+
{ skbaddr: ffff8800d62a1300 } hitcount: 3 len: 138
1881+
{ skbaddr: ffff88002e37a100 } hitcount: 4 len: 184
1882+
{ skbaddr: ffff880064504400 } hitcount: 4 len: 184
1883+
{ skbaddr: ffff8800a8bfec00 } hitcount: 4 len: 184
1884+
{ skbaddr: ffff88000ad53700 } hitcount: 5 len: 230
1885+
{ skbaddr: ffff8800d2bcdb00 } hitcount: 5 len: 196
1886+
{ skbaddr: ffff8800a1f90000 } hitcount: 6 len: 276
1887+
{ skbaddr: ffff88006a54f900 } hitcount: 6 len: 276
1888+
1889+
Totals:
1890+
Hits: 81
1891+
Entries: 42
1892+
Dropped: 0
1893+
# event histogram
1894+
#
1895+
# trigger info: hist:name=foo:keys=skbaddr.hex:vals=hitcount,len:sort=hitcount:size=2048 [active]
1896+
#
1897+
1898+
{ skbaddr: ffff88000ad53500 } hitcount: 1 len: 46
1899+
{ skbaddr: ffff8800af5a1500 } hitcount: 1 len: 76
1900+
{ skbaddr: ffff8800d62a1900 } hitcount: 1 len: 46
1901+
{ skbaddr: ffff8800d2bccb00 } hitcount: 1 len: 468
1902+
{ skbaddr: ffff8800d3c69900 } hitcount: 1 len: 46
1903+
{ skbaddr: ffff88009ff09100 } hitcount: 1 len: 52
1904+
{ skbaddr: ffff88010f13ab00 } hitcount: 1 len: 168
1905+
{ skbaddr: ffff88006a54f400 } hitcount: 1 len: 46
1906+
{ skbaddr: ffff8800d2bcc500 } hitcount: 1 len: 260
1907+
{ skbaddr: ffff880064505000 } hitcount: 1 len: 46
1908+
{ skbaddr: ffff8800baf24e00 } hitcount: 1 len: 32
1909+
{ skbaddr: ffff88009fe0ad00 } hitcount: 1 len: 46
1910+
{ skbaddr: ffff8800d3edff00 } hitcount: 1 len: 44
1911+
{ skbaddr: ffff88009fe0b400 } hitcount: 1 len: 168
1912+
{ skbaddr: ffff8800a1c55a00 } hitcount: 1 len: 40
1913+
{ skbaddr: ffff8800d2bcd100 } hitcount: 1 len: 40
1914+
{ skbaddr: ffff880064505f00 } hitcount: 1 len: 174
1915+
{ skbaddr: ffff8800a8bff200 } hitcount: 1 len: 160
1916+
{ skbaddr: ffff880044e3cc00 } hitcount: 1 len: 76
1917+
{ skbaddr: ffff8800a8bfe700 } hitcount: 1 len: 46
1918+
{ skbaddr: ffff8800d2bcdc00 } hitcount: 1 len: 32
1919+
{ skbaddr: ffff8800a1f64800 } hitcount: 1 len: 46
1920+
{ skbaddr: ffff8800d2bcde00 } hitcount: 1 len: 988
1921+
{ skbaddr: ffff88006a5dea00 } hitcount: 1 len: 46
1922+
{ skbaddr: ffff88002e37a200 } hitcount: 1 len: 44
1923+
{ skbaddr: ffff8800a1f32c00 } hitcount: 2 len: 676
1924+
{ skbaddr: ffff88000ad52600 } hitcount: 2 len: 107
1925+
{ skbaddr: ffff8800a1f91e00 } hitcount: 2 len: 92
1926+
{ skbaddr: ffff8800af5a0200 } hitcount: 2 len: 142
1927+
{ skbaddr: ffff8800d2bcc600 } hitcount: 2 len: 220
1928+
{ skbaddr: ffff8800ba36f500 } hitcount: 2 len: 92
1929+
{ skbaddr: ffff8800d021f800 } hitcount: 2 len: 92
1930+
{ skbaddr: ffff8800a1f33600 } hitcount: 2 len: 675
1931+
{ skbaddr: ffff8800a8bfff00 } hitcount: 3 len: 138
1932+
{ skbaddr: ffff8800d62a1300 } hitcount: 3 len: 138
1933+
{ skbaddr: ffff88002e37a100 } hitcount: 4 len: 184
1934+
{ skbaddr: ffff880064504400 } hitcount: 4 len: 184
1935+
{ skbaddr: ffff8800a8bfec00 } hitcount: 4 len: 184
1936+
{ skbaddr: ffff88000ad53700 } hitcount: 5 len: 230
1937+
{ skbaddr: ffff8800d2bcdb00 } hitcount: 5 len: 196
1938+
{ skbaddr: ffff8800a1f90000 } hitcount: 6 len: 276
1939+
{ skbaddr: ffff88006a54f900 } hitcount: 6 len: 276
1940+
1941+
Totals:
1942+
Hits: 81
1943+
Entries: 42
1944+
Dropped: 0
1945+
1946+
And here's an example that shows how to combine histogram data from
1947+
any two events even if they don't share any 'compatible' fields
1948+
other than 'hitcount' and 'stacktrace'. These commands create a
1949+
couple of triggers named 'bar' using those fields:
1950+
1951+
# echo 'hist:name=bar:key=stacktrace:val=hitcount' > \
1952+
/sys/kernel/debug/tracing/events/sched/sched_process_fork/trigger
1953+
# echo 'hist:name=bar:key=stacktrace:val=hitcount' > \
1954+
/sys/kernel/debug/tracing/events/net/netif_rx/trigger
1955+
1956+
And displaying the output of either shows some interesting if
1957+
somewhat confusing output:
1958+
1959+
# cat /sys/kernel/debug/tracing/events/sched/sched_process_fork/hist
1960+
# cat /sys/kernel/debug/tracing/events/net/netif_rx/hist
1961+
1962+
# event histogram
1963+
#
1964+
# trigger info: hist:name=bar:keys=stacktrace:vals=hitcount:sort=hitcount:size=2048 [active]
1965+
#
1966+
1967+
{ stacktrace:
1968+
_do_fork+0x18e/0x330
1969+
kernel_thread+0x29/0x30
1970+
kthreadd+0x154/0x1b0
1971+
ret_from_fork+0x3f/0x70
1972+
} hitcount: 1
1973+
{ stacktrace:
1974+
netif_rx_internal+0xb2/0xd0
1975+
netif_rx_ni+0x20/0x70
1976+
dev_loopback_xmit+0xaa/0xd0
1977+
ip_mc_output+0x126/0x240
1978+
ip_local_out_sk+0x31/0x40
1979+
igmp_send_report+0x1e9/0x230
1980+
igmp_timer_expire+0xe9/0x120
1981+
call_timer_fn+0x39/0xf0
1982+
run_timer_softirq+0x1e1/0x290
1983+
__do_softirq+0xfd/0x290
1984+
irq_exit+0x98/0xb0
1985+
smp_apic_timer_interrupt+0x4a/0x60
1986+
apic_timer_interrupt+0x6d/0x80
1987+
cpuidle_enter+0x17/0x20
1988+
call_cpuidle+0x3b/0x60
1989+
cpu_startup_entry+0x22d/0x310
1990+
} hitcount: 1
1991+
{ stacktrace:
1992+
netif_rx_internal+0xb2/0xd0
1993+
netif_rx_ni+0x20/0x70
1994+
dev_loopback_xmit+0xaa/0xd0
1995+
ip_mc_output+0x17f/0x240
1996+
ip_local_out_sk+0x31/0x40
1997+
ip_send_skb+0x1a/0x50
1998+
udp_send_skb+0x13e/0x270
1999+
udp_sendmsg+0x2bf/0x980
2000+
inet_sendmsg+0x67/0xa0
2001+
sock_sendmsg+0x38/0x50
2002+
SYSC_sendto+0xef/0x170
2003+
SyS_sendto+0xe/0x10
2004+
entry_SYSCALL_64_fastpath+0x12/0x6a
2005+
} hitcount: 2
2006+
{ stacktrace:
2007+
netif_rx_internal+0xb2/0xd0
2008+
netif_rx+0x1c/0x60
2009+
loopback_xmit+0x6c/0xb0
2010+
dev_hard_start_xmit+0x219/0x3a0
2011+
__dev_queue_xmit+0x415/0x4f0
2012+
dev_queue_xmit_sk+0x13/0x20
2013+
ip_finish_output2+0x237/0x340
2014+
ip_finish_output+0x113/0x1d0
2015+
ip_output+0x66/0xc0
2016+
ip_local_out_sk+0x31/0x40
2017+
ip_send_skb+0x1a/0x50
2018+
udp_send_skb+0x16d/0x270
2019+
udp_sendmsg+0x2bf/0x980
2020+
inet_sendmsg+0x67/0xa0
2021+
sock_sendmsg+0x38/0x50
2022+
___sys_sendmsg+0x14e/0x270
2023+
} hitcount: 76
2024+
{ stacktrace:
2025+
netif_rx_internal+0xb2/0xd0
2026+
netif_rx+0x1c/0x60
2027+
loopback_xmit+0x6c/0xb0
2028+
dev_hard_start_xmit+0x219/0x3a0
2029+
__dev_queue_xmit+0x415/0x4f0
2030+
dev_queue_xmit_sk+0x13/0x20
2031+
ip_finish_output2+0x237/0x340
2032+
ip_finish_output+0x113/0x1d0
2033+
ip_output+0x66/0xc0
2034+
ip_local_out_sk+0x31/0x40
2035+
ip_send_skb+0x1a/0x50
2036+
udp_send_skb+0x16d/0x270
2037+
udp_sendmsg+0x2bf/0x980
2038+
inet_sendmsg+0x67/0xa0
2039+
sock_sendmsg+0x38/0x50
2040+
___sys_sendmsg+0x269/0x270
2041+
} hitcount: 77
2042+
{ stacktrace:
2043+
netif_rx_internal+0xb2/0xd0
2044+
netif_rx+0x1c/0x60
2045+
loopback_xmit+0x6c/0xb0
2046+
dev_hard_start_xmit+0x219/0x3a0
2047+
__dev_queue_xmit+0x415/0x4f0
2048+
dev_queue_xmit_sk+0x13/0x20
2049+
ip_finish_output2+0x237/0x340
2050+
ip_finish_output+0x113/0x1d0
2051+
ip_output+0x66/0xc0
2052+
ip_local_out_sk+0x31/0x40
2053+
ip_send_skb+0x1a/0x50
2054+
udp_send_skb+0x16d/0x270
2055+
udp_sendmsg+0x2bf/0x980
2056+
inet_sendmsg+0x67/0xa0
2057+
sock_sendmsg+0x38/0x50
2058+
SYSC_sendto+0xef/0x170
2059+
} hitcount: 88
2060+
{ stacktrace:
2061+
_do_fork+0x18e/0x330
2062+
SyS_clone+0x19/0x20
2063+
entry_SYSCALL_64_fastpath+0x12/0x6a
2064+
} hitcount: 244
2065+
2066+
Totals:
2067+
Hits: 489
2068+
Entries: 7
2069+
Dropped: 0

‎kernel/trace/trace.c

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3842,6 +3842,7 @@ static const char readme_msg[] =
38423842
"\t [:sort=<field1[,field2,...]>]\n"
38433843
"\t [:size=#entries]\n"
38443844
"\t [:pause][:continue][:clear]\n"
3845+
"\t [:name=histname1]\n"
38453846
"\t [if <filter>]\n\n"
38463847
"\t When a matching event is hit, an entry is added to a hash\n"
38473848
"\t table using the key(s) and value(s) named, and the value of a\n"
@@ -3854,13 +3855,18 @@ static const char readme_msg[] =
38543855
"\t specified using the 'sort' keyword. The sort direction can\n"
38553856
"\t be modified by appending '.descending' or '.ascending' to a\n"
38563857
"\t sort field. The 'size' parameter can be used to specify more\n"
3857-
"\t or fewer than the default 2048 entries for the hashtable size.\n\n"
3858+
"\t or fewer than the default 2048 entries for the hashtable size.\n"
3859+
"\t If a hist trigger is given a name using the 'name' parameter,\n"
3860+
"\t its histogram data will be shared with other triggers of the\n"
3861+
"\t same name, and trigger hits will update this common data.\n\n"
38583862
"\t Reading the 'hist' file for the event will dump the hash\n"
38593863
"\t table in its entirety to stdout. If there are multiple hist\n"
38603864
"\t triggers attached to an event, there will be a table for each\n"
3861-
"\t trigger in the output. The default format used to display a\n"
3862-
"\t given field can be modified by appending any of the following\n"
3863-
"\t modifiers to the field name, as applicable:\n\n"
3865+
"\t trigger in the output. The table displayed for a named\n"
3866+
"\t trigger will be the same as any other instance having the\n"
3867+
"\t same name. The default format used to display a given field\n"
3868+
"\t can be modified by appending any of the following modifiers\n"
3869+
"\t to the field name, as applicable:\n\n"
38643870
"\t .hex display a number as a hex value\n"
38653871
"\t .sym display an address as a symbol\n"
38663872
"\t .sym-offset display an address as a symbol and offset\n"

0 commit comments

Comments
 (0)