DTrace based on BPF Implementation Plan

Introduction

This document provides details on the implementation of DTrace on top of existing Linux kernel tracing facilities. It is meant as a guideline for project planning, and provides (to the extent possible) the order of implementation of various components, features, and capabilities of DTrace on BPF.

The document is organized by high level features, providing more detailed implementation information for each of these features. Generally, the implementation will follow this break-up of functionality, but it is certainly anticipated that some deviations will occur. One concrete example would be the string datatype. While the implementation of values of this datatype can be isolated to specific parts of the DTrace on Linux source code, support for this datatype is present in many different areas. Therefore, the implementation of the string datatype can either be done by providing all functionality as a single deliverable, or it can be done by implementing parts of the string datatype support for features that are planned as a specific deliverable, thereby providing a partial string datatype implementation. This document will be updated as these practical decisions about deliverables are made.

User Visible Features

Built-in variables

There are two approaches possible for implementing built-in variables. We can either generate code directly from the compiler whenever a built-in variable is encountered (which makes it impossible to cache the values unless we make the generated code even more fancy), or we can implement a get_bvar() function to which the compiler will generate calls whenever a built-in variable value is needed. That function could provide for caching of values. The latter approach is currently preferred.

Name	Target Release	In Release	Description	Comments
arg0 .. arg9		2.0.0-1.2	Raw probe argument	dctx.argv[0] .. dctx.argv[9]
args			Mapped (possibly translated) arguments
caller		2.0.0-1.6	Function in which the probe fired
curcpu		2.0.0-1.2	CPU on which the probe fired	Use BPF helper: bpf_get_smp_processor_id()
curthread		2.0.0-1.2	Task in which the probe fired	Use BPF helper: bpf_get_current_task()
epid		2.0.0-1.2	Enabled probe ID	dctx->mst→epid (resolved at program load time)
errno		2.0.0-1.6	Return code of the current system call. It will only have a non-zero value during the processing of the syscall return probe. In all other cases it is 0.	Ensure that the current probe is a syscall return probe, and retrieve the value as dctx->mst→argv[0]. Since it is (currently) difficult to determine which probe we're executing for, it might be easier to populate an errno field in dctx→mst from the prologue of a syscall return probe (in all other cases it needs to be 0), and just return that.
execname	2.0.0-1.8		Executable name of the current task	Use BPF helper: bpf_get_current_comm() Depends on reading kernel structures (not implemented yet)
gid		2.0.0-1.2	Group ID of the current task	Use BPF helper: bpf_get_current_uid_gid()
id		2.0.0-1.3	Probe ID	dctx->mst→prid (resolved at program load time)
ipl			Interrupt level
pid		2.0.0-1.2	Process ID of current task	Use BPF helper: bpf_get_current_pid_tgid()
ppid		2.0.0-1.3	Parent process ID of current task	BPF code to read task->real_parent→pid
probefunc		2.0.0-1.6	Probe description: function name	The plan is to provide a BPF map that maps EPID to probe ID and probe description names
probemod		2.0.0-1.6	Probe description: module name	The plan is to provide a BPF map that maps EPID to probe ID and probe description names
probename		2.0.0-1.6	Probe description: probe name	The plan is to provide a BPF map that maps EPID to probe ID and probe description names
probeprov		2.0.0-1.6	Probe description: provider name	The plan is to provide a BPF map that maps EPID to probe ID and probe description names
stackdepth		2.0.0-1.6
tid		2.0.0-1.2		Use BPF helper: bpf_get_current_pid_tgid()
timestamp		2.0.0-1.2		Use BPF helper: bpf_ktime_get_ns()
ucaller		2.0.0-1.6	Userspace function calling function where probe fires
uid		2.0.0-1.2		Use BPF helper: bpf_get_current_uid_gid()
uregs
ustackdepth		2.0.0-1.6
vtimestamp
walltimestamp		2.0.0-1,6

Clause-local variables

Name	Target Release	In Release	Description	Comments
Initial value		2.0.0-1.0		DTrace documentation specifies that the initial value of clause-local variables is unspecified. BPF enforces a strict store-before-load policy. In DTrace v2 clause-local variables are always initialized as 0.
		2.0.0-1.5		DESIGN CHANGE: No longer applicable.
Allocate space		2.0.0-1.5
. (1) size <= 8 bytes		2.0.0-1.0		Clause-local variables of 8 bytes or less are allocated on the stack as a 64-bit (8 bytes) value.
		2.0.0-1.5		DESIGN CHANGE: Clause-local variables of any size are stored as a sequence of bytes in a BPF map value.
. (2) size > 8 bytes		2.0.0-1.5		Clause-local variables of more than 8 bytes are allocated as a sequence of bytes in the 'lvars' BPF map value, each aligned at a 64-bit (8 byte) boundary.
. Size-align data		2.0.0-1.6		DESIGN CHANGE: Ensure that variable values are stored at their proper alignment rather than all aligned at 8-byte boundaries.
Get value		2.0.0-1.5
. (1) by value		2.0.0-1.0		Only for values of 8 bytes or less.
		2.0.0-1.5		For values of any size.
. (2) by reference		2.0.0-1.5
. . (a) size <= 8		2.0.0-1.5		DESIGN CHANGE: Reference to the storage location of the clause-local variable in the 'lvars' BPF map value.
. . (b) size > 8		2.0.0-1.5		Reference to the storage location of the clause-local variable in the' lvars' BPF map value.
Set value		2.0.0-1.5
. (1) size <= 8		2.0.0-1.0		Direct store to the stack location.
		2.0.0-1.5		DESIGN CHANGE: Direct store to the storage location of the clause-local -variable in the 'lvars' BPF map value.
. (2) size > bytes		2.0.0-1.5		memcpy() to the storage location of the clause-local -variable in the 'lvars' BPF map value.

Global variables

Name	Target Release	In Release	Description	Comments
Initial value		2.0.0-1.0		DTrace documentation specifies that the initial value of clause-local variables is unspecified. BPF enforces a strict store-before-load policy. In DTrace v2 clause-local variables are always initialized as 0.
		2.0.0-1.5		DESIGN CHANGE: No longer applicable.
Allocate space		2.0.0-1.5
. (1) size <= 8 bytes		2.0.0-1.0		Clause-local variables of 8 bytes or less are allocated on the stack as a 64-bit (8 bytes) value.
		2.0.0-1.5		DESIGN CHANGE: Clause-local variables of any size are stored as a sequence of bytes in a BPF map value.
. (2) size > 8 bytes		2.0.0-1.5		Clause-local variables of more than 8 bytes are allocated as a sequence of bytes in the 'gvars' BPF map value, each aligned at a 64-bit (8 byte) boundary.
. Size-align data		2.0.0-1.6		DESIGN CHANGE: Ensure that variable values are stored at their proper alignment rather than all aligned at 8-byte boundaries.
Get value		2.0.0-1.5
. (1) by value		2.0.0-1.0		Only for values of 8 bytes or less.
		2.0.0-1.5		For values of any size.
. (2) by reference		2.0.0-1.5
. . (a) size <= 8		2.0.0-1.5		DESIGN CHANGE: Reference to the storage location of the clause-local variable in the 'gvars' BPF map value.
. . (b) size > 8		2.0.0-1.5		Reference to the storage location of the clause-local variable in the' gvars' BPF map value.
Set value		2.0.0-1.5
. (1) size <= 8		2.0.0-1.0		Direct store to the stack location.
		2.0.0-1.5		DESIGN CHANGE: Direct store to the storage location of the clause-local -variable in the 'gvars' BPF map value.
. (2) size > bytes		2.0.0-1.5		memcpy() to the storage location of the clause-local -variable in the 'gvars' BPF map value.

Thread-local-storage variables

Name	Target Release	In Release	Description	Comments
Key calculation	2.0.0-1.x			This requires a formula that converts the tuple of key values into a unique index into the thread-local storage variable table.
Initial value	2.0.0-1.x			Per DTrace documentation, the initial value is undefined.
Allocate space
. (1) size <= 8 bytes	2.0.0-1.x			DESIGN CHANGE: All sizes will be handled the same (see below).
. (2) size > 8 bytes	2.0.0-1.x
Get value
. (1) by value	2.0.0-1.x
. (2) by reference
. . (a) size <= 8	2.0.0-1.x			DESIGN CHANGE: All sizes will be handled the same (see below).
. . (b) size > 8	2.0.0-1.x
Set value
. (1) size <= 8	2.0.0-1.x			DESIGN CHANGE: All sizes will be handled the same (see below).
. (2) size > bytes	2.0.0-1.x

Strings:

Due to limitations in the BPF implementation and the strict validation rules of the BPF verifier, string handling requires custom BPF functions. DTrace is designed around a concept of fixed length string space which makes the implementation easier but it also wastes a lot of space. In addition, it would be very expensive to have to perform a strlen() operation as part of string manipulation operations.
Strings are stored as a variable-length integer encoding its length, followed by a 0-terminated sequence of data bytes. This provides a compact format that eliminates the need to recalculate the length of the string.

Name	Target Release	In Release	Description	Comments
Consolidated string table		2.0.0-1.6	While every DIFO (compiled clause) has its own string table, a global one is necessary to avoid duplicating string constants between BPF programs.
Add probe description components to string table		2.0.0-1.6	Support for probeprov, probemod, probefun, and probename requires those strings (for all enabled probes) to be added to the string table.
Create 'probes' BPF map		2.0.0-1.6	The mapping from probe ID to probeprov, probemod, probefun, and probename is to be stored in a BPF map so clauses can retrieve the correct values.
Variable-length integers		2.0.0-1.6	String length are stored as variable-length integers ahead of the actual string data bytes.
String constants
. Inline constants		2.0.0-1.6	Support for string constant explicitly included in source code (i.e. "...")
. Dynamic ref'd		2.0.0-1.6	Support for dynamically referenced string constants (e.g. probename)
Dynamic strings
. Kernel strings	2.0.0-1.x		Support to retrieve (load) strings from kernel addresses
. Userspace strings	2.0.0-1.x		Support to retrieve (load) strings from userspace addresses
Functions
. index()	2.0.0-1.8		Same as strstr() blocked by nested loops not accepted by BPF verifier (hard limit on # of instructions)
. rindex()	2.0.0-1.8		Same as strstr() blocked by nested loops not accepted by BPF verifier (hard limit on # of instructions)
. strchr()	2.0.0-1.8		limited functionality , needs tricks in code generation
. strjoin()		2.0.0-1.7	Store sum of lengths, memcpy() 1st string (omit NUL), and memcpy() 2nd string (incl. NUL)
. strlen()		2.0.0-1.6	Use dt_vint2int()
. strrchr()	2.0.0-1.8		limited functionality , needs tricks in code generation
. strstr()	2.0.0-1.8		Same as strstr() blocked by nested loops not accepted by BPF verifier (hard limit on # of instructions)
. strtok()	2.0.0-1.8		special version of strstr, same issues w/ verifier
. substr()		2.0.0-1.7	Store length of slice, memcpy() slice, and append NUL

Associative arrays

The implementation of associative arrays in DTrace based on BPF can learn a lot from the legacy implementation. The limiting conditions are near identical. We must work with a limited storage pool that is set side ahead of time, and we need to be able to support dynamic allocation and deallocation without sleeping. Static analysis of the D code being compiled allows us to determine the maximum element size. The legacy implementation limits allocations for associative array elements to the calculated maximum element size so that all allocations are of the same size.

Aggregations

Name	Target Release	In Release	Description	Comments
Implement aggregations as variabes		2.0.0-1.4	Aggregations were implemented as a special kind of action. In the new design, they are a special kind of variables. This requires compiler and disassembler changes.
Allocate space		2.0.0-1.4	Aggregations are allocated as one or more 8-byte chunks (each storing a uint64_t) in the aggs BPF per-CPU map value.
Data update mechanism		2.0.0-1.4	Aggregation data is to be updated on a per-CPU basis (i.e. the CPU a probe executes on) in a lock-free, wait-free manner.  We use a latch sequence protected data-pair approach.
Consumer processing		2.0.0-1.4	Processing aggregation data requires retrieving the agg BPF map and aggregating the data across the active CPUs.
Functions		2.0.0-1.4
. avg()		2.0.0-1.4
. count()		2.0.0-1.4
. llquantize()		2.0.0-1.4
. lquantize()		2.0.0-1.4
. max()		2.0.0-1.4
. min()		2.0.0-1.4
. quantize()		2.0.0-1.4
. stddev()		2.0.0-1.4
. sum()		2.0.0-1.4
Implement tuple indexed aggregations

Aggregations used to be implemented with their own specific per-cpu output buffer (alongside the regular per-cpu trace data output buffer). Space was allocated in the per-cpu buffer for the different aggregation data items (one or more uint64_t values), and the execution of aggregation actions caused these values to be updated. When the consumer retrieved the aggregation buffer for a specific cpu, a buffer swap would take place to ensure that further probe firing would record aggregation data in a cleared buffer while the consumer processed the data in the now inactive buffer.

In the BPF-based design, aggregations are stored in the singleton map value of a per-CPU BPF map (index 0). The map value will be allocated with enough space to hold all aggregations that are in use. Aggregation actions will update the values allocated to the aggregation they operate on. When the consumer is ready to process aggregation data, it will perform a bpf_map_lookup_elem() to retrieve the data for all CPUs at once.

There is a potential complexity: the legacy implementation for DTrace could ensure that the buffer swap did not happen in the midst of probe processing on that specific CPU. The new design needs to guard against processing incomplete data (e.g. for aggregation functions that use 2 data items, one has been updated while the other has not).

Speculations

Name	Target Release	In Release	Description	Comments
Speculation buffer management	2.0.0-1.8		New functionality for userspace.
Speculation ID implementation	2.0.0-1.8
Speculation actions
. commit()	2.0.0-1.8			Action
. discard()	2.0.0-1.8			Action
. speculate()	2.0.0-1.8			Action
. speculation()	2.0.0-1.8			Subroutine

Speculative tracing in the legacy implementation used kernel-side buffers tracing data, and it would copy the tracing data to the main output buffer upon commit (or discard the buffer). This model is difficult to implement without kernel modifications because of how much work is require at the kernel level.

A new implementation will be used for speculative tracing where the handling of speculations is managed at the userspace level instead. When a speculation is started, a unique tag ID will be generated and all tracing data that is generated from that point forward will be tagged with this ID. As userspace processes the output buffer, it will append any tagged data to its respective speculation buffer. When a commit() is encountered, the stored data will be processed as part of the main tracing data stream. If a discard() is encountered, the stored data is dropped.

Actions

Name	Target Release	In Release	Description	Comments
breakpoint()
chill(int)
clear(@)
commit(int)				See .
denormalize(@)		2.0.0-1.5
discard(int)				See .
exit(int)		2.0.0-1.0
freopen(@, ...)		2.0.0-1.3	Variant of the printf() action.
ftruncate()	2.0.0-1.8
func(uintptr_t)
jstack([uint32_t], [])
mod(uintptr_t)
normalize(@, [uint64])		2.0.0-1.5
panic()
pcap(void*, int)			It is unclear how we can do this without kernel support, unless BPF already provides access to this data.
printa(@, ...)		2.0.0-1.4
printf(string, ...)		2.0.0-1.2		Numeric values only.
raise(int)		2.0.0-1.2	Send a signal to the running task. Use BPF helper: bpf_send_signal()
setopt(const char *, [])
speculate(int)				See .
stack([uint32_t])		2.0.0-1.6
stop()
sym(uintptr_t)
system(@, ...)		2.0.0-1.3	Variant of the printf() action.
trace(@)		2.0.0-1.0 2.0.0-1.2	2.0.0-1.0: Raw numeric values as signed 64-bit integers 2.0.0-1.2: [Orabug: 31407534] Enhancement of DTrace v1 behaviour by providing consistent output based on sign and bit-width of the value.	Numeric values only.
tracemem(@, size_t, [])
trunc(@, [uint64_t])
uaddr(uintptr_t)		2.0.0-1.6
ufunc(uintptr_t)
umod(uintptr_t)		2.0.0-1.6
ustack([uint32_t]. [uint32_t]))		2.0.0-1.6
usym(uintptr_t)		2.0.0-1.6

Subroutines

Name	Target Release	In Release	Description	Comments
alloca
basename
bcopy	2.0.0-1.9		See BPF function memcpy(). Depends on alloca(), which depends on dynamic memory management
cleanpath
copyin
copyinstr
copyinto
d_path			Added for Linux - we need to investigate if it is still needed.
dirname
getmajor
getminor
htonl	2.0.0-1.8
htonll	2.0.0-1.8
htons	2.0.0-1.8
index	2.0.0-1.8		See #Strings
inet_ntoa
inet_ntoa6
inet_ntop
lltostr
msgdsize			Specific to Solaris - won't get implemented.
msgsize			Specific to Solaris - won't get implemented.
mutex_owned
mutex_owner
mutex_type_adaptive
mutex_type_spin
ntohl	2.0.0-1.8
ntohll	2.0.0-1.8
ntohs	2.0.0-1.8
progenyof
rand	2.0.0-1.8
rindex	2.0.0-1.8		See #Strings
rw_iswriter
rw_read_held
rw_write_held
speculation			See .
strchr	2.0.0-1.8		See #Strings
strjoin		2.0.0-1.7	See #Strings
strlen		2.0.0-1.6	See #Strings
strrchr	2.0.0-1.8		See #Strings
strstr	2.0.0-1.8		See #Strings
strtok	2.0.0-1.8		See #Strings
substr		2.0.0-1.7	See #Strings

Stack Traces

They are working as well as in V1. We use the kernel backtrace capability (BPF helper function).

DONE (2.0.0-1.6)

Management of Scratch Space

Name	Target Release	In Release	Description	Comments
stack space		2.0.0-1.6
string space		2.0.0-1.7
alloca space	2.0.0-1.9

Various internal functions such as string manipulation functions and stack trace retrieval need temporary storage.  The D language supports explicit allocation of some temporary storage (for the duration of a clause) using the alloca() function.  This represents two distinct cases of scratch space needs.  The legacy implementation used a rather simplistic but guaranteed approach: all storage requests were cumulative.  This means that there was no reuse of any scratch space within the execution of an action.  This made space management very easy but it also wasted space.

Since DTrace based on BPF uses entire clauses rather than actions as compilation units the waste of scratch space is more significant.  On the other hand, the fact that strings have a known maximum size and given an upper limit for stack trace depth, we can determine a fixed storage size that is sufficient to satisfy the needs of string functions (even in the common case of nested function calls) and of stack trace retrieval functions.  We also know that an expression cannot combine string functions and stack trace functions, so the same scratch space can be used to support both needs.

IN PROGRESS

Output formatting

Converting dtrace_eprobedesc_t into dtrace_datadesc_t

Description needed DONE (2.0.0-1.2)

Flow indent

Description needed DONE (2.0.0-1.2)

Providers

dtrace (BEGIN, END, and ERROR)

BEGIN and END probes

Name	Target Release	In Release	Description	Comments
Create probes		2.0.0-1.0
Determine probe info		2.0.0-1.0
Implement trampoline		2.0.0-1.0
Probe cleanup		2.0.0-1.0
BEGIN probe semantics		2.0.0-1.3	BEGIN probe must fire before all other probes
END probe semantics		2.0.0-1.3	END probe must fire after all other probes

ERROR probe

Name	Target Release	In Release	Description	Comments
Create probe		2.0.0-1.5		Kris
Determine probe info		2.0.0-1.5		Kris
Implement trampoline		2.0.0-1.5		Kris

Timer based tracing (profile and tick)

profile probes

Name	Target Release	In Release	Description	Comments
Create probe		2.0.0-1.2
Determine probe info		2.0.0-1.2
Implement trampoline		2.0.0-1.2

tick probes

Name	Target Release	In Release	Description	Comments
Create probe		2.0.0-1.2
Determine probe info		2.0.0-1.2
Implement trampoline		2.0.0-1.2

System call tracing (syscall)

Name	Target Release	In Release	Description	Comments
Discover probe points		2.0.0-0.8
Create probes		2.0.0-0.8
Determine probe info		2.0.0-0.8
Implement trampoline		2.0.0-0.8

Function Boundary Tracing (FBT)

Name	Target Release	In Release	Description	Comments
Discover probe points		2.0.0-0.8
Create probes		2.0.0-0.8
Determine probe info		2.0.0-0.8
Implement trampoline		2.0.0-0.8
Probe cleanup		2.0.0-1.0

Statically Defined Tracing (SDT)

• Discover probe points: DONE in release 2.0.0-0.8
• Create probes: DONE in release 2.0.0-0.8
• Determine probe info: DONE in release 2.0.0-0.8
• Implement trampoline: DONE in release 2.0.0-0.8
• Standard DTrace probes: Provide the standard set of DTrace SDT probes. This requires a kernel patch. Where the first implementation of DTrace on Linux provided a DTrace-specific way to define SDT probes and kernel tracepoints were exposed using that same mechanism, DTrace in the current form will need to do the opposite: implement the DTrace SDT probes as kernel tracepoints. It may be possible to re-use some of the existing kernel tracepoints (possibly requiring argument mapping and translators).

Userspace Process Tracing (pid and USDT)

pid probes

The implementation of the PID provider (function boundary tracing for userspace and arbitrary instruction tracing for userspace) is build upon the uprobes support provided by the Linux kernel through the perf event sub-system. Because PID probes are grouped in process-specific providers whereas the underlying uprobes are inode based, a level of indirection is needed to implement PID probes in DTrace. This is done by associating the process-specific probes with their corresponding uprobes based "real" probe (belonging to a generic, system-wide "pid" provider). In other words, when a pid probe is requested for pid1234:a.out:func:entry, we first create a pid-provider probe pid:0x12ab:func:entry (representing the underlying uprobe) if it does not exist yet. We then associate the pid1234:a.out:func:entry probe with that pid:0x12ab:func:entry probe.
When the final probe program for that pid:0x12ab:func:entry probe is being constructed, clauses for all associated probes are included in the program, with a conditional preceding each to ensure the correct clauses are executed based on the PID of the process that triggered the probe.
Additional changes that are needed for the pid provider include changing the notification mechanism for process death to use an eventfd so that the epoll_wait() that waits for trace buffer data will also be triggered for process death notifications.

• Discover probe points DONE in release 2.0.0-1.5
• Create probes: DONE in release 2.0.0-1.5
• Implement dynamic provider creating: DONE in release 2.0.0-1.5
• Implement dynamic probe creation: DONE in release 2.0.0-1.5
• Implement probe and provider cleanup upon error during compilation: DONE in release 2.0.0-1.5
• Implement meta-probe support : DONE in release 2.0.0-1.5
• Determine probe info : DONE in release 2.0.0-1.5
• Implement trampoline : DONE in release 2.0.0-1.5
• Implement provider-controlled clause call code generation: DONE in release 2.0.0-1.5
• Probe cleanup : DONE in release 2.0.0-1.5

USDT Probes

• Discover probe points
• Create probes
• Determine probe info
• Implement trampoline
• Probe cleanup

Language implementation features

• Compilation of D clauses into BPF code: DONE in release 2.0.0-0.1
• BPF probe program as trampoline to compiled D clauses: DONE in release 2.0.0-0.1
• Linking generated BPF code with pre-compiled BPF support functions: DONE in release 2.0.0-1.0
• Integration of the clause predicate as conditional into the main clause code: DONE in release 2.0.0-1.2
• Support probe specifications with wildcards: DONE in release 2.0.0-1.2
• Support more than one clause for a given probe: DONE in release 2.0.0-1.2
• Store the DTrace BPF context somewhere other than the BPF stack: DONE in release 2.0.0-1.2
• Provide correct buffer consumption semantics: DONE release 2.0.0-1.3
• Use default action only for empty clause: DONE release 2.0.0-1.3

Compilation of D clauses into BPF code

DTrace on Linux is no longer using the in-kernel support code for DTrace and instead uses existing Linux kernel tracing features such as BPF. The D compiler (primarily the code generator and the assembler) will now generate BPF code. Whereas before clauses were divided into actions and each action was compiled into its own D program, the entire clause is now being compiled into BPF code.

The assembler requires modifications as well to handle BPF specific relocations and to account for the compilation of complete clauses. Also, the disassembler needs to be modified to support and output BPF code. DONE (2.0.0-0.1)

BPF probe program as trampoline to compiled D clauses

BPF programs are invoked from trace event triggers with a probe type specific BPF context. DTrace D clauses are expected to execute in a consistent context, especially because a single clause can be associated with probes of different types. In order to support this configuration, a probe-specific trampoline is generated by the provider code, using the BPF context to populate a generic DTrace context. Then it calls a predicate function (if the clause has a predicate), and if the predicate function returns true, the generated clause function is called.

DONE (2.0.0-0.1)

Linking generated BPF code with pre-compiled BPF support functions

There are common code sequences that are necessary for the implementation of D code using BPF. Things like retrieving the value of a variable of a particular kind, setting the value of a variable of a particular kind, string manipulation functions, etc. Many of these can be implemented in C and compiled into BPF code. These functions are linked together into the bpf_dlib.o ELF object. These pre-compiled BPF support functions are referenced by dynamically generated BPF code as function calls to external functions. This means that entries are added into the BPF symbol relocation table. These are resolved during a special linking stage that is executed prior to loading the BPF programs into the kernel.

During this linking stage the relocation records that reference external symbols are processed. The following high level mechanism is implemented:

1. For each relocation record, determine the pre-compiled BPF support function symbol it references
2. If the symbol has already been seen, continue with the next relocation record (goto step 1)
3. If the symbol is new, do the following:
   a. Mark the symbol as seen
   b. Append the executable code for the symbol to the BPF program we are processing
   c. Process the relocation records for this symbol per the algorithm described here

At the conclusion of this recursive process, there should not be any unresolved symbols left. The BPF program will contain all executable code necessary for executing the entire program.

DONE (2.0.0-1.0)

Integration of the clause predicate as conditional into the main clause code

The predicate can perform most operations that the main clause code can do, including accessing variables of all three kinds and calling subroutines. Many of these operations require access to the DTrace BPF context. However, only the main program (dt_program) was being generated with the appropriate prologue code to ensure that BPF support functions would be able to operate in a consistent execution context. While it would be possible to resolve this issue by generating the predicate as a function with prologue and epilogue as well, the more logical solution is to simply include the predicate in the main clause code as a conditional. The main program becomes:

PROLOGUE if (!predicate), goto exit MAIN PROGRAM CODE EPILOGUE exit: return 0

DONE (2.0.0-1.2)

Support probe specifications with wildcards

(See: DTrace D Compiler redesign) DONE (2.0.0-1.2)

Support more than one clause for a given probe

(See: DTrace D Compiler redesign) DONE (2.0.0-1.2)

Store the DTrace BPF context somewhere other than the BPF stack

The BPF JIT compiler generates code that depends on a single BPF stack (maximum size right now limited to 512 bytes) whereas the BPF interpreter provides a BPF stack for each function that is executing in a call chain. So, for interpreted code, the BPF stack for each function in the call chain The quite limited stack size for JIT-compiled code caused stack overrun conditions when executing D clauses because the DTrace BPF context (stored on the stack) was consuming too much precious stack space.

The solution is to decouple the DTrace BPF context from the DTrace machine state. While the DTrace BPF context must still reside on the stack, the DTrace machine state can be stored elsewhere. The choice has been made to use the 'mem' BPF map that is used to provide memory for the output buffer. The first portion of the memory used for the map value is set aside to hole the DTrace machine state, and the remainder is for the output buffer.

DONE (2.0.0-1.2)

Provide correct buffer consumption semantics

The processing of the trace buffers must adhere to specific DTrace buffer consumption semantics. The buffer that contains the BEGIN probe trace data must be processed first, so that the BEGIN probe will always be reported first in output. This means that we need to know what CPU the BEGIN probe fired on. Similarly, the buffer that contains the END probe trace data must be processed last, so that the END probe will always be reported last in output. This means we need to know what CPU the END probe fired on.

When processing the buffer for the BEGIN probe trace data it is important to also process any ERROR probe firings that relate to the BEGIN probe. On the other hand, ERROR probe trace data that is found in that same buffer but that does not correspond to the BEGIN probe firing must be processed after the BEGIN probe processing completes.

DONE (2.0.0-1.3)

Use default action only for empty clause

The "default action" is used for empty clauses. After the BPF port, it was also used for clauses that were not empty but simply had no data-recording actions. One can argue whether or not that is reasonable, but it is a significant departure from established precedent. Rather reasonable scripts would perform very differently with the new behavior. So, revert to the legacy behavior.

DONE (2.0.0-1.3)