Skip to content

Commit 8972979

Browse files
RoseZhang03RoseZhang03
authored
[libc] Updated header_generation.rst (#99712)
Added new headergen documentation.
1 parent 3cb82f4 commit 8972979

File tree

1 file changed

+164
-85
lines changed

1 file changed

+164
-85
lines changed

‎libc/docs/dev/header_generation.rst

Lines changed: 164 additions & 85 deletions
Original file line numberDiff line numberDiff line change
@@ -1,120 +1,199 @@
11
Generating Public and Internal headers
22
======================================
33

4-
.. warning::
5-
This page is severely out of date. Much of the information it contains may be
6-
incorrect. Please only remove this warning once the page has been updated.
4+
This is a new implementation of the previous libc header generator. The old
5+
header generator (libc-hdrgen aka "Headergen") was based on TableGen, which
6+
created an awkward dependency on the rest of LLVM for our build system. By
7+
creating a new standalone Headergen we can eliminate these dependencies for
8+
easier cross compatibility.
79

8-
Other libc implementations make use of preprocessor macro tricks to make header
9-
files platform agnostic. When macros aren't suitable, they rely on build
10-
system tricks to pick the right set of files to compile and export. While these
11-
approaches have served them well, parts of their systems have become extremely
12-
complicated making it hard to modify, extend or maintain. To avoid these
13-
problems in llvm-libc, we use a header generation mechanism. The mechanism is
14-
driven by a *header configuration language*.
10+
There are 3 main components of the new Headergen. The first component are the
11+
YAML files that contain all the function header information and are separated by
12+
header specification and standard. The second component are the classes that are
13+
created for each component of the function header: macros, enumerations, types,
14+
function, arguments, and objects. The third component is the Python script that
15+
uses the class representation to deserialize YAML files into its specific
16+
components and then reserializes the components into the function header. The
17+
Python script also combines the generated header content with header definitions
18+
and extra macro and type inclusions from the .h.def file.
1519

16-
Header Configuration Language
17-
-----------------------------
1820

19-
Header configuration language consists of few special *commands*. The header
20-
generation mechanism takes an input file, which has an extension of
21-
``.h.def``, and produces a header file with ``.h`` extension. The header
22-
configuration language commands are listed in the input ``.h.def`` file. While
23-
reading a ``.h.def`` file, the header generation tool does two things:
21+
Instructions
22+
------------
2423

25-
1. Copy the lines not containing commands as is into the output ``.h`` file.
26-
2. Replace the line on which a command occurs with some other text as directed
27-
by the command. The replacement text can span multiple lines.
24+
Required Versions:
25+
- Python Version: 3.11.8
2826

29-
Command syntax
30-
~~~~~~~~~~~~~~
27+
1. Keep full-build mode on when building, otherwise headers will not be
28+
generated.
29+
2. Once the build is complete, enter in the command line within the build
30+
directory ``ninja check-newhdrgen`` to ensure that the integration tests are
31+
passing.
32+
3. Then enter in the command line ``ninja libc`` to generate headers. Headers
33+
will be in ``build/projects/libc/include`` or ``build/libc/include`` in a
34+
runtime build. Sys spec headers will be located in
35+
``build/projects/libc/include/sys``.
3136

32-
A command should be listed on a line by itself, and should not span more than
33-
one line. The first token to appear on the line is the command name prefixed
34-
with ``%%``. For example, a line with the ``include_file`` command should start
35-
with ``%%include_file``. There can be indentation spaces before the ``%%``
36-
prefix.
3737

38-
Most commands typically take arguments. They are listed as a comma separated
39-
list of named identifiers within parenthesis, similar to the C function call
40-
syntax. Before performing the action corresponding to the command, the header
41-
generator replaces the arguments with concrete values.
38+
New Headergen is turned on by default, but if you want to use old Headergen,
39+
you can include this statement when building: ``-DLIBC_USE_NEW_HEADER_GEN=OFF``
4240

43-
Argument Syntax
44-
~~~~~~~~~~~~~~~
41+
To add a function to the YAML files, you can either manually enter it in the
42+
YAML file corresponding to the header it belongs to or add it through the
43+
command line.
4544

46-
Arguments are named indentifiers but prefixed with ``$`` and enclosed in ``{``
47-
and ``}``. For example, ``${path_to_constants}``.
45+
To add through the command line:
4846

49-
Comments
50-
~~~~~~~~
47+
1. Make sure you are in the llvm-project directory.
5148

52-
There can be cases wherein one wants to add comments in the .h.def file but
53-
does not want them to be copied into the generated header file. Such comments
54-
can be added by beginning the comment lines with the ``<!>`` prefix. Currently,
55-
comments have to be on lines of their own. That is, they cannot be suffixes like
56-
this:
49+
2. Enter in the command line:
5750

58-
```
59-
%%include_file(a/b/c) <!> Path to c in b of a. !!! WRONG SYNTAX
60-
```
51+
.. code-block:: none
6152
62-
Available Commands
63-
------------------
53+
python3 libc/newhdrgen/yaml_to_classes.py
54+
libc/newhdrgen/yaml/[yaml_file.yaml] --add_function "<return_type>" <function_name> "<function_arg1, function_arg2>" <standard> <guard> <attribute>
6455
65-
Sub-sections below describe the commands currently available. Under each command
66-
is the description of the arguments to the command, and the action taken by the
67-
header generation tool when processing a command.
56+
Example:
6857

69-
``include_file``
70-
~~~~~~~~~~~~~~~~
58+
.. code-block:: none
7159
72-
This is a replacement command which should be listed in an input ``.h.def``
73-
file.
60+
python3 libc/newhdrgen/yaml_to_classes.py
61+
libc/newhdrgen/yaml/ctype.yaml --add_function "char" example_function
62+
"int, void, const void" stdc example_float example_attribute
7463
75-
Arguments
64+
Keep in mind only the return_type and arguments have quotes around them. If
65+
you do not have any guards or attributes you may enter "null" for both.
7666

77-
* **path argument** - An argument representing a path to a file. The file
78-
should have an extension of ``.h.inc``.
67+
3. Check the YAML file that the added function is present. You will also get a
68+
generated header file with the new addition in the newhdrgen directory to
69+
examine.
7970

80-
Action
8171

82-
This command instructs that the line on which the command appears should be
83-
replaced by the contents of the file whose path is passed as argument to the
84-
command.
72+
Testing
73+
-------
8574

86-
``begin``
87-
~~~~~~~~~
75+
New Headergen has an integration test that you may run once you have configured
76+
your CMake within the build directory. In the command line, enter the following:
77+
``ninja check-newhdrgen``. The integration test is one test that ensures the
78+
process of YAML to classes to generate headers works properly. If there are any
79+
new additions on formatting headers, make sure the test is updated with the
80+
specific addition.
8881

89-
This is not a replacement command. It is an error to list it in the input
90-
``.h.def`` file. It is normally listed in the files included by the
91-
``include_file`` command (the ``.h.inc`` files). A common use of this command it
92-
mark the beginning of what is to be included. This prevents copying items like
93-
license headers into the generated header file.
82+
Integration Test can be found in: ``libc/newhdrgen/tests/test_integration.py``
9483

95-
Arguments
84+
File to modify if adding something to formatting:
85+
``libc/newhdrgen/tests/expected_output/test_header.h``
9686

97-
None.
9887

99-
Action
88+
Common Errors
89+
-------------
90+
1. Missing function specific component
10091

101-
The header generator will only include content starting from the line after the
102-
line on which this command is listed.
92+
Example:
10393

104-
``public_api``
105-
~~~~~~~~~~~~~~
94+
.. code-block:: none
10695
107-
This is a replacement command which should be listed in an input ``.h.def``
108-
file. The header file generator will replace this command with the public API of
109-
the target platform. See the build system document for more information on the
110-
relevant build rules. Also, see "Mechanics of public_api" to learn the mechanics
111-
of how the header generator replaces this command with the public API.
96+
"/llvm-project/libc/newhdrgen/yaml_to_classes.py", line 67, in yaml_to_classes function_data["return_type"]
11297
113-
Arguments
98+
If you receive this error or any error pertaining to
99+
``function_data[function_specific_component]`` while building the headers
100+
that means the function specific component is missing within the YAML files.
101+
Through the call stack, you will be able to find the header file which has
102+
the issue. Ensure there is no missing function specific component for that
103+
YAML header file.
114104

115-
None.
105+
2. CMake Error: require argument to be specified
116106

117-
Action
107+
Example:
118108

119-
The header generator will replace this command with the public API to be exposed
120-
from the generated header file.
109+
.. code-block:: none
110+
111+
CMake Error at:
112+
/llvm-project/libc/cmake/modules/LLVMLibCHeaderRules.cmake:86 (message):
113+
'add_gen_hdr2' rule requires GEN_HDR to be specified.
114+
Call Stack (most recent call first):
115+
/llvm-project/libc/include/CMakeLists.txt:22 (add_gen_header2)
116+
/llvm-project/libc/include/CMakeLists.txt:62 (add_header_macro)
117+
118+
If you receive this error, there is a missing YAML file, h_def file, or
119+
header name within the ``libc/include/CMakeLists.txt``. The last line in the
120+
error call stack will point to the header where there is a specific component
121+
missing. Ensure the correct style and required files are present:
122+
123+
| ``[header_name]``
124+
| ``[../libc/newhdrgen/yaml/[yaml_file.yaml]``
125+
| ``[header_name.h.def]``
126+
| ``[header_name.h]``
127+
| ``DEPENDS``
128+
| ``{Necessary Depend Files}``
129+
130+
3. Command line: expected arguments
131+
132+
Example:
133+
134+
.. code-block:: none
135+
136+
usage: yaml_to_classes.py [-h] [--output_dir OUTPUT_DIR] [--h_def_file H_DEF_FILE]
137+
[--add_function RETURN_TYPE NAME ARGUMENTS STANDARDS GUARD ATTRIBUTES]
138+
[--e ENTRY_POINTS] [--export-decls]
139+
yaml_file
140+
yaml_to_classes.py:
141+
error: argument --add_function: expected 6 arguments
142+
143+
In the process of adding a function, you may run into an issue where the
144+
command line is requiring more arguments than what you currently have. Ensure
145+
that all components of the new function are filled. Even if you do not have a
146+
guard or attribute, make sure to put null in those two areas.
147+
148+
4. Object has no attribute
149+
150+
Example:
151+
152+
.. code-block:: none
153+
154+
File "/llvm-project/libc/newhdrgen/header.py", line 60, in __str__ for
155+
function in self.functions: AttributeError: 'HeaderFile' object has no
156+
attribute 'functions'
157+
158+
When running ``ninja libc`` in the build directory to generate headers you
159+
may receive the error above. Essentially this means that in
160+
``libc/newhdrgen/header.py`` there is a missing attribute named functions.
161+
Make sure all function components are defined within this file and there are
162+
no missing functions to add these components.
163+
164+
5. Unknown type name
165+
166+
Example:
167+
168+
.. code-block:: none
169+
170+
/llvm-project/build/projects/libc/include/sched.h:20:25: error: unknown type
171+
name 'size_t'; did you mean 'time_t'?
172+
20 | int_sched_getcpucount(size_t, const cpu_set_t*) __NOEXCEPT
173+
| ^
174+
/llvm-project/build/projects/libc/include/llvm-libc-types/time_t.h:15:24:
175+
note: 'time_t' declared here
176+
15 | typedef __INT64_TYPE__ time_t;
177+
| ^
178+
179+
During the header generation process errors like the one above may occur
180+
because there are missing types for a specific header file. Check the YAML
181+
file corresponding to the header file and make sure all the necessary types
182+
that are being used are input into the types as well. Delete the specific
183+
header file from the build folder and re-run ``ninja libc`` to ensure the
184+
types are being recognized.
185+
186+
6. Test Integration Errors
187+
188+
Sometimes the integration test will fail but that
189+
still means the process is working unless the comparison between the output
190+
and expected_output is not showing. If that is the case make sure in
191+
``libc/newhdrgen/tests/test_integration.py`` there are no missing arguments
192+
that run through the script.
193+
194+
If the integration tests are failing due to mismatching of lines or small
195+
errors in spacing that is nothing to worry about. If this is happening while
196+
you are making a new change to the formatting of the headers, then
197+
ensure the expected output file
198+
``libc/newhdrgen/tests/expected_output/test_header.h`` has the changes you
199+
are applying.

0 commit comments

Comments
 (0)