Skip to content

Commit ac227cd

Browse files
RoseZhang03RoseZhang03
committed
[libc] Updated header_generation.rst
Added new headergen documentation.
1 parent 3cb82f4 commit ac227cd

File tree

1 file changed

+136
-96
lines changed

1 file changed

+136
-96
lines changed

‎libc/docs/dev/header_generation.rst

Lines changed: 136 additions & 96 deletions
Original file line numberDiff line numberDiff line change
@@ -1,120 +1,160 @@
1+
.. role:: raw-html(raw)
2+
:format: html
3+
14
Generating Public and Internal headers
25
======================================
36

4-
.. warning::
5-
This page is severely out of date. Much of the information it contains may be
6-
incorrect. Please only remove this warning once the page has been updated.
7-
8-
Other libc implementations make use of preprocessor macro tricks to make header
9-
files platform agnostic. When macros aren't suitable, they rely on build
10-
system tricks to pick the right set of files to compile and export. While these
11-
approaches have served them well, parts of their systems have become extremely
12-
complicated making it hard to modify, extend or maintain. To avoid these
13-
problems in llvm-libc, we use a header generation mechanism. The mechanism is
14-
driven by a *header configuration language*.
15-
16-
Header Configuration Language
17-
-----------------------------
18-
19-
Header configuration language consists of few special *commands*. The header
20-
generation mechanism takes an input file, which has an extension of
21-
``.h.def``, and produces a header file with ``.h`` extension. The header
22-
configuration language commands are listed in the input ``.h.def`` file. While
23-
reading a ``.h.def`` file, the header generation tool does two things:
24-
25-
1. Copy the lines not containing commands as is into the output ``.h`` file.
26-
2. Replace the line on which a command occurs with some other text as directed
27-
by the command. The replacement text can span multiple lines.
28-
29-
Command syntax
30-
~~~~~~~~~~~~~~
31-
32-
A command should be listed on a line by itself, and should not span more than
33-
one line. The first token to appear on the line is the command name prefixed
34-
with ``%%``. For example, a line with the ``include_file`` command should start
35-
with ``%%include_file``. There can be indentation spaces before the ``%%``
36-
prefix.
7+
This is a new implementation of the previous libc header generator. The old
8+
header generator (libc-hdrgen aka “headergen”) is based on tablegen, which
9+
creates an awkward dependency on the rest of LLVM for our build system. By
10+
creating a new standalone headergen we can eliminate these dependencies for
11+
easier cross compatibility.
3712

38-
Most commands typically take arguments. They are listed as a comma separated
39-
list of named identifiers within parenthesis, similar to the C function call
40-
syntax. Before performing the action corresponding to the command, the header
41-
generator replaces the arguments with concrete values.
13+
There are 3 main components of the new Headergen. The first component are the
14+
yaml files that contain all the function header information and are separated by
15+
header specification and standard. The second component are the classes that are
16+
created for each component of the function header: macros, enumerations, types,
17+
function, arguments, and objects. The third component is the python script that
18+
uses the class representation to deserialize yaml files into its specific
19+
components and then reserializes the components into the function header. The
20+
python script also combines the generated header content with header definitions
21+
and extra macro and type inclusions from the .h.def file.
4222

43-
Argument Syntax
44-
~~~~~~~~~~~~~~~
4523

46-
Arguments are named indentifiers but prefixed with ``$`` and enclosed in ``{``
47-
and ``}``. For example, ``${path_to_constants}``.
48-
49-
Comments
50-
~~~~~~~~
51-
52-
There can be cases wherein one wants to add comments in the .h.def file but
53-
does not want them to be copied into the generated header file. Such comments
54-
can be added by beginning the comment lines with the ``<!>`` prefix. Currently,
55-
comments have to be on lines of their own. That is, they cannot be suffixes like
56-
this:
57-
58-
```
59-
%%include_file(a/b/c) <!> Path to c in b of a. !!! WRONG SYNTAX
60-
```
61-
62-
Available Commands
63-
------------------
64-
65-
Sub-sections below describe the commands currently available. Under each command
66-
is the description of the arguments to the command, and the action taken by the
67-
header generation tool when processing a command.
24+
Instructions
25+
-----------------------------
6826

69-
``include_file``
70-
~~~~~~~~~~~~~~~~
27+
Required Versions:
7128

72-
This is a replacement command which should be listed in an input ``.h.def``
73-
file.
29+
- Python Version: 3.11.8 [subject to be lower]
30+
- CMake Version: 3.20.0
7431

75-
Arguments
32+
1. Make sure to have `LLVM <https://llvm.org/docs/GettingStarted.html>`_ on your
33+
system.
34+
2. Make sure you have created a build directory within your LLVM directory.
35+
3. When generating the necessary files by your build make sure to include:
36+
``-DLLVM_LIBC_FULL_BUILD=ON`` within the CMake command since new headergen
37+
only runs on full-build.
38+
4. Once the build is complete, enter in the command line within the build
39+
directory ``ninja check-newhdrgen`` to ensure that the integration tests are
40+
passing.
41+
5. Then enter in the command line ``ninja libc`` to generate headers. Headers
42+
will be in “build/projects/libc/include” or “build/libc/include” in a runtime
43+
build. Sys spec headers will be located in
44+
``build/projects/libc/include/sys``.
7645

77-
* **path argument** - An argument representing a path to a file. The file
78-
should have an extension of ``.h.inc``.
7946

80-
Action
47+
New Headergen is turned on by default, but if you wanted to use old headergen,
48+
you can include this statement when building: ``-DLIBC_USE_NEW_HEADER_GEN=OFF``
8149

82-
This command instructs that the line on which the command appears should be
83-
replaced by the contents of the file whose path is passed as argument to the
84-
command.
50+
To add a function to the yaml files, you can either manually enter it in the
51+
yaml file depending on the header spec or enter it through the command line.
8552

86-
``begin``
87-
~~~~~~~~~
53+
To enter through the command line:
8854

89-
This is not a replacement command. It is an error to list it in the input
90-
``.h.def`` file. It is normally listed in the files included by the
91-
``include_file`` command (the ``.h.inc`` files). A common use of this command it
92-
mark the beginning of what is to be included. This prevents copying items like
93-
license headers into the generated header file.
55+
1. Make sure you are in the llvm-project directory.
9456

95-
Arguments
57+
2. Enter in the command line:
58+
``python3 libc/newhdrgen/yaml_to_classes.py
59+
libc/newhdrgen/yaml/[yaml_file.yaml] --add_function “<return_type>”
60+
<function_name> “<function_arg1, function_arg2>” <standard> <guard>
61+
<attribute>``
9662

97-
None.
63+
Example:
64+
``python3 libc/newhdrgen/yaml_to_classes.py
65+
libc/newhdrgen/yaml/ctype.yaml --add_function “char” example_funtion “int,
66+
void, const void” stdc example_float example_attribute``
67+
68+
Keep in mind only the return_type and arguments have quotes around them. If
69+
you do not have any guards or attributes you may enter “null” for both.
9870

99-
Action
71+
3. Check the yaml file that the added function is present. You will also get a
72+
generated header file with the new addition in the newhdrgen directory to
73+
examine.
10074

101-
The header generator will only include content starting from the line after the
102-
line on which this command is listed.
10375

104-
``public_api``
105-
~~~~~~~~~~~~~~
76+
Testing
77+
-----------------------------
10678

107-
This is a replacement command which should be listed in an input ``.h.def``
108-
file. The header file generator will replace this command with the public API of
109-
the target platform. See the build system document for more information on the
110-
relevant build rules. Also, see "Mechanics of public_api" to learn the mechanics
111-
of how the header generator replaces this command with the public API.
79+
New Headergen has an integration test that you may run once you have configured
80+
your CMake within the build directory. In the command line, enter the following:
81+
“ninja check -newhdrgen”. The integration test is one test that ensures the
82+
process of yaml to classes to generate headers works properly. If there are any
83+
new additions on formatting headers, make sure the test is updated with the
84+
specific addition.
11285

113-
Arguments
86+
Integration Test can be found in: ``libc/newhdrgen/tests/test_integration.py``
11487

115-
None.
88+
File to modify if adding something to formatting:
89+
``libc/newhdrgen/tests/expected_output/test_header.h``
11690

117-
Action
11891

119-
The header generator will replace this command with the public API to be exposed
120-
from the generated header file.
92+
Common Errors
93+
-----------------------------
94+
1. ``"/llvm-project/libc/newhdrgen/yaml_to_classes.py", line 67, in yaml_to_classes function_data["return_type"]``
95+
96+
If you receive this error or any error pertaining to
97+
``function_data[function_specific_component]`` while building the headers that
98+
means the function specific component is missing within the yaml files.
99+
Through the call stack, you will be able to find the header file which has the
100+
issue. Ensure there is no missing function specific component for that yaml
101+
header file.
102+
103+
2. ``CMake Error at:
104+
/llvm-project/libc/cmake/modules/LLVMLibCHeaderRules.cmake:86 (message):
105+
'add_gen_hdr2' rule requires GEN_HDR to be specified.
106+
Call Stack (most recent call first):
107+
/llvm-project/libc/include/CMakeLists.txt:22 (add_gen_header2)
108+
/llvm-project/libc/include/CMakeLists.txt:62 (add_header_macro)``
109+
110+
If you receive this error, there is a missing yaml_file, h_def file, or
111+
header name within the ``libc/include/CMakeLists.txt``. The last line in the
112+
error call stack will point to the header where there is a specific
113+
component missing. Ensure the correct style and required files are present:
114+
115+
| ``[header_name]``
116+
| ``[../libc/newhdrgen/yaml/[yaml_file.yaml]``
117+
| ``[header_name.h.def]``
118+
| ``[header_name.h]``
119+
| ``DEPENDS``
120+
| ``{Necessary Depend Files}``
121+
122+
3. ``usage: yaml_to_classes.py [-h] [--output_dir OUTPUT_DIR]
123+
[--h_def_file H_DEF_FILE] [--add_function RETURN_TYPE NAME ARGUMENTS
124+
STANDARDS GUARD ATTRIBUTES][--e ENTRY_POINTS] [--export-decls] yaml_file
125+
yaml_to_classes.py: error: argument --add_function: expected 6 arguments``
126+
127+
In the process of adding a function, you may run into an issue where the
128+
command line is requiring more arguments than what you currently have.Ensure that all components of the new function are filled. Even if you do
129+
not have a guard or attribute, make sure to put null in those two areas.
130+
131+
4. ``File "/llvm-project/libc/newhdrgen/header.py", line 60, in __str__ for function in self.functions: AttributeError: 'HeaderFile' object has no attribute 'functions'``
132+
133+
When running ninja libc in the build directory to generate headers you may
134+
receive the error above. Essentially this means that in
135+
``libc/newhdrgen/header.py`` there is a missing attribute named functions.
136+
Make sure all function components are defined within this file and there are
137+
no missing functions to add these components.
138+
139+
5. ``/llvm-project/build/projects/libc/include/sched.h:20:25: error: unknown type name 'size_t'; did you mean 'time_t'?``
140+
:raw-html:`<br />` ``20 | int_sched_getcpucount(size_t, const cpu_set_t*) __NOEXCEPT``
141+
:raw-html:`<br />` ``/llvm-project/build/projects/libc/include/llvm-libc-types/time_t.h:15:24: note: 'time_t' declared here``
142+
:raw-html:`<br />` ``15 | typedef __INT64_TYPE__ time_t;``
143+
144+
During the header generation process errors like the one above may occur
145+
because there are missing types for a specific header file. Check the yaml
146+
file corresponding to the header file and make sure all the necessary types
147+
that are being used are input into the types as well. Delete the specific
148+
header file from the build folder and re-run ninja libc to ensure the types
149+
are being recognized.
150+
151+
6. Test Integration Errors: Sometimes the integration test will fail but that
152+
still means the process is working unless the comparison between the output
153+
and expected_output is not showing. If that is the case make sure in
154+
``libc/newhdrgen/tests/test_integration.py`` there are no missing arguments
155+
that run through the script.
156+
If the integration tests are failing due to mismatching of lines or small
157+
errors in spacing that is nothing to worry about. If this is happening
158+
while you are making a new change to the formatting of the headers, then
159+
ensure the expected output file ``libc/newhdrgen/tests/expected_output/test_header.h``
160+
has the changes you are applying.

0 commit comments

Comments
 (0)