Skip to content

[libc] rename newhdrgen to just hdrgen #118545

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion libc/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ if(LIBC_BUILD_GPU_LOADER OR NOT LLVM_RUNTIMES_BUILD)
endif()
endif()

add_subdirectory(newhdrgen)
add_subdirectory(hdrgen)

option(LIBC_CMAKE_VERBOSE_LOGGING
"Log details warnings and notifications during CMake configuration." OFF)
Expand Down
4 changes: 2 additions & 2 deletions libc/cmake/modules/LLVMLibCHeaderRules.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ function(add_gen_header target_name)

add_custom_command(
OUTPUT ${out_file}
COMMAND ${Python3_EXECUTABLE} ${LIBC_SOURCE_DIR}/newhdrgen/yaml_to_classes.py
COMMAND ${Python3_EXECUTABLE} ${LIBC_SOURCE_DIR}/hdrgen/yaml_to_classes.py
${yaml_file}
--h_def_file ${def_file}
${entry_points}
Expand All @@ -126,7 +126,7 @@ function(add_gen_header target_name)
set(decl_out_file ${LIBC_INCLUDE_DIR}/llvm-libc-decls/${relative_path})
add_custom_command(
OUTPUT ${decl_out_file}
COMMAND ${Python3_EXECUTABLE} ${LIBC_SOURCE_DIR}/newhdrgen/yaml_to_classes.py
COMMAND ${Python3_EXECUTABLE} ${LIBC_SOURCE_DIR}/hdrgen/yaml_to_classes.py
${yaml_file}
--export-decls
${entry_points}
Expand Down
66 changes: 30 additions & 36 deletions libc/docs/dev/header_generation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,15 @@
Generating Public and Internal headers
======================================

This is a new implementation of the previous libc header generator. The old
header generator (libc-hdrgen aka "Headergen") was based on TableGen, which
created an awkward dependency on the rest of LLVM for our build system. By
creating a new standalone Headergen we can eliminate these dependencies for
easier cross compatibility.

There are 3 main components of the new Headergen. The first component are the
YAML files that contain all the function header information and are separated by
header specification and standard. The second component are the classes that are
created for each component of the function header: macros, enumerations, types,
function, arguments, and objects. The third component is the Python script that
uses the class representation to deserialize YAML files into its specific
components and then reserializes the components into the function header. The
Python script also combines the generated header content with header definitions
and extra macro and type inclusions from the .h.def file.
There are 3 main components of the Headergen. The first component are the YAML
files that contain all the function header information and are separated by
header specification and standard. The second component are the classes that
are created for each component of the function header: macros, enumerations,
types, function, arguments, and objects. The third component is the Python
script that uses the class representation to deserialize YAML files into its
specific components and then reserializes the components into the function
header. The Python script also combines the generated header content with
header definitions and extra macro and type inclusions from the .h.def file.


Instructions
Expand All @@ -30,7 +24,7 @@ Required Versions:
1. Keep full-build mode on when building, otherwise headers will not be
generated.
2. Once the build is complete, enter in the command line within the build
directory ``ninja check-newhdrgen`` to ensure that the integration tests are
directory ``ninja check-hdrgen`` to ensure that the integration tests are
passing.
3. Then enter in the command line ``ninja libc`` to generate headers. Headers
will be in ``build/projects/libc/include`` or ``build/libc/include`` in a
Expand All @@ -50,41 +44,41 @@ To add through the command line:

.. code-block:: none

python3 libc/newhdrgen/yaml_to_classes.py
libc/newhdrgen/yaml/[yaml_file.yaml] --add_function "<return_type>" <function_name> "<function_arg1, function_arg2>" <standard> <guard> <attribute>
python3 libc/hdrgen/yaml_to_classes.py
libc/hdrgen/yaml/[yaml_file.yaml] --add_function "<return_type>" <function_name> "<function_arg1, function_arg2>" <standard> <guard> <attribute>

Example:

.. code-block:: none

python3 libc/newhdrgen/yaml_to_classes.py
libc/newhdrgen/yaml/ctype.yaml --add_function "char" example_function
python3 libc/hdrgen/yaml_to_classes.py
libc/hdrgen/yaml/ctype.yaml --add_function "char" example_function
"int, void, const void" stdc example_float example_attribute

Keep in mind only the return_type and arguments have quotes around them. If
you do not have any guards or attributes you may enter "null" for both.

3. Check the YAML file that the added function is present. You will also get a
generated header file with the new addition in the newhdrgen directory to
generated header file with the new addition in the hdrgen directory to
examine.

If you want to sort the functions alphabetically you can check out libc/newhdrgen/yaml_functions_sorted.py.
If you want to sort the functions alphabetically you can check out libc/hdrgen/yaml_functions_sorted.py.


Testing
-------

New Headergen has an integration test that you may run once you have configured
your CMake within the build directory. In the command line, enter the following:
``ninja check-newhdrgen``. The integration test is one test that ensures the
process of YAML to classes to generate headers works properly. If there are any
new additions on formatting headers, make sure the test is updated with the
specific addition.
Headergen has an integration test that you may run once you have configured
your CMake within the build directory. In the command line, enter the
following: ``ninja check-hdrgen``. The integration test is one test that
ensures the process of YAML to classes to generate headers works properly. If
there are any new additions on formatting headers, make sure the test is
updated with the specific addition.

Integration Test can be found in: ``libc/newhdrgen/tests/test_integration.py``
Integration Test can be found in: ``libc/hdrgen/tests/test_integration.py``

File to modify if adding something to formatting:
``libc/newhdrgen/tests/expected_output/test_header.h``
``libc/hdrgen/tests/expected_output/test_header.h``


Common Errors
Expand All @@ -95,7 +89,7 @@ Common Errors

.. code-block:: none

"/llvm-project/libc/newhdrgen/yaml_to_classes.py", line 67, in yaml_to_classes function_data["return_type"]
"/llvm-project/libc/hdrgen/yaml_to_classes.py", line 67, in yaml_to_classes function_data["return_type"]

If you receive this error or any error pertaining to
``function_data[function_specific_component]`` while building the headers
Expand Down Expand Up @@ -123,7 +117,7 @@ Common Errors
missing. Ensure the correct style and required files are present:

| ``[header_name]``
| ``[../libc/newhdrgen/yaml/[yaml_file.yaml]``
| ``[../libc/hdrgen/yaml/[yaml_file.yaml]``
| ``[header_name.h.def]``
| ``[header_name.h]``
| ``DEPENDS``
Expand Down Expand Up @@ -153,13 +147,13 @@ Common Errors

.. code-block:: none

File "/llvm-project/libc/newhdrgen/header.py", line 60, in __str__ for
File "/llvm-project/libc/hdrgen/header.py", line 60, in __str__ for
function in self.functions: AttributeError: 'HeaderFile' object has no
attribute 'functions'

When running ``ninja libc`` in the build directory to generate headers you
may receive the error above. Essentially this means that in
``libc/newhdrgen/header.py`` there is a missing attribute named functions.
``libc/hdrgen/header.py`` there is a missing attribute named functions.
Make sure all function components are defined within this file and there are
no missing functions to add these components.

Expand Down Expand Up @@ -190,12 +184,12 @@ Common Errors
Sometimes the integration test will fail but that
still means the process is working unless the comparison between the output
and expected_output is not showing. If that is the case make sure in
``libc/newhdrgen/tests/test_integration.py`` there are no missing arguments
``libc/hdrgen/tests/test_integration.py`` there are no missing arguments
that run through the script.

If the integration tests are failing due to mismatching of lines or small
errors in spacing that is nothing to worry about. If this is happening while
you are making a new change to the formatting of the headers, then
ensure the expected output file
``libc/newhdrgen/tests/expected_output/test_header.h`` has the changes you
``libc/hdrgen/tests/expected_output/test_header.h`` has the changes you
are applying.
6 changes: 3 additions & 3 deletions libc/docs/dev/source_tree_layout.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ directories::
- examples
- fuzzing
- hdr
- hdrgen
- include
- lib
- newhdrgen
- src
- startup
- test
Expand Down Expand Up @@ -89,13 +89,13 @@ The ``lib`` directory
This directory contains a ``CMakeLists.txt`` file listing the targets for the
public libraries ``libc.a``, ``libm.a`` etc.

The ``newhdrgen`` directory
The ``hdrgen`` directory
---------------------------

This directory contains the sources and specifications for the types, macros
and entrypoint functions. These definitions are organized in the ``yaml``
subdirectory and match the organization of the ``*.h.def`` files. This folder
also contains the python sources for new headergen, which is what generates the
also contains the python sources for headergen, which is what generates the
headers.

The ``src`` directory
Expand Down
17 changes: 17 additions & 0 deletions libc/hdrgen/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
if(LLVM_LIBC_FULL_BUILD)
enable_testing()

set(NEWHDGEN_TESTS_DIR ${CMAKE_CURRENT_SOURCE_DIR}/tests)
set(TEST_OUTPUT_DIR ${CMAKE_BINARY_DIR}/hdrgen/output)

add_test(
NAME hdrgen_integration_test
COMMAND python3 ${NEWHDGEN_TESTS_DIR}/test_integration.py --output_dir ${TEST_OUTPUT_DIR}
)

add_custom_target(check-hdrgen
COMMAND ${CMAKE_CTEST_COMMAND} -R hdrgen_integration_test --output-on-failure
)

message(STATUS "Integration test for hdrgen added.")
endif()
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
class TestHeaderGenIntegration(unittest.TestCase):
def setUp(self):
self.output_dir = Path(
args.output_dir if args.output_dir else "libc/newhdrgen/tests/output"
args.output_dir if args.output_dir else "libc/hdrgen/tests/output"
)

self.maxDiff = None
Expand All @@ -21,7 +21,7 @@ def run_script(self, yaml_file, h_def_file, output_dir, entry_points):
h_def_file = self.source_dir / h_def_file
command = [
"python3",
str(self.source_dir / "libc/newhdrgen/yaml_to_classes.py"),
str(self.source_dir / "libc/hdrgen/yaml_to_classes.py"),
str(yaml_file),
"--h_def_file",
str(h_def_file),
Expand Down Expand Up @@ -51,10 +51,10 @@ def compare_files(self, generated_file, expected_file):
self.assertEqual(gen_content, exp_content)

def test_generate_header(self):
yaml_file = "libc/newhdrgen/tests/input/test_small.yaml"
h_def_file = "libc/newhdrgen/tests/input/test_small.h.def"
yaml_file = "libc/hdrgen/tests/input/test_small.yaml"
h_def_file = "libc/hdrgen/tests/input/test_small.h.def"
expected_output_file = (
self.source_dir / "libc/newhdrgen/tests/expected_output/test_header.h"
self.source_dir / "libc/hdrgen/tests/expected_output/test_header.h"
)
output_file = self.output_dir / "test_small.h"
entry_points = {"func_b", "func_a", "func_c", "func_d", "func_e"}
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loading
Loading