[llvm] Add a tool to check mustache compliance against the public spec #142813
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This stack of pull requests is managed by Graphite. Learn more about stacking. |
|
@llvm/pr-subscribers-llvm-binary-utilities Author: Paul Kirth (ilovepi) ChangesThis is a cli tool to that tests the conformance of LLVM's mustache Co-authored-by: Peter Chou <[email protected]> Full diff: https://github.com/llvm/llvm-project/pull/142813.diff 6 Files Affected:
diff --git a/llvm/CMakeLists.txt b/llvm/CMakeLists.txt
index ed44b16bf9aeb..cb5c9f10e419d 100644
--- a/llvm/CMakeLists.txt
+++ b/llvm/CMakeLists.txt
@@ -1302,6 +1302,7 @@ if( LLVM_INCLUDE_UTILS )
add_subdirectory(utils/yaml-bench)
add_subdirectory(utils/split-file)
add_subdirectory(utils/mlgo-utils)
+ add_subdirectory(utils/llvm-mustachespec)
if( LLVM_INCLUDE_TESTS )
set(LLVM_SUBPROJECT_TITLE "Third-Party/Google Test")
add_subdirectory(${LLVM_THIRD_PARTY_DIR}/unittest ${CMAKE_CURRENT_BINARY_DIR}/third-party/unittest)
diff --git a/llvm/docs/CommandGuide/index.rst b/llvm/docs/CommandGuide/index.rst
index 643951eca2a26..ae8fff1574ad0 100644
--- a/llvm/docs/CommandGuide/index.rst
+++ b/llvm/docs/CommandGuide/index.rst
@@ -87,6 +87,7 @@ Developer Tools
llvm-exegesis
llvm-ifs
llvm-locstats
+ llvm-mustachespec
llvm-pdbutil
llvm-profgen
llvm-tli-checker
diff --git a/llvm/docs/CommandGuide/llvm-mustachespec.rst b/llvm/docs/CommandGuide/llvm-mustachespec.rst
new file mode 100644
index 0000000000000..498928c12e4f2
--- /dev/null
+++ b/llvm/docs/CommandGuide/llvm-mustachespec.rst
@@ -0,0 +1,13 @@
+llvm-mustachespec - LLVM tool to test Mustache Compliance Library
+=================================================================
+
+llvm-mustachespec test the mustache spec conformance of the LLVM
+mustache library. The spec can be found here https://github.com/mustache/spec
+
+ $ llvm-mustachespec input-file
+
+.. program:: llvm-mustachespec
+
+Outputs the number of tests failures and success in the spec
+
+
diff --git a/llvm/utils/llvm-mustachespec/CMakeLists.txt b/llvm/utils/llvm-mustachespec/CMakeLists.txt
new file mode 100644
index 0000000000000..2a94eda32c9bb
--- /dev/null
+++ b/llvm/utils/llvm-mustachespec/CMakeLists.txt
@@ -0,0 +1,5 @@
+add_llvm_utility(llvm-mustachespec
+ llvm-mustachespec.cpp
+ )
+
+target_link_libraries(llvm-mustachespec PRIVATE LLVMSupport)
diff --git "a/llvm/utils/llvm-mustachespec/\\" "b/llvm/utils/llvm-mustachespec/\\"
new file mode 100644
index 0000000000000..2161410e06428
--- /dev/null
+++ "b/llvm/utils/llvm-mustachespec/\\"
@@ -0,0 +1,108 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Simple drivers to test the mustache spec found here
+// https://github.com/mustache/
+// It is used to verify that the current implementation conforms to the spec
+// Simply download the spec and pass the test files to the driver
+//
+// Currently Triple Mustache is not supported, so we expect the following spec
+// test to fail:
+// Triple Mustache
+// Triple Mustache Integer Interpolation
+// Triple Mustache Decimal Interpolation
+// Triple Mustache Null Interpolation
+// Triple Mustache Context Miss Interpolation
+// Dotted Names - Triple Mustache Interpolation
+// Implicit Iterators - Triple Mustache
+// Triple Mustache - Surrounding Whitespace
+// Triple Mustache - Standalone
+// Triple Mustache With Padding
+// Standalone Indentation
+// Implicit Iterator - Triple mustache
+//
+// Usage:
+// mustache path/to/test/file/test.json path/to/test/file/test2.json ...
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Mustache.h"
+#include "llvm/Support/WithColor.h"
+#include <string>
+
+using namespace llvm;
+using namespace llvm::json;
+using namespace llvm::mustache;
+
+cl::list<std::string> InputFiles(cl::Positional, cl::desc("<input files>"),
+ cl::OneOrMore);
+
+static ExitOnError ExitOnErr;
+
+void runThroughTest(StringRef InputFile) {
+ outs() << "Running Tests: " << InputFile << "\n";
+ std::unique_ptr<MemoryBuffer> Buffer =
+ ExitOnErr(errorOrToExpected(MemoryBuffer::getFile(InputFile)));
+
+ StringRef FileContent = Buffer->getBuffer();
+ json::Value Json = ExitOnErr(parse(FileContent));
+
+ // Get test
+ Array *Obj = Json.getAsObject()->getArray("tests");
+ size_t Total = 0;
+ size_t Success = 0;
+ for (Value V : *Obj) {
+ Object *TestCase = V.getAsObject();
+ StringRef TemplateStr = TestCase->getString("template").value();
+ StringRef ExpectedStr = TestCase->getString("expected").value();
+ StringRef Name = TestCase->getString("name").value();
+ Value *Data = TestCase->get("data");
+ Value *Partials = TestCase->get("partials");
+
+ if (!Data)
+ continue;
+
+ Template T = Template(TemplateStr);
+ if (Partials) {
+ for (auto &PartialPairs : *Partials->getAsObject()) {
+ const auto &[Partial, Str] = PartialPairs;
+ T.registerPartial(Str.getAsString()->str(), Partial.str());
+ }
+ }
+
+ std::string ActualStr;
+ raw_string_ostream OS(ActualStr);
+ T.render(*Data, OS);
+ if (ExpectedStr == ActualStr) {
+ Success++;
+ } else {
+ llvm::errs() << "Template: " << TemplateStr <<"\n";
+ if (Partials)
+ Partials->print(llvm::errs());
+ llvm::errs() << "JSON Data: "; Data->print(errs()); errs() << "\n";
+ outs() << "Test Failed: " << Name << "\n"
+ << " Expected: \'" << ExpectedStr << "\'\n"
+ << " Actual: \'" << ActualStr << "\'\n";
+ llvm::errs() << "==========\n";
+ }
+ Total++;
+ }
+
+ outs() << "Result " << Success << "/" << Total << " succeeded\n";
+}
+
+int main(int argc, char **argv) {
+ ExitOnErr.setBanner(std::string(argv[0]) + " error:");
+ cl::ParseCommandLineOptions(argc, argv);
+ for (const auto &FileName : InputFiles) {
+ runThroughTest(FileName);
+ }
+ return 0;
+}
diff --git a/llvm/utils/llvm-mustachespec/llvm-mustachespec.cpp b/llvm/utils/llvm-mustachespec/llvm-mustachespec.cpp
new file mode 100644
index 0000000000000..acd090a6d3abe
--- /dev/null
+++ b/llvm/utils/llvm-mustachespec/llvm-mustachespec.cpp
@@ -0,0 +1,157 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Simple drivers to test the mustache spec found at
+// https://github.com/mustache/
+//
+// It is used to verify that the current implementation conforms to the spec.
+// Simply download the spec and pass the test files to the driver.
+//
+// The current implementation only supports non-optional parts of the spec, so
+// we do not expect any of the dynamic-names, inheritance, or lambda tests to
+// pass. Additionally, Triple Mustache is not supported, so we expect the
+// following tests to fail:
+// Triple Mustache
+// Triple Mustache Integer Interpolation
+// Triple Mustache Decimal Interpolation
+// Triple Mustache Null Interpolation
+// Triple Mustache Context Miss Interpolation
+// Dotted Names - Triple Mustache Interpolation
+// Implicit Iterators - Triple Mustache
+// Triple Mustache - Surrounding Whitespace
+// Triple Mustache - Standalone
+// Triple Mustache With Padding
+// Standalone Indentation
+// Implicit Iterator - Triple mustache
+//
+// Usage:
+// llvm-mustachespec path/to/test/file.json path/to/test/file2.json ...
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Mustache.h"
+#include <string>
+
+using namespace llvm;
+using namespace llvm::json;
+using namespace llvm::mustache;
+
+#define DEBUG_TYPE "llvm-mustachespec"
+
+static cl::OptionCategory Cat("llvm-mustachespec Options");
+
+cl::list<std::string> InputFiles(cl::Positional, cl::desc("<input files>"),
+ cl::OneOrMore);
+
+cl::opt<bool> ReportErrors("report-errors",
+ cl::desc("Report errors in spec tests"),
+ cl::cat(Cat));
+
+static ExitOnError ExitOnErr;
+
+struct TestData {
+ static Expected<TestData> createTestData(json::Object *TestCase,
+ StringRef InputFile) {
+ // If any of the needed elements are missing, we cannot continue.
+ // NOTE: partials are optional in the test schema.
+ if (!TestCase || !TestCase->getString("template") ||
+ !TestCase->getString("expected") || !TestCase->getString("name") ||
+ !TestCase->get("data"))
+ return createStringError(
+ llvm::inconvertibleErrorCode(),
+ "invalid JSON schema in test file: " + InputFile + "\n");
+
+ return TestData{TestCase->getString("template").value(),
+ TestCase->getString("expected").value(),
+ TestCase->getString("name").value(), TestCase->get("data"),
+ TestCase->get("partials")};
+ }
+
+ TestData() = default;
+
+ StringRef TemplateStr;
+ StringRef ExpectedStr;
+ StringRef Name;
+ Value *Data;
+ Value *Partials;
+};
+
+static void reportTestFailure(const TestData &TD, StringRef ActualStr) {
+ LLVM_DEBUG(dbgs() << "Template: " << TD.TemplateStr << "\n");
+ if (TD.Partials) {
+ LLVM_DEBUG(dbgs() << "Partial: ");
+ LLVM_DEBUG(TD.Partials->print(dbgs()));
+ LLVM_DEBUG(dbgs() << "\n");
+ }
+ LLVM_DEBUG(dbgs() << "JSON Data: ");
+ LLVM_DEBUG(TD.Data->print(dbgs()));
+ LLVM_DEBUG(dbgs() << "\n");
+ outs() << "Test Failed: " << TD.Name << "\n";
+ if (ReportErrors) {
+ outs() << " Expected: \'" << TD.ExpectedStr << "\'\n"
+ << " Actual: \'" << ActualStr << "\'\n"
+ << " ====================\n";
+ }
+}
+
+static void registerPartials(Value *Partials, Template &T) {
+ if (!Partials)
+ return;
+ for (const auto &[Partial, Str] : *Partials->getAsObject())
+ T.registerPartial(Partial.str(), Str.getAsString()->str());
+}
+
+static json::Value readJsonFromFile(StringRef &InputFile) {
+ std::unique_ptr<MemoryBuffer> Buffer =
+ ExitOnErr(errorOrToExpected(MemoryBuffer::getFile(InputFile)));
+ return ExitOnErr(parse(Buffer->getBuffer()));
+}
+
+static void runTest(StringRef InputFile) {
+ outs() << "Running Tests: " << InputFile << "\n";
+ json::Value Json = readJsonFromFile(InputFile);
+
+ json::Object *Obj = Json.getAsObject();
+ Array *TestArray = Obj->getArray("tests");
+ // Even though we parsed the JSON, it can have a bad format, so check it.
+ if (!TestArray)
+ ExitOnErr(createStringError(
+ llvm::inconvertibleErrorCode(),
+ "invalid JSON schema in test file: " + InputFile + "\n"));
+
+ const size_t Total = TestArray->size();
+ size_t Success = 0;
+
+ for (Value V : *TestArray) {
+ auto TestData =
+ ExitOnErr(TestData::createTestData(V.getAsObject(), InputFile));
+ Template T(TestData.TemplateStr);
+ registerPartials(TestData.Partials, T);
+
+ std::string ActualStr;
+ raw_string_ostream OS(ActualStr);
+ T.render(*TestData.Data, OS);
+ if (TestData.ExpectedStr == ActualStr)
+ ++Success;
+ else
+ reportTestFailure(TestData, ActualStr);
+ }
+
+ outs() << "Result [" << Success << "/" << Total << "] succeeded\n";
+}
+
+int main(int argc, char **argv) {
+ ExitOnErr.setBanner(std::string(argv[0]) + " error: ");
+ cl::ParseCommandLineOptions(argc, argv);
+ for (const auto &FileName : InputFiles)
+ runTest(FileName);
+ return 0;
+}
|
nikic
reviewed
Jun 5, 2025
ilovepi
force-pushed
the
users/ilovepi/mustache-spec-tool
branch
2 times, most recently
from
b05caf5 to
5469d49
Compare
ilovepi
force-pushed
the
users/ilovepi/mustache-spec-tool
branch
2 times, most recently
from
eec5619 to
22e099f
Compare
nikic
approved these changes
Jun 7, 2025
ilovepi
force-pushed
the
users/ilovepi/mustache-spec-tool
branch
from
22e099f to
dab7e62
Compare
This is a cli tool to that tests the conformance of LLVM's mustache implementation against the public Mustache spec, hosted at https://github.com/mustache/spec. This is a revised version of the patches in #111487. Co-authored-by: Peter Chou <[email protected]>
ilovepi
force-pushed
the
users/ilovepi/mustache-spec-tool
branch
from
dab7e62 to
7c18f69
Compare
|
@nikic I realized the doc file wasn't following the format we use for most other tools, so I made it match more closely. I wouldn't have noticed, but the |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/10/builds/7148 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/108/builds/13950 Here is the relevant piece of the build log for the reference |
|
Fix for c++20 in #143801, though I'm not aware that we support c++20 in tree at all yet. The other bot failure is unrelated. |
tomtor
pushed a commit
to tomtor/llvm-project
that referenced
this pull request
Jun 14, 2025
llvm#142813) This is a cli tool to that tests the conformance of LLVM's mustache implementation against the public Mustache spec, hosted at https://github.com/mustache/spec. This is a revised version of the patches in llvm#111487. Co-authored-by: Peter Chou <[email protected]>
akuhlens
pushed a commit
to akuhlens/llvm-project
that referenced
this pull request
Jun 24, 2025
llvm#142813) This is a cli tool to that tests the conformance of LLVM's mustache implementation against the public Mustache spec, hosted at https://github.com/mustache/spec. This is a revised version of the patches in llvm#111487. Co-authored-by: Peter Chou <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a cli tool to that tests the conformance of LLVM's mustache
implementation against the public Mustache spec, hosted at
https://github.com/mustache/spec. This is a revised version of the
patches in #111487.
Co-authored-by: Peter Chou [email protected]