Skip to content

TBDGen: Introduce option to emit API descriptor as supplementary output #68994

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Oct 5, 2023
Merged

TBDGen: Introduce option to emit API descriptor as supplementary output #68994

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e1f2e25
Frontend: Introduce -emit-api-descriptor flag.
tshortli Oct 5, 2023
e91580b
TBDGen: Teach APIGenRecorder to emit API descriptors during -emit-mod…
tshortli Oct 5, 2023
778532d
TBDGen: Don't emit pretty-printed API descriptor JSON by default.
tshortli Oct 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion include/swift/AST/DiagnosticsFrontend.def
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,9 @@ ERROR(error_mode_cannot_emit_module_summary,none,
ERROR(error_mode_cannot_emit_symbol_graph,none,
"this mode does not support emitting symbol graph files", ())
ERROR(error_mode_cannot_emit_abi_descriptor,none,
"this mode does not support emitting ABI descriptor", ())
"this mode does not support emitting ABI descriptor files", ())
ERROR(error_mode_cannot_emit_api_descriptor,none,
"this mode does not support emitting API descriptor files", ())
ERROR(error_mode_cannot_emit_const_values,none,
"this mode does not support emitting extracted const values", ())
ERROR(error_mode_cannot_emit_module_semantic_info,none,
Expand Down Expand Up @@ -313,6 +315,10 @@ ERROR(tbd_not_supported_with_cmo,none,
"Test-Based InstallAPI (TBD) is not support with cross-module-optimization",
())

WARNING(api_descriptor_only_supported_in_whole_module,none,
"API descriptor generation is only supported when the whole module can be seen",
())

ERROR(previous_installname_map_missing,none,
"cannot open previous install name map from %0",
(StringRef))
Expand Down
1 change: 1 addition & 0 deletions include/swift/Basic/FileTypes.def
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ TYPE("pch", PCH, "pch", "")
TYPE("none", Nothing, "", "")

TYPE("abi-baseline-json", SwiftABIDescriptor, "abi.json", "")
TYPE("api-json", SwiftAPIDescriptor, "", "")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The extension right now for partial SDKDB files is .sdkdb . Should this be listed in the extension field?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could not identify a reason it was important to specify an extension here. There are some utilities in the compiler to take a filename and look up its corresponding file_types entry using the contents of this table, but AFAIK, we don't need to do that with an API descriptor since the path to one is always specified explicitly in the arguments to the frontend. We could certainly add .sdkdb here but it feels slightly over-specified to me since ultimately this output is just JSON to the compiler, it doesn't otherwise know about SDKDB as a concept.

@artemcm @xymus @nkcsgexi Am I missing any reasons this needs to be specified?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I don't remember any specific reasons for being explicit about file extensions other than inferring output kinds. However, it seems to be good documentation, no?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I named it originally partial.sdkdb for the intermediate JSON format. Up for a better name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think given what I've heard that we do not need to encode the suffix of this file in the Swift compiler. From the compiler's point of view it's just another type of supplementary file that it knows how to write to a given path when requested, but the filename can be opaque to the compiler.

TYPE("fixit", SwiftFixIt, "", "")
TYPE("module-semantic-info", ModuleSemanticInfo, "", "")
TYPE("cached-diagnostics", CachedDiagnostics, "", "")
Expand Down
3 changes: 3 additions & 0 deletions include/swift/Basic/SupplementaryOutputPaths.def
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,9 @@ OUTPUT(ModuleSummaryOutputPath, TY_SwiftModuleSummaryFile)
/// The output path to generate ABI baseline.
OUTPUT(ABIDescriptorOutputPath, TY_SwiftABIDescriptor)

/// The output path to the module's API description.
OUTPUT(APIDescriptorOutputPath, TY_SwiftAPIDescriptor)

/// The output path for extracted compile-time-known value information
OUTPUT(ConstValuesOutputPath, TY_ConstValues)

Expand Down
5 changes: 5 additions & 0 deletions include/swift/Frontend/Frontend.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -428,6 +428,11 @@ class CompilerInvocation {
std::string getModuleInterfaceOutputPathForWholeModule() const;
std::string getPrivateModuleInterfaceOutputPathForWholeModule() const;

/// APIDescriptorPath only makes sense in whole module compilation mode,
/// so return the APIDescriptorPath when in that mode and fail an assert
/// if not in that mode.
std::string getAPIDescriptorPathForWholeModule() const;

public:
/// Given the current configuration of this frontend invocation, a set of
/// supplementary output paths, and a module, compute the appropriate set of
Expand Down
1 change: 1 addition & 0 deletions include/swift/Frontend/FrontendInputsAndOutputs.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,7 @@ class FrontendInputsAndOutputs {
bool hasModuleInterfaceOutputPath() const;
bool hasPrivateModuleInterfaceOutputPath() const;
bool hasABIDescriptorOutputPath() const;
bool hasAPIDescriptorOutputPath() const;
bool hasConstValuesOutputPath() const;
bool hasModuleSemanticInfoOutputPath() const;
bool hasModuleSummaryOutputPath() const;
Expand Down
1 change: 1 addition & 0 deletions include/swift/Frontend/FrontendOptions.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -569,6 +569,7 @@ class FrontendOptions {
static bool canActionEmitModuleSummary(ActionType);
static bool canActionEmitInterface(ActionType);
static bool canActionEmitABIDescriptor(ActionType);
static bool canActionEmitAPIDescriptor(ActionType);
static bool canActionEmitConstValues(ActionType);
static bool canActionEmitModuleSemanticInfo(ActionType);

Expand Down
14 changes: 13 additions & 1 deletion include/swift/Option/Options.td
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -626,7 +626,19 @@ def emit_const_values :
def emit_const_values_path : Separate<["-"], "emit-const-values-path">,
Flags<[FrontendOption, NoInteractiveOption, ArgumentIsPath,
SupplementaryOutput, CacheInvariant]>,
MetaVarName<"<path>">, HelpText<"Emit the extracted compile-time known values to <path>">;
MetaVarName<"<path>">,
HelpText<"Emit the extracted compile-time known values to <path>">;

def emit_api_descriptor :
Flag<["-"], "emit-api-descriptor">,
Flags<[NoInteractiveOption, SupplementaryOutput, CacheInvariant]>,
HelpText<"Output a JSON file describing the module's API">;
def emit_api_descriptor_path :
Separate<["-"], "emit-api-descriptor-path">,
Flags<[FrontendOption, NoInteractiveOption, ArgumentIsPath,
SupplementaryOutput, CacheInvariant]>,
MetaVarName<"<path>">,
HelpText<"Output a JSON file describing the module's API to <path>">;

def emit_objc_header : Flag<["-"], "emit-objc-header">,
Flags<[FrontendOption, NoInteractiveOption, SupplementaryOutput, CacheInvariant]>,
Expand Down
3 changes: 3 additions & 0 deletions lib/Basic/FileTypes.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ bool file_types::isTextual(ID Id) {
case file_types::TY_JSONDependencies:
case file_types::TY_JSONFeatures:
case file_types::TY_SwiftABIDescriptor:
case file_types::TY_SwiftAPIDescriptor:
case file_types::TY_ConstValues:
return true;
case file_types::TY_Image:
Expand Down Expand Up @@ -164,6 +165,7 @@ bool file_types::isAfterLLVM(ID Id) {
case file_types::TY_JSONFeatures:
case file_types::TY_IndexUnitOutputPath:
case file_types::TY_SwiftABIDescriptor:
case file_types::TY_SwiftAPIDescriptor:
case file_types::TY_ConstValues:
case file_types::TY_SwiftFixIt:
case file_types::TY_ModuleSemanticInfo:
Expand Down Expand Up @@ -220,6 +222,7 @@ bool file_types::isPartOfSwiftCompilation(ID Id) {
case file_types::TY_JSONFeatures:
case file_types::TY_IndexUnitOutputPath:
case file_types::TY_SwiftABIDescriptor:
case file_types::TY_SwiftAPIDescriptor:
case file_types::TY_ConstValues:
case file_types::TY_SwiftFixIt:
case file_types::TY_ModuleSemanticInfo:
Expand Down
1 change: 1 addition & 0 deletions lib/Driver/Driver.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2090,6 +2090,7 @@ void Driver::buildActions(SmallVectorImpl<const Action *> &TopLevelActions,
case file_types::TY_JSONDependencies:
case file_types::TY_JSONFeatures:
case file_types::TY_SwiftABIDescriptor:
case file_types::TY_SwiftAPIDescriptor:
case file_types::TY_ConstValues:
case file_types::TY_SwiftFixIt:
case file_types::TY_ModuleSemanticInfo:
Expand Down
2 changes: 2 additions & 0 deletions lib/Driver/ToolChains.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -733,6 +733,7 @@ const char *ToolChain::JobContext::computeFrontendModeForCompile() const {
case file_types::TY_SwiftOverlayFile:
case file_types::TY_IndexUnitOutputPath:
case file_types::TY_SwiftABIDescriptor:
case file_types::TY_SwiftAPIDescriptor:
case file_types::TY_ConstValues:
case file_types::TY_SwiftFixIt:
case file_types::TY_ModuleSemanticInfo:
Expand Down Expand Up @@ -997,6 +998,7 @@ ToolChain::constructInvocation(const BackendJobAction &job,
case file_types::TY_SwiftOverlayFile:
case file_types::TY_IndexUnitOutputPath:
case file_types::TY_SwiftABIDescriptor:
case file_types::TY_SwiftAPIDescriptor:
case file_types::TY_ConstValues:
case file_types::TY_SwiftFixIt:
case file_types::TY_ModuleSemanticInfo:
Expand Down
5 changes: 5 additions & 0 deletions lib/Frontend/ArgsToFrontendOptionsConverter.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -765,6 +765,11 @@ bool ArgsToFrontendOptionsConverter::checkUnusedSupplementaryOutputPaths()
Diags.diagnose(SourceLoc(), diag::error_mode_cannot_emit_abi_descriptor);
return true;
}
if (!FrontendOptions::canActionEmitAPIDescriptor(Opts.RequestedAction) &&
Opts.InputsAndOutputs.hasAPIDescriptorOutputPath()) {
Diags.diagnose(SourceLoc(), diag::error_mode_cannot_emit_api_descriptor);
return true;
}
if (!FrontendOptions::canActionEmitConstValues(Opts.RequestedAction) &&
Opts.InputsAndOutputs.hasConstValuesOutputPath()) {
Diags.diagnose(SourceLoc(), diag::error_mode_cannot_emit_const_values);
Expand Down
11 changes: 10 additions & 1 deletion lib/Frontend/ArgsToFrontendOutputsConverter.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -331,6 +331,8 @@ SupplementaryOutputPathsComputer::getSupplementaryOutputPathsFromArguments()
options::OPT_emit_module_summary_path);
auto abiDescriptorOutput = getSupplementaryFilenamesFromArguments(
options::OPT_emit_abi_descriptor_path);
auto apiDescriptorOutput = getSupplementaryFilenamesFromArguments(
options::OPT_emit_api_descriptor_path);
auto constValuesOutput = getSupplementaryFilenamesFromArguments(
options::OPT_emit_const_values_path);
auto moduleSemanticInfoOutput = getSupplementaryFilenamesFromArguments(
Expand Down Expand Up @@ -365,6 +367,7 @@ SupplementaryOutputPathsComputer::getSupplementaryOutputPathsFromArguments()
sop.ModuleSourceInfoOutputPath = (*moduleSourceInfoOutput)[i];
sop.ModuleSummaryOutputPath = (*moduleSummaryOutput)[i];
sop.ABIDescriptorOutputPath = (*abiDescriptorOutput)[i];
sop.APIDescriptorOutputPath = (*apiDescriptorOutput)[i];
sop.ConstValuesOutputPath = (*constValuesOutput)[i];
sop.ModuleSemanticInfoOutputPath = (*moduleSemanticInfoOutput)[i];
sop.YAMLOptRecordPath = (*optRecordOutput)[i];
Expand Down Expand Up @@ -475,8 +478,13 @@ SupplementaryOutputPathsComputer::computeOutputPathsForOneInput(

// There is no non-path form of -emit-abi-descriptor-path
auto ABIDescriptorOutputPath = pathsFromArguments.ABIDescriptorOutputPath;

// There is no non-path form of -emit-api-descriptor-path
auto APIDescriptorOutputPath = pathsFromArguments.APIDescriptorOutputPath;

// There is no non-path form of -emit-module-semantic-info-path
auto ModuleSemanticInfoOutputPath = pathsFromArguments.ModuleSemanticInfoOutputPath;
auto ModuleSemanticInfoOutputPath =
pathsFromArguments.ModuleSemanticInfoOutputPath;

ID emitModuleOption;
std::string moduleExtension;
Expand Down Expand Up @@ -513,6 +521,7 @@ SupplementaryOutputPathsComputer::computeOutputPathsForOneInput(
sop.ModuleSourceInfoOutputPath = moduleSourceInfoOutputPath;
sop.ModuleSummaryOutputPath = moduleSummaryOutputPath;
sop.ABIDescriptorOutputPath = ABIDescriptorOutputPath;
sop.APIDescriptorOutputPath = APIDescriptorOutputPath;
sop.ConstValuesOutputPath = constValuesOutputPath;
sop.ModuleSemanticInfoOutputPath = ModuleSemanticInfoOutputPath;
sop.YAMLOptRecordPath = YAMLOptRecordPath;
Expand Down
8 changes: 8 additions & 0 deletions lib/Frontend/Frontend.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,14 @@ CompilerInvocation::getPrivateModuleInterfaceOutputPathForWholeModule() const {
.SupplementaryOutputs.PrivateModuleInterfaceOutputPath;
}

std::string CompilerInvocation::getAPIDescriptorPathForWholeModule() const {
assert(
getFrontendOptions().InputsAndOutputs.isWholeModule() &&
"APIDescriptorPath only makes sense when the whole module can be seen");
return getPrimarySpecificPathsForAtMostOnePrimary()
.SupplementaryOutputs.APIDescriptorOutputPath;
}

SerializationOptions CompilerInvocation::computeSerializationOptions(
const SupplementaryOutputPaths &outs, const ModuleDecl *module) const {
const FrontendOptions &opts = getFrontendOptions();
Expand Down
6 changes: 6 additions & 0 deletions lib/Frontend/FrontendInputsAndOutputs.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -511,6 +511,12 @@ bool FrontendInputsAndOutputs::hasABIDescriptorOutputPath() const {
return outs.ABIDescriptorOutputPath;
});
}
bool FrontendInputsAndOutputs::hasAPIDescriptorOutputPath() const {
return hasSupplementaryOutputPath(
[](const SupplementaryOutputPaths &outs) -> const std::string & {
return outs.APIDescriptorOutputPath;
});
}
bool FrontendInputsAndOutputs::hasConstValuesOutputPath() const {
return hasSupplementaryOutputPath(
[](const SupplementaryOutputPaths &outs) -> const std::string & {
Expand Down
42 changes: 42 additions & 0 deletions lib/Frontend/FrontendOptions.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -743,6 +743,48 @@ bool FrontendOptions::canActionEmitInterface(ActionType action) {
llvm_unreachable("unhandled action");
}

bool FrontendOptions::canActionEmitAPIDescriptor(ActionType action) {
switch (action) {
case ActionType::NoneAction:
case ActionType::Parse:
case ActionType::DumpParse:
case ActionType::DumpInterfaceHash:
case ActionType::DumpAST:
case ActionType::PrintAST:
case ActionType::PrintASTDecl:
case ActionType::EmitImportedModules:
case ActionType::EmitPCH:
case ActionType::DumpScopeMaps:
case ActionType::DumpTypeRefinementContexts:
case ActionType::DumpTypeInfo:
case ActionType::EmitSILGen:
case ActionType::EmitSIBGen:
case ActionType::CompileModuleFromInterface:
case ActionType::TypecheckModuleFromInterface:
case ActionType::Immediate:
case ActionType::REPL:
case ActionType::EmitPCM:
case ActionType::DumpPCM:
case ActionType::ScanDependencies:
case ActionType::PrintFeature:
return false;
case ActionType::ResolveImports:
case ActionType::Typecheck:
case ActionType::MergeModules:
case ActionType::EmitModuleOnly:
case ActionType::EmitSIL:
case ActionType::EmitSIB:
case ActionType::EmitIRGen:
case ActionType::EmitIR:
case ActionType::EmitBC:
case ActionType::EmitAssembly:
case ActionType::EmitObject:
case ActionType::PrintVersion:
return true;
}
llvm_unreachable("unhandled action");
}

bool FrontendOptions::doesActionProduceOutput(ActionType action) {
switch (action) {
case ActionType::Parse:
Expand Down
32 changes: 32 additions & 0 deletions lib/FrontendTool/FrontendTool.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -809,6 +809,34 @@ static bool writeTBDIfNeeded(CompilerInstance &Instance) {
Instance.getOutputBackend(), tbdOpts);
}

static bool writeAPIDescriptor(ModuleDecl *M, StringRef OutputPath,
llvm::vfs::OutputBackend &Backend) {
return withOutputPath(M->getDiags(), Backend, OutputPath,
[&](raw_ostream &OS) -> bool {
writeAPIJSONFile(M, OS, /*PrettyPrinted=*/false);
return false;
});
}

static bool writeAPIDescriptorIfNeeded(CompilerInstance &Instance) {
const auto &Invocation = Instance.getInvocation();
const auto &frontendOpts = Invocation.getFrontendOptions();
if (!frontendOpts.InputsAndOutputs.hasAPIDescriptorOutputPath())
return false;

if (!frontendOpts.InputsAndOutputs.isWholeModule()) {
Instance.getDiags().diagnose(
SourceLoc(), diag::api_descriptor_only_supported_in_whole_module);
return false;
}

const std::string &APIDescriptorPath =
Invocation.getAPIDescriptorPathForWholeModule();

return writeAPIDescriptor(Instance.getMainModule(), APIDescriptorPath,
Instance.getOutputBackend());
}

static bool performCompileStepsPostSILGen(CompilerInstance &Instance,
std::unique_ptr<SILModule> SM,
ModuleOrSourceFile MSF,
Expand Down Expand Up @@ -975,6 +1003,10 @@ static bool emitAnyWholeModulePostTypeCheckSupplementaryOutputs(
hadAnyError |= writeTBDIfNeeded(Instance);
}

{
hadAnyError |= writeAPIDescriptorIfNeeded(Instance);
}

{
hadAnyError |= writeModuleSemanticInfoIfNeeded(Instance);
}
Expand Down
Loading