Merge pull request #28344 from apple/tensorflow-merge

Merging swift-DEVELOPMENT-SNAPSHOT-2019-11-11-a into tensorflow.

Author: dan-zheng

CHANGELOG.md

@@ -26,6 +26,39 @@ CHANGELOG
 Swift 5.2
 ---------
 
+* [SR-2790][]:
+
+  The compiler will now emit a warning when attempting to pass a temporary
+  pointer argument produced from an array, string, or inout argument to a
+  parameter which is known to escape it. This includes the various initializers 
+  for the `UnsafePointer`/`UnsafeBufferPointer` family of types, as well as
+  memberwise initializers.
+
+  ```swift
+  struct S {
+    var ptr: UnsafePointer<Int8>
+  }
+
+  func foo() {
+    var i: Int8 = 0
+    let ptr = UnsafePointer(&i)
+    // warning: initialization of 'UnsafePointer<Int8>' results in a 
+    // dangling pointer
+    
+    let s1 = S(ptr: [1, 2, 3]) 
+    // warning: passing '[Int8]' to parameter, but argument 'ptr' should be a
+    // pointer that outlives the call to 'init(ptr:)'
+    
+    let s2 = S(ptr: "hello")
+    // warning: passing 'String' to parameter, but argument 'ptr' should be a
+    // pointer that outlives the call to 'init(ptr:)'
+  }
+  ```
+
+  All 3 of the above examples are unsound because each argument produces a
+  temporary pointer only valid for the duration of the call they are passed to.
+  Therefore the returned value in each case references a dangling pointer.
+
 * [SR-2189][]:
 
   The compiler now supports local functions whose default arguments capture
@@ -7824,6 +7857,7 @@ Swift 1.0
 [SR-2608]: <https://bugs.swift.org/browse/SR-2608>
 [SR-2672]: <https://bugs.swift.org/browse/SR-2672>
 [SR-2688]: <https://bugs.swift.org/browse/SR-2688>
+[SR-2790]: <https://bugs.swift.org/browse/SR-2790>
 [SR-4206]: <https://bugs.swift.org/browse/SR-4206>
 [SR-4248]: <https://bugs.swift.org/browse/SR-4248>
 [SR-5581]: <https://bugs.swift.org/browse/SR-5581>

CMakeLists.txt

@@ -472,7 +472,7 @@ if(MSVC OR "${CMAKE_SIMULATE_ID}" STREQUAL MSVC)
   include(ClangClCompileRules)
 endif()
 
-if(CMAKE_C_COMPILER_ID STREQUAL Clang)
+if(CMAKE_C_COMPILER_ID MATCHES Clang)
   add_compile_options($<$<OR:$<COMPILE_LANGUAGE:C>,$<COMPILE_LANGUAGE:CXX>>:-Werror=gnu>)
 endif()
 
@@ -1151,6 +1151,8 @@ endif()
 
 add_subdirectory(utils)
 
+add_subdirectory(userdocs)
+
 if ("${CMAKE_SYSTEM_NAME}" STREQUAL "Darwin")
   if(SWIFT_BUILD_PERF_TESTSUITE)
     add_subdirectory(benchmark)

README.md

@@ -33,7 +33,7 @@ Please make sure you use Python 2.x. Python 3.x is not supported currently.
 
 #### macOS
 
-To build for macOS, you need [Xcode 11 beta 6](https://developer.apple.com/xcode/downloads/).
+To build for macOS, you need [Xcode 11.2](https://developer.apple.com/xcode/downloads/).
 The required version of Xcode changes frequently, and is often a beta release.
 Check this document for the current required version.
 

cmake/modules/AddSwift.cmake

@@ -2024,6 +2024,9 @@ function(add_swift_target_library name)
         DEPLOYMENT_VERSION_WATCHOS "${SWIFTLIB_DEPLOYMENT_VERSION_WATCHOS}"
         GYB_SOURCES ${SWIFTLIB_GYB_SOURCES}
       )
+    if(NOT SWIFT_BUILT_STANDALONE AND NOT "${CMAKE_C_COMPILER_ID}" MATCHES "Clang")
+      add_dependencies(${VARIANT_NAME} clang)
+    endif()
 
       if(sdk STREQUAL WINDOWS)
         if(SWIFT_COMPILER_IS_MSVC_LIKE)

docs/DebuggingTheCompiler.rst

@@ -214,6 +214,11 @@ debugging press <CTRL>-C on the LLDB prompt.
 Note that this only works in Xcode if the PATH variable in the scheme's
 environment setting contains the path to the dot tool.
 
+swift/Basic/Debug.h includes macros to help contributors declare these methods
+with the proper attributes to ensure they'll be available in the debugger. In
+particular, if you see ``SWIFT_DEBUG_DUMP`` in a class declaration, that class
+has a ``dump()`` method you can call.
+
 Debugging and Profiling on SIL level
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/Diagnostics.md

@@ -90,6 +90,25 @@ The Swift compiler has a setting (under LangOptions) called `DiagnosticsEditorMo
 
 Most diagnostics have no reason to change behavior under editor mode. An example of an exception is the "protocol requirements not satisfied diagnostic"; on the command line, it may be better to show all unsatisfied requirements, while in an IDE a single multi-line fix-it would be preferred.
 
+### Educational Notes ###
+
+**Note**: This feature is currently experimental. It can be enabled by passing the `-Xfrontend -enable-descriptive-diagnostics` flag.
+
+Educational notes are small snippets of documentation attached to a diagnostic which explain relevant language concepts. They are intended to further Swift's goal of progressive disclosure by providing a learning resource at the point of use for users encountering a new error message for the first time. In very limited circumstances, they also allow the main diagnostic message to use more precise and correct terminology (e.g. nominal types) which would otherwise be too unfriendly for beginners.
+
+When outputting diagnostics on the command line, educational notes will be printed after the main diagnostic body if descriptive diagnostics are enabled. When presented in an IDE, it's expected they will be collapsed under a disclosure arrow, info button, or similar to avoid cluttering output.
+
+Generally speaking, a diagnostic should try to provide educational notes for any concepts/terminology which is difficult to understand from context or is especially subtle. Educational notes should:
+- Explain a single language concept. This makes them easy to reuse across diagnostics and helps keep them clear, concise, and easy to understand.
+- Be written in unabbreviated English. These are longer form messages compared to the main diagnostic, so there is no need to omit needless words and punctuation.
+- Not generally exceed 3-4 paragraphs. Educational notes should be clear and easily digestible. Messages which are too long also have the potential to create diagnostics UX issues in some contexts.
+- Be accessible. Educational notes should be beginner friendly and avoid assuming unnecesary prior knowledge. The goal is not only to help users understand what a diagnostic is telling them, but also to turn errors and warnings into "teachable moments".
+- Include references to relevant chapters of _The Swift Programming Language_ if applicable.
+- Be written in Markdown, but avoid excessive markup to avoid impacting the terminal UX. 
+
+To add a new educational note:
+1. Add a new Markdown file in the `userdocs/diagnostics/` directory containing the contents of the note. 
+2. Associate the note with one or more diagnostics in EducationalNotes.def.
 
 ### Format Specifiers ###
 

docs/WindowsBuild.md

@@ -1,10 +1,14 @@
 # Building Swift on Windows
 
-Visual Studio 2017 or newer is needed to build swift on Windows.
+Visual Studio 2017 or newer is needed to build swift on Windows.  The following must take place in the developer command prompt (provided by Visual Studio).  This shows up as "x64 Native Tools Command Prompt for VS2017" (or VS2019, VS2019 Preview depending on the Visual Studio that you are using) in the Start Menu.
 
 ## Install dependencies
 - Install the latest version of [Visual Studio](https://www.visualstudio.com/downloads/)
-- Make sure to include "Programming Languages|Visual C++" and "Windows and Web Development|Universal Windows App Development|Windows SDK" in your installation.
+- Make sure to include "Programming Languages|Visual C++" and "Windows and Web Development|Universal Windows App Development|Windows SDK" in your installation.  The following components are required:
+
+1. Microsoft.VisualStudio.Component.Windows10SDK
+1. Microsoft.VisualStudio.Component.Windows10SDK.17763
+1. Microsoft.VisualStudio.Component.VC.Tools.x86.x64 
 
 ## Clone the repositories
 1. Clone `apple/llvm-project` into a directory for the toolchain
@@ -49,54 +53,29 @@ git clone https://github.com/compnerd/windows-swift windows-swift
       ┕ usr/...
 ```
 
-## Get ready
-- From within a **NATIVE developer** command prompt (not PowerShell nor cmd, but the [Visual Studio Developer Command Prompt](https://msdn.microsoft.com/en-us/library/f35ctcxw.aspx)), execute the following command if you have an x64 PC (The Native Developer command prompt is situated at "C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Visual Studio 2019\Visual Studio Tools\VC\x64 Native Tools Command Prompt for VS 2019.lnk").
-
-Run this as administrator the first time, for setting up the symlinks below. 
-
-If instead you're compiling for a 32-bit Windows target, adapt the `arch` argument to `x86` and run
-
-```cmd
-VsDevCmd -arch=x86
-```
-
-- Decide whether you want to build a release or debug version of Swift on Windows and 
-  replace the `CMAKE_BUILD_TYPE` parameter in the build steps below with the correct value 
-  (`Debug`, `RelWithDebInfoAssert` or `Release`) to avoid conflicts between the debug and 
-  non-debug version of the MSCRT library.
-
 ## One-time Setup (re-run on Visual Studio upgrades)
 - Set up the `ucrt`, `visualc`, and `WinSDK` modules by copying  `ucrt.modulemap` located at
   `swift/stdlib/public/Platform/ucrt.modulemap` into
   `${UniversalCRTSdkDir}/Include/${UCRTVersion}/ucrt` as `module.modulemap`, copying `visualc.modulemap` located at `swift/stdlib/public/Platform/visualc.modulemap` into `${VCToolsInstallDir}/include` as `module.modulemap`, and copying `winsdk.modulemap` located at `swift/stdlib/public/Platform/winsdk.modulemap` into `${UniversalCRTSdkDir}/Include/${UCRTVersion}/um` and setup the `visualc.apinotes` located at `swift/stdlib/public/Platform/visualc.apinotes` into `${VCToolsInstallDir}/include` as `visualc.apinotes`
 
 ```cmd
-mklink "%UniversalCRTSdkDir%\Include\%UCRTVersion%\ucrt\module.modulemap" S:\swift\stdlib\public\Platform\ucrt.modulemap
-mklink "%UniversalCRTSdkDir%\Include\%UCRTVersion%\um\module.modulemap" S:\swift\stdlib\public\Platform\winsdk.modulemap
-mklink "%VCToolsInstallDir%\include\module.modulemap" S:\swift\stdlib\public\Platform\visualc.modulemap
-mklink "%VCToolsInstallDir%\include\visualc.apinotes" S:\swift\stdlib\public\Platform\visualc.apinotes
+mklink "%UniversalCRTSdkDir%\Include\%UCRTVersion%\ucrt\module.modulemap" S:\toolchain\swift\stdlib\public\Platform\ucrt.modulemap
+mklink "%UniversalCRTSdkDir%\Include\%UCRTVersion%\um\module.modulemap" S:\toolchain\swift\stdlib\public\Platform\winsdk.modulemap
+mklink "%VCToolsInstallDir%\include\module.modulemap" S:\toolchain\swift\stdlib\public\Platform\visualc.modulemap
+mklink "%VCToolsInstallDir%\include\visualc.apinotes" S:\toolchain\swift\stdlib\public\Platform\visualc.apinotes
 ```
 
 Warning: Creating the above links usually requires administrator privileges. The quick and easy way to do this is to open a second developer prompt by right clicking whatever shortcut you used to open the first one, choosing Run As Administrator, and pasting the above commands into the resulting window. You can then close the privileged prompt; this is the only step which requires elevation.
 
 ## Build the toolchain
-- This must be done from within a developer command prompt. Make sure that the build type for LLVM/Clang is compatible with the build type for Swift. That is, either build everything `Debug` or some variant of `Release` (e.g. `Release`, `RelWithDebInfo`).
 
 ```cmd
 md "S:\b\toolchain"
-cmake -B "S:\b\toolchain" -G Ninja -S S:\toolchain\llvm -C S:\windows-swift\cmake\caches\Windows-x86_64.cmake -C S:\windows-swift\cmake\caches\org.compnerd.dt.cmake -DLLVM_ENABLE_ASSERTIONS=YES -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;cmark;swift;lldb;lld" -DLLVM_EXTERNAL_PROJECTS="cmark;swift" -DSWIFT_PATH_TO_LIBDISPATCH_SOURCE=S:\llvm-project\swift-corelibs-libdispatch -DLLVM_ENABLE_PDB=YES -DLLDB_DISABLE_PYTHON=YES -DSWIFT_WINDOWS_x86_64_ICU_UC_INCLUDE="S:/Library/icu-64/usr/include" -DSWIFT_WINDOWS_x86_64_ICU_UC="S:/Library/icu-64/usr/lib/icuuc64.lib" -DSWIFT_WINDOWS_x86_64_ICU_I18N_INCLUDE="S:/Library/icu-64/usr/include" -DSWIFT_WINDOWS_x86_64_ICU_I18N="S:/Library/icu-64/usr/lib/icuin64.lib" -DCMAKE_INSTALL_PREFIX="C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr" -DPYTHON_EXECUTABLE=C:\Python27\python.exe -DSWIFT_BUILD_DYNAMIC_STDLIB=YES -DSWIFT_BUILD_DYNAMIC_SDK_OVERLAY=YES
+cmake -B "S:\b\toolchain" -G Ninja -S S:\toolchain\llvm -C S:\windows-swift\cmake\caches\Windows-x86_64.cmake -C S:\windows-swift\cmake\caches\org.compnerd.dt.cmake -DLLVM_ENABLE_ASSERTIONS=YES -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;cmark;swift;lldb;lld" -DLLVM_EXTERNAL_PROJECTS="cmark;swift" -DSWIFT_PATH_TO_LIBDISPATCH_SOURCE=S:\toolchain\swift-corelibs-libdispatch -DLLVM_ENABLE_PDB=YES -DLLDB_DISABLE_PYTHON=YES -DSWIFT_WINDOWS_x86_64_ICU_UC_INCLUDE="S:/Library/icu-64/usr/include" -DSWIFT_WINDOWS_x86_64_ICU_UC="S:/Library/icu-64/usr/lib/icuuc64.lib" -DSWIFT_WINDOWS_x86_64_ICU_I18N_INCLUDE="S:/Library/icu-64/usr/include" -DSWIFT_WINDOWS_x86_64_ICU_I18N="S:/Library/icu-64/usr/lib/icuin64.lib" -DCMAKE_INSTALL_PREFIX="C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr" -DPYTHON_EXECUTABLE=C:\Python27\python.exe -DSWIFT_BUILD_DYNAMIC_STDLIB=YES -DSWIFT_BUILD_DYNAMIC_SDK_OVERLAY=YES
 ninja -C S:\b\toolchain
 ```
 
-- Update your path to include the toolchain.
-
-```cmd
-path S:\b\toolchain\bin;%PATH%
-```
-
-## Running tests on Windows
-
-Running the testsuite on Windows has additional external dependencies.
+## Running Swift tests on Windows
 
 ```cmd
 path S:\Library\icu-64\usr\bin;S:\b\toolchain\bin;S:\b\toolchain\libdispatch-prefix\bin;%PATH%;%ProgramFiles%\Git\usr\bin
@@ -106,15 +85,10 @@ ninja -C S:\b\toolchain check-swift
 ## Build swift-corelibs-libdispatch
 
 ```cmd
-cmake -B S:\b\libdispatch -G Ninja -S S:\toolchain\swift-corelibs-libdispatch -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_COMPILER=clang-cl -DCMAKE_CXX_COMPILER=clang-cl -DCMAKE_Swift_COMPILER=S:\b\toolchain\bin\swiftc.exe -DENABLE_SWIFT=YES
+cmake -B S:\b\libdispatch -G Ninja -S S:\toolchain\swift-corelibs-libdispatch -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_COMPILER=clang-cl -DCMAKE_CXX_COMPILER=clang-cl -DCMAKE_Swift_COMPILER=S:/b/toolchain/bin/swiftc.exe -DENABLE_SWIFT=YES
 ninja -C S:\b\libdispatch
 ```
 
-- Add libdispatch to your path:
-```cmd
-path S:\b\libdispatch;S:\b\libdispatch\src;%PATH%
-```
-
 ## Test swift-corelibs-libdispatch
 
 ```cmd
@@ -237,12 +211,12 @@ cd S:\b\spm
 C:\Python27\python.exe S:\swift-package-manager\Utilities\bootstrap --foundation S:\b\foundation --libdispatch-build-dir S:\b\libdispatch --libdispatch-source-dir S:\swift-corelibs-libdispatch --llbuild-build-dir S:\b\llbuild --llbuild-source-dir S:\llbuild --sqlite-build-dir S:\b\sqlite --sqlite-source-dir S:\sqlite-amalgamation-3270200
 ```
 
-## Install Swift on Windows
+## Install the Swift toolchain on Windows
 
 - Run ninja install:
 
 ```cmd 
-ninja -C S:\b\swift install
+ninja -C S:\b\toolchain install
 ```
 
 - Add the Swift on Windows binaries path (`C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin`)  to the `PATH` environment variable.

docs/conf.py

@@ -13,6 +13,7 @@
 
 import sys
 from datetime import date
+from sphinx.highlighting import lexers
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
@@ -263,9 +264,7 @@
 # Enable this if you want TODOs to show up in the generated documentation.
 todo_include_todos = True
 
-#
-# Monkeypatch pygments so it will know about the Swift lexers
-#
+# -- Patch pygments so it will know about the Swift lexers ---------------
 
 # Pull in the Swift lexers
 from os.path import abspath, dirname, join as join_paths  # noqa (E402)
@@ -277,22 +276,6 @@
 
 sys.path.pop(0)
 
-# Monkeypatch pygments.lexers.get_lexer_by_name to return our lexers. The
-# ordering required to allow for monkeypatching causes the warning
-# "I100 Import statements are in the wrong order." when linting using
-# flake8-import-order. "noqa" is used to suppress this warning.
-from pygments.lexers import get_lexer_by_name as original_get_lexer_by_name  # noqa (E402)
-
-
-def swift_get_lexer_by_name(_alias, *args, **kw):
-    if _alias == 'swift':
-        return swift_pygments_lexers.SwiftLexer()
-    elif _alias == 'sil':
-        return swift_pygments_lexers.SILLexer()
-    elif _alias == 'swift-console':
-        return swift_pygments_lexers.SwiftConsoleLexer()
-    else:
-        return original_get_lexer_by_name(_alias, *args, **kw)
-
-import pygments.lexers  # noqa (I100 Import statements are in the wrong order.)
-pygments.lexers.get_lexer_by_name = swift_get_lexer_by_name
+lexers['swift'] = swift_pygments_lexers.SwiftLexer()
+lexers['sil'] = swift_pygments_lexers.SILLexer()
+lexers['swift-console'] = swift_pygments_lexers.SwiftConsoleLexer()

include/swift/AST/ASTContext.h

@@ -70,7 +70,6 @@ namespace swift {
   class LazyContextData;
   class LazyIterableDeclContextData;
   class LazyMemberLoader;
-  class LazyResolver;
   class PatternBindingDecl;
   class PatternBindingInitializer;
   class SourceFile;
@@ -102,6 +101,7 @@ namespace swift {
   class SourceManager;
   class ValueDecl;
   class DiagnosticEngine;
+  class TypeChecker;
   class TypeCheckerDebugConsumer;
   struct RawComment;
   class DocComment;
@@ -273,6 +273,9 @@ class ASTContext final {
   /// Cache of remapped types (useful for diagnostics).
   llvm::StringMap<Type> RemappedTypes;
 
+  /// The # of times we have performed typo correction.
+  unsigned NumTypoCorrections = 0;
+
   /// Cache of autodiff-associated vector spaces.
   llvm::DenseMap<Type, Optional<VectorSpace>> AutoDiffVectorSpaces;
 
@@ -415,28 +418,14 @@ class ASTContext final {
   /// Set a new stats reporter.
   void setStatsReporter(UnifiedStatsReporter *stats);
 
-  /// Creates a new lazy resolver by passing the ASTContext and the other
-  /// given arguments to a newly-allocated instance of \c ResolverType.
-  ///
-  /// \returns true if a new lazy resolver was created, false if there was
-  /// already a lazy resolver registered.
-  template<typename ResolverType, typename ... Args>
-  bool createLazyResolverIfMissing(Args && ...args) {
-    if (getLazyResolver())
-      return false;
-
-    setLazyResolver(new ResolverType(*this, std::forward<Args>(args)...));
-    return true;
-  }
-
-  /// Retrieve the lazy resolver for this context.
-  LazyResolver *getLazyResolver() const;
-
 private:
-  /// Set the lazy resolver for this context.
-  void setLazyResolver(LazyResolver *resolver);
+  friend class TypeChecker;
 
+  void installGlobalTypeChecker(TypeChecker *TC);
 public:
+  /// Retrieve the global \c TypeChecker instance associated with this context.
+  TypeChecker *getLegacyGlobalTypeChecker() const;
+
   /// getIdentifier - Return the uniqued and AST-Context-owned version of the
   /// specified string.
   Identifier getIdentifier(StringRef Str) const;
@@ -498,17 +487,13 @@ class ASTContext final {
   /// Retrieve the type Swift.Never.
   CanType getNeverType() const;
 
-  /// Retrieve the declaration of ObjectiveC.ObjCBool.
-  StructDecl *getObjCBoolDecl() const;
-
-  /// Retrieve the declaration of Foundation.NSCopying.
-  ProtocolDecl *getNSCopyingDecl() const;
-  /// Retrieve the declaration of Foundation.NSError.
-  ClassDecl *getNSErrorDecl() const;
-  /// Retrieve the declaration of Foundation.NSNumber.
-  ClassDecl *getNSNumberDecl() const;
-  /// Retrieve the declaration of Foundation.NSValue.
-  ClassDecl *getNSValueDecl() const;
+#define KNOWN_OBJC_TYPE_DECL(MODULE, NAME, DECL_CLASS) \
+  /** Retrieve the declaration of MODULE.NAME. */ \
+  DECL_CLASS *get##NAME##Decl() const; \
+\
+  /** Retrieve the type of MODULE.NAME. */ \
+  Type get##NAME##Type() const;
+#include "swift/AST/KnownObjCTypes.def"
 
   // Declare accessors for the known declarations.
 #define FUNC_DECL(Name, Id) \
@@ -894,6 +879,11 @@ class ASTContext final {
   /// This guarantees that resulted \p names doesn't have duplicated names.
   void getVisibleTopLevelModuleNames(SmallVectorImpl<Identifier> &names) const;
 
+  /// Whether to perform typo correction given the pre-configured correction limit.
+  /// Increments \c NumTypoCorrections then checks this against the limit in
+  /// the language options.
+  bool shouldPerformTypoCorrection();
+  
 private:
   /// Register the given generic signature builder to be used as the canonical
   /// generic signature builder for the given signature, if we don't already